Deep-change at AXOLOTL-24: Orchestrating WSD and WSI Models for Semantic Change Modeling (2024)

Abstract: This paper describes our solution of the first subtask from the AXOLOTL-24 shared task on Semantic Change Modeling. The goal of this subtask is to distribute a given set of usages of a polysemous word from a newer time period between senses of this word from an older time period and clusters representing gained senses of this word. We propose and experiment with three new methods solving this task. Our methods achieve SOTA results according to both official metrics of the first substask. Additionally, we develop a model that can tell if a given word usage is not described by any of the provided sense definitions. This model serves as a component in one of our methods, but can potentially be useful on its own.

heading: 1 Introduction
text: The shared task on explainable Semantic Change Modeling (SCM) AXOLOTL-24 (Fedorova et al., 2024) is related to automation of Lexical Semantic Change (LSC) studies, i.e. linguistic studies on how word meanings change over time. It consists of two subtasks, however, we focus on the first one and skip the definition generation subtask. Unlike other shared tasks LSC held before, the first subtask of AXOLOTL-24 requires automatic annotation of individual usages of target words instead of target words as a whole. An example of the provided data and required outputs is shown on Figure 1. Namely, for each target word, two sets of usages from an older and a newer period are given (we will call them old and new usages). Additionally, a set of glosses describing word senses in the older time period (old senses) are provided, and the old usages are annotated with these sense glosses. Senses occurring among the new usages (new senses) should be discovered automatically. To be precise, the goal is to annotate each new usage with one of the given old sense glosses, or a unique sense identifier if none of them is applicable. We will refer to those
senses that occur only among old and only among new usages as lost and gained senses, and all other senses as stable senses.
To solve the task, we experiment with three types of models. Word Sense Disambiguation (WSD) models for a given word usage select among given glosses the most suitable one. Word Sense Induction (WSI) models group word usages into clusters corresponding to word senses, they are applicable even when sense descriptions are not available. Finally, Novel Sense Detection (NSD) models find usages corresponding to unknown word senses, the ones that are not covered by the provided definitions. We propose three methods that solve the task. Our best solution denoted as Outlier2Cluster combines all three types of models in a novel way, essentially using an NSD model for each usage to decide whether to return a definition selected by a WSD model, or an identifier of a cluster this usage was put into by a WSI model. On average across languages, this solution achieves SOTA results among all participants of the first substask of AXOLOTL-24 according to both official metrics.
An important additional contribution is the pro-
168
posed NSD model and the related experiments. We study the importance of different features of the NSD model and its effect on SCM quality. Our experiments suggest that improving NSD quality is the most promising direction for the future.

heading: 2 Related work
text: LSCD methods. Several shared tasks related to LSCD were organized in the past, including Schlechtweg et al. (2020); Kutuzov and Pivovarova (2021); Zamora-Reina et al. (2022). Unlike AXOLOTL-24 (Fedorova et al., 2024), they required word-level predictions from their participants, either in the form of word ranking or binary word classification. This type of task setup is generally mentioned under the name Lexical Semantic Change Detection / Discovery (LSCD). In the earlier shared tasks the best results were achieved by solutions that employed non-contextualized word-level embeddings such as word2vec (Mikolov et al., 2013) and vector alignment methods such as Canonical Correlation Analysis and Orthogonal Procrustes Alignment (Pömsl and Lyapin, 2020; Pražák et al., 2020). However, recently tokenlevel methods (Laicher et al., 2021; Rachinskiy and Arefyev, 2021a, 2022) have surpassed them. These methods rely on masked language models finetuned on existing datasets for various tasks of lexical semantics. For instance, solutions relying on the contextualized embeddings from GlossReader, which is a WSD system, have shown SOTA results in the shared tasks on LSCD in Russian and Spanish (Rachinskiy and Arefyev, 2021a, 2022). Methods proposed in this work exploit GlossReader too, both as a WSD model and as a source of contextualized embeddings well-suited for LSC-related tasks.
GlossReader is a multilingual gloss-based WSD model originally developed to solve the Word-in-Context task (Rachinskiy and Arefyev, 2021b). It modifies the English WSD model BEM (Blevins and Zettlemoyer, 2020) replacing the backbone with the multilingual XLM-R language model (Conneau et al., 2020). The model consists of a gloss encoder and a context encoder, both initialized with the XLM-R weights and finetuned jointly learning to select among all glosses of a target word the one describing its sense in a given context. Specifically, the dot product between the context embedding and the correct gloss embedding is maximized.
NSD methods. Several methods were proposed to solve the NSD task. Some of them perform WSI internally. For instance, Lau et al. (2012); Cook et al. (2014) employ a topic modelling approach to jointly cluster old and new usages using the Hierarchical Dirichlet Process. Clusters are ranked based on the novelty score (the difference between estimated probabilities of a cluster appearing in the new and the old corpus). While the method was originally designed for LSCD, the novelty ranking of senses can be combined with a static threshold to identify novel senses.
Alternatively, Mitra et al. (2015) performs WSI separately for an old and a new corpus on graphs, where an edge weight between two words is proportional to the number of words appearing in bigrams with both of them. A cluster in the new corpus is labeled as a novel sense if words in this cluster have weak links with the target word in the graph for the old corpus. A recent method by Ma et al. (2024) uses BERT (Devlin et al., 2019) to build contextualized representations. It employs agglomerative clustering to perform WSI and then matches old and new clusters based on their centroids. The new clusters that are not matched are considered novel senses. Similarly to this method we use agglomerative clustering for WSI, but employing GlossReader to obtain contextualized embeddings.
In Erk (2006) several NSD methods were proposed to detect word senses that are not described in FrameNet (Baker et al., 1998). Instead of relying on WSI, similarly to our NSD method their best method formulates the task as an outlier detection problem. They employ distances between old and new usages requiring a significant number of old usages for each sense, which are not always available in AXOLOTL-24. Thus, we rely on distances between new usages and old glosses instead. Another similar method is introduced in Lautenschlager et al. (2024). They use the XL-LEXEME model (Cassotti et al., 2023) to build representations for usages and senses. Sense representations are built from glosses or example usages of senses taken from dictionaries. They do not always contain the target word, which makes application of XL-LEXEME non-trivial. Authors attempt to solve this problem by modifying glosses and example usages to include the target word. For each usage its nearest sense is found based on the cosine similarity or the Spearman’s correlation between their embeddings. If the similarity is above a threshold, the usage is considered to belong to some non-
described sense. Our methods also rely on usage and sense representations, but we use GlossReader which has a separate gloss encoder and does not require any preprocessing for glosses. We experiment with many measures of similarity between a sense and a usage embedding, and found the manhattan distance between l1-normalized embeddings to outperform other measures and a classifier on a combination of measures to perform best. However, we did not experiment with example usages from sense inventories. When such usages are available, being combined with glosses they may potentially improve sense representations.

heading: 3 Methods

heading: 3.1 Target word positions
text: All our methods assume that a usage is represented as a string and two character-level indices pointing to a target word occurrence inside this string. However, for the Russian subsets these indices were absent. To find them, we first generated all grammatical forms for each target lemma using Pymorphy2 (Korobov, 2015). Then retrieved all occurrences of these forms as separate tokens in the provided usages employing regular expressions.1 For usages with several occurrences of the target word we selected one of them that has both left and right context of reasonable length.2 We inspected new usages from the development and the test sets that did not contain any of the automatically generated word forms and added absent forms manually, then reran retrieval.3

heading: 3.2 WSD methods
text: The first group of methods in our experiments include pure WSD methods, which select one of the provided definitions of old senses for each new usage, and thus, cannot discover gained senses.
1E.g. ’\b(cat|cats)\b’, where \b denotes a word boundary. Matching is case-insensitive.
2This idea is based on our observations that a word occurrence is encoded sub-optimally when it is either the first or the last token, which is probably related to confusion of Transformer heads that have learnt to attend to the adjacent tokens (Voita et al., 2019). The heuristic implemented takes the second to last occurrence if there are more than two of them. For two occurrences it takes argmaxu∈{u1,u2}min(lu, ru), where lu, ru are the lengths of the left and the right contexts.
3Repeating this manual procedure for all Russian data requires significantly more efforts and would have few benefits for our methods. Thus, all old usages having this issue were left without indices and new usages from the training set we dropped.
GlossReader. We employ the original GlossReader model (Rachinskiy and Arefyev, 2021b) as the baseline. For a given new usage u of a target word w its usage representation ru is built with the context encoder. Then gloss representations rg are built for each gloss g of the target word w using the gloss encoder. Finally, the gloss with the highest dot product similarity to the usage is selected.
To improve the results, we further fine-tune the GlossReader model on the data of AXOLOTL-24.
GlossReader FiEnRu is fine-tuned following the original GlossReader training procedure on three datasets: the train sets of the shared task in Finnish and Russian, and the English WSD dataset SemCor (Miller et al., 1994) which GlossReader was originally trained on. We employ all old and new usages from the Russian and Finnish datasets along with their sense definitions. We fined-tuned for 3 epochs using 90/10% train/validation split to select the best checkpoint.4
GlossReader Ru is fine-tuned exactly the same way, but only on the train set in Russian.
GlossReader Fi SG is fine-tuned on the Finnish train set only. Unlike two previous models, we made an attempt to teach this model how to discover novel senses. Specifically, we replaced all glosses of gained senses with a Special Gloss (SG) "the sense of the word is unknown" in Finnish5 and fine-tuned the model as before. For inference we tried adding the special gloss to the provided old glosses, essentially extending the WSD model with NSD abilities. However, this resulted in a noticeable decrease of the metrics on the Finnish development set. Thus, we decided to use the special gloss for training only.6

heading: 3.3 WSI methods
text: Unlike WSD methods, WSI methods do not use definitions or any other descriptions of word senses. Instead they discover senses of a word from an unlabeled set of its usages by splitting this set into clusters hopefully corresponding to word senses. WSI methods cannot attribute usages to the provided old glosses, but can potentially group usages
4The last checkpoint was selected, though after ≈0.5 epochs metrics improve very slowly.
5"sanan merkitystä ei tunneta" as translated by Google Translate
6The majority of words in the Finnish dataset have one sense only, see Section 4.2. Pure WSD methods always return perfect predictions for such cases, thus, it is very hard to compete with them on this dataset. In the future we plan to experiment with this model on the Russian dataset having much smaller proportion of such words.
of the same sense, including gained senses, into a separate cluster.
Agglomerative is the only WSI method we propose and experiment with. For each new usage its representation ru is built using the context encoder of the original GlossReader model. Then we perform agglomerative clustering of old usages using the cosine distance and average linkage on these representations. This clustering algorithm was successfully used to cluster vectors of lexical substitutes, another kind of word sense representations, in several substitution-based WSI methods (Amrami and Goldberg, 2018, 2019; Arefyev et al., 2020; Kokosinskii and Arefyev, 2024), as well as for LSCD (Laicher et al., 2021; Ma et al., 2024).
Agglomerative clustering starts with each usage in a separate cluster, then iteratively merges two closest clusters. The distance between two clusters is the average pairwise cosine distance from the usages in the first cluster to the usages in the second one. Merging stops when the predefined number of clusters is reached. We range the number of clusters between 2 and 9 and select a clustering with the highest Calinski-Harabasz score (Caliński and Harabasz, 1974).7

heading: 3.4 SCM methods
text: WSD and WSI methods provide only partial solutions of the semantic change modeling task, the former cannot discover novel senses, and the latter cannot annotate usages with the old glosses provided. We propose three new methods developed to fully solve the task.

heading: 3.4.1 AggloM
text: Our first SCM method modifies the Agglomerative WSI method by incorporating old usages and senses into the clustering process. We perform agglomerative clustering of a set containing both old and new usages of a target word. Initially, each new usage is assigned to a separate cluster. The old usages are clustered according to the provided sense annotations. Then at each iteration we compute the distances from each cluster containing only new usages to all other clusters. The distance between two clusters is defined as the minimum cosine distance between the usage representations from the first and the second cluster.8 We then merge two
7For one or two usages the Calinski-Harabasz score is not defined. We return a single cluster in such cases.
8This is known as single linkage.
nearest clusters, one of which contains new usages only. This iterative merging process stops when the number of clusters is larger than the number of old senses by k ≥ 0. Therefore AggloM returns exactly k novel senses, where k is a hyperparameter.9 We do not use this method on the Russian datasets because for most senses there are no old usages there.
AggloM FiEnRu is identical to AggloM but relies on the fine-tuned GlossReader FiEnRu.

heading: 3.4.2 Cluster2sense
text: In the second SCM method we first independently cluster new usages using the Agglomerative WSI method and annotate them with the old senses using GlossReader FiEnRu. We then keep the clustering obtained from WSI, but relabel those clusters that overlap heavily with one of the predicted senses. Specifically, we label a cluster c with a sense s if c has the highest Jaccard similarity to s among all the old senses of the target word, and at the same time s has the highest similarity to c among all the clusters built for new usages of this word. Notably, two clusters cannot be labeled with a single sense, thus the clustering of usages is identical to the one originally predicted by WSI. Some clusters will not be labeled with any sense, thus, Cluster2sense can discover gained senses. At the same time, some senses will not be assigned to any cluster, which means the potential to discover lost senses as well.

heading: 3.4.3 Outlier2Cluster
text: Unlike Cluster2Sense which relabels whole clusters, Outlier2Cluster relabels individual usages. Figure 2 shows the processing pipeline. First WSD and WSI predictions are independently made
9In the preliminary experiments on the Finnish development set we selected k = 0, which means that all new usages are eventually merged into clusters representing old senses. This is likely related to the low proportion of gained senses in this dataset and noisy usages which make them hard to discover.
by GlossReader FiEnRu and Agglomerative respectively. Then we discover usages of gained senses. For that we propose a Novel Sense Detection (NSD) model finding usages of those senses that we do not have definitions for.10 Finally, we return WSI predictions for all these discovered usages, and WSD predictions for all other usages.
Novel sense detection. We treat the NSD task as an outlier detection problem, essentially finding those usages that are distant enough from all the provided definitions. Since GlossReader selects the most similar definition for a given usage, it is enough to check if this definition is distant enough to conclude that the usage is an outlier. To check this we employ a logistic regression classifier. Each input example corresponds to a single usage and a gloss selected for this usage by the WSD model. The output is 1 if this usage is an outlier, i.e. does not belong to the predicted sense, and 0 otherwise.
We use distances (computed with several distance functions) between GlossReader representations of the new usages and the glosses for old senses as features for logistic regression.
For the new usage u and the selected definition g we take the corresponding representations ru and rg from a gloss encoder and a context encoder respectively. We take these representations from two different GlossReader models, the original one and GlossReader FiEnRu, and calculate distances from ru to rg using different distance and normalization functions. This gives 10 different features presented in Table 1. We also include three extra features: the number of old usages, old senses, and new usages for the target word in the dataset. We employ the Standard Scaler to normalize features and train the logistic regression with L2 regularization of C = 1.
Thus, the trained logistic regression can be used for each usage to decide whether the WSD method has assigned a correct sense or should be replaced with some cluster corresponding to a gained sense. If the score is above a threshold of 0.65, which was selected on the development sets of the shared task, the usage is considered an outlier.
We train two NSD models on the Russian and the Finnish development sets separately and use the trained models for the corresponding test sets.
10In the context of the shared task these are gained senses. However, the approach is general enough to discover lost senses when a modern dictionary and old usages are given, or just senses from the same time period as the dictionary but not covered by it.
For the surprise language, we do not have labeled data to select one of two models or train a separate model, thus, we simply report the results of both models.
Outlier relabeling. We experiment with two ways of assigning clusters to the detected outliers. Our first approach (w/o WSI) groups all outliers into a single new cluster. Alternatively, w/ WSI approach assigns the clusters predicted by the WSI method to outliers. We use the first option for the Finnish test set, as we observed that the words in the Finnish development set rarely have more than one gained sense. On the contrary, the words in the Russian development set have many gained senses, therefore, we employ w/ WSI for the Russian test set. For the surprise language Outlier2Clusterfi employs w/o WSI and Outlier2Clusterru employs w/ WSI.
All of the described methods are briefly summarized in Table 2.

heading: 4 Evaluation setup
text: The first AXOLOTL-24 subtask evaluates semantic change modeling systems in three diachronic datasets in Finnish, Russian, and German (Fedorova et al., 2024). Train and development sets are provided for the first two, but not for the last. We will now describe the datasets in more detail.

heading: 4.1 Data sources
text: The source for the Finnish dataset of the shared task is resource (1997). The usages are divided into two groups: before 1700 and after 1700. The usages in the dataset are not complete sentences but short phrases. Some parts of the phrase can be missing and replaced with double hyphens, presumably due to OCR errors. Furthermore, the usages from both the old and the new corpus exhibit no-
table differences from modern Finnish. They often feature characters (such as c, z, w, and x), that are not commonly found in contemporary Finnish. It is important to highlight that the glosses provided for word senses are in modern Finnish.
Two data sources used to create the Russian dataset are Dahl (1909) processed by Mikhaylov and Shershneva (2019) and Mickus et al. (2022). The first one was the source of old usages and glosses, and the latter provided new usages and glosses. However, the specific procedure used to map senses between these two sources was undisclosed at the time of the competition. Some old senses are not accompanied by old usages in the Russian datasets. Consequently, our methods for the Russian datasets do not rely on the old usages. Notably, the Russian datasets lack information regarding the position of a target word within a usage or the actual word form of the target word. As a result, we incorporate the identification of the target word’s position within a usage as a preprocessing step in our solution.
The shared task also includes a test dataset in a surprise language revealed only at the test phase of competition with no development or train sets. The source of this dataset is a German diachronic corpus with sense annotations (Schlechtweg et al., 2020; Schlechtweg, 2023).

heading: 4.2 Data Statistics
text: To get insights into the data we categorize the target words within the train and the development sets based on several characteristics:
• Has lost senses: does the word have old senses for which there are no new usage?
• Number of gained senses: how many senses are there having new usages only?
67% 63% 64% 66% 64% 55% 63% 33% 37% 36% 34% 36% 45% 38%Has lost senses? true false
95% 94% 95% 100% 79% 85% 96% 21% Disjoint senses? true false
38% 39% 40% 72% 99% 100% 88% 62% 61% 60% 28%New hasone sense? true false
52% 54% 54% 79% 99% 100% 100%
48% 46% 46% 21%Has one sense? true false
fi train fi dev fi test ru trainru dev ru tes tde test
76% 70% 74% 49% 67% 21% 51% 21% 96% 97% Number of gained senses
Total number of words 4289 254 275 924 201 211 24
2+ 1 0
The number of target words in each category for all11 the datasets of the shared task is presented on Figure 3.
In the Finnish datasets, almost half of target words have only one sense and approximately 70% of words have no gained senses. Therefore, the conservative methods that rarely discover gained senses are preferable for the Finnish datasets.
The main observation for the Russian datasets is the dramatic differences in proportions of almost all categories between the train and the development
11This information for the test sets was not available during the competition.
set. We can see that the statistics of the test set are similar to those of the development set. Contrary to the Finnish sets, almost all words in the Russian development set have gained senses. Therefore, methods which are prone to predict new senses rather than old ones are preferable for the Russian development set.
The German dataset is relatively small and contains 8 times fewer words than the other test sets. We can see that it is similar to the Finnish datasets in the proportion of gained and lost senses.

heading: 4.3 Metrics
text: The shared task employs two metrics to evaluate the systems, the Adjusted Rand Index (ARI) and the F1 score.
ARI (Hubert and Arabie, 1985) is a wellestablished clustering metric employed to evaluate how well new usages are clustered by a system. In the subtask, ARI is computed for all the new usages of a target word, the ground truth clusters correspond to senses. Notably, cluster labels are not taken into account by ARI. It means that old senses and gained senses are indistinguishable from each other in terms of ARI.
The F1 score is used in the first subtask to estimate how well a system can discriminate between old senses. It is computed only for the new usages of the old senses, and not for the usages of the gained senses. The F1 score for a target word is the average of the F1 scores for all old senses. If a target word does not have any new usages with the old senses, it is arbitrarily assigned the F1 score of 1 if old senses are not predicted for any of its usages and 0 otherwise. Thus, in this edge case a system is heavily penalized when even a single usage is misclassified as one of the old senses.
All new usages of the old senses which are (incorrectly) predicted as belonging to a gained sense are considered to belong to a single auxiliary "novel" class when calculating the F1 score. The F1 score for this class is zero as it has zero precision. For this reason, even a single usage misclassified as a gained sense can dramatically affect the overall score for a target word independently of the total number of its usages.12
12Assume the target word has k old senses. In case when only old senses are predicted: F = F1+...+Fk
k . If we replace
one of the correct predictions of sense 1 with an incorrect prediction of a gained sense: F ′ = F ′ 1+...+Fk+0
k+1 < F1+...+Fk+0 k+1 .
The drop in this metric is F F ′ > k+1 k E.g. in the case k = 1, which is a frequent case in the Finnish AXOLOTL-24 dataset,

heading: 5 Results

heading: 5.1 Our submissions
text: The number of submissions for the test sets per team was not limited in the competition. We evaluate ten models on the test sets: four WSD models (based on GlossReader, GlossReader FiEnRu, GlossReader Ru, and GlossReader Fi SG), one WSI model (Agglomerative with GlossReader representations), two AggloM models (based on GlossReader and GlossReader FiEnRu representations), Cluster2Sense, and Outlier2Cluster with different configurations for the German dataset: Outlier2Clusterru and Outlier2Clusterfi. Table 3 demonstrates the evaluation results. We also include the best submissions from other teams for comparison.
WSD and WSI. The best results in terms of the F1 score are achieved by pure WSD methods. The F1 score is calculated only for the usages of old senses, this gives a huge advantage to WSD methods because incorrect prediction of old senses for usages of gained senses is not penalized, while the opposite reduces the F1 score severely as explained in Section 4.3.
WSD methods have notably higher ARI than Agglomerative and Cluster2Sense (both of them predict the same clusters but label them differently) for the Finnish and German datasets. On the contrary, Agglomerative and Cluster2Sense are the best-performing methods for the Russian dataset. Our explanation for this fact comes from the analysis in Section 4.2. The sets of senses of the new and the old usages in the Finnish and German datasets overlap heavily, which is beneficial for WSD methods. The overlap is much smaller for the Russian dataset, which hurts ARI of the WSD methods. Discovering gained senses is crucial for the Russian dev and test set.
AggloM. The AggloM method with the hyperparameter k = 0 (never predicts gained senses) does not fall far behind pure WSD methods. The main reasons for that probably are the usage of the same underlying context encoder and prediction of only old senses. Therefore, AggloM is a viable alternative to the GlossReader models when word senses are described with usage examples instead of sense definitions.
Outlier2Cluster. Outlier2Cluster achieves
an incorrect prediction of a gained sense for a single usage results in more than 2x decrease of the F1 score.
SOTA or near-SOTA ARI13 for Russian and Finnish, but falls behind WSD methods for German, which has no labeled data to train a dedicated NSD model. However, Outlier2Cluster can discover gained senses unlike WSD methods. Thus, we consider Outlier2Cluster to be preferable for the SCM task and suggest training the NSD model for each language of interest.14
The important hyperparameter of the NSD model, and consequently the Outlier2Cluster model exploiting it as a component, is the threshold dividing usages into outliers and normal usages. Figure 4 shows the dependence of the metrics on the threshold value for the Finnish and Russian development sets. Both w/ WSI and w/o WSI versions of Outlier2cluster are included. We also compute the results of Outlier2Cluster with the WSI oracle which perfectly clusters the detected outliers according to their ground truth senses, and the NSD oracle which perfectly detects usages of gained senses. The methods we study in this Section are briefly summarized in Table 4.
We can see that the F1 score (computed only
13We made Outlier2Clusterfi submissions in the competition separately for different datasets. For this reason, it was not selected as our best submission by the competition organizers.
14We used only small development sets with ≈ 200 target words to train novel sense detection models.
over new usages with old senses) monotonically increases with the increasing threshold, i.e. with fewer outliers detected. This again shows that trying to detect usages of gained senses and clean the old senses from them hurts the F1 score, supporting the criticism of this metric in Section 4.3.
ARI reaches a peak at the threshold of 0.65 for the Russian dataset with F1 being close to maximum as well. We therefore set the threshold at 0.65 for the Russian NSD model. This gives the SCM model that almost achieves the ARI of pure WSI predictions (threshold of 0) while having only a bit smaller F1 score compared to the best WSD model.
For the Finnish dataset, higher ARI monotonically increase with the threshold, i.e. with the proportion of predictions taken from the WSD model. This agrees with the observations from Table 3 that the pure WSD models give the best ARI for Finnish. We can also see that the threshold values in the middle, where neither WSI nor WSD predictions are dominant, result in a significant decrease in ARI. It means, that our NSD model cannot be used effectively to combine the predictions for Finnish. We select a high threshold of 0.65 for the Finnish dataset, resulting in a low number of outliers. Consequently, the novel sense detector predicts less than 1% of usages to be outliers in the Finnish test set, compared to 42% of usages predicted as outliers for the Russian test dataset.
We can observe that according to the F1 score, the NSD oracle performs better than the pure WSD method, especially on the Russian development set. The reason lies in the words with disjointed senses. Since there are no new usages of old senses for such words, the ordinary F1 score and it is arbitrarily defined as 1 if all usages are recognized as usages of gained senses, i.e. put into new clusters, and 0 otherwise. Thus, the ideal processing of these edge cases is crucial for the F1 score, but can hardly be achieved unless the NSD oracle is employed. For other words it does not help. Considering ARI, the NSD oracle performs much better than w/WSI on the Russian dataset. It means that better NSD models may help greatly improve clustering.
According to the results of w/ WSI oracle on the Finnish development set, it is impossible to increase ARI with better WSI method without a huge drop in the F1 score. For the Russian dataset situation is the opposite. The main reason is likely the average number of gained senses per word in these datasets as described in Section 4.2. Only 7% of words in the Finnish dataset have gained two or more senses, therefore the perfect clustering of the gained senses does not increase the results significantly compared to merging all gained senses into a single cluster. On the contrary, 97% of the word in the Russian have two or more gained senses, making WSI necessary.

heading: 6 Conclusion
text: We have proposed three new methods that solve the SCM task. Our solution achieves SOTA results among all participants of the first subtask of the AXOLOTL-24 shared task. Additional experiments propose directions of further improvement of the developed models, NSD being potentially the most promising one.

heading: 7 Limitations
text: While our methods can in theory be applied to any SCM dataset, we acknowledge that they may be overspecified for the first subtask of AXOLOTL-24. Notably, we extensively use the train sets provided for the competition in Finnish and Russian to train the embedding model and to optimize the hyperparameters. While we also evaluate on the German dataset in a zero-shot fashion, the results may be unreliable due to relatively small size of the dataset.
Semantic change modeling may be of particular interest in studies of older time periods, where the language is quite different from its modern state. The underlying model, GlossReader, is a finetuned version of XLM-R, which was not specifically designed to handle old languages. In this case datasetspecific finetuning of the base GlossReader may become even more relevant.

heading: 8 Acknowledgements
text: Nikolay Arefyev has received funding from the European Union’s Horizon Europe research and innovation program under Grant agreement No 101070350 (HPLT).

heading: A Ablation study of the NSD model
text: In this section we provide an ablation study of our NSD model. In order to get insights about the importance of the chosen features, we compare our trained classifiers and several pure similarity measures such as the predicted probability, the dot product and the distance from a usage to the gloss selected by GlossReader FiEnRu (Figure 5). It turns out that the probabilities and dot products are far behind the classifiers, and on the Finnish dev set they even perform no better than a random classifier. The manhattan distance with l1 normalization is a bit worse than the trained classifiers. The extra
features consistently help on the Finnish dev set, but are almost useless on the Russian dev set.
In Table 5 we compare different NSD models using the average precision on the dev sets. To understand which quality can be achieved using the minimal number of features, we evaluate all single distance-based features. Furthermore, we train classifiers on all possible pairs of features, where each pair contains distances only from the same GlossReader. Also we compare classifiers with or without extra features.
We observe that the manhattan distance with l1 normalization, which is the best single feature, works poorly on the Finnish dataset, especially for the embeddings from GlossReader that was not fine-tuned on the Finnish train set. However, on the Russian dev set it closely follows the best classifier. As for the classifiers, we found that including nondistance features is important for Finnish. What is more interesting, when using the original GlossReader model among all pairs of features the best one does not contain embedding-based features at all, only the number of old senses and the number of new usages. This signals that for the Finish dataset GlossReader provides poor embeddings without fine-tuning.

heading: Future Work
text: In the future we plan to experiment with this model on the Russian dataset having much smaller proportion of such words. of the same sense, including gained senses, into a separate cluster. Agglomerative is the only WSI method we propose and experiment with. For each new usage its representation ru is built using the context encoder of the original GlossReader model. Then we perform agglomerative clustering of old usages using the cosine distance and average linkage on these representations. This clustering algorithm was successfully used to cluster vectors of lexical substitutes, another kind of word sense representations, in several substitution-based WSI methods (Amrami and Goldberg, 2018, 2019; Arefyev et al. , 2020; Kokosinskii and Arefyev, 2024), as well as for LSCD (Laicher et al. , 2021; Ma et al. , 2024). Agglomerative clustering starts with each usage in a separate cluster, then iteratively merges two closest clusters. The distance between two clusters is the average pairwise cosine distance from the usages in the first cluster to the usages in the second one. Merging stops when the predefined number of clusters is reached. We range the number of clusters between 2 and 9 and select a clustering with the highest Calinski-Harabasz score (Caliński and Harabasz, 1974). 7