Substitution-based Semantic Change Detection using Contextual Embeddings
Year: 2023

Abstract:
Measuring semantic change has thus far remained a task where methods using contextual embeddings have struggled to improve upon simpler techniques relying only on static word vectors. Moreover, many of the previously proposed approaches suffer from downsides related to scalability and ease of interpretation. We present a simplified approach to measuring semantic change using contextual embeddings, relying only on the most probable substitutes for masked terms. Not only is this approach directly interpretable, it is also far more efficient in terms of storage, achieves superior average performance across the most frequently cited datasets for this task, and allows for more nuanced investigation of change than is possible with static word vectors.

heading: 1 Introduction
text: Measuring semantic change is one of the few areas of NLP where contextual embeddings have not yet led to a definitive improvement over previous methods. In particular, the commonly used approach of aligning static embeddings trained on different time periods (Hamilton et al., 2016b) continues to be a surprisingly hard to beat baseline.
Given that contextual embeddings provide a representation for each occurrence of a word in context, they would seem to be ideally suited to a more nuanced investigation of semantic change. Most attempts to leverage them for this purpose, however, produce quantitatively worse results, while being less interpretable and requiring more resources.
Here, we present a simplified and improved approach to scalable, interpretable, semantic change detection using contextual embeddings. Inspired by Eyal et al. (2022), we work only with the most probable replacements for masked words, and measure semantic change in terms of the distributions of replacements in each time period. Not only does this better match human judgements, it is highly space
efficient, works seamlessly for out-of-vocabulary words, and helps intuitively characterize meaning change and variation.

heading: 2 Background
text: Measuring semantic change involves a set of tasks related to determining if and how a term’s meaning has changed over time. Here, we focus on the task of measuring the amount of change that has occurred from one time period to another (Gulordava and Baroni, 2011; Schlechtweg et al., 2020).1
Existing approaches to this task are mostly of two types. The first is associating each term with a single vector per time period and measuring the distance between vectors, of which we take Hamilton et al. (2016b) to be representative. As a variation on this, several authors have proposed averaging the output of contextual embedding models to get a single vector per term in each time period, but this has generally not led to an improvement over using static vectors (Martinc et al., 2020a; Kurtyigit et al., 2021; Liu et al., 2021). A related approach is to represent words in terms of their nearest neighbors using static word vectors (Hamilton et al., 2016a; Gonen et al., 2020), but this does not show a clear improvement over other static embedding methods (Montariol et al., 2021).
A second type of approach begins with various methods for word sense induction, then measures change in terms of the relative prevalence of a term’s different senses (Frermann and Lapata, 2016; Hu et al., 2019; Arefyev and Zhikov, 2020; Arefyev and Bykov, 2021). In some cases, authors simply cluster contextual representations for each term, and measure differences in the distributions of clusters between two time periods, rather than dealing with explicit word senses (Giulianelli et al., 2020; Martinc et al., 2020b; Montariol et al., 2021).
1For surveys of computational approaches to lexical semantic change detection, see Kutuzov et al. (2018), Tang (2018), and Tahmasebi et al. (2021).
590
Despite the additional information provided by contextual embedding models, methods using type embeddings (as opposed to token), continue to be competitive. For example, on the recent SemEval multilingual semantic change detection task, none of the top four systems used token embeddings (Schlechtweg et al., 2020). Methods using contextual embeddings have done better on some more recent mono-lingual shared tasks (Kutuzov and Pivovarova, 2021; Zamora-Reina et al., 2022), but have not yet been evaluated with a consistent setup across multiple languages.

heading: 3 Methods
text: Building on Eyal et al. (2022), we represent each token in the corpus (or a sufficiently large sample of them) by a small set of probable replacement terms from a contextual embedding model. However, whereas Eyal et al. (2022) did this for the purpose of word sense disambiguation, we do so for the purpose of measuring semantic change.
For each sampled occurrence of each term, we mask the term of interest, feed the masked context through a model, and obtain the predicted token probabilities corresponding to the mask token.2 From these, we save only the top-k most probable words (excluding stopwords and partial word pieces), and discard the rest.
For a given term in a particular time period, we then count how many times each word in the model vocabulary has appeared as a top-k replacement for that term, and normalize this by its sum, giving us a distribution over replacements. To obtain a raw score of semantic change between two time periods, we compute the Jensen-Shannon Divergence (JSD) between the two distributions representing the same term in different time periods. However, as we show below, the raw JSD scores are strongly correlated with term frequency. Thus, to obtain a scaled metric, we convert the raw JSD scores into a quantile, comparing the raw score for a term of interest to other terms with similar frequency.
Compared to saving the full output vector per token, this approach only requires a miniscule amount of storage per token, and thus does not require the kind of heuristic dropping of tokens employed by Montariol et al. (2021). In addition, the dominant meanings of a word in each context can be summarized by the terms which occur most fre-
2Words that get tokenized into multiple word pieces are replaced by a single mask token.
quently among the top-k replacements. Although such replacements are limited to the terms which exist in the model vocabulary, in practice this is sufficient to represent a nuanced set of meanings, and works even for words which get tokenized into multiple word pieces, as we show below.
More formally, given two corpora C1 and C2, let the count of token v as a top-k replacement for term t in corpus c be:
count(v, t, c) = ΣNc(t)i=1 I[v ∈ R(t, i, k)], (1) where R(t, i, k) is the set of top-k most probable replacements for occurrence i of term t (excluding stopwords and partial word pieces in the model vocabulary), and Nc(t) is the number of sampled occurrence of term t in corpus c.3
Let ∆ct by the distribution of top-k replacement counts for term t in corpus c, obtained by dividing the corresponding vector of counts (i.e., [count(·, t, c)]) by the sum over the model vocabulary. The raw change score for term t is given by the JSD between the two distributions:
raw(t) = JSD ( ∆C1t ,∆ C2 t ) . (2)
Finally, we correct for frequency effects by rescaling the raw JSD scores against the scores for terms with similar frequency as the target term, giving us a quantile scaled in [0, 1]:
scaled(t) = Σs∈T (t)I[raw(t) ≥ raw(s)]/|T (t)|, (3) where T (t) is the set of terms with similar frequency to term t (excluding term t itself). More specifically, we compare against all terms within a fixed factor of the target frequency:
T (t) = {s : fr(t)/F ≤ fr(s) ≤ fr(t)× F, s ̸= t}, (4) where fr(t) is the frequency of term t in the corpus, with window factor F .

heading: 4 Experiments
text: To evaluate our method we make use of datasets for which there have been prior evaluations of methods across multiple languages, following standards established by past work for the sake of a head-tohead comparison.4
3Unlike Eyal et al. (2022), we do not combine probabilities for different forms of the same lemmas in the model vocabulary. In addition, we do not exclude the target term from the top-k replacements, except implicitly for terms which get split into multiple word pieces.
4Code to replicate these experiments is available at https://github.com/dallascard/SBSCD

heading: 4.1 Data
text: We use five datasets with words labeled in terms of semantic change between two time periods. Four of these are from SemEval 2020 Task 1: Unsupervised Lexical Semantic Change Detection (SE; Schlechtweg et al., 2020). These datasets contain 31 to 48 terms from four languages, graded in terms of change by human raters, along with accompanying corpora to be used in estimating the amount of change. The fifth dataset (GEMS) comes from Gulordava and Baroni (2011), and contains 100 words labeled in terms of semantic change from the 1960s to 1990s. As with most recent papers which use this dataset, we use the Corpus of Historical American English (COHA; Davies, 2010) for measuring change in the GEMS words.

heading: 4.2 Experimental Details
text: For each dataset, we fine tune an appropriate BERT model to the union of the two associated unlabeled corpora using continued masked language model training with the HuggingFace transformers package. We then index the corpora to find all occurrences of each word. For all target words, along with a random set of 10,000 background terms, we randomly sample up to 4,000 occurrences of each from the associated corpora. We process all sampled tokens as described above to obtain and store the top-k replacements for each, with k = 5. Using the replacements obtained from the model, we compute raw JSD scores for each term. Finally, we convert these to scaled scores by comparing to the background terms that have frequency within a factor of two of the target term (i.e., F = 2).
Following past work, we evaluate using Spearman correlation with human ratings, comparing against the best results from recent papers. In particular, we include two results based on slight variations on Hamilton et al. (2016b), one of which was the best performing method in the SemEval competition (Pömsl and Lyapin, 2020), as well as methods using contextual embeddings (Martinc et al., 2020b; Montariol et al., 2021). For fully experimental details, please refer to Appendix A.

heading: 4.3 Results
text: Full results are given in Table 1. Although our method is not uniformly better than all previous methods on all dataset, it does produce the best result on average, as well as improvements on GEMS, SE English and SE Latin.
As an example to better understand these results, the raw JSD scores from our method are shown in Figure 1 (top) for the SE English data, with select terms labeled. As can be seen, there is a strong relationship between term frequency and raw JSD, hence the need to rescale the raw scores relative to terms with similar frequency. After rescaling, we see a strong correlation between our final semantic change scores and the human ratings, as shown in Figure 1 (bottom) for the SE English data.
As with the approach of Hamilton et al. (2016b), our method supports direct interpretation of semantic change. To understand the change in a word’s typical usage, we can look at the overall most common replacements from each time period. Table 2 shows the scores and rankings of several selected terms from SE English, along with the most common substitutes from each time period.
Looking at the results, we can see, for example, strong agreement with human annotators on a dramatic change in the meaning of plane (comparing 1810–1860 vs. 1960–2010), from the geometric concept to the flying machine. On the other hand, our results suggest that human raters may have slightly underestimated the amount of change in
the meaning of graft, which was previously used mostly in reference to vegetation, but now most commonly refers to corruption.5
By contrast, ounce may be a case where our method has underestimated the change that has taken place. Older usages seem to map more generically to a wider range of quantities (hence the appearance among the early substitutes of hour, acre, and dollars), whereas modern usage seems more restricted. Indeed, we do find some difference in the distribution of substitutes between the two time periods, but less of a difference than is typical for words with similar frequency, hence the low final score from our method (see Figure 1).
Although we do not emphasize it in this paper, of our method can easily be combined with the approach of Eyal et al. (2022) to further investigate meaning changes, by inferring senses from the term replacements, and looking at how their usage varies by time period. In particular, for each target term, we can construct a graph from the set of term substitutes (as nodes), where edge weights represent the number of top-k clusters in which two
5Note that because graft is not a term in the BERT vocabulary, the term itself does not appear as a potential substitute, but the results remain interpretable nonetheless.
substitutes co-occur. Following Eyal et al. (2022), we experiment with Louvain community detection to identify sense clusters from these graphs for each term of interest, and use Jaccard similarity to associate each mention with a sense cluster, based on substitute overlap (see Appendix A for details).
Inspecting the distribution of these senses over time helps to distinguish the gradual adoption of existing senses from the creation of new ones. For example, the most common sense of plane is captured by the sense cluster {aircraft, jet, airplane, car}, and as expected, this sense is not found in the 1810–1860 English data, except for two instances which appear to be errors in the inferred sense. By contrast, the second most common sense—{planes, line, point, surface}—appears in both time periods, but is much more common in the earlier time.
This approach also provides more insight into how the meaning of graft has changed. The most common sense cluster is the horticultural meaning {tree, plant, stock, vine}, and this meaning occurs in both time periods, but is much more common in the earlier one. A second cluster, corresponding to illicit activity—{corruption, violence, bribery, fraud}—occurs only in the later time period. This clustering method also surfaces a third sense with
a medical meaning—{transplant, surgery, disease, drug}—which is not revealed by the top few overall most common replacements given in Table 2.

heading: 5 Discussion and Related Work
text: As noted by others, new and larger datasets for rigorously evaluating semantic change are badly needed (Tahmasebi et al., 2021). Existing datasets are relatively small, and are mostly based on inspecting a limited number of examples per term. Unfortunately, determining ground truth for semantic change is challenging, and producing such resources is costly. Ideally, future datasets for evaluation should be larger, both to allow for more robust evaluation, and to have sufficient targets for both hyperparameter tuning and evaluation.
In addition to the dataset we have used in this paper, two others are available from shared tasks on Spanish and Russian, respectively (Kutuzov and Pivovarova, 2021; Zamora-Reina et al., 2022). Both of these are comparable in size to the GEMS dataset used here. Unfortunately, they are less useful for evaluation because most submissions to these shared tasks only evaluated on the task data, and not on other datasets. As shown by the replication of Martinc et al. (2020b) in Montariol et al. (2021), a method can sometimes perform well on one language but fail to generalize to others. As such, we have based our evaluation on datasets for which there has been a consistent evaluation of methods across multiple languages. As future work, a careful replication study of all methods from each competition on all available datasets, including an assessment of sensitivity to hyperparameters, would be highly informative.
Besides Eyal et al. (2022), The closest prior work to ours is Kudisov and Arefyev (2022), who use dynamic patterns to generate many variations on example usages sampled from the given corpora. These variations are then used to generate hundreds of replacement terms from a masked language model with associated probabilities. These probabilities are averaged (heuristically combining replacements with differing numbers of word pieces) to obtain a mean vector for each sampled instance. Finally, semantic change is computed as the average cosine distance between all pairs of vectors across corpora. This method was evaluated as part of the LSCDiscovery shared task on Spanish (Zamora-Reina et al., 2022). Preliminary work on this method was described in Arefyev and
Bykov (2021), where a slightly different version of it was evaluated on the RuShiftEval shared task on Russian (Kutuzov and Pivovarova, 2021).
Compared to Kudisov and Arefyev (2022), our approach is considerably simpler, and better suited to storing representations of a complete corpus for subsequent analysis and exploration. In particular, we only consider a small number of substitutes for each example (storing only the top-k most probable terms, without the associated probabilities). We do not use dynamic patterns, and only consider terms in the model vocabulary as potential substitutes. We also associate each term with a single distribution over the model vocabulary per time period (not per mention), and use Jensen-Shannon divergence to more naturally measure the distance between distributions. Importantly, we also correct for frequency effects, as described above.
Although our approach avoids the onerous storage requirements of methods which save full contextual vectors, it still requires considerable processing time to obtain the top-k replacements for all tokens. Future work could explore smaller or more efficient models for this purpose.6
Finally, despite its simplicity, measuring the cosine distance between aligned static vectors remains a strong and efficient baseline (Hamilton et al., 2016b). More work is needed to determine where contextual embeddings can offer sufficient advantage in measuring semantic change to justify their greater computational cost.
Compared to static embeddings, our approach is weakest on the German and Swedish datasets, which could relate to the quality of the pretrained models that are available for those languages, the data used for pretraining, or perhaps issues that arise in tokenization of the reference corpora. For a tentative exploration of some possible factors, please refer to Appendix C.

heading: 6 Conclusion
text: We have presented a simplified and improved approach to measuring semantic change using contextual embeddings, based on the Jensen-Shannon Divergence between the distributions of the most probable replacements for masked tokens in different time periods, corrected for frequency effects. This approach achieves superior performance on average, while remaining directly interpretable, with vastly reduced storage requirements.
6See Appendix B for results using various model sizes.
Limitations
There are several limitations to this work which should be kept in mind. First and foremost, the datasets for evaluating the measurement of semantic change are relatively small, meaning that any estimates of correlation with human judgements will be relatively high variance. In addition, although the SemEval data includes text from four languages, there is no guarantee that these methods will work as well as they do on other languages or other time periods. Moreover, our approach depends on the use of pretrained language models, and the quality (or existence) of these and other relevant resources will vary by language.
In addition, like all methods, our approach involves numerous small choices, such as the number of background terms to sample, the number of samples taken, and the value of k in choosing top substitutes. We have kept our choices for these consistent across all five datasets, and these values have not been tuned. As such, different choices could result in better or worse correlation with human judgements. It is also worth noting that the human judgements collected by the creators of these datasets may involve errors or noise. It is possible that a different sample of data, or having different people evaluate the same data, would produce different judgements.
For exploring the variation in word meanings, we have used the approach of Eyal et al. (2022) directly, with the only differences being that we mask terms of interest (allowing us to work with terms that do not exist in the model vocabulary), and do not combine multiple forms of lemmas when getting the top-k terms. We adopt this approach because it is especially easy to combine with our own work, but different methods for word sense induction might lead to different conclusions about the different meanings of a term that existed in any particular time period. In addition, any conclusions drawn are necessarily limited to the corpora that are used, most of which will be a highly biased sample of all text that was produced by all people for any given period of time.
Ethical Considerations
This work only uses well established datasets for the purposes for which they were designed (studying changes in languages and evaluating measurement of semantic change), thus poses few ethical concerns that did not already exist for these data.
Nevertheless, it is worth emphasizing that all of methods discussed in this paper only return, at best, a noisy estimate of semantic change. Words are used differently by different people, and attempts to measure changes in language inevitably simplify the diversity of uses into a single number, which discards a great deal of nuance. As such, any work applying these methods to measure semantic change should be aware of their limitations and proceed carefully.

heading: Acknowledgements
text: Many thanks to Kaitlyn Zhou and anonymous reviewers for helpful comments and suggestions.

heading: A Experimental Details
text: For each dataset, we use a BERT model, preferring a high quality monolingual model where available. For GEMS and SE English, we use bert-large-uncased. For SE Latin we use bert-base-multilingual-uncased, deepset/gbert-large for SE German, and KB/bert-base-swedish-cased for SE Swedish, with all models available through HuggingFace. In all cases, we first adapt the model to the dataset by doing continued masked language model training for five epochs on the union of the two associated corpora.
For the SemEval data, the corpora are provided in both raw and lemmatized formats, with the target terms given as lemmas. Because the contextual embedding models have been trained on nonlemmatized text, we prefer to embed mentions using the raw (non-lemmatized data). However, because of uncertainty about how the raw text was lemmatized, we begin by aligning the lemmatized data to the non-lemmatized text. We then index terms in the lemmatized data (for both target terms and random background terms), and then map these indices to indices in the corresponding non-lemmatized data, which we then sample to get replacements.
To do the alignment, we begin by tokenizing the text, and then removing the punctuation from both the lemmatized and non-lemmatized text, storing indices to allow mapping back to the original token sequences in the non-lemmatized data. For each pair of texts (a raw and a lemmatized form), we first identify tokens that occur exactly once in each, and align the positions of these to each other, as long as the ordering of these tokens is consistent. We then recursively do this for the subsequences between each adjacent pair of aligned tokens. Given these landmark alignments, (using exact matches), we then attempt to align all remaining substrings between each pair of aligned tokens, (adding padding tokens as necessary), using Levenshtein distance as a heuristic way to evaluate possible token alignments. Finally, we do a post-alignment correction to consider inserting a padding token in each position to correct for occasional off-by-one errors, and taking the best scoring overall alignment.
By inspecting the target tokens in the raw (nonlemmatized text) that are obtained using this alignment (based on indexing target terms in the lemmatized version, then mapping these indices to
the non-lemmatized text using the alignment), we find that the vast majority of mentions are properly aligned. To eliminate the small number of alignment errors, we only keep tokens that are at least two characters in length where the non-lemmatized form comprises at least 0.02% of the total number of indexed terms for a given lemma, and where the first letter of the indexed token matches the first letter of the target lemma. To account for a small number of special cases (such as examples in SE Latin where a word sometimes starts with “j” and sometimes with “i”, (presumably due to OCR errors), we create a handful of exceptions to the first letter rule. For full details of this alignment process and exceptions, please refer to replication code.7
In addition, for the SE English data, target terms (only) are given with specific part of speech tags. However, to better match a random sample of background lemmas, we ignore part of speech in our experiments, and index all occurrences of each target term in the lemmatized data. Future work could explore the impact of restricting measurements to certain parts of speech, both for target and background terms.
For GEMS, where the targets are not lemmatized, we ignore lemmatization and simply sample from all exact matches of the target terms as tokens in the raw text. As with past work, we combine the multiple annotations for the GEMS data by averaging their scores.
All masked tokens are fed into the appropriate model with up to 50 tokens to either side from the original context, which returns a probability distribution over the model vocabulary. When computing the top-k most probable substitutes, we follow Eyal et al. (2022) and exclude stopwords and partial word pieces (i.e., those that start with ##). For GEMS and SE English, we use the stopword list from the Snowball stemmer.8 For SE Latin, we use a Latin stopword list from the Perseus Digital Library.9 For SE German and SE Swedish, we use the respective stopword lists from NLTK.10
For the exploration of sense clusters in the main paper using Louvain community detection, we use the same data as used in measuring semantic change, keeping k = 5, but we exclude the target
7https://github.com/dallascard/SBSCD 8http://snowball.tartarus.org/
algorithms/english/stop.txt 9https://www.perseus.tufts.edu/hopper/ stopwords 10https://www.nltk.org/
term itself when gathering the top-k substitutes.11 We then construct a weighted graph for each target term, where nodes represent substitutes, and edge weights correspond to the number of top-k replacement sets in which each pair of replacements appear together.
To obtain sense clusters, we use the implementation of Louvain community detection in networkx with default parameter settings, to detect clusters in the graph.12 Finally, we associate each instance of a target term with a corresponding cluster using Jaccard similarity between the instance’s set of top-k replacements and the terms in the cluster.
All of these experiments were run on either an NVidia RTX A6000 or A5000 GPU.

heading: B Alternative Models
text: In order to investigate the effect of model size on the performance of our approach to measuring semantic change, we try a range of model sizes for BERT on the English datasets, all available from HuggingFace. The results are shown in Table 3. As can be seen, there is a clear correlation between model size and task performance for the SE English data, but this is not the case for the GEMS dataset, perhaps because the COHA corpora used for GEMS provides longer contexts for term mentions (see Appendix C).
We also demonstrate the effect of using a multilingual model, rather than a language specific model, for all datasets other than SE Latin (for which we are already using a multilingual model in the main paper). As can be seen in Table 4, the multilingual model uniformly results in worse performance, demonstrating the importance of having a strong language-specific model for measuring semantic change in this way.

heading: C Exploring Performance Differences Across Languages
text: Using the method presented in the main paper, our results were better than using static word vectors for English and Latin, but worse for German and Swedish. Unfortunately, we do not yet have a satisfactory explanation for this discrepancy in performance. Notably, other approaches using contextual
11In practice, this is done by initially saving the top-(k+ 1) substitutes, and dropping the target term for the purpose of clustering, where necessary.
12https://networkx.org/
embeddings (e.g., Montariol et al., 2021), have also performed worse on these languages (relative to approaches based on Hamilton et al., 2016b).
Several possible explanations suggest themselves for why methods based on contextual embeddings might struggle. For example, tokenization used for these models breaks some words into multiple word pieces, which is not an issue for static embeddings. Another consideration is the amount of context in which the examples occur in the reference corpora (since static vectors typically only use very small context windows, whereas contextual embedding models are capable of using much longer contexts). We might also consider factors relevant to all methods, such as the number of examples given for each target term, or the number of different word forms in which each lemma occurs in the corpora provided.
Although several of these factors perhaps help to explain why performance on English is especially good (relative to static vectors), they do not provide a convincing way to explain the differences in performance observed on the other languages. In particular, the SE English data has the highest proportion of target words that occur in the model vocabulary (without being broken into multiple word pieces), and these lemmas occur in text using the fewest number of surface forms per target.
By contrast, the other languages tend to have more surface forms, on average, with fewer of the target terms occurring in the corresponding model vocabulary, but Swedish is mid-range on the later (with German being lowest). Latin, by contrast, tends to have more examples of target terms per corpus in both time periods (with German again the lowest), but Swedish is between English and Latin. The Swedish model does have a larger vocabulary, but it is not as large as the multilingual model we used for Latin. Quantitative summaries of these factors are presented for reference in Table 5.
Ultimately, perhaps the best explanation has to do with the quality of the underlying pretrained models available for each language. Given that different models for different languages were trained on entirely different data, this seems like a highly relevant source of potential differences. Unfortunately, is it difficult to assess the overall quality of pretrained models across languages, so all of these explanations essentially remain no more than hypotheses for further investigation.

heading: D Additional Results on GEMS
text: The GEMS dataset has been used for evaluation by many additional papers, beyond those discussed in the main body of this paper. However, these have not all used consistent metrics and corpora, making comparison difficult. For completeness, we include additional results here, as shown in Table 6.
The GEMS dataset was originally introduced by Gulordava and Baroni (2011), from whom we obtained the labeled data. These authors reported results in terms of Pearson correlation, and used multiple datasets for measuring semantic change, including the Google Books Corpus. Frermann and Lapata (2016) also used this dataset for evaluation, but used different additional data (beyond COHA), and reported results in terms of Spearman correlation.
More recent papers using this dataset (from Giulianelli et al., 2020 onwards) have tended to make use of the COHA data from the 1960s and 1990s as the corpus in which to measure change, to correspond to the periods used in the annotation process, which we also use for our results in this paper. Martinc et al. (2020b) reported very strong
results on this dataset, but subsequent work from the same authors (Montariol et al., 2021) revealed that this method performed relatively poorly on the SemEval datasets, as reported in Table 1 in the main paper.
Different authors have excluded different numbers of words from the 100 target terms in evaluation. Giulianelli et al. (2020) excluded extracellular due to insufficient occurrences in COHA during the 1960 and 1990s, which we also exclude for the same reason. Martinc et al. (2020b) and Montariol et al. (2021) excluded assay, extracellular, mediaeval, and sulphate because they were split into multiple tokens by BERT. Because we mask the target terms, multi-piece words are not a problem, but for completeness we evaluate using the exclusions of both Giulianelli et al. (2020) and Martinc et al. (2020b) and report both in Table 6.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 5 and the Limitations Section on Page 5
7 A2. Did you discuss any potential risks of your work? Yes, in the Ethical Considerations
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.1
3 B1. Did you cite the creators of artifacts you used? Section 4.1
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We are not creating new artifacts
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We are only using well established resources for their intended purposes
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We are only using well established resources for their intended purposes
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We are only using well established resources for their intended purposes
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Number of words per dataset in Table 1, along with pointers to dataset descriptions
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Partially; we report the computing infrastructure used in Appendix B; the relevant parameter counts are all based on standard models
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Yes, we note that we did not tune the relevant hyperparameters in the Limitations section
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? The datasets that are available are too small to be able to provide meaningful estimates for these.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.2 and Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Future work could explore the impact of restricting measurements to certain parts of speech, both for target and background terms. For GEMS, where the targets are not lemmatized, we ignore lemmatization and simply sample from all exact matches of the target terms as tokens in the raw text. As with past work, we combine the multiple annotations for the GEMS data by averaging their scores. All masked tokens are fed into the appropriate model with up to 50 tokens to either side from the original context, which returns a probability distribution over the model vocabulary. When computing the top-k most probable substitutes, we follow Eyal et al. (2022) and exclude stopwords and partial word pieces (i. e. , those that start with ##). For GEMS and SE English, we use the stopword list from the Snowball stemmer. 8 For SE Latin, we use a Latin stopword list from the Perseus Digital Library. 9 For SE German and SE Swedish, we use the respective stopword lists from NLTK. 10
For the exploration of sense clusters in the main paper using Louvain community detection, we use the same data as used in measuring semantic change, keeping k = 5, but we exclude the target
7https://github. com/dallascard/SBSCD 8http://snowball. tartarus. org/
algorithms/english/stop. txt 9https://www. perseus. tufts. edu/hopper/ stopwords 10https://www. nltk. org/
term itself when gathering the top-k substitutes. 11 We then construct a weighted graph for each target term, where nodes represent substitutes, and edge weights correspond to the number of top-k replacement sets in which each pair of replacements appear together. To obtain sense clusters, we use the implementation of Louvain community detection in networkx with default parameter settings, to detect clusters in the graph. 12 Finally, we associate each instance of a target term with a corresponding cluster using Jaccard similarity between the instance’s set of top-k replacements and the terms in the cluster. All of these experiments were run on either an NVidia RTX A6000 or A5000 GPU