Venue: NAACL 2024

# A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change

Francesco Periti

University of Milan

Via Celoria, 18

20133 Milano, Italy francesco.periti@unimi.it Nina Tahmasebi

University of Gothenburg

Renstromsgatan 6

40530 Goteborg, Sweden nina.tahmasebi@gu.se

###### Abstract

Contextualized embeddings are the preferred tool for modeling Lexical Semantic Change (LSC). Current evaluations typically focus on a specific task known as Graded Change Detection (GCD). However, performance comparison across work are often misleading due to their reliance on diverse settings. In this paper, we evaluate state-of-the-art models and approaches for GCD under equal conditions. We further break the LSC problem into Word-in-Context (WiC) and Word Sense Induction (WSI) tasks, and compare models across these different levels. Our evaluation is performed across different languages on eight available benchmarks for LSC, and shows that (i) APD outperforms other approaches for GCD; (ii) XL-LEXEME outperforms other contextualized models for WiC, WSI, and GCD, while being comparable to GPT-4; (iii) there is a clear need for improving the modeling of word meanings, as well as focus on _how_, _when_, and _why_ these meanings change, rather than solely focusing on the extent of semantic change.

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4262–4282 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Lexical Semantic Change (LSC) is the problem of automatically identifying words that change their meaning over time Montanelli and Periti (2023); Tahmasebi et al. (2021); Kutuzov et al. (2018); Tang (2018). The interest in this problem has been significantly fueled by the advent of word embeddings and modern language models. After more than a decade of ad hoc evaluation, a new evaluation framework was recently introduced, aimed at assessing and comparing the performance of different models and approaches Schlechtweg et al. (2020). This framework was adopted to create benchmarks in different languages. Each benchmark includes a diachronic corpus spanning two time periods, along with a list of target words and tasks aimed at detecting word meaning change over time. The most popular task, known as Graded Change Detection (GCD), consists of ranking a list of target words based on their degree of change.

The initial excitement for word embeddings prompted researchers and practitioners to solve the GCN task by using static embedding models Schlechtweg et al. (2020); Shoemark et al. (2019). However, the shift towards more advanced Transformer architectures has established the use of contextualized embedding models as the preferred tool for addressing GCD Montanelli and Periti (2023); Kutuzov et al. (2022b). On one hand, these models distinguish the different meanings of a word by contextualizing each occurrence with a different embedding. On the other hand, the generation and processing of contextualized embeddings across entire corpora pose scalability challenges, both in terms of time and memory consumption Periti et al. (2022); Montariol et al. (2021). Different strategies have been adopted to tackle these challenges, leading to a proliferation of evaluations across diverse settings (e.g., limited samples of benchmarks) and conditions (e.g., pre-trained vs. fine-tuned models). As a result, these evaluations on GCD hinder a fair comparison among the performance of different models and approaches, thereby deviating from the original goal of the framework.

Moreover, while the GCD task is attracting more and more evaluations, it addresses only a partial complexity inherent to the established framework. Notably, the framework includes three distinct aspects also discussed in Schlechtweg et al. (2024):

* **semantic proximity judgments** of word _inn context_,
* **word sense induction** based on proximity judgments,
* **quantification of semantic change** from induced senses.



As a matter of fact, when contextualized embedding models are used to address GCD, cosine similarities among word embeddings serve as surrogate for **(i)**, without evaluation focused on this aspect. Additionally, most approaches to GCD, pass from **(i)** to **(iii)**, sidestepping the intermediate aspect **(ii)**. That is, they quantify semantic change as overall proximity variation, without inducing word senses. Consequently, while these approaches can be evaluated through GDC, they preclude the interpretation of which meaning(s) have changed.

We argue that **(i)** and **(ii)** are equally relevant aspects as **(iii)**, constituting a fundamental aspect of the LSC problem. Their evaluation can provide valuable insights into the current state of LSC modeling, while offering a broader perspective on contextualized embedding models in Natural Language Processing (NLP).1

Footnote 1: Software is available at [https://github.com/FrancescoPeriti/CSSDetection](https://github.com/FrancescoPeriti/CSSDetection).

Original contribution of our work
* We systematically evaluate and compare various models and approaches for GCD under equal settings and conditions. Our evaluation for GCD spans eight different languages. Importantly, we perform the first evaluation over Chinese and the second evaluation for Norwegian within the existing literature. Our results show superior performance of the recent state-of-the-art model for GCD, namely XL-LEXEME, over various approaches.
* We are the first to evaluate contextualized embedding models for **(i)** and **(ii)** within the existing literature. Our evaluation of **(i)** and **(ii)** relies on two well-known tasks in NLP, namely Word-in-Context (WiC), and Word Sense Induction (WSI). Importantly, we evaluate various models as _computational annotators_.
* We compare GPT-4 to contextualized models through the WiC, WSI, and GCD tasks. Our evaluation reveals that GPT-4 obtains comparable performance to XL-LEXEME. In contrast to the limited accessibility2 and high associated cost3 of GPT-4, XL-LEXEME is a considerably smaller, open-source model. Thus, we argue that the use of GPT-4 is not justified for modeling the LSC problem. Footnote 2: [https://platform.openai.com/docs/guides/rate-limits](https://platform.openai.com/docs/guides/rate-limits)

Footnote 3: [https://openai.com/pricing](https://openai.com/pricing)

## 2 Background and related work

The established LSC framework adheres to the novel annotation paradigm for word senses and encompasses **(i-iii)**Schlechtweg et al. (2021). **(i)** Human annotators provide semantic proximity judgments for pairs of word usages _sampled_ from a diachronic corpus spanning two time periods. **(ii)** Word usages and judgments are represented as nodes and edges in a weighted, _diachronic_ graph, known as Diachronic Word Usage Graph (DWUG). This graph is then clustered with a graph clustering algorithm and the resulting clusters are interpreted as word senses (see Figure 1), thus sidestepping the need for explicit word sense definitions. Finally, **(iii)** given a word, a ground truth score of semantic change is computed by comparing the probability distributions of clusters in different time periods, e.g., a cluster with most of its usages from one time period indicates a substantial semantic change.

Originally, the framework was proposed in a shared task at SemEval-2020, including benchmarks for four languages, namely English (EN), German (DE), Swedish (SW), and Latin (LA) Schlechtweg et al. (2020). Benchmarks for Italian Basile et al. (2020), Russian (RU) Kutuzov and Pivovarova (2021b, c), Spanish (SP) Zamorae Reina et al. (2022b), Norwegian (NO) Kutuzov et al. (2022a), and Chinese (ZH) Chen et al. (2023a, 2022) have recently been introduced. Each benchmark4 consists of a diachronic corpus and a set of target words over which the human annotation was conducted. The evaluation over a benchmark is typically conducted through the GCD task where the goal is to rank the targets by degree of semantic change across the corpus. The Spearman correlation between _predicted_ and _ground truth scores is used to evaluate models and approaches.

Figure 1: DWUG for the German word _Etintagsfliese_. Nodes represent word usages. Edges represent the relatedness between usages. Colors indicate clusters (senses) inferred from the full graph Laicher et al. (2021).

 



### Approaches to Graded Change Detection

GCD is typically addressed using two kinds of approaches for modeling word meanings: _form-_ and _sense_-based Montanelli and Periti (2023); Giulianelli et al. (2020). The former capture signals of change by analysing how the dominant meaning, or the degree of polysemy of a word, changes over time (e.g., Giulianelli et al. (2020); Martinc et al. (2020)). The latter cluster word usages according to their meanings and then estimate the semantic change of a word by comparing the cluster distribution of its usages over time (e.g., Periti et al. (2023); Martinc et al. (2020)). Form- and sense-based approaches can be further distinguished into _supervised_, which leverage external knowledge (e.g., dictionaries, Rachinskiy and Arefyev (2022) or other forms of supervision (e.g., Word-in-Context datasets, Cassotti et al. (2023), and _unsupervised_, which rely solely on the knowledge encoded in pretrained models (e.g., Aida and Bollegala (2023).

### Comparison of approaches

Models and approaches for GCD have been evaluated under different settings and conditions. For example, some studies utilized the _entire_ diachronic corpus to estimate the change of each target (e.g., Periti et al. (2022), while others relied on smaller _samples_(e.g., Rodina et al. (2021), or solely on the annotated word usages (e.g., Laicher et al. (2021). Also, different versions of the ground truth are used (e.g., Schlechtweg et al. (2022). In the current literature, some studies fine-tune the models on the corpus (e.g., Rosin et al. (2022), while others directly use pre-trained models (e.g., Kudisov and Arefyev (2022). Performance comparison are conducted across different models such as BERT (e.g., Laicher et al. (2021), mBERT (e.g., Beck (2020), and XLM-R (e.g., Giulianelli et al. (2022). However, even when the same model is employed, different layer aggregations are used, such as concatenating the output of the last four encoder layers (e.g., Kanjirangat et al. (2020), or summing the output of all the encoder layers (e.g., Giulianelli et al. (2022). Moreover, sense-based approaches are compared with different clustering algorithms such as Affinity Propagation (e.g., Martinez et al. (2020), A Posteriori affinity Propagation (e.g., Periti et al. (2022), and K-Means (e.g., Montariol et al. (2021).

As a results, comparing Spearman correlation across different evaluations is often **misleading**.

### Current modeling of LSC

Current modeling of LSC overlooks the procedure **(i-iii)** used to generate the ground truth. Mostly, only **(iii)** is evaluated by relying on form-based approaches. However, these approaches capture only the _degree_ of semantic change, preventing its interpretation. Sense-based approaches could fill this gap by explaining _how_ and _what_ has changed, but currently suffer from lower performance on **(iii)** and are therefore less pursued. As a results, it is not clear which meanings these models and approaches are capturing. There is thus a need to carefully evaluate their ability in both **(i)** and **(ii)**.

Thus far, this evaluation is missing. To the best of our knowledge, only Laicher et al. (2021) evaluate **(ii)** through the WSI task. This evaluation needs to be extended beyond a single model, using the same procedure used to generate the ground truth.

A systematic comparisonunder equal settings and conditions is necessary to evaluate different models and approaches. Thus, we first evaluate standard form- and sense-based approaches to provide a fair performance comparison on GCD across eight languages. We then assess different models as _computational annotators_ by evaluating them on **(i-iii)** through WiC, WSI, and GCD. Aligning with Karjus (2023), if computational models perform close to human-level, their usage would represent an unprecedented opportunity to scale up semantic change studies in the humanities and social sciences.

## 3 Evaluation setup

We consider benchmarks for eight different languages: EN, LA, DE, SV, ES, RU, NO, and ZH (see Table 6). For each benchmark, we evaluate four different models: BERT Devlin et al. (2019), mBERT, XLM-R Conneau et al. (2020), and XL-LEXEME Cassotti et al. (2023). BERT is a monolingual model, mBERT, XLM-R, and XL-LEXEME are multilingual models. BERT, mBERT, and XLM-R are pre-trained masked language models. XL-LEXEME is a fine-tuned XLM-R model that leverages the SBERT architecture to solve the WiC task, thus serving as a WiC pre-trained model.

Aligning with the _unsupervised_ nature of the LSC framework, we compare pre-trained models without performing additional fine-tuning (see Table 7). For each model and each target word in a benchmark, we collect contextualized embeddings for all its word usages in both time periods

Specifically, we generate the sets of embeddings \(\Phi^{1}=\{a_{1},...,a_{n}\}\) and \(\Phi^{2}=\{b_{1},...,b_{m}\}\) for the word usages associated to time periods \(t_{1}\) and \(t_{2}\), respectively.

### Standard Graded Change Detection

We compare the use of different models with four standard approaches to GCD, specifically two form-based and two sense-based. Similar to Laicher et al. (2021), we consider the raw data originally used to derive ground truth scores, instead of considering the associated corpora. This ensures an accurate evaluation under a controlled setting.

### Computational annotators

We assess different models as computational annotators by using cosine similarities between embeddings as a surrogate of human judgments. In our evaluation, we consider word usage pairs where human judgments are available, instead of considering all potential usage pairs (as in Section 3.1). Specifically, we adhere to the framework **(i-iii)** and evaluate different models through the WiC, WSI, and GCD tasks.

Inspired by Periti et al. (2024); Laskar et al. (2023); Kocon et al. (2023); Karjus (2023), we evaluate GPT-4 and compare its use to contextualized models. However, the limited accessibility and high associated cost constraint our extension only to the EN benchmark.

## 4 Comparing approaches for GCD

We evaluate different approaches for GCD using Spearman correlation Spearman (1904) between computational predictions and ground truth scores. Specifically, we process the embeddings of each target using the following approaches.

### Form-based approaches

In the most recent survey on LSC by Montanelli and Periti (2023), it was observed that cosine distance over word prototype (PRT) and the average pairwise distance (APD) consistently demonstrated superior performance compared to alternative approaches. Thus, we employ these approaches:

Prtcomputes the degree of change of a word \(w\) as the cosine distance between the average embeddings \(\mu_{1}\) and \(\mu_{2}\) (also know as _prototype_ embeddings) of \(w\) in the time periods \(t_{1}\) and \(t_{2}\)Martinc et al. (2020); Kutuzov and Giulianelli (2020). Formally, given a word \(w\), we compute its degree of change by computing:

\[\text{PRT}(\Phi^{1},\Phi^{2})=1-cosine(\mu_{1},\mu_{2}) \tag{1}\]

The intuition behind PRT is that a prototype embedding encodes the dominant meaning of a word, and as such, the semantic change is computed as a shift in the dominant meaning over time.

Apdcomputes the degree of change of a word \(w\) as the average pairwise distance between the word embeddings in \(\Phi^{1}\) and \(\Phi^{2}\)Giulianelli et al. (2020); Kutuzov and Giulianelli (2020). Formally, given a word \(w\), we compute its degree of change, where \(d\) is cosine distance, as follows:

\[\text{APD}(\Phi^{1},\Phi^{2})=\frac{1}{|\Phi^{1}||\Phi^{2}|}~{}\cdot\sum_{a\in \leavevmode\nobreak\ \Phi^{1},\leavevmode\nobreak\ b\in\leavevmode\nobreak\ \Phi^{2}}d(a,b) \tag{2}\]

The intuition behind APD is that different word embeddings encode the polysemy of a word, and as such, the semantic change is computed as a shift in the word's degree of polysemy.

### Sense-based approaches

We choose two state-of-the-art sense-based approaches Montanelli and Periti (2023). The first utilizes the unsupervised clustering algorithm Affinity Propagation (AP, Frey and Dueck, 2007) combined with the Jensen Shannon divergence (JSD, Lin, 1991). Additionally, we employ the evolutionary extension of Affinity Propagation, called A Posterior affinity Propagation (APP, Castano et al., 2024), combined with the average pairwise distances between sense prototypes (APDP). This approach is called WiDiD (Periti et al., 2022).

Apr+jsd leverages the AP clustering to distinguish the different contextual usages of a given word \(w\). Specifically, the embeddings \(\Phi^{1}\), and \(\Phi^{2}\) are _collectively_ clustered to generate clusters comprising embeddings from both time periods (i.e., \(t_{1}\) and \(t_{2}\)), or embeddings exclusive from a time period (i.e., \(t_{1}\) or \(t_{2}\)). The semantic change of \(w\) is computed as the JSD between the probability distributions \(p_{1}\) and \(p_{2}\) of clusters in time periods \(t_{1}\) and \(t_{2}\). These distributions represent the relative number of embeddings from \(\Phi^{1}\) and \(\Phi^{2}\) grouped in each cluster, respectively Martinc et al. (2020). Formally, the degree of semantic change is:

\[\text{JSD}(p_{1},p_{2})=\frac{1}{2}\left(KL(p_{1}||M)+KL(p_{2}||M)\right)\] (3) 

where \(KL\) stands for Kullback-Leibler divergence and \(M=\frac{(p^{1}+p^{2})}{2}\). The intuition behind AP+JSD is that different clusters encode nuanced word meanings, and as such, the semantic change is computed as an overall measure of the differences in the prominence of each sense over time.

WiDiDleverages the APP clustering to distinguish the usages of a given word \(w\). Specifically, the embeddings \(\Phi^{1}\), and \(\Phi^{2}\) are _individually_ clustered to generate incremental clusters of embeddings that evolve with each clustering iteration. The semantic change of \(w\) is computed as the average pairwise distances between the _sense prototypes_\(\Psi^{1}\) and \(\Psi^{2}\) of \(w\) in the time periods \(t_{1}\) and \(t_{2}\), where \(\Psi^{1}\) and \(\Psi^{2}\) are the set of embeddings obtained by averaging the embeddings \(\Phi^{1}\) and \(\Phi^{2}\) in each cluster, respectively [30; 4]. Formally, given a word \(w\), the degree of semantic change is computed as follows:5

Footnote 5: Following Periti et al. (2023), we use the Canberra distance instead of the cosine distance

\[\text{APDP}(\Phi^{1},\Phi^{2})=\text{APD}(\Psi^{1},\Psi^{2}) \tag{4}\]

The intuition behind WiDiD is similar to AP+JSD. However, while the latter considers change as the difference between the amount of probability for a sense over time, WiDiD is similar to APD in computing the shift in prototypical word meanings.

### Evaluation results - Table 1

We present the results of our evaluation in Table 1 for both form- and sense-based approaches. For the sake of comparison, we include state-of-the-art (SOTA) results in Table 5.6 As a general remark, we note instances where our results surpass SOTA (e.g., XL-LEXEME+APD for EN). We attribute this to the controlled setting established in our experiments. We note also instances where our results are lower than SOTA (e.g., BERT+APD for SV). This discrepancy may be influenced by various factors such as _different versions_ of the benchmarks (e.g., 37 vs 46 targets for EN in DWUG version 2.0.1, Schlechtweg et al., 2020). Additionally, _variations in text pre-processing_ can play a beneficial role. For instance, Laicher et al. (2021) demonstrate the effectiveness of lemmatization to mitigate word form biases, while Martinc et al. (2020c) suggest that filtering Named Entities can help models avoid inflating semantic change. Moreover, some studies _fine-tune or utilize different embedding layers_, whereas we adhere to the standard, generally adopted procedures without fine-tuning, considering embeddings generated from the last (i.e., 12th) layer of the models. Finally, there are sometimes significantly different results reported by different studies under similar conditions. For instance, Zhou et al. (2023) achieve a correlation of .706 using pre-trained BERT and APD, whereas others typically report correlations ranging between .400 and .600 (e.g., .489, Keidar et al., 2022; .514, Giulianelli et al., 2020; .546, Kutuzov and Giulianelli, 2020; .571, Laicher et al., 2021). This disparity cannot currently be explained.

Languages.We obtain strong correlations with all benchmarks but LA. Our results show a _weighted average_ correlation of **.751** when employing XL-LEXEME + APD. In this calculation, we assign weights based on the number of targets in each benchmark, considering larger sets more reliable than smaller ones. For LA, it can be argued that the models were not directly tailored or fine-tuned for Latin. However, XL-LEXEME demonstrates optimal performance in GCD in SV and medium performance in SP and NO without specific training on either [20]. This leads us to consider that the quality of the LA benchmark potentially is lower than other benchmarks, as it was developed using a different procedure [30].

Form-based vs Sense-based.We note that formulated approaches significantly outperform sense-based approaches. Our results consistently highlight APD as the most effective approach, regardless of the skewness in the distribution of judgments, as previously argued by Kutuzov and Giulianelli (2020). In addition, WiDiD consistently demonstrate superior performance over AP+JSD. This can be attributed to the use of i) an evolutionary clustering algorithm, which enables to consider the time dimension of text in a dynamic way; or, alternatively ii) APD over sense-prototypes, as APD has demonstrated high effectiveness.

Our **laderboard** is as follows: APD, PRT, WiDiD, AP+JSD. Although form-based approaches exhibit superior effectiveness, they fall short in capturing word meanings and interpreting detected semantic changes. In contrast, although sense-based approaches theoretically facilitate such modeling and interpretation, they obtain poor results in GCD, 

raising concerns about their reliability and whether they capture meaningful patterns or produce noisy aggregation. We will investigate this in Section 5.

Supervised vs Unsupervised.We note that the use of supervision significantly improves the modeling of semantic change for both form- and sense-based approaches. While Cassotti et al. (2023) have previously evaluated XL-LEXEME + APD, we extend the evaluation to sense-based approaches, demonstrating that _supervision_ enhances the performance of AP+JSD and WiDiD.

Models.We note that the use of XL-LEXEME significantly improves the modeling of LSC compared to standard BERT, mBERT, and XLM-R. However, we observe a pattern in performance, indicating that on average, BERT performs better than mBERT, which, in turn, performs better than XLM-R for form-based approaches. This suggests that the use of XLM-R models is not more effective than BERT models for LSC, confirming the medium-low correlation coefficients obtained by Giulianelli et al. (2022) using XLM-R.

Layers.As different works employ different embedding layers, we repeat our evaluation by considering embeddings generated by each layer of BERT, mBERT, and XLM-R (see Appendix C). Our evaluation aligns with recent findings on other downstream tasks Ma et al. (2019); Reif et al. (2019); Liang and Shi (2023) and shows that using early layers consistently in higher performance. For example, we note a correlation of \(.747\) for ZH by using layer 4, compared to \(.656\) obtained by using the last layer of BERT. On average, and in line with Periti and Dubossarsky (2023), we find that the best results for each language are obtained by leveraging embeddings from layers \(8-10\).

Furthermore, since previous studies aggregated outputs from different layers, we also use aggregated embeddings extracted from different layers through sum and concatenation (see Appendix C). Specifically, our evaluation covers all possible layer combinations with lengths of 2 (e.g., layers 1 and 2), 3 (e.g., layers 6, 7, and 8), and 4 (e.g., layers 9, 10, 11, 12). We find no improvement in aggregating the output of the last four layers for addressing GCD. By employing alternative layer combinations, we obtain higher correlation compared to both the last layer and the last four layers. For instance, for EN, using the sum of layers 2, 4, 5, and 8 for APD+BERT, or the concatenation of layers 4, 5, 6, and 11 for WiDiD+BERT, results in correlation of \(.692\) and \(.760\), respectively; compared to \(.563\) (APD) and \(.385\) (WiDiD) by using the last BERT layer. However, no combination consistently emerges as the optimal choice across various benchmarks or models. Instead, we observe that using a middle layer, such as layer 8, tends to be advantageous across benchmarks and models compared to the last layer or the aggregation of the last

\begin{table}
\begin{tabular}{|c|c|r r r|r r r|r r r r|r r r|}  & **EN** & **LA** & **DE** & **SV** & **ES** & **RU** & \multicolumn{1}{l|}{**NOomi**} & **ELi** & **Aviz.** \\ \cline{8-13} \multirow{8}{*}{**APD**} & & \(C_{1}-C_{2}-C_{3}\) & \(C_{1}-C_{2}-C_{3}\) & \(C_{1}-C_{2}-C_{3}\) & \(C_{1}-C_{2}-C_{3}\) & \(C_{1}-C_{2}-C_{3}\) & \(C_{1}-C_{2}-C_{3}\) & \(C_{1}-C_{3}\) & \(C_{1}-C_{2}-C_{3}\) & \(C_{1}-C_{2}-C_{3}\) & \(C_{1}-C_{2}-C_{3}\) & \(C_{1}-C_{2}\) & \(C_{1}-C_{3}\) & \(C_{1}-C_{2}\) & \(C_{1}-C_{3}\) & \(C_{1}-C_{2}\) \\ \hline \multirow{8}{*}{**PRT**} & BERT & 363 & . & 2. & 2. & 2. & 3. & 3. & 518 & . & 4. & 4. & 4. & 4. & 4. & 6. & 6. & 4. \\  & mBERT & 363 & . & 1.02 & . & 3. & 3. & 3. & 3. & 3. & 3. & 3. & 3. & 6. & 5. & 3. & 3. \\  & xLF-LREME & **. & **. & **. & 2. & 2. & 2. & 3.

 

four layers (see Figure 2 and 3).

## 5 Computational annotation

We evaluate different models on reproducing human judgments **(i)**, the inferred word senses **(ii)**, and the resulting change scores **(iii)**.

We leverage models as annotators, hence the term _computational annotator_, using the same procedure employed for benchmark construction Schlechtweg et al. (2021); Schlechtweg and Schulte im Walde (2020); Schlechtweg et al. (2018). However, we cannot evaluate LA as the benchmark was developed differently nor **(ii)** for the RU benchmark since no word senses were provided Kutuzov and Pivovarova (2021b, c).

### **(i) - Word-in-Context**

Given a benchmark, a word usage pair is associated with two contexts, \(c_{1}\) and \(c_{2}\), along with the average judgment of multiple annotators (see Example A). We thus use the cosine similarity between the embeddings of \(w\) in the contexts \(c_{1}\) and \(c_{2}\) as computational proximity judgement.

Our evaluation is grounded in the Word-in-Context (WiC) task Loureiro et al. (2022); Raganato et al. (2020); Pilehvar and Camacho-Collados (2019). In contrast to the original WiC definition, our WiC evaluation aligns with the continuous framework introduced by Armendariz et al. (2020) in the Graded Word Similarity in Context task. Specifically, we evaluate the quality of computational predictions by computing the Spearman correlation with human judgments.

### **(ii) - Word Sense Induction**

We first create a DWUG using the computational annotations in Section 5.1. Then, we derive sense clusters through a variation of correlation clustering Bansal et al. (2004) on the DWUG.

Our evaluation is grounded in the Word Sense Induction (WSI) task Aksenova et al. (2022); Manandhar et al. (2010); Agirre and Soroa (2007). We evaluate the quality of clusters from computationally annotated DWUGs against clusters from human-annotated DWUGs. Specifically, we use Adjusted Rand Index (ARI, Hubert and Arabie (1985) and Purity (PUR, Manning (2009) as metrics to quantify the cluster agreement. ARI comprehensively evaluates the similarity among clustering result. However, it may yield low scores when a clustering result contains numerous small, yet coherent clusters. This does not necessarily indicate poor clustering quality, especially when the clusters are semantically meaningful. PUR assigns each cluster to the class that is most frequent in the cluster, measuring the accuracy of this assignment by counting the relative number of correctly assigned elements.

### **(iii) - Graded Change Detection**

Given a word \(w\), we split its DWUG into two subgraphs representing nodes from the two time periods (see Figure 1) and quantify the semantic change of \(w\) by computing the \(\sqrt{JSD}\) between the two time-specific cluster distributions. In contrast, for RU, we adhere to the RuShiftEval procedure and quantify semantic change through the application of the COMPARE metric that directly measures the

\begin{table}
\begin{tabular}{|r|l||c|c|c|c|c|c|c|c|c|c|c|}  & \multicolumn{2}{c|}{**EN**} & \multicolumn{2}{c|}{**DE**} & \multicolumn{2}{c|}{**SV**} & \multicolumn{2}{c|}{**ES**} & \multicolumn{2}{c|}{**RU**} & \multicolumn{2}{c|}{**NO**} & \multicolumn{2}{c|}{**ZII**} & \multicolumn{2}{c|}{**Avg.**} \\ \cline{6-13} \multirow{6}{*}{} & \multicolumn{2}{c|}{\(C_{1}-C_{2}\)} & \(C_{1}-C_{2}\) & \(C_{1}-C_{2}\) & \(C_{1}-C_{2}\) & \(C_{1}-C_{2}\) & \(C_{2}-C_{3}\) & \(C_{1}-C_{3}\) & \(C_{1}-C_{3}\) & \(C_{1}-C_{2}\) & \(C_{2}-C_{3}\) & \(C_{1}-C_{2}\) & \(C_{1}-C_{2}\) & \(C_{1}-C_{2}\) & \(C_{1}-C_{2}\) & \(C_{1}-C_{2}\) \\ \cline{2-13}  & BERT & .503 & .350 & .221 & .319 & .314 & .344 & .350 & .429 & .406 & .516 & .358 \\  & mBERT & .332 & .344 & .284 & .289 & .280 & .273 & .293 & .283 & .333 & .413 & .301 \\  & XLM-R & .352 & .289 & .255 & .288 & .212 & .250 & .251 & .317 & .261 & .392 & .272 \\  & XL-LEXME & **.626** & **.628** & **.631** & **.547** & **.549** & **.558** & **.564** & **.484** & **.521** & **.630** & **.568** \\  & GPT+4.0 & .606 & - & - & - & - & - & - & - & - & - & - & - \\  & _Agreement_ & _.633_ & _.666_ & _.672_ & _.531_ & _.567_ & _.564_ & _.761_ & _.667_ & _.662_ & _.691_ \\ \hline \multirow{6}{*}{} & BERT & .316 /.700 & .047 /.662 & .023 /.596 & .189 /.695 & - & _- & _- & - & .251 /.771 & .247 /.758 & .279 /.759 & .166 /.702 \\  & mBERT & .067 /.644 & .054 /.679 & .024 /.648 & .288 /.700 & - & _- & - & - & .241 /.759 & .159 /.753 & .173 /.712 & .114 /.696 \\  & XLM-R & .068 /.737 & .024 /.725 & .031 /.680 & .164 /.755 & - & _- & - & - & **.17** & **.179** & **.775** & **.131** & **.729** \\  & XL-LEXEME & .273 /.334 & **.300** /.**788** & **.249** /.**766** & **.400** /.**820** & - & _- & - & - & **.337** & **.306** & **.304** /.808** & **.448** & **.336** & **.339** /.810** \\  & GPT+4.0 & **.340** /.**877** & _- & _- & _- & - & - & - & - & - & - & - & - \\ \hline \multirow{6}{*}{} & BERT & .425 & .116 & .148 & .284 & .487 & .452 & .469 & .571 & .521 & **.808** & .422 \\  & mBERT & .120 & .205 & .234 & .394 & .372 & .325 & .408 & .290 & .454 & .737 & .357 \\ \cline{1-1}  & XLM-R & .219 & .069 & .143 & .464 & .284 & .301 & .375 & .395 & .345 & .557 & .324 \\ \cline{1-1}  & XL-LENEXME & .801 & **.799** & **.721** & **.655** & **.780** & **.824** & **.851** & **.620** & **.567** & .716 & **.754** \\ \cline{1-1}  & GPT+4.0 & **.818** & - & - & - & - & - & - & - & - & - & - \\ \end{tabular}
\end{table}
Table 2: **Evaluation of contextualized models as computational annotators**: Spearman correlation for WiC and GCD, Adjusted Random Index and Purity (ARI / PUR) for WSI. Top score for each approach and benchmark is highlighted in **bold**. Avg is a weighted average based on the number of targets in each benchmark test set. For the sake of comparison, we report the Krippendorff’s \(\alpha\) score for inter-human annotator _agreement_ in WiC (_italic_).

 

mean relatedness of annotated word usage pairs as semantic change scores (Schlechtweg et al., 2018). Our evaluation is based on the GCD task and thus use Spearman correlation as evaluation metric between predicted ranking and ground truth rankings.

### Evaluation results - Table 2

* Word-in-Context** Our evaluation reveals that pre-trained models such as BERT, mBERT, and XLM-R demonstrate a low average correlation with human judgments (.358, .301, .272). In contrast, XL-LEXEME and GPT-4 emerge as powerful solutions for scaling up and aiding human annotations. For EN, they obtain a moderately strong correlation (.626, .606) with human judgments, only marginally lower than the Krippendorf \(\alpha\) human agreement (.633). In particular, XL-LEXEME slightly outperforms a considerably larger model like GPT-4 in terms of parameters, at a considerable lower cost. In contrast to previous cross-lingual evaluation (Conneau et al., 2020) and in line with the finding in Table 1, mBERT consistently outperforms XLM-R. However, our results highlight the advantageous use of monolingual BERT models over the multilingual ones, for assessing **(i)**

We consider the WiC evaluation to be the most valuable as it involves a direct comparison between computational predictions and human judgments.

* Word Sense Induction** Our evaluation indicates that moderate performance in **(i)**-WiC leads to moderately _low_ performance in inferring word sense. We obtain low ARI scores across all models and benchmarks, with XL-LEXEME and GPT-4 exhibiting the highest values. Specifically, GPT-4 outperforms XL-LEXEME (with .340 compared to .273) in ARI for EN. However, we highlight that even such low scores represent a moderately _high_ result, given an inter-annotator agreement of .633.

XL-LEXEME consistently demonstrates high PUR scores across all benchmarks, while other models yield slightly lower PUR scores, suggesting that some word sense patterns are captured when using contextualized models. Previous studies highlight that contextualized models tend to produce a large number of clusters (Martinc et al., 2020; Periti et al., 2022), thereby influencing PUR scores. Therefore, it is crucial to interpret PUR in conjunction with ARI.

* Graded Change Detection** As for GCD, we obtain average results for BERT, mBERT, XLM-R, and XL-LEXEME equal to .422, .357, .324, .754, respectively. These results are consistent with those presented in Table 1, when compared to form-based approaches (.316
- .751). We observe that employing more word usage pairs, as in Table 1, proves beneficial for certain benchmarks in the GCD tasks (e.g., XL-LEXEME+APD for EN and DE). However, we note that these results for **(ii)**
- WSI are significantly higher to those obtained by sense-based approaches (.077
- .422). This can likely be attributed the fact that here we are using the same clustering algorithm that was used for obtaining the ground truth clusters, or to the fact that the clustering algorithm is more able to capture nuanced word meaning than AP and APP. In contrast, for RU, following the RuShiftEval procedure does not improve the performance and results between Table 1 and 2 are somewhat comparable.

## 6 Concluding remarks

We have performed a first-ever evaluation of models and approaches for modeling LSC under equal settings and conditions, over eight different languages. First, we evaluated different models combined with standard approaches to the popular GCD task. In particular, we consider BERT, mBERT, XLM-R, XL-LEXEME as pre-trained models, APD and PRT as form-based approaches, and AP+JSD and WiDiD as sense-based approaches. We find that the XL-LEXEME consistently outperforms other models across all approaches, and thus should be used as the defacto standard. We also find that form-based approaches significantly outperform sense-based approaches, with APD as the best approach for GCD. Among the sense-based approaches, we find that _evolutionary_ clustering is advantageous in contrast to static clustering and should be a focus of future work. We additionally extended the evaluation to includes the WiC and WSI tasks, both inherently crucial to solve the complex task of LSC. We compare GPT-4 to the previous models and find that GPT-4 and XL-LEXEME both perform close to human-level while the other models obtain only low-moderate performance. Due to the costs associated with using GPT-4, it is not affordable to evaluate it on the remaining languages. Since XL-LEXEME obtains results close those of GPT-4, even beating it for the WiC task, we argue that XL-LEXEME can be used for LSC tasks as a affordable, scalable solution.

All in all, considering the current state of the LSC modeling, we argue that **only obtaining state 

of-the-art performance on GCD does not solve the LSC problem, as there is a clear need to **distinguish the different senses of a word and how these evolve over time**[145, 203, 9]. GCD maintains relevance for identifying words that have changed across multiple time periods in need of further _sense-based_ modeling. GCD also serves to quantify the change on the level of vocabulary. In conclusion, we offer a first comparable evaluation of contextualized word embeddings for LSC and establish clear settings that should be used for future comparison and evaluation. With this work, we want to raise awareness of the current trend of the community in modeling only the GCD task. Our aim is to shift the focus from merely assessing _how much_ to _how_[76], _when_, and _why_, prompting the development of both _unsupervised_ and _supervised_ approaches for addressing the full spectrum of LSC.

## 7 Limitations

There are limitations we had to consider in the making of this paper. Firstly, we could not evaluate GPT-4 across all languages due to both price and API limitations. This means that while the results are comparable with XL-LEXEME for EN, we do not know how GPT-4 will behave for the other languages. Although we are aware of open source solution such as LLaMA, our initial experiments, revealed that its performance does not match that of GPT-4. As LLaMA still necessitates expensive research infrastructure, we chose to focus only GPT-4. Our decision to use GPT-4 over the cheaper GPT-3 is based on recent studies showing conflicting results across different tasks. Notably, Karjus (2023) reported high scores for GPT-4 in the GCD task. However, Periti et al. (2024); Laskar et al. (2023); Kocon et al. (2023) reported low scores for the WiC task when employing GPT-3. As a result, we opted for GPT-4 to ensure relevance and accuracy in our evaluations.

Since the instruction-tuning datasets of OpenAI models are unknown, the datasets used for evaluation may or may not be part of the instruction-tuning training data of OpenAI. We also acknowledge that OpenAI continues to train and release new models, which could potentially affect the reproducibility of our results, as well as invalidate future evaluations Balloccu et al. (2024).

In this paper, we evaluate different contextualized models utilizing the popular Transformers library for deep learning maintained by Hugging Face Wolf et al. (2020). We specifically excluded the evaluation of a BERT model for Latin, opting instead to focus on mBERT, XLM-R, and XL-LEXME. At the beginning of our evaluation, we were not aware of any experiments using Latin BERT models to address GCD, nor were we aware of an open BERT version for Latin on the Hugging Face platform. As we have only recently become aware of novel BERT models that are exclusively trained and fine-tuned for Latin Riemenschneider and Frank (2023); Lendvai and Wick (2022), we plan to further test and utilize these models in our future work.

To make a fair comparison between different contextualized models, we employed the same procedure across all benchmarks and languages. However, different languages have different structures and hence different requirements. It would be equally fair to have different processing of the different benchmarks (e.g., lemmatization for German, Laicher et al., 2021). We opted to reduce the number of open variables to be able to make this first evaluation. Future work could optimize each language and then compare model performance.

Lastly, the models compared in this study, despite sharing similar architectures, tokenize text sequences differently based on their reference vocabulary. Consequently, a word may be split into different subtokens by one model and represented as a single token by another. Additionally, when contexts exceed the maximum input size, different models may truncate them at various points. Adhering to standard procedures in the field of LSC, we use the average embeddings of sub-words when a word is split into multiple sub-words. However, the impact of different truncation methods was not evaluated.

This work has in part been funded by the project Towards Computational Lexical Semantic Change Detection supported by the Swedish Research Council (2019-2022; contract 2018-01184), and in part by the research program Change is Key! supported by Riksbankens Jubileumsfond (under reference number M21-0021). The computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through 

grant agreement no. 2022-06725.

## References

* Agire and Soroa (2007) Eneko Agirre and Aitor Soroa. 2007. SemEval-2007 Task 02: Evaluating Word Sense Induction and Discrimination Systems. In _Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)_, pages 7-12, Prague, Czech Republic. Association for Computational Linguistics.
* Predicting the Semantic Changes in Words across Corpora by Context Swapping. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 7753-7772, Singapore. Association for Computational Linguistics.
* Aksenova et al. (2022) Anna Aksenova, Ekaterina Gavrishina, Elisei Rykov, and Andrey Kutuzov. 2022. RuDSI: Graph-based Word Sense Induction Dataset for Russian. In _Proceedings of TextGraphs-16: Graph-based Methods for Natural Language Processing_, pages 77-88, Gyeongju, Republic of Korea. Association for Computational Linguistics.
* Armendariz et al. (2020) Carlos Santos Armendariz, Matthew Purver, Senja Pollak, Nikola Ljubesic, Matej Ulcar, Ivan Vulic, and Mohammad Taher Pilehvar. 2020. SemEval-2020 Task 3: Graded Word Similarity in Context. In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_, pages 36-49, Barcelona (online). International Committee for Computational Linguistics.
* Balloccu et al. (2024) Simone Balloccu, Patricia Schmidtova, Mateusz Lango, and Ondrej Dusek. 2024. Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 67-93, St. Julian's, Malta. Association for Computational Linguistics.
* Bansal et al. (2004) Nikhil Bansal, Avrim Blum, and Shuchi Chawla. 2004. Correlation clustering. _Machine learning_, 56:89-113.
* Basile et al. (2020) Pierpaolo Basile, Annalina Caputo, Tommaso Caselli, Pierluigi Cassotti, and Rossella Varvara. 2020. DIACR-Ita@ EVALITA2020: Overview of the EVALITA2020 DiachronicLexical Semantics (DIACR-Ita) Task. In _Proceedings of the Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA)_, Online. CEUR-WS.
* Beck (2020) Christin Beck. 2020. DiaSense at SemEval-2020 Task 1: Modeling Sense Change via Pre-trained BERT Embeddings. In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_, pages 50-58, Barcelona (online). International Committee for Computational Linguistics.
* Cassotti et al. (2023) Pierluigi Cassotti, Lucia Siciliani, Marco DeGemmis, Giovanni Semeraro, and Pierpaolo Basile. 2023. XLLEXMEE: WiC Pretrained Model for Cross-Lingual LZAcial SEMatic changeE. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 1577-1585, Toronto, Canada. Association for Computational Linguistics.
* Castano et al. (2024) Silvana Castano, Alfio Ferrara, Stefano Montanelli, and Francesco Periti. 2024. Incremental Affinity Propagation based on Cluster Consolidation and Stratification.
* Chen et al. (2022) Jing Chen, Emmanuele Chersoni, and Chu-ren Huang. 2022. Lexicon of Changes: Towards the Evaluation of Diachronic Semantic Shift in Chinese. In _Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change_, pages 113-118, Dublin, Ireland. Association for Computational Linguistics.
* Chen et al. (2023a) Jing Chen, Emmanuele Chersoni, Dominik Schlechtweg, Jelena Prokic, and Chu-Ren Huang. 2023a. ChiWUG: A Graph-based Evaluation Dataset for Chinese Lexical Semantic Change Detection. In _Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change_, pages 93-99, Singapore. Association for Computational Linguistics.
* Chen et al. (2023b) Jing Chen, Emmanuele Chersoni, Dominik Schlechtweg, Jelena Prokic, and Chu-Ren Huang. 2023b. ChiWUG: A Graph-based Evaluation Dataset for Chinese Lexical Semantic Change Detection. In _Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change_, pages 93-99, Singapore. Association for Computational Linguistics.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Camboim de Sa et al. (2024) Jader Martins Cambloim de Sa, Marcos Da Silveira, and Cedric Pruski. 2024. Survey in Characterization of Semantic Change.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.

 

Brendan J Frey and Delbert Dueck. 2007. Clustering by Passing Messages Between Data Points. _science_, 315(5814):972-976.
* Giulianelli et al. [2020] Mario Giulianelli, Marco Del Tredici, and Raquel Fernandez. 2020. Analysing Lexical Semantic Change with Contextualised Word Representations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3960-3973, Online. Association for Computational Linguistics.
* Giulianelli et al. [2022] Mario Giulianelli, Andrey Kutuzov, and Lidia Pivovaova. 2022. Do Not Fire the Linguistic Grammatical Profiles Help Language Models Detect Semantic Change. In _Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change_, pages 54-67, Dublin, Ireland. Association for Computational Linguistics.
* Hubert and Arabie [1985] Lawrence Hubert and Phipps Arabie. 1985. Comparing Partitions. _Journal of classification_, 2:193-218.
* Kanjirangat et al. [2020] Vani Kanjirangat, Sandra Mitrovic, Alessandro Antonucci, and Fabio Rinaldi. 2020. SST-BERT at SemEval-2020 Task 1: Semantic Shift Tracing by Clustering in BERT-based Embedding Spaces. In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_, pages 214-221, Barcelona (online). International Committee for Computational Linguistics.
* Karjus [2023] Andres Karjus. 2023. Machine-assisted Mixed Methods: Augmenting Humanities and Social Sciences with Artificial Intelligence.
* Kashleva et al. [2022] Kseniia Kashleva, Alexander Shein, Elizaveta Takhtina, and Svetlana Vydrina. 2022. HSE at LSCDiscovery in Spanish: Clustering and Profiling for Lexical Semantic Change Discovery. In _Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change_, pages 193-197, Dublin, Ireland. Association for Computational Linguistics.
* Keidar et al. [2022] Daphna Keidar, Andreas Opedal, Zhijing Jin, and Mrinamaya Sachan. 2022. Slangvolution: A Causal Analysis of Semantic Change and Frequency Dynamics in Shang. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1422-1442, Dublin, Ireland. Association for Computational Linguistics.
* Kocon et al. [2023] Jan Kocon, Igor Cichecki, Olivier Kaszyca, Mateusz Kochanek, Dominika Szyldo, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Kocon, Bartlomiej Koptyma, Wiktoria Mielexczenko-Kowszewiczu, Piotr Mikowski, Marcin Oleksky, Maciej Piasecki, Lukasz Radliiski, Konrad Wojtasik, Stanislaw Rozniak, and Przesmyslow Kazienko. 2023. ChatGPT: Jack of All Trades, Master of None. _Information Fusion_, 99:101861.
* Kadiosov and Arefyev [2022] Artem Kudiosov and Nikolay Arefyev. 2022. BOS at LSCDiscovery: Lexical Substitution for Interpretable Lexical Semantic Change Detection. In _Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change_, pages 165-172, Dublin, Ireland. Association for Computational Linguistics.
* Kutuzov and Giulianelli [2020]Andrey Kutuzov and Mario Giulianelli. 2020. UiOvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection. In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_, pages 126-134, Barcelona (online). International Committee for Computational Linguistics.
* Kutuzov et al. [2018] Andrey Kutuzov, Lilja Ovrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic Word Embeddings and Semantic Shifts: a Survey. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 1384-1397, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Kutuzov and Pivovarova [2021a] Andrey Kutuzov and Lidia Pivovarova. 2021a. RuShiftEval.
* Kutuzov and Pivovarova [2021b] Andrey Kutuzov and Lidia Pivovarova. 2021b. RuShiftEval: A Shared Task on Semantic Shift Detection for Russian. In _Proceedings of the Conference on Computational Linguistics and Intellectual Technologies (Dialogue)_, 20, (online). RSUH.
* Kutuzov and Pivovarova [2021c] Andrey Kutuzov and Lidia Pivovarova. 2021c. Three-part Diachronic Semantic Change Dataset for Russian. In _Proceedings of the 2nd International Workshop on Computational Approaches to Historical Language Change 2021_, pages 7-13, Online. Association for Computational Linguistics.
* Kutuzov et al. [2022a] Andrey Kutuzov, Samia Touileb, Peter Mehlum, Tita Enstad, and Alexandra Wittemann. 2022a. NorDiChange: Diachronic Semantic Change Dataset for Norwegian. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 2563-2572, Marseille, France. European Language Resources Association.
* Kutuzov et al. [2021] Andrey Kutuzov, Samia Touileb, Petter Mehlum, Tita Ranveig Enstad, and Alexandra Wittemann. 2021. NorDiAnChange.
* Kutuzov et al. [2022b] Andrey Kutuzov, Erik Velldal, and Lilja Ovrelid. 2022b. Contextualized Embeddings for Semantic Change Detection: Lessons Learned. In _Northern European Journal of Language Technology, Volume 8_, Copenhagen, Denmark. Northern European Association of Language Technology.
* Laicher et al. [2021] Severin Laicher, Sinan Kurtyijit, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. Explaining and Improving BERT Performance on Lexical Semantic Change Detection. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop_, pages 192-202, Online. Association for Computational Linguistics.
* Laskar et al. [2023] Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossein Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023. A Systematic Study and 

Comprehensive Evaluation of ChatGPT on Benchmark Datasets. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 431-469, Toronto, Canada. Association for Computational Linguistics.
* Ledvai and Wick (2022) Piroska Ledvai and Claudia Wick. 2022. Finetuning Latin BERT for Word Sense Disambiguation on the Thesaurus Language Latinate. In _Proceedings of the Workshop on Cognitive Aspects of the Lexicon_, pages 37-41, Taipei, Taiwan. Association for Computational Linguistics.
* Li et al. (2023) Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. 2023. Large Language Models Understand and Can be Enhanced by Emotional Stimuli.
* Liang and Shi (2023) Meng Liang and Yao Shi. 2023. Named Entity Recognition Method Based on BERT-whitening and Dynamic Fusion Model. In _2023 5th International Conference on Natural Language Processing (ICNLP)_, pages 191-197.
* Lin (1991) Jianhua Lin. 1991. Divergence Measures Based on the Shannon Entropy. _IEEE Transactions on Information Theory_, 37(1):145-151.
* Loureiro et al. (2022) Daniel Loureiro, Aminette D'Souza, Areej Nasser Muhajab, Isabella A. White, Gabriel Wong, Luis Espinosa-Anke, Leonardo Neves, Francesco Barbieri, and Jose Camacho-Collados. 2022. TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 3353-3359, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Ma et al. (2019) Xiaofei Ma, Zhiguo Wang, Patrick Ng, Ramesh Nallapati, and Bing Xiang. 2019. Universal Text Representation from BERT: An Empirical Study.
* Manandhar et al. (2010) Suresh Manandhar, Ioannis Klapaftis, Dmitriy Dilgach, and Sameer Pradhan. 2010. SemEval-2010 Task 14: Word Sense Induction & Disambiguation. In _Proceedings of the 5th International Workshop on Semantic Evaluation_, pages 63-68, Uppsala, Sweden. Association for Computational Linguistics.
* Manning (2009) Christopher D Manning. 2009. _An Introduction to Information Retrieval_. Cambridge university press.
* Martinc et al. (2020a) Matej Martinc, Petra Kralj Novak, and Senja Pollak. 2020a. Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 4811-4819, Marseille, France. European Language Resources Association.
* Martinez et al. (2020b) Matej Martinc, Syrielle Montariol, Elaine Zosa, and Ladia Pivovarova. 2020b. Capturing Evolution in Word Usage: Just Add More Clusters? In _Companion Proceedings of the Web Conference 2020_, WWW '20, page 343-349, Taipei, Taiwan. Association for Computing Machinery.
* Martinez et al. (2020c) Matej Martinc, Syrielle Montariol, Elaine Zosa, and Lidia Pivovarova. 2020c. Discovery Team at SemEval-2020 Task 1: Context-sensitive Embeddings Not Always Better than Static for Semantic Change Detection. In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_, pages 67-73, Barcelona (online). International Committee for Computational Linguistics.
* McGillivray et al. (2021) Barbara McGillivray, Dominik Schlechtweg, Haim Dubossarsky, Nina Tahmasebi, and Simon Hengchen. 2021. DWUG LA: Diachronic Word Usage Graphs for Latin.
* Montanelli and Periti (2023) Stefano Montanelli and Francesco Periti. 2023. A Survey on Contextualised Semantic Shift Detection.
* Montaroli et al. (2021) Syrielle Montaroli, Matej Martinc, and Lidia Pivovarova. 2021. Scalable and Interpretable Semantic Change Detection. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4642-4652, Online. Association for Computational Linguistics.
* Periti and Dubossarsky (2023) Francesco Periti and Haim Dubossarsky. 2023. The Time-Embedding Travelers@WiC-ITA. In _Proceedings of the Eighth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALLTA 2023)_, Parma, Italy. CEUR.org.
* Periti et al. (2024) Francesco Periti, Haim Dubossarsky, and Nina Tahmasebi. 2024. (Chat)GPT v BERT Dawn of Justice for Semantic Change Detection. In _Findings of the Association for Computational Linguistics: EACL 2024_, pages 420-436, St. Julian's, Malta. Association for Computational Linguistics.
* Periti et al. (2022) Francesco Periti, Alfio Ferrara, Stefano Montanelli, and Martin Ruskov. 2022. What is Done is Done: an Incremental Approach to Semantic Shift Detection. In _Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change_, pages 33-43, Dublin, Ireland. Association for Computational Linguistics.
* Periti et al. (2023) Francesco Periti, Sergio Picascia, Stefano Montanelli, Alfio Ferrara, and Nina Tahmasebi. 2023. Studying Word Meaning Evolution through Incremental Semantic Shift Detection: A Case Study of Italian Parliamentary Speeches.
* Piehlvar and Camacho-Collados (2019) Mohammad Taber Pilehlvar and Jose Camacho-Collados. 2019. WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1267-1273, Minneapolis, Minnesota. Association for Computational Linguistics.
* Pomsl and Lyapin (2020) Martin Pomsl and Roman Lyapin. 2020. CIRCE at SemEval-2020 Task 1: Ensembling Context-Free and 

Context-Dependent Word Representations. In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_, pages 180-186, Barcelona (online). International Committee for Computational Linguistics.
* Discover Lexical Semantic Change in Spanish. In _Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change_, pages 198-203, Dublin, Ireland. Association for Computational Linguistics.
* Raganato et al. (2020) Alessandro Raganato, Tommaso Pasini, Jose Camachocollados, and Mohammad Taher Pilehvar. 2020. XLWiC: A Multilingual Benchmark for Evaluating Semantic Contextualization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7193-7206, Online. Association for Computational Linguistics.
* Reif et al. (2019) Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and Measuring the Geometry of BERT. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Riemenschneider and Frank (2023) Frederick Riemenschneider and Anette Frank. 2023. Exploring Large Language Models for Classical Philology. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15181-15199, Toronto, Canada. Association for Computational Linguistics.
* Rodina et al. (2021) Julia Rodina, Yuliya Trofimova, Andrey Kutuzov, and Ekaterina Artemova. 2021. ELMo and BERT in Semantic Change Detection for Russian. In _Analysis of Images, Social Networks and Texts_, pages 175-186, Cham. Springer International Publishing.
* Rosin et al. (2022) Guy D. Rosin, Ido Guy, and Kira Radinsky. 2022. Time Masking for Temporal Language Models. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_, WSDM '22, page 833-841, Virtual Event, AZ, USA. Association for Computing Machinery.
* Schlechtweg (2023) Dominik Schlechtweg. 2023. _Human and Computational Measurement of Lexical Semantic Change_. Ph.D. thesis, University of Stuttgart.
* Schlechtweg et al. (2022a) Dominik Schlechtweg, Haim Dubossarsky, Simon Hengchen, Barbara McGillivray, and Nina Tahmasebi. 2022a. DWUG EN: Diachronic Word Usage Graphs for English.
* Schlechtweg et al. (2020) Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection. In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_, pages 1-23, Barcelona (online). International Committee for Computational Linguistics.
* Schlechtweg et al. (2022b) Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2022b. DWUG DE: Diachronic Word Usage Graphs for German.
* Schlechtweg and Walde (2020) Dominik Schlechtweg and Sabine Schulte im Walde. 2020. Simulating Lexical Semantic Change from Sense-Annotated Data. In _Proceedings of the 13th International Conference on the Evolution of Language (EvoLang13)_, Brussels, Belgium.
* Schlechtweg et al. (2018) Dominik Schlechtweg, Sabine Schulte im Walde, and Stefanie Eckmann. 2018. Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 169-174, New Orleans, Louisiana. Association for Computational Linguistics.
* Schlechtweg et al. (2021) Dominik Schlechtweg, Nina Tahmasebi, Simon Hengchen, Haim Dubossarsky, and Barbara McGillivray. 2021. DWUG: A large Resource of Diachronic Word Usage Graphs in Four Languages. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7079-7091, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Schlechtweg et al. (2024) Dominik Schlechtweg, Shafaqat Mumtaz Virk, and Nikolay Arefyev. 2024. The lscd benchmark: a testbed for diachronic word meaning tasks.
* Schlechtweg et al. (2023) Dominik Schlechtweg, Shafaqat Mumtaz Virk, Pauline Sander, Emma Skoldberg, Lukas Treuer Linke, Tuo Zhang, Nina Tahmasebi, Jonas Kuhn, and Sabine Schulte im Walde. 2023. The DURel Annotation Tool: Human and Computational Measurement of Semantic Proximity, Sense Clusters and Semantic Change.
* Shoemark et al. (2019) Philippa Shoemark, Farhana Ferdousi Liza, Dong Nguyen, Scott Hale, and Barbara McGillivray. 2019. Room to Glo: A Systematic Comparison of Semantic Change Detection Approaches with Word Embeddings. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 66-76, Hong Kong, China. Association for Computational Linguistics.
* Spearman (1904) Charles Spearman. 1904. The Proof and Measurement of Association Between Two Things. _American Journal of Psychology_, 15:88-103.
* Tahmasebi et al. (2021) Nina Tahmasebi, Lars Borin, and Adam Jatowt. 2021. _Survey of Computational Approaches to Lexical Semantic Change Detection_, pages 1-91. Language Science Press, Berlin.
* Tahmasebi et al. (2022) Nina Tahmasebi, Simon Hengchen, Dominik Schlechtweg, Barbara McGillivray, and Haim Dubossarsky. 2022. DWUG SV: Diachronic Word Usage Graphs for Swedish 

* Tang (2018) Xuri Tang. 2018. A State-of-the-art of Semantic Change Computation. _Natural Language Engineering_, 24(5):649-676.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvian Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Zamora-Reina et al. (2022a) Frank D. Zamora-Reina, Felipe Bravo-Marquez, and Dominik Schlechtweg. 2022a. Dwug es: Diachronic word usage graphs for spanish.
* Zamora-Reina et al. (2022b) Frank D. Zamora-Reina, Felipe Bravo-Marquez, and Dominik Schlechtweg. 2022b. LSCDiscovery: A Shared Task on Semantic Change Discovery and Detection in Spanish. In _Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change_, pages 149-164, Dublin, Ireland. Association for Computational Linguistics.
* Zhou and Li (2020) Jinan Zhou and Jiaxin Li. 2020. TemporalTeller at SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection with Temporal Referencing. In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_, pages 222-231, Barcelona (online). International Committee for Computational Linguistics.
* Zhou et al. (2023) Wei Zhou, Nina Tahmasebi, and Haim Dubossarsky. 2023. The Finer They Get: Combining Fine-Tuned Models For Better Semantic Change Detection. In _Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)_, pages 518-528, Torshavn, Faroe Islands. University of Tartu Library.

 



## Appendix A Semantic proximity

As an example, consider the following word usage pair \(\langle w,c_{1},c_{2}\rangle\) extracted by the English benchmark for the word \(w=\)_plane_.

* \(c_{1}\): But we are most familiar with the exhibitions of gravity in bodies descending inclined **planes**, as in the avalanche and the cataract.
* \(c_{2}\): Over the next several years, he said, the Coast Guard will get 60 more people, two new 270-foot vessels and al twin-engine **planes**.

Following the DURel relatedness scale (see Table 3), the pair is annotated with an average judgment of **1** by human annotators.

## Appendix B State-of-the-art for Graded Change Detection

In Table 5, we report the current top scores for GCD in the state-of-the-art with a reference to the paper from where the result is taken. Notably, we report results for different benchmarks using four different approaches evaluated in this paper. However, some benchmarks might have been assessed using other approaches that are excluded from this table.

## Appendix C Graded Change Detection across layers

In Table 4, we report correlation scores for GCD across benchmarks. Specifically, we report results for BERT, mBERT, and XLM-R (separated by slash, i.e. "/") by utilizing all layers of the models (1-12), _individually_.

In Figure 2 and 3, we report correlation scores distribution for GCD obtained by using all possible layer _combinations_ of length 2 (e.g., Layer 1 and 2), length 3 (e.g., Layer 10, 11, 12), and length 4 (e.g., Layer 1, 10, 11, 12) for BERT, mBERT, and XLM-R.

For the sake of comparison, we report in Table 8 the overall top score for GCD obtained using BERT, mBERT, and XLM-R. Specifically, we present results for the optimal combination and the outcome obtained by summing the last four layers, separated by a slash. Additionally, we include the standard result obtained using the last layer individually.

## Appendix D Benchmarks

In Table 6, we report the benchmarks used in this work. Specifically, for each benchmark, we report time periods, diachronic corpus composition, number of targets, and benchmark versions.

## Appendix E BERT, mBERT, XLM-R, XL-LEXEME

In Table 7, we report the BERT, mBERT, XLM-R, and XL-LEXEME models employed in our evaluation. All the models are base versions with 12 encoder layers and can be accessed on huggingface.co.



GPT-4 evaluation

We evaluate GPT-4 as computational annotator by relying on computational proximity judgments gathered through the following method.

Model initialization.We initialized the model with the following prompt (guideline):

Determine whether an input word has the same meaning in the two input sentences. Answer with 'Same', 'Related', 'Linked', or 'Distinct'. This is very important to my career.

Notably, we combine and refine two different prompts used in previous works. We drew inspiration from the prompt utilized by Karjus (2023) to assess GPT-4 in addressing the Graded Change Detection task. Additionally, we drew inspiration from the prompt utilized by Li et al. (2023), called _EmotionPrompt_, which combines the original prompt with emotional stimuli to enhance the performance of Large Language Models.

Model template.For each word usage pair, we used the following prompt:

Determine whether [Target word] has the same meaning in the following sentences. Do they refer to roughly the Same, different but closely Related, distant/figuratively Linked or unrelated Distinct word meanings?
**Sentence 1**: _[Context 1]_
**Sentence 2**: _[Context 2]_

Notably, drawing inspiration from the OpenAI documentation7 and the prompts utilized in previous work for the Word-in-Context task (Kocoi et al., 2023; Laskar et al., 2023), we structured our prompt in a format that facilitates parsing and comprehension. For each usage pair \(\langle w,c_{1},c_{2}\rangle\) of a word \(w\), we substitute [Target word] with the actual target \(w\) and [Context 1] and [Context 2] with \(c_{1}\) and \(c_{2}\), respectively.

Footnote 7: platform.openai.com/docs/guides/prompt-engineering

We prompt GPT-4 without providing any message history. This means that, for each usage pair \(\langle w,c_{1},c_{2}\rangle\), we re-initialize the model with the initial prompt (guideline) and subsequently prompt the model to gather a semantic proximity judgment for the pair \(\langle w,c_{1},c_{2}\rangle\). This approach ensures that the model relies solely on its pre-trained knowledge, preventing potential biases stemming from previously prompted pairs.

 



\begin{table} \end{tabular}

\end{table}
Table 4: **Comprehensive evaluation of standard approaches to GCD by using the layers 1-12 of BERT / mBERT / NLM-R. Top score for each approach, model, and benchmark in bold. Avg is the weighted average score based on the number of targets in each benchmark.**



\begin{table}
\begin{tabular}{|c c||c|c|c|c|c|c|c|c|c|c|c|c|c|}  & **EN** & **LA** & **DE** & **EW** & **NE** & **NN** & **EM** & **EM** & **N0** & **EM** \\ \cline{2-13}  & C\({}_{1}-C_{2}\) & C\({}_{3}-C_{3}\) & C\({}_{4}-C_{5}\) & C\({}_{6}-C_{7}\) & C\({}_{8}-C_{9}\) & C\({}_{1}-C_{2}\) & C\({}_{3}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) \\ \hline Time & C\({}_{1}-C_{2}\) & C\({}_{3}-C_{3}\) & C\({}_{4}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) \\ \hline Time & C\({}_{1}-C_{2}\) & C\({}_{3}-C_{3}\) & C\({}_{4}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) \\ \hline Periods & C\({}_{1}-C_{2}\) & C\({}_{3}-C_{3}\) & C\({}_{4}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) & C\({}_{1}-C_{5}\) \\ \end{tabular}
\end{table}
Table 6: **LSC benchmark for Graded Change Detection. Overview of time periods, diachronic corpus composition, number of targets, and benchmark versions used in this study.**

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c|c|c|c|c|c}  & **BERT** & **mBERT** & **XLM-R** & **XL-LEXME** & & & & & & & & & & & \\ \cline{2-13}
**English** & bart-base-uncased & bart-base-multilingual-cased & xlm-robustr-baste & bart-base-multilingual-cased & xlm-robustr-baste & bart-bigel/xl-known & & & & & & & & & \\ \cline{2-13}
**Latin** & bert-base-partial-casd & bart-base-multilingual-cased & bart-base-multilingual-cased & xlm-robustr-baste & bart-bigel/xl-learning & & & & & & & & & & \\ \cline{2-13}
**German** & bert-base-partial-casd & bart-base-multilingual-cased & bart-base-multilingual-cased & xlm-robustr-baste & bart-bigel/xl-learning & & & & & & & & & \\ \cline{2-13}
**Swefehi** & af-al-center/bert-base-aswellsh-un- Uncased & bart-base-multilingual-cased & xlm-robustr-baste & bart-bigel/xl-learning & & & & & & & & & & \\ \cline{2-13}
**Spanish** & dccouiliite/bert-base-spanish-wuns- Uncased & bart-base-multilingual-cased & bart-base-multilingual-cased & xlm-robustr-baste & bart-bigel/xl-learning & & & & & & & & \\ \cline{2-13}
**Russian** & Deepraviov/robert-base-cased & bart-base-multilingual-cased & bart-base-multilingual-cased & xlm-robustr-baste & bart-bigel/xl-learning & & & & & & & & \\ \cline{2-13}
**Norwegian** & hkila/nb-bert-base & bart-base-multilingual-cased & xlm-robustr-baste & bart-base-multilingual-cased & xlm-robustr-baste & bart-bigel/xl-learning & & & & & & & & \\ \cline{2-13}
**Chinese** & bert-base-embines & bart-base-multilingual-cased & xlm-robustr-baste & bart-bigel/xl-learning & & & & & & & & & & \\ \end{tabular}
\end{table}
Table 7: BERT, mBERT, XLM-R, and XL-LEXME models employed in our evaluation. All models are available at huggingface.co.

\begin{table}
\begin{tabular}{|p{42.7pt}|p{42.7pt} p{42.



* [14] M. C. Bader, J.-M. Chen, and S. G. L. Gladman, "A new approach to the classification of deep neural networks for image recognition," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 2573-2580, June 2016, pp. 191-202.
* [15] A. D. Krizhevsky, I. Sutskever, and E. Rabinovich, "Dropout: a simple way to prevent overfitting," in _Advances in Neural Information Processing Systems (NIPS'06)_, volume 1, pages 1-10, 2006.
* [16] R. Girshick, V. Ramanan, and Y. Bengio, "Deep residual learning with convolutional layers," in _Neural information processing systems_, vol. 30, no. 1, p. 115002, 2006.

[MISSING_PAGE_POST]

 

* [14] M. C. Bender, J. S. Lee, and D. R. K. Smith, "A survey of the population-based surveys for the detection of large numbers of sources in a sample of 2008," _Journal of Statistical Planning Research_**53**(6):79-90, 2008.

[MISSING_PAGE_POST]

 

* [14] M. C. Bender, A. K. Sridharan, and J. R. Friedman, "A new approach to the classification of a class of objects in a class of classes of classes of classes of classes of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class of class class of class of class of class class of class of class class of class class of class class of class class of class class of class class of class class of class class of class class of class class of class class of class class of class class of class class of class class of class class class of class class of class class class of class class class of class class class of class class class of class class class of class class class class of class class class class of class