SlowGuess/NAACL_2021_OCR huggingface dataset:
Syrielle Montariol*  
LISN - CNRS, Univ. Paris-Saclay  
Société Générale
syrielle.montariol@limsi.fr
Matej Martinc* Jozef Stefan Institute matej.martinc@ijs.si
Lidia Pivovarova  
University of Helsinki  
lidia.pivovarova@helsinki.fi
Several cluster-based methods for semantic change detection with contextual embeddings emerged recently. They allow a fine-grained analysis of word use change by aggregating embeddings into clusters that reflect the different usages of the word. However, these methods are unscalable in terms of memory consumption and computation time. Therefore, they require a limited set of target words to be picked in advance. This drastically limits the usability of these methods in open exploratory tasks, where each word from the vocabulary can be considered as a potential target. We propose a novel scalable method for word usage-change detection that offers large gains in processing time and significant memory savings while offering the same interpretability and better performance than unscalable methods. We demonstrate the applicability of the proposed method by analysing a large corpus of news articles about COVID-19.
Studying language evolution is important for many applications, since it can reflect changes in the political and social sphere. In the literature, the study of language evolution either focuses on long-term changes in the meaning of a word, or on more common short-term evolutionary phenomena, such as the word suddenly appearing in a new context, while keeping its meaning unchanged in a lexicographic sense. We refer to all types of language evolution—short- or long-term, with or without meaning change—as word usage change, a broad category that includes semantic change, but also any shifts in the context in which a word appears.
Recent studies (Giulianelli et al., 2020; Martinc et al., 2020a) show that clustering of contextual embeddings could be a proxy for word usage change: if clusters, which in theory capture distinct word usages, are distributed differently across time periods,
it indicates a possible change in word's context or even loss or gain of a word sense. Thus, the cluster-based approach offers a more intuitive interpretation of word usage change than alternative methods, which look at the neighborhood of a word in each time period to interpret the change (Gonen et al., 2020; Martinc et al., 2020b) and ignore the fact that a word can have more than one meaning. The main limitation of the cluster-based methods is the scalability in terms of memory consumption and time: clustering is applied to each word in the corpus separately and all occurrences of a word need to be aggregated into clusters. For large corpora with large vocabularies, where some words can appear millions of times, the use of these methods is severely limited.
To avoid the scalability issue, cluster-based methods are generally applied to a small set of less than a hundred manually pre-selected words (Giulianelli et al., 2020; Martinc et al., 2020a). This drastically limits the application of the methods in scenarios such as identification of the most changed words in a large corpus or measuring of usage change of extremely frequent words, since clustering of all of word's contextual embeddings requires large computational resources. One way to solve the scalability problem using contextual embeddings is to average a set of contextual representations for each word into a single static representation (Martinc et al., 2020b). Averaging, while scalable, loses a lot on the interpretability aspect, since word usages are merged into a single representation.
The method we propose in this paper tackles scalability and interpretability at the same time. The main contributions of the paper are the following:
- A scalable method for contextual embeddings clustering that generates interpretable representations and outperforms other cluster-based methods.
- A method of measuring word usage change between periods with the Wasserstein distance. As far as we are aware, this is the first paper leveraging
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4642-4652 June 6-11, 2021. ©2021 Association for Computational Linguistics
ing optimal transport for lexical semantic change detection.
- A cluster filtering step, which balances the deficiencies of clustering algorithms and consistently improves performance.
- An interpretation pipeline that automatically labels word senses, allowing a domain expert to find the most changing concepts and to understand how those changes happened.
The practical abilities of our method are demonstrated on a large corpus of news articles related to COVID-19, the Aylien Coronavirus News Dataset<sup>1</sup>. We compute the degree of usage change of almost 8,000 words, i.e., all words that appear more than 50 times in every time slice of the corpus, in the collection of about half a million articles in order to find the most changing words and interpret their drift<sup>2</sup>.
Diachronic word embedding models have undergone a surge of interest in the last two years with the successive publications of three articles dedicated to a literature review of the domain (Kutuzov et al., 2018; Tahmasebi et al., 2018; Tang, 2018). Most approaches build static embedding models for each time slice of the corpus and then make these representations comparable by either employing incremental updating (Kim et al., 2014) or vector space alignment (Hamilton et al., 2016b). The alignment method has proved superior on a set of synthetic semantic drifts (Shoemark et al., 2019) and has been extensively used (Hamilton et al., 2016b; Dubossarsky et al., 2017) and improved (Dubossarsky et al., 2019) in the literature. The recent SemEval Task on Unsupervised lexical semantic change detection has shown that this method is most stable and yields the best averaged performance across four SemEval corpora (Schlechtweg et al., 2020).
Yet another approach (Hamilton et al., 2016a; Yin et al., 2018) is based on comparison of neighbors of a target word in different time periods. This approach has been recently used to tackle the scalability problem (Gonen et al., 2020).
In all these methods, each word has only one representation within a time slice, which limits the sensitivity and interpretability of these techniques.
The recent rise of contextual embeddings such as BERT (Devlin et al., 2019) and ELMO (Peters et al., 2018) introduced significant changes to word representations. Contextual embeddings can be used for usage change detection by aggregating the information from the set of token embeddings. This can be done either through averaging of all vectors within a time slice and then computing averaged vector similarity (Martinc et al., 2020b), by computing a pairwise distance between vectors from different time slices (Kutuzov and Giulianielli, 2020), or by clustering all token representations to approximate its set of senses (Giulianelli et al., 2020). The analysis in this paper derives from this last set of methods, which demonstrate a higher performance than static embeddings methods at least on some datasets (Martinc et al., 2020a).
Automatic semantic shift detection has been used for text stream monitoring tasks, such as event detection (Kutuzov et al., 2017) viewpoint analysis (Azarbonyad et al., 2017) or monitoring of rapid discourse changes during crisis events (Stewart et al., 2017). None of these applications use clustering techniques and, as far as we are aware, only Martinc et al. (2020b) uses contextual embeddings for news stream analysis. In this paper we demonstrate the large potential of contextual embeddings for the interpretable tracking of short-term changes in word usage, which has a practical application for crisis-related news monitoring.
3 Scalability and Interpretability Limitations of Previous Methods
The main motivation for this research are the scalability or interpretability issues of previous methods for word usage change detection. The ones using contextual embeddings are either interpretable but unscalable (Giulianelli et al., 2020; Martinc et al., 2020a) or scalable but uninterpretable (Martinc et al., 2020b). The scalability issues of interpretable methods can be divided into two problems.
Memory consumption: Giulianelli et al. (2020) and Martinc et al. (2020a) apply clustering on all embeddings of each target word. This procedure becomes unfeasible for large sets of target words or if the embeddings need to be generated on a large corpus, since too many embeddings need to be saved into memory for further processing. To give an example, single-precision floating-point in Python requires 4 bytes of memory. Each contextual embedding contains 768 floats (Devlin et al.,
$^{1}$ https://blog.aylien.com/free-coronavirus-news-dataset/  $^{2}$ The code can be found at https://github.com/matejMartinc/scalable_semantichift
2019), leading each embedding to occupy 3072 bytes<sup>3</sup>. To use the previous methods on the Aylien Coronavirus News Dataset, which contains 250M tokens, about 768 Gb RAM would be necessary to store the embeddings for the entire corpus. If we limit our vocabulary to the 7,651 words that appear at least 50 times in every time slice and remove the stopwords (as we do in this work), we still need to generate contextual embeddings for 120M tokens, which is about 369 Gb of RAM.
Complexity of clustering algorithms: For the complexity analyses, we denote by  $d$  the dimension of the embedding,  $k$  is the number of clusters and  $n$  is the number of contextual embeddings, i.e., the number of word occurrences in the corpus. The time complexity of the affinity propagation algorithm (the best performing algorithm according to Martinc et al. (2020a)) is  $O(n^{2}td)$ , with  $t$  being the predefined maximum number of iterations of the data point message exchange. The time complexity of the simpler k-means algorithm<sup>4</sup> can be stated as  $O(tkd)$ , where  $t$  is the number of iterations of Lloyd's algorithm (Lloyd, 1982). As an example, consider the word coronavirus, which appears in the Aylien corpus about 1,2M times. For k-means with  $k = 5$  and a maximal number of iterations set to 300 (the Scikit library default), about  $300 * 5 * 1$ , 300,000 * 768 ≈ 1.5 × 10<sup>12</sup> operations are conducted for the clustering. With affinity propagation with the maximum number of iterations set to 200 (the default), clustering of the word coronavirus would require 1,300,000<sup>2</sup> * 200 * 768 ≈ 2.6 × 10<sup>17</sup> operations, which is impossible to conduct in a reasonable amount of time on a high end desktop computer.
Contextual Embeddings Method with Interpretability Limitations: The averaging approach (Martinc et al., 2020b) eliminates the scalability problems: token embeddings for each word are not collected in a list but summed together in an element-wise fashion, which means that only 768 floats need to be saved for each word in the vocabulary. The averaged word representation is obtained for each time slice by dividing the sum by the word count. A single embedding per word is
saved, leading to only  $23.5\mathrm{Mb}$  of RAM required to store the embeddings for 7,651 words. These representations lose on the interpretability aspect, since all word usages are merged into a single averaged representation. It makes the method inappropriate for some tasks such as automatic labelling of word senses, and in some cases affects the overall performance of the method (Martinc et al., 2020a).
Our word usage change detection pipeline follows the procedure proposed in the previous work (Martinc et al., 2020a; Giulianelli et al., 2020): for each word, we generate a set of contextual embeddings using BERT (Devlin et al., 2019). These representations are clustered using k-means or affinity propagation and the derived cluster distributions are compared across time slices by either using Jensen-Shannon divergence (JSD) (Lin, 2006) or the Wasserstein distance (WD) (Solomon, 2018). Finally, words are ranked according to the distance measure, assuming that the ranking resembles a relative degree of usage shift.
The primary contributions of this work lay in the embedding generation step, which improves the scalability of the method, and in leveraging WD to compute the distance between clusters. We also propose post-processing steps, which domain experts could use for the interpretation of results. We now describe the pipeline in more details.
4.1 Embeddings Generation
We use a pre-trained BERT model for each language of the evaluation corpora<sup>5</sup>. All models have 12 attention layers and a hidden layer of size 768. We fine-tune them for domain adaptation on each corpus as a masked language model for 5 epochs. Then, we extract token embeddings from the finetuned models. Each corpus is split into time slices. The models are fed 256 tokens long sequences in batches of 16 sequences at once. We generate sequence embeddings by summing the last four encoder output layers of BERT, following Devlin et al. (2019). Next, we split each sequence into 256 subparts to obtain a separate contextual embedding of size 768 for each token. Since one token does not necessarily correspond to one word due to byte
<sup>3</sup>If we ignore the additional memory of a Python container—e.g., a Numpy list or a Pytorch tensor—required for storing this data.
4Here we are referring to the Scikit implementation of the algorithm employed in this work: https://scikit-learn.org/ stable/modules/generated/sklearn.cluster.KMeans.html.
5For German: bert-base-german-cased (https://deepset. ai/german-bert, for English: bert-base-uncased model, for Latin: bert-base-multilingual-uncased model from the huggingface library, for Swedish: bert-base-swedish- uncased (https://github.com/af-ai-center/SweBERT).
pair tokenization, we average embeddings for each byte-pair token constituting a word to obtain embeddings for each occurrence of a word.
Next, after obtaining a contextual embedding vector for each target word in a specific sequence, we decide whether this vector should be saved to the list or merged with one of the previously obtained vectors for the same word in the same time slice. To improve the scalability, we limit the number of contextual embeddings that are kept in the memory for a given word and time slice to a predefined threshold. The threshold of 200 was chosen empirically from a set of threshold candidates (20, 50, 100, 200, 500) and offers a reasonable compromise between scalability and performance. The new vector is merged if it is too similar—i.e., a duplicate or a near-duplicate—to one of the saved vectors or if the list already contains a predefined maximum number of vectors (200 in our case).
More formally, we add the new embedding  $e_{new}$  to the list of word embeddings  $L = \{e_i, \dots, e_n\}$  if:
$$
\left| L \right| <   2 0 0 \& \forall e _ {i} \in L: s \left(e _ {\text {n e w}}, e _ {i}\right) <   1 - \varepsilon
$$
where  $s$  is the cosine similarity and  $\varepsilon$  is a threshold set to 0.01.
If  $|L| \geq 200$  or if any vector in the list  $L$  is a near duplicate to  $e_{\text{new}}$ , we find a vector  $e_m$  in the list which is the closest to  $e_{\text{new}}$  in terms of cosine similarity:
$$
e_{m} = \operatorname *{arg  max}_{e_{i}\in L}s(e_{i},e_{\mathrm{new}})
$$
This element  $e_m$  is then modified by summing it with  $e_{\text{new}}$ :
$$
e _ {m} \gets e _ {m} + e _ {\mathrm {n e w}}
$$
The number of summed-up elements for each of the 200 groups in the list is stored besides their summed-up representations. Once the model has been fed with all the sequences in the time slice, the final summed-up vector is divided by this number to obtain an averaged embedding.
By having only 200 merged word embeddings per word per time slice, and by limiting the vocabulary of the corpus to 7,651 target words, we require up to  $4.7\mathrm{Gb}$  of space for each time slice, no matter the size of the corpus. While this is still 200 times more space than if the averaging method was used (Martinc et al., 2020b), the conducted experiments show that the proposed method nevertheless keeps the bulk of the interpretability of the less scalable method proposed by Giulianielli et al. (2020), and offers competitive performance on several corpora.
4.2 Clustering
After collecting 200 vectors for each word in each time slice, we conduct clustering on these lists to extract the usage distribution of the word at each period. Clustering for a given word is performed on the set of all vectors from all time slices jointly.
We use two clustering methods previously applied for this task, namely k-means used in Giulianielli et al. (2020) and affinity propagation in Martinc et al. (2020a). The main strength of affinity propagation is that the number of clusters is not defined in advance but inferred during training. The clustering is usually skewed: a limited number of large clusters is accompanied with many clusters consisting of only a couple of instances. Thus, affinity propagation allows to pick out the core senses of a word. K-means tends to produce more even clusters. Appearance of small clusters that contain only few instances and do not represent a specific sense or usage of the word is nevertheless relatively common, since BERT is sensitive to syntax and pragmatics, which are not necessarily relevant for usage change detection. Another limitation of the k-means algorithm is that the number of clusters needs to be set in advance. This means that if the number of actual word usages is smaller than a predefined number of clusters, k-means will generate more than one cluster for each word usage.
To compensate for these deficiencies, we propose an additional filtering and merging step. A cluster is considered to be a legitimate representation of a usage of the word, if it contains at least 10 instances<sup>6</sup>. We compute the average embedding inside each cluster, and measure the cosine distance (1 - cosine similarity) between the average embeddings in each pair of legitimate clusters for a given word. If the distance between two clusters is smaller than a threshold, the clusters are merged. The threshold is defined as  $avg_{cd} - 2 * std_{cd}$ , where  $avg_{cd}$  is the average pairwise cosine distance between all legitimate clusters and  $std_{cd}$  is the standard deviation of that distance. This merging procedure is applied recursively until the minimum distance between the two closest clusters is larger than the threshold. After that, the merging proce
The threshold of 10 was derived from the procedure for manual labelling employed in the SemEval Task (Schlechtweg et al., 2020), where a constraint was enforced that the specific sense is attested at least 5 times in a specific time period in order to contribute word senses. We set the overall threshold of 10, which roughly translates to 5 per time period, since all of our test corpora (besides Aylien) contain two time periods.
dure is applied to illegitimate clusters (that contain less than 10 instances), using the same threshold. Illegitimate clusters could be added into one of the legitimate clusters or merged together to form a legitimate cluster with more than 10 instances. If there is no cluster that is close enough to be merged with, the illegitimate cluster is removed.
4.3 Change Detection and Interpretation
After the clustering procedure described above, for each word in each time slice, we extract its cluster distribution and normalise it by the word frequency in the time slice. Then target words are ranked according to the usage divergence between successive time slices, measured with the JSD or the  $\mathrm{WD}^7$ . If a ground-truth ranking exists, the method can be evaluated using the Spearman Rank Correlation to compare the true and the outputted ranking. In the exploratory scenario, the ranking is used to detect the most changing words and then investigate the most unevenly distributed clusters over time for the interpretation of the change.
JSD has been used for semantic shift detection in several recent papers, e.g. (Martinc et al., 2020a; Giulianelli et al., 2020; Kutuzov and Giulianelli, 2020). Since this is the first paper applying WD for this purpose, we describe it in more details.
The motivation for using the WD (Solomon, 2018) is to take into account the position of the clusters in the semantic space when comparing them. The JSD leverages semantic information encoded in the embeddings indirectly, distilled into two time-specific cluster distributions that JSD receives as an input. In addition to cluster distributions, WD accesses characteristics of the semantic space explicitly, through a matrix of cluster averages (obtained by averaging embeddings in each cluster) of size  $T \times k \times 768$ , where  $k$  is a number of clusters,  $T$  is a number of time slices and 768 is the embedding dimension.
This setup is a classical problem that can be solved using optimal transport (Peyre et al., 2019). We denote with  $\mu_{1}$  and  $\mu_{2}$  the sets of  $k$  average embedding points in the two vector spaces, and with  $c_{1}$  and  $c_{2}$  the associated clusters distributions. Thus,  $c_{1}$  and  $c_{2}$  are histograms on the simplex (positive and sum to 1) that represent the weights of each embedding in the source  $(\mu_{1})$  and target  $(\mu_{2})$  distributions. The task is to quantify the effort of moving one unit of mass from  $\mu_{1}$  to  $\mu_{2}$  using a cho
sen cost function, in our case the cosine distance. It is solved by looking for the transport plan  $\gamma$ , which is the minimal effort required to reconfigure  $c_{1}$ 's mass distribution into that of  $c_{2}$ . The WD is the sum of all travels that have to be made to solve the problem:
$$
\begin{array}{l} \mathrm {W D} \left(c _ {1}, c _ {2}\right) = \min  _ {\gamma} \sum_ {i, j} \gamma_ {i, j} M _ {i, j} \\ \text {w i t h} \gamma 1 = c _ {1}; \gamma^ {\intercal} 1 = c _ {2}; \gamma \geq 0 \\ \end{array}
$$
Where  $M \in \mathbb{R}_{m \times n}^{+}$  is the cost matrix defining the cost to move mass from  $\mu_{1}$  to  $\mu_{2}$ . We use the cosine similarity  $s$ , with  $M = 1 - s(\mu_{1}, \mu_{2})$ .
Interpretation. Once the most changing words are detected, the next step is to understand how they change between two time slices by interpreting their clusters of usages.
Cluster distributions can be used directly to identify the clusters that are unevenly distributed across a time dimension. However, a cluster itself may consist of several hundreds or thousands of word usages, i.e. sentences. Interpreting the underlying sense behind each cluster by manually looking at the sentences is time-consuming. To reduce human work, we extract the most discriminating words and bigrams for each cluster: by considering a cluster as a single document and all clusters as a corpus, we compute the term frequency - inverse document frequency (tfidf) score of each word and bigram in each cluster. The stopwords and the words appearing in more than  $80\%$  of the clusters are excluded to ensure that the selected keywords are the most discriminant. Thus, a ranked list of keywords for each cluster is obtained and top-ranked keywords are used for the interpretation of the cluster.
We use six existing manually annotated datasets for evaluation. The first dataset, proposed by Gulordava and Baroni (2011), consists of 100 English words labelled by five annotators according to the level of semantic change between the 1960s and  $1990\mathrm{s}^8$ . To build the dataset, the annotators evaluated semantic change using their intuition, without looking at the context. This procedure is problematic since an annotator may forget or not be aware of a particular sense of the word.
In order to make the proposed approach comparable to previous work, we remove four words that do not appear in the BERT vocabulary from the evaluation dataset, same as in Martinc et al. (2020a).
7Using the POT package https://pythonot.github.io/.