SlowGuess/ACL_2020_OCR huggingface dataset:
Mario Giulianielli Marco Del Tredici Raquel Fernandez
Institute for Logic, Language and Computation
{m.giulianelli|m.deltredici|raquel.fernandez}@uva.nl
This paper presents the first unsupervised approach to lexical semantic change that makes use of contextualised word representations. We propose a novel method that exploits the BERT neural language model to obtain representations of word usages, clusters these representations into usage types, and measures change along time with three proposed metrics. We create a new evaluation dataset and show that the model representations and the detected semantic shifts are positively correlated with human judgements. Our extensive qualitative analysis demonstrates that our method captures a variety of synchronic and diachronic linguistic phenomena. We expect our work to inspire further research in this direction.
In the fourteenth century the words boy and girl referred respectively to a male servant and a young person of either sex (Oxford English Dictionary). By the fifteenth century a narrower usage had emerged for girl, designating exclusively female individuals, whereas by the sixteenth century boy had lost its servile connotation and was more broadly used to refer to any male child, becoming the masculine counterpart of girl (Bybee, 2015). Word meaning is indeed in constant mutation and, since correct understanding of the meaning of individual words underpins general machine reading comprehension, it has become increasingly relevant for computational linguists to detect and characterise lexical semantic change—e.g., in the form of laws of semantic change (Dubossarsky et al., 2015; Xu and Kemp, 2015; Hamilton et al., 2016)—with the aid of quantitative and reproducible evaluation procedures (Schlechtweg et al., 2018).
Most recent studies have focused on shift detection, the task of deciding whether and to what extent the concept evoked by a word has changed
between time periods (e.g., Gulordava and Baroni, 2011; Kim et al., 2014; Kulkarni et al., 2015; Del Tredici et al., 2019; Hamilton et al., 2016; Bamler and Mandt, 2017; Rosenfeld and Erk, 2018). This line of work relies mainly on distributional semantic models, which produce one abstract representation for every word form. However, aggregating all senses of a word into a single representation is particularly problematic for semantic change as word meaning hardly ever shifts directly from one sense to another, but rather typically goes through polysemous stages (Hopper et al., 1991). This limitation has motivated recent work on word sense induction across time periods (Lau et al., 2012; Cook et al., 2014; Mitra et al., 2014; Frermann and Lapata, 2016; Rudolph and Blei, 2018; Hu et al., 2019). Word senses, however, have shortcomings themselves as they are a discretisation of word meaning, which is continuous in nature and modulated by context to convey ad-hoc interpretations (Brugman, 1988; Kilgarriff, 1997; Paradis, 2011).
In this work, we propose a usage-based approach to lexical semantic change, where sentential context modulates lexical meaning "on the fly" (Ludlow, 2014). We present a novel method that (1) exploits a pre-trained neural language model (BERT; Devlin et al., 2019) to obtain contextualised representations for every occurrence of a word of interest, (2) clusters these representations into usage types, and (3) measures change along time. More concretely, we make the following contributions:
- We present the first unsupervised approach to lexical semantic change that makes use of state-of-the-art contextualised word representations.
- We propose several metrics to measure semantic change with this type of representation. Our code is available at https://github.com/glnmario/cwr41sc.
- We create a new evaluation dataset of human sim
3960
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3960-3973 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ilarity judgements on more than 3K word usage pairs across different time periods, available at https://doi.org/10.5281/zenodo.3773250.
- We show that both the model representations and the detected semantic shifts are positively correlated with human intuitions.
- Through in-depth qualitative analysis, we show that the proposed approach captures synchronic phenomena such as word senses and syntactic functions, literal and metaphorical usage, as well as diachronic linguistic processes related to narrowing and broadening of meaning across time.
Overall, our study demonstrates the potential of using contextualised word representations for modelling and analysing lexical semantic change and opens the door to further work in this direction.
Semantic change modelling Lexical semantic change models build on the assumption that meaning change results in the modification of a word's linguistic distribution. In particular, with the exception of a few methods based on word frequencies and parts of speech (Michel et al., 2011; Kulkarni et al., 2015), lexical semantic change detection has been addressed following two main approaches: form-based and sense-based (for an overview, see Kutuzov et al., 2018; Tang, 2018).
In form-based approaches independent models are trained on the time intervals of a diachronic corpus and the distance between representations of the same word in different intervals is used as a semantic change score (Gulordava and Baroni, 2011; Kulkarni et al., 2015). Representational coherence between word vectors across different periods can be guaranteed by incremental training procedures (Kim et al., 2014) as well as by post hoc alignment of semantic spaces (Hamilton et al., 2016). More recent methods capture diachronic word usage by learning dynamic word embeddings that vary as a function of time (Bamler and Mandt, 2017; Rosenfeld and Erk, 2018; Rudolph and Blei, 2018). Form-based models depend on a strong simplification: that a single representation is sufficient to model the different usages of a word.
Time-dependent representations are also created in sense-based approaches: in this case word meaning is encoded as a distribution over word senses. Several Bayesian models of sense change have been proposed (Wijaya and Yeniterzi, 2011; Lau
et al., 2012, 2014; Cook et al., 2014). Among these is the recent SCAN model (Frermann and Lapata, 2016), which represents (1) the meaning of a word in a time interval as a multinomial distribution over word senses and (2) word senses as probability distributions over the vocabulary. The main limitation of sense-based models is that they rely on a bag-of-words representation of context. Furthermore, many of these models keep the number of senses constant across time intervals and require this number to be manually set in advance.
Unsupervised approaches have been proposed that do not rely on a fixed number of senses. For example, the method for novel sense identification by Mitra et al. (2015) represents senses as clusters of short dependency-labelled contexts. Like ours, this method analyses word forms within the grammatical structures they appear. However, it requires syntactically parsed diachronic corpora and focuses exclusively on nouns. None of these restrictions limit our proposed approach, which leverages neural contextualised word representations.
Contextualised word representations Several approaches to context-sensitive word representations have been proposed in the past. Schütze (1998) introduced a clustering-based disambiguation algorithm for word usage vectors, Erk and Padó (2008) proposed creating multiple vectors for the same word and Erk and Padó (2010) proposed to directly learn usage-specific representations based on the set of exemplary contexts within which the target word occurs.
Recently, neural contextualised word representations have gained widespread use in NLP, thanks to deep learning models which learn usage-dependent representations while optimising tasks such as machine translation (CoVe; McCann et al., 2017) and language modelling (Dai and Le, 2015, ULMFiT; Howard and Ruder, 2018, ELMo; Peters et al., 2018, GPT; Radford et al., 2018, 2019, BERT; Devlin et al., 2019). State-of-the-art language models typically use stacked attention layers (Vaswani et al., 2017), they are pre-trained on a very large amount of textual data, and they can be fine-tuned for specific downstream tasks (Howard and Ruder, 2018; Radford et al., 2019; Devlin et al., 2019).
Contextualised representations have been shown to encode lexical meaning dynamically, reaching high accuracy on, e.g., the binary usage similarity judgements of the WiC evaluation set (Pilehvar and Camacho-Collados, 2019), performing on a
par with state-of-the-art word sense disambiguation models (Wiedemann et al., 2019), and proving useful for the supervised derivation of time-specific sense representation (Hu et al., 2019). In this work, we investigate the potential of contextualised word representations to detect and analyse lexical semantic change, without any lexicographic supervision.
3 Method: A Usage-based Approach to Lexical Semantic Change
We introduce a usage-based approach to lexical semantic change analysis which relies on contextualised representations of unique word occurrences (usage representations). First, given a diachronic corpus and a list of words of interest, we use the BERT language model (Devlin et al., 2019) to compute usage representations for each occurrence of these words. Then, we cluster all the usage representations collected for a given word into an automatically determined number of partitions (usage types) and organise them along the temporal axis. Finally, we propose three metrics to quantify the degree of change undergone by a word.
3.1 Language Model
We produce usage representations using the BERT language model (Devlin et al., 2019), a multi-layer bidirectional Transformer encoder trained on masked token prediction and next sentence prediction, on the BooksCorpus (800M words) (Zhu et al., 2015) and on English text passages extracted from Wikipedia (2,500M words). There are two versions of BERT. For space and time efficiency, we use the smaller base-uncased version, with 12 layers, 768 hidden dimensions, and 110M parameters. $^{1}$
3.2 Usage Representations
Given a word of interest  $w$  and a context of occurrence  $s = (v_{1},\dots,v_{i},\dots,v_{n})$  with  $w = v_{i}$ , we extract the activations of all of BERT's hidden layers for sentence position  $i$  and sum them dimension-wise. We use addition because neither concatenation nor selecting a subset of the layers produced notable differences in the relative geometric distance between word representations.
The set of  $N$  usage representations for  $w$  in a given corpus can be expressed as the usage matrix  $\mathbf{U}_w = (\mathbf{w}_1,\dots ,\mathbf{w}_N)$ . For each usage representation in the usage matrix  $\mathbf{U}_w$ , we store the context of
(a) PCA visualisation of the usage representations.
(b) Probability-based usage type distributions along time.
Figure 1: Usage representations and usage type distributions generated with occurrences of the word atom in COHA (Davies, 2012). Colours encode usage types.
occurrence (a 128-token window around the target word) as well as a temporal label  $\mathbf{t}_w$  indicating the time interval of the usage.
3.3 Usage Types
Once we have obtained a word-specific matrix of usage vectors  $\mathbf{U}_w$ , we standardise it and cluster its entries using  $K$ -Means. This step partitions usage representations into clusters of similar usages of the same word, or usage types (see Figure 1a), and thus it is directly related to automatic word sense discrimination (Schütze, 1998; Pantel and Lin, 2002; Manandhar et al., 2010; Navigli and Vannella, 2013, among others).
For each word independently, we automatically select the number of clusters  $K$  that maximises the silhouette score (Rousseeuw, 1987), a metric of cluster quality which favours intra-cluster coherence and penalises inter-cluster similarity, without the need for gold labels. For each value of  $K$ , we execute 10 iterations of Expectation Maximization to alleviate the influence of different initialisation values (Arthur and Vassilvitskii, 2007). The final clustering for a given  $K$  is the one that yields the minimal distortion value across the 10 runs, i.e., the minimal sum of squared distances of each data point from its closest centroid. We experiment with  $K \in [2,10]$ . We choose the range [2,10] heuristically: we forgo  $K = 1$  as  $K$ -Means and the silhouette score are ill-defined for this case, while keeping the number of possible clusters manageable computationally. This excludes the possibility that a word has a single usage type. Alternatively, we could use a measure of intra-cluster dispersion for  $K = 1$ , and consider a word monosemous if its dispersion value is below a threshold  $d$  (if the dispersion is higher than  $d$ , we would discard  $K = 1$ ).
2Other clustering methods are also possible. For this first study, we choose the widely used  $K$ -Means (scikit-learn).
<sup>1</sup>We rely on Hugging Face's implementation of BERT (available at https://github.com/huggingface/transformers).
and use the silhouette score to find the best  $K \geq 2$ . There also exist clustering methods that select the optimal  $K$  automatically, e.g. DBSCAN or Affinity Propagation (Martinc et al., 2020). They nevertheless require method-specific parameter choices which indirectly determine the number of clusters.
By counting the number of occurrences of each usage type  $k$  in a given time interval  $t$  (we refer to this count as  $\text{freq}(k, t)$ ), we obtain frequency distributions  $\mathbf{f}_w^t$  for each interval under scrutiny:
$$
\mathbf {f} _ {w} ^ {t} \in \mathbb {N} ^ {K _ {w}}: \mathbf {f} _ {w} ^ {t} [ k ] = f r e q (k, t) \quad k \in [ 1, K _ {w} ] \tag {1}
$$
When normalised, frequency distributions can be interpreted as probability distributions over usage types  $\mathbf{u}_w^t:\mathbf{u}_w^t [k] = \frac{1}{N_t}\mathbf{f}_w^t [k]$ . Figure 1b illustrates the result of this process.
3.4 Quantifying Semantic Change
We propose three metrics for the automatic quantification of lexical semantic change using contextualised word representations. The first two (entropy difference and Jensen-Shannon divergence) are known metrics for comparing probability distributions. In our approach, we apply them to measure variations in the relative prominence of coexisting usage types. We conjecture that these kinds of metric can help detect semantic change processes that, e.g., lead to broadening or narrowing (i.e., to increase or decrease, respectively, in the number or relative distribution of usage types).
The third metric (average pairwise distance) only requires a usage matrix  $\mathbf{U}_w$  and the temporal labels  $\mathbf{t}_w$  (Section 3.2). Since it does not rely on usage type distributions, it is not sensitive to possible errors stemming from the clustering process.
Entropy difference (ED) We propose measuring the uncertainty (e.g., due to polysemy) in the interpretation of a word  $w$  in interval  $t$  using the normalised entropy of its usage distribution  $\mathbf{u}_w^t$ :
$$
\eta \left(\mathbf {u} _ {w} ^ {t}\right) = \log_ {K _ {w}} \left(\prod_ {k = 1} ^ {K _ {w}} \mathbf {u} _ {w} ^ {t} [ k ] ^ {- \mathbf {u} _ {w} ^ {t} [ k ]}\right) \tag {2}
$$
To quantify how uncertainty over possible interpretations varies across time intervals, we compute the difference in entropy between the two usage type distributions in these intervals:  $\mathrm{ED}(\mathbf{u}_w^t,\mathbf{u}_w^{t'}) = \eta (\mathbf{u}_w^{t'}) - \eta (\mathbf{u}_w^t)$ . We expect high ED values to signal the broadening of a word's interpretation and negative values to indicate narrowing.
Jensen-Shannon divergence (JSD) The second metric takes into account not only variations in the size of usage type clusters but also which clusters have grown or shrunk. It is the Jensen-Shannon divergence (Lin, 1991) between usage type distributions:
$$
\begin{array}{l} \operatorname {J S D} \left(\mathbf {u} _ {w} ^ {t}, \mathbf {u} _ {w} ^ {t ^ {\prime}}\right) = \mathrm {H} \left(\frac {1}{2} \left(\mathbf {u} _ {w} ^ {t} + \mathbf {u} _ {w} ^ {t ^ {\prime}}\right)\right) \tag {3} \\ - \frac {1}{2} \left(\mathrm {H} \left(\mathbf {u} _ {w} ^ {t}\right) - \mathrm {H} \left(\mathbf {u} _ {w} ^ {t ^ {\prime}}\right)\right) \\ \end{array}
$$
where  $\mathrm{H}$  is the Boltzmann-Gibbs-Shannon entropy. Very dissimilar usage distributions yield high JSD whereas low JSD values indicate that the proportions of usage types barely change across periods.
Average pairwise distance (APD) While the previous two metrics rely on usage type distributions, it is also possible to quantify change bypassing the clustering step into usage types, e.g. by calculating the average pairwise distance between usage representations in different periods  $t$  and  $t'$ :
$$
\operatorname {A P D} \left(\mathbf {U} _ {w} ^ {t}, \mathbf {U} _ {w} ^ {t ^ {\prime}}\right) = \frac {1}{N ^ {t} \cdot N ^ {t ^ {\prime}}} \sum_ {\mathbf {x} _ {i} \in \mathbf {U} _ {w} ^ {t}, \mathbf {x} _ {j} \in \mathbf {U} _ {w} ^ {t ^ {\prime}}} d \left(\mathbf {x} _ {i}, \mathbf {x} _ {j}\right) \tag {4}
$$
where  $\mathbf{U}_w^t$  is a usage matrix constructed with occurrences of  $w$  only in interval  $t$ . We experiment with cosine, Euclidean, and Canberra distance.
Generalisation to multiple time intervals The presented metrics quantify semantic change across pairs of temporal intervals  $(t, t')$ . When more than two intervals are available, we measure change across all contiguous intervals  $(m(\mathbf{U}_w^t, \mathbf{U}_w^{t+1})$ , where  $m$  is one of the metrics), and collect these values into vectors. We then transform each vector into a scalar change score by computing the vector's mean and maximum values. Whereas the mean is indicative of semantic change across the entire period under consideration, the max pinpoints the pair of successive intervals where the strongest shift has occurred.
3The Jensen-Shannon divergence can also be measured with respect to  $T > 2$  probability distributions (Ré and Azad, 2014): JSD  $\left(\mathbf{u}_w^1,\dots ,\mathbf{u}_w^T\right) = \mathrm{H}\left(\frac{1}{T}\sum_{i = 1}^{T}\mathbf{u}_w^i\right) - \frac{1}{T}\sum_{i = 1}^{T}\mathrm{H}\left(\mathbf{u}_w^i\right)$ . However, this definition of the JSD is insensitive to the order of the temporal intervals and yields lower correlation with human semantic change ratings (cfr. Section 5.2) than the pairwise metrics.
4 Data
We examine word usages in a large diachronic corpus of English, the Corpus of Historical American English (COHA, Davies, 2012), which covers two centuries (1810-2009) of language use and includes a variety of genres, from fiction to newspapers and popular magazines, among others. In this study, we focus on texts written between 1910 and 2009, for which a minimum of 21M words per decade is available, and discard previous decades, where data are less balanced per decade.
We use the 100 words annotated with semantic shift scores by Gulordava and Baroni (2011) as our target words. These scores are human judgements collected by asking five annotators to quantify the degree of semantic change undertaken by each word (shown out of context) from the 1960's to the 1990's. We exclude extracellular as in COHA this word only appears in three decades; all other words appear in at least 8 decades, with a minimum and maximum frequency of 191 and 108,796, respectively. We refer to the resulting set of 99 words and corresponding shift scores as the 'GEMS dataset' or the 'GEMS words', as appropriate.
We collect a contextualised representation for each occurrence of these words in the second century of COHA, using BERT as described in Section 3.2. This results in a large set of usage representations,  $\sim 1.3\mathrm{M}$  in total, which we cluster into usage types using  $K$ -Means and silhouette coefficients (Section 3.3). We use these usage representations and usage types in the evaluation and the analyses offered in the remaining of the paper.
5 Correlation with Human Judgements
Before using our proposed method to analyse language change, we assess how its key components compare with human judgements. We test whether the clustering into usage types reflects human similarity judgements (Section 5.1) and to what extent the degree of change computed with our metrics correlates with shift scores provided by humans (Section 5.2).
5.1 Evaluation of Usage Types
The clustering of contextualised representations into usage types is one of the main steps in our method (see Section 3.3). It relies on the similarity values between pairs of usage representations created by the language model. To quantitatively evaluate the quality of these similarity values (and thus,
by extension, the quality of usage representations and usage types), we compare them to similarity judgements by human raters.
New dataset of similarity judgements We create a new evaluation dataset, following the annotation approach of Erk et al. (2009, 2013) for rating pairs of usages of the same word. Since we need to collect human judgements for pairs of usages, annotating the entire GEMS dataset would be extremely costly and time consuming. Therefore, to limit the scope of the annotation, we select a subset of words. For each shift score value  $s$  in the GEMS dataset, we sample a word uniformly at random from the words annotated with  $s$ . This results in 16 words. To ensure that our selection of usages is sufficiently varied, for each of these words, we sample five usages from each of their usage types (the number of usage types is word-specific) along different time intervals, one usage per 20-year period over the century. All possible pairwise combinations are generated for each target word, resulting in a total of 3,285 usage pairs.
We use the crowdsourcing platform Figure Eight to collect five similarity judgements for each of these usage pairs. Annotators are shown pairs of usages of the same word: each usage shows the target word in its sentence, together with the previous and the following sentences (67 tokens on average). Annotators are asked to assign a similarity score on a 4-point scale, ranging from unrelated to identical, as defined by Brown (2008) and used e.g., by Schlechtweg et al. (2018). A total of 380 annotators participated in the task. The interrater agreement, measured as the average pairwise Spearman's correlation between common annotation subsets, is 0.59. This is in line with previous approaches such as Schlechtweg et al. (2018), who report agreement scores between 0.57 and 0.68.
Results To obtain a single human similarity judgement per usage pair, we average the scores given by five annotators. We encode all averaged human similarity judgements for a given word in a square matrix. We then compute similarity scores over pairs of usage vectors output by BERT to
${}^{4}$  https://www_figure-eight.com,recently acquired by Appen (https://appen.com).
5The full instructions with examples given to the annotators are available in Appendix A.1.
For this evaluation, BERT is given the same variable-size context as the human annotators. Vector similarity values are computed as the inverse of Euclidean distance, because  $K$ -means relies on this metric for cluster assignments.
obtain analogous matrices per word and measure Spearman's rank correlation between the human- and the machine-generated matrices using the Mantel test (Mantel, 1967).
We observe a significant  $(p < 0.05)$  positive correlation for 10 out of 16 words, with  $\rho$  coefficients ranging from 0.13 to 0.45. $^{7}$  This is an encouraging result, which indicates that BERT's word representations and similarity scores (as well as our clustering methods which build on them) correlate, to a substantial extent, with human similarity judgements. We take this to provide a promising empirical basis for our approach.
5.2 Evaluation of Semantic Change Scores