SlowGuess/ACL_2020_OCR huggingface dataset:
We now quantitatively assess the semantic change scores yielded by the metrics described in Section 3.4 when applied to BERT usage representations and the usage types created with our approach. We do so by comparing them to the human shift scores in the GEMS dataset. For consistency with this dataset, which quantifies change from the 1960's to the 1990's as explained in Section 4, we only consider these four decades when calculating our scores. Using each of the metrics on representations from these time intervals, we assign a semantic change score to all the GEMS words. We then compute Spearman's rank correlation between the automatically generated change scores and the gold standard shift values.
Results Table 1 shows the Spearman's correlation coefficients obtained using our metrics, together with a frequency baseline (the difference between the normalised frequency of a word in the 1960's and in the 1990's). The three proposed metrics yield significant positive correlations. This is again a very encouraging result regarding the potential of contextualised word representations for capturing lexical semantic change.
As a reference, we report the correlation coefficients with respect to GEMS shift scores documented by the authors of two alternative approaches: the count-based model by Gulordava and Baroni (2011) themselves (trained on two time slices from the Google Books corpus with texts from the 1960's and the 1990's) and the sense-based SCAN model by Frermann and Lapata (2016) (trained on the DATE corpus with texts from the 1960's through the 1990's).
Frequency difference	0.068
Entropy difference (max)	0.278
Jensen-Shannon divergence (max)	0.276
Average pairwise distance (Euclidean, max)	0.285
Gulordava and Baroni (2011)	0.386
Frermann and Lapata (2016)	0.377
Table 1: Spearman's  $\rho$  correlation coefficients between the gold standard scores in the GEMS dataset and the change scores assigned by our three metrics and a relative frequency baseline. For reference, correlation coefficients reported by previous works using different approaches are also given. All correlations are significant  $(p < 0.05)$  except for the frequency difference baseline.
For all our metrics, the max across the four time intervals—i.e., identifying the pair of successive intervals where the strongest shift has occurred (cfr. end of Section 3.4)—is the best performing aggregation strategy. Table 1 only shows values obtained with max and Euclidean distance for APD, as they are the best-performing options.
It is interesting to observe that APD can prove as informative as JSD and ED, although it does not depend on the clustering of word occurrences into usage types. Yet, computing usage types offers a powerful tool for analysing lexical change, as we will see in the next section.
In this section, we provide an in-depth qualitative analysis of the linguistic properties that define usage types and the kinds of lexical semantic change we observe. More quantitative methods (such as taking the top  $n$  words with highest JSD, APD and ED and checking, e.g., how many cases of broadening each metric captures) are difficult to operationalise (Tang et al., 2016) because there exist no well-established formal notions of semantic change types in the linguistic literature. To carry out this analysis, for each GEMS word, we identify the most representative usages in a given usage type cluster by selecting the five closest vectors to the cluster centroid, and take the five corresponding sentences as usage examples.
6.1 What do Usage Types Capture?
We first leave the temporal variable aside and present a synchronous analysis of usage types. Our
However, to allow for direct comparison, Frermann and Lapata (2016) computed Spearman correlation for that work (see their footnote 7), which is the value we report.
<sup>7</sup>Scores per target word are given in Appendix A.2.
Gulordava and Baroni (2011) report Pearson correlation.
goal is to assess the interpretability and internal coherence of the obtained usage clusters.
We observe that usage types can discriminate between underlying senses of polysemous (and homonymous) words, between literal and figurative usages, and between usages that fulfil different syntactic roles; plus they can single out phrasal collocations as well as named entities.
Polysemy and homonymy Distinctions often occur between underlying senses of polysemous and homonymous words. For example, the vectors collected for the polysemous word curious are grouped together into two usage types, depending on whether curious is used to describe something that excites attention as odd, novel, or unexpected ('a wonderful and curious and unbelievable story') or rather to describe someone who is marked by a desire to investigate and learn ('curious and amazed and innocent'). The same happens for the homonymous usages of the word coach, for instance, which can denote vehicles as well as instructors (see Figure 2a for a diachronic view of the usage types).
Metaphor and metonymy In several cases, literal and metaphorical usages are also separated. For example, occurrences of curtain are clustered into four usage types (Figure 2c): two of these correspond to a literal interpretation of the word as a hanging piece of cloth ('curtainless windows', 'pulled the curtain closed') whereas the other two indicate metaphorical interpretations of curtain as any barrier that excludes the free exchange of information or communication ('the curtain on the legal war is being raised'). Similarly, we obtain two usage types for sphere: one for literal usages that denote a round solid figure ('the sphere of the moon'), and the other for metaphorical interpretations of the word as an area of knowledge or activity ('a certain sphere of autonomy') as well as metonymical usages that refer to the planet Earth ('land and peoples on the top half of the sphere').
Syntactic roles and argument structure Further distinctions are observed between word usages that fulfil a different syntactic functionality: not only is part-of-speech ambiguity detected (e.g., 'the cost-tapered average tariff' vs. 'cost less to make') but contextualised representations also capture regularities in syntactic argument structures. For example, usages of refuse are clustered into nominal usages ('society's emotional refuse', 'the amount of refuse'), verbal transitive and intransi
tive usages ('fall, give up, refuse, kick'), as well as verbal usages with infinitive complementation ('refuse to go', 'refuse for the present to sign a treaty').
Collocations and named entities Specific clusters are also assigned to lexical items that are parts of phrasal collocations (e.g., 'iron curtain') or of named entities ('alexander graham bell' vs. 'bell-like whistle').
Other distinctions Some distinctions are interpretable but unexpected. As an example, the word doubt does not show the default noun-verb separation but rather a distinction between usages in affirmative contexts ('there is still doubt', 'the benefit of the doubt') and in negative contexts ('there is not a bit of doubt', 'beyond a reasonable doubt').
Observed errors For some words, we find that usages which appear to be identical are separated into different usage types. In a handful of cases, this seems due to the setup we have used for experimentation, which sets the minimum number of clusters to 2 (see Section 3.3). This leads to distinct usage types for words such as maybe, for which a single type is expected. In other cases, a given interpretation is not identified as an independent type, and its usages appear in different clusters. This holds, for example, for the word tenure, whose usages in phrases such as 'tenure-track faculty position' are present in two distinct usage types (see Figure 2b).
Finally, we see that in some cases a usage type ends up including two interpretations which arguably should have been distinguished. For example, two of the usage types identified for address are interpretable and coherent: one includes usages in the sense of formal speech and the other one includes verbal usages. The third usage type, however, includes a mix of nominal usages of the word as in 'disrespectful manners or address' as well as in 'network address'.
6.2 What Kinds of Change are Observed?
Here we consider usage types diachronically. Different kinds of change, driven by cultural and technological innovation as well as by historical events, emerge from a qualitative inspection of usage distributions along the temporal dimension. We describe the most prominent kinds—narrowing and broadening, including metaphorisation—and discuss the extent to which our metrics are able to detect them.
3966
(a)coach
(b) tenure
(c) curtain
(d) disk
Figure 2: Evolution of usage type distributions in the period 1910-2009, generated with occurrences of coach, tenure, curtain and disk in COHA (Davies, 2012). The legends show sample usages per identified usage type.
Narrowing Examination of the dynamics of usage distributions allows us to see that for a few words certain usage types disappear or become less common over time (i.e., the interpretation of the word becomes 'narrower', less varied). This is the case, for example, for coach, where the frequency decrease of a usage type is gradual and caused by technological evolution (see Figure 2a).
Negative mean ED (see Section 3.4) reliably indicates this kind of narrowing. Indeed coach is assigned one of the lowest ED score among the GEMS words. In contrast, ED fails to detect the obsolescence of a usage type when new usage types emerge simultaneously (since this may lead to no entropy reduction). This is the case, e.g., of tenure. The usage type capturing tenure of a landed property becomes obsolete; however, we obtain a positive mean ED caused by the appearance of a new usage type (the third type in Figure 2b).
Broadening For a substantial amount of words, we observe the emergence of new usage types (i.e., a 'broadening' of their use). This may be due to
technological advances as well as to specific historical events. As an example, Figure 2d shows how, starting from the 1950's and as a result of technological innovation, the word disk starts to be used to denote also optical disks while beforehand it referred only to generic flat circular objects.
A special kind of broadening is metaphorisation. As mentioned in Section 6.1, the usage types for the word curtain include metaphorical interpretations. Figure 2c allows us to see when the metaphorical meaning related to the historically charged expression iron curtain is acquired. This novel usage type is related to a specific historical period: it emerges between the 1930's and the 1940's, reaches its peak in the 1950's, and remains stably low in frequency starting from the 1970's.
The metrics that best capture broadening are JSD and APD—e.g., disk is assigned a high semantic change score by both metrics. Yet, sometimes these metrics generate different score rankings. For example, curtain yields a rather low APD score due to the low relative frequency of the novel usage (Figure 2c). In contrast, even though the novel us
age type is not very prominent in some decades, JSD can still discriminate it and measure its development. On the other hand, the word address, for which we also observe broadening, is assigned a low score by JSD due to the errors in its usage type assignments pointed out in Section 6.1. As APD does not rely on usage types, it is not affected by this issue and does indeed assign a high change score to the word.
Finally, although our metrics help us identify the broadening of a word's meaning, they cannot capture the type of broadening (i.e., the nature of the emerging interpretations). Detecting metaphorisation, for example, may require inter-cluster comparisons to identify a metaphor's source and target usage types, which we leave to future work.
We have introduced a novel approach to the analysis of lexical semantic change. To our knowledge, this is the first work that tackles this problem using neural contextualised word representations and no lexicographic supervision. We have shown that the representations and the detected semantic shifts are aligned to human interpretation, and presented a new dataset of human similarity judgements which can be used to measure said alignment. Finally, through extensive qualitative analysis, we have demonstrated that our method allows us to capture a variety of synchronic and diachronic linguistic phenomena.
Our approach offers several advantages over previous methods: (1) it does not rely on a fixed number of word senses, (2) it captures morphosyntactic properties of word usage, and (3) it offers a more effective interpretation of lexical meaning by enabling the inspection of particular example sentences. In recent work, we have experimented with alternative ways of obtaining usage representations (using a different language model, fine-tuning, and various layer selection strategies) and we have obtained very promising results in detecting semantic change across four languages (Kutuzov and Giulianelli, 2020). In the future, we plan to investigate whether usage representations can provide an even finer grained account of lexical meaning and its dynamics, e.g., to automatically discriminate between different types of meaning change. We expect our work to inspire further analyses of variation and change which exploit the expressiveness of contextualised word representations.
This paper builds upon the preliminary work presented by Giulianelli (2019). We would like to thank Lisa Beinborn for providing useful feedback as well as the three anonymous ACL reviewers for their helpful comments. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 819455).
David Arthur and Sergei Vassilvitskii. 2007. k-means++: The Advantages of Careful Seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 1027-1035. Society for Industrial and Applied Mathematics.
Robert Bamler and Stephan Mandt. 2017. Dynamic Word Embeddings. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 380-389. JMLR.org.
Susan Windisch Brown. 2008. Choosing Sense Distinctions for WSD: Psycholinguistic Evidence. In Proceedings of ACL-08: HLT, Short Papers, pages 249-252, Columbus, Ohio. Association for Computational Linguistics.
Claudia Marlea Brugman. 1988. The Story of Over: Polysemy, Semantics, and the Structure of the Lexicon. Garland, New York.
Joan Bybee. 2015. Language Change. Cambridge University Press.
Paul Cook, Joy Han Lau, Diana McCarthy, and Timothy Baldwin. 2014. Novel Word-Sense Identification. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1624-1635.
Andrew M Dai and Quoc V Le. 2015. Semi-supervised Sequence Learning. In Advances in Neural Information Processing Systems, pages 3079-3087.
Mark Davies. 2012. Expanding Horizons in Historical Linguistics with the 400-Million Word Corpus of Historical American English. Corpora, 7(2):121-157.
Marco Del Tredici, Raquel Fernandez, and Gemma Boleda. 2019. Short-Term Meaning Shift: A Distributional Exploration. In Proceedings of NAACL-HLT 2019 (Annual Conference of the North American Chapter of the Association for Computational Linguistics).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Haim Dubossarsky, Yulia Tsvetkov, Chris Dyer, and Eitan Grossman. 2015. A Bottom Up Approach to Category Mapping and Meaning Change. In *Word Structure and Word Usage*. Proceedings of the NetWordS Final Conference, pages 66-70.
Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2009. Investigations on Word Senses and Word Usages. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 10-18, Suntec, Singapore. Association for Computational Linguistics.
Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2013. Measuring Word Meaning in Context. Computational Linguistics, 39(3):511-554.
Katrin Erk and Sebastian Padó. 2008. A Structured Vector Space Model for Word Meaning in Context. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 897-906.
Katrin Erk and Sebastian Padó. 2010. Exemplar-Based Models for Word Meaning in Context. In Proceedings of the ACL 2010 Conference (Short Papers), pages 92-97.
Lea Frermann and Mirella Lapata. 2016. A Bayesian Model of Diachronic Meaning Change. Transactions of the Association for Computational Linguistics, 4:31-45.
Mario Giulianielli. 2019. Lexical Semantic Change Analysis with Contextualised Word Representations. Master's thesis, University of Amsterdam, July.
Kristina Gulordava and Marco Baroni. 2011. A Distributional Similarity Approach to the Detection of Semantic Change in the Google Books Ngram Corpus. In Proceedings of the GEMS 2011 Workshop on Geometrical Models of Natural Language Semantics, pages 67-71.
William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1501.
Paul J Hopper et al. 1991. On Some Principles of Grammaticization. Approaches to Grammaticalization, 1:17-35.
Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339.
Renfen Hu, Shen Li, and Shichen Liang. 2019. Diachronic Sense Modeling with Deep Contextualized Word Embeddings: An Ecological View. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3899-3908, Florence, Italy. Association for Computational Linguistics.
Adam Kilgarriff. 1997. I Don't Believe in Word Senses. Computers and the Humanities, 31(2):91-113.
Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal Analysis of Language through Neural Language Models. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 61-65.
Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically Significant Detection of Linguistic Change. In Proceedings of the 24th International Conference on World Wide Web, pages 625-635. International World Wide Web Conferences Steering Committee.
Andrey Kutuzov and Mario Giulianelli. 2020. UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection. Forthcoming.
Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic Word Embeddings and Semantic Shifts: A Survey. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1384-1397.
Jey Han Lau, Paul Cook, Diana McCarthy, Spandana Gella, and Timothy Baldwin. 2014. Learning Word Sense Distributions, Detecting Unattested Senses and Identifying Novel Senses Using Topic Models. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 259-270.
Jey Han Lau, Paul Cook, Diana McCarthy, David Newman, and Timothy Baldwin. 2012. Word Sense Induction for Novel Sense Detection. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 591-601. Association for Computational Linguistics.
Jianhua Lin. 1991. Divergence Measures Based on the Shannon Entropy. IEEE Transactions on Information theory, 37(1):145-151.
Peter Ludlow. 2014. Living Words: Meaning Underdetermination and the Dynamic Lexicon. OUP Oxford.
Suresh Manandhar, Ioannis P Klapaftis, Dmitriy Dligach, and Sameer S Pradhan. 2010. SemEval-2010 Task 14: Word Sense Induction & Disambiguation. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 63-68. Association for Computational Linguistics.
Nathan Mantel. 1967. The Detection of Disease Clustering and a Generalized Regression Approach. Cancer Research, 27(2):209-220.
Matej Martinc, Syrielle Montariol, Elaine Zosa, and Lidia Pivovarova. 2020. Capturing Evolution in Word Usage: Just Add More Clusters? In *Companion Proceedings of the International World Wide Web Conference*, pages 20-24.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in Translation: Contextualized Word Vectors. In Advances in Neural Information Processing Systems, pages 6294-6305.
Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K Gray, Joseph P Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, et al. 2011. Quantitative Analysis of Culture Using Millions of Digitized Books. Science, 331(6014):176-182.
Sunny Mitra, Ritwik Mitra, Suman Kalyan Maity, Martin Riedl, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2015. An Automatic Approach to Identify Word Sense Changes in Text Media across Timescales. *Natural Language Engineering*, 21(5):773-798.
Sunny Mitra, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal. 2014. That's Sick Dude! Automatic Identification of Word Sense Change across Different Timescales. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1020-1029.
Roberto Navigli and Daniele Vannella. 2013. SemEval-2013 Task 11: Word Sense Induction and Disambiguation within an End-User Application. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 193-201.
Patrick Pantel and Dekang Lin. 2002. Discovering Word Senses from Text. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '02, page 613-619, New York, NY, USA. Association for Computing Machinery.
Carita Paradis. 2011. Metonymization: A Key Mechanism in Semantic Change. Defining Metonymy in Cognitive Linguistics: Towards a Consensus View, pages 61-98.