SlowGuess/ACL_2020_OCR huggingface dataset:
English (diachronic, 90y) Top-100 words identified by our method cover all the words attested as real semantic shift in Hamilton et al. (2016b)'s top-10 except the word 'wanting'. Specifically, three attested words, 'gay', 'major' and check' are present in our top-10, which also has more interesting words not present in Hamilton et al. (2016b)'s top-10 (1900 vs. 1990): van (captain vs. vehicle), press (printing vs. places), oxford (location vs. university). In addition, interesting words that came up in the top-30 list are the following: headed (body part vs. move in a direction), mystery (difficulty in understanding vs. book genre).
Occupation (performer vs. sports) Interesting words found at the top-10 list are the following: cc (carbon copy vs. country club), duo (duet vs. pair of people), wing (politics vs. football player position). In addition, interesting words that came up in the top-30 list are the following: jazz (music
genre vs. basketball team), worlds (general meaning vs. championships), stages (platforms vs. company(bikes)), record (music record vs. achievement), harlem (neighborhood vs. basketball team).
7.2 Quantitative Evaluation: Stability
We compare the stability of our method to that of the AlignCos method (Hamilton et al., 2016b) using the intersection@  $k$  metric, as defined in Section 3. We use  $k \in \{10, 20, 50, 100, 200, 500, 1000\}$ .
In Figure 1(a) we plot the intersection@  $k$  for different values of  $k$  for all splits, with solid lines for the results of our method and dashed lines for the results of AlignCos method. It is clear that our method is significantly more stable, for all  $k$  values and across all splits. To better understand the parameters that affect the stability of the different methods, we also examine how the intersection changes with different values of frequency cut-off. In Figure 1(b) we plot intersection@100 as a function of the frequency cut-off (minimum word occurrences required for a word to be included in the ranking). Here, our method is again more stable for all corpus splits. In addition, our method is similarly stable, regardless the frequency cut-off, unlike the AlignCos method. We also examine how the size of NN lists considered for the intersection
(a) Change in intersection@k w.r.t  $k$
(b) Change in intersection@100 w.r.t word frequency cut-off.
(c) Change in intersection@100 w.r.t number of neighbors to consider.
Figure 1: Stability plots. Solid lines: our method, dashed lines: AlignCos method.
— NN Age	— NN Hebrew	— NN Time of week	— NN Occupation (Creator vs Performer)
— — AlignCos Age	— — AlignCos Hebrew	— — AlignCos Time of week	— — AlignCos Occupation (Creator vs Performer)
— NN Gender	— NN Occupation (Performer vs Sports)	— NN Occupation (Creator vs Sports)	— NN French
— — AlignCos Gender	— — AlignCos Occupation (Performer vs Sports)	— — AlignCos Occupation (Creator vs Sports)	— — AlignCos French
affects the stability. In Figure 1(c) we plot the intersection@100 against number of neighbors taken into consideration using our method. We get that from around  $k = 250$ , our method is substantially more stable for all splits.
7.3 Quantitative Evaluation: DUREl and SUREl datasets
This field of semantic change suffers from lack of proper evaluation datasets, and there is no common benchmark that is being used. Two new datasets were recently introduced, and used to extensively compare between previous methods (Schlechtweg et al., 2019): the DUREl dataset (Schlechtweg et al., 2018) focuses on diachronic changes, while the SUREl dataset (Hatty et al., 2019) focuses on domain-based semantic changes. We use them to verify the quality of our results and compare against AlignCos (Hamilton et al., 2016b).
Both datasets include a limited number of German words, along with human annotations of the degrees of semantic relatedness between contexts of the words (across the different texts). However, they are not ideal as they are extremely limited (22 words each)<sup>7</sup>.
Evaluation Metrics Spearman correlation is the standard measure used in this field to compare between methods with respect to gold rankings. However, it is extremely important to note its limitations in this setting, since comparing to a very small gold ranking might be tricky. Specifically, it does
method	measure	SURel	DURel
AlignCos	spearman	0.800	0.814
NN	spearman	0.859	0.59
AlignCos	DCG	-4.5	-4.31
NN	DCG	-4.54	-4.3
Table 3: Results on DUREl and SURel with NN and with AlignCos.
not take into account the global ranking of each method, but only the relative position of each of the gold words in each method's ranking. For example, a method that ranks all the gold words at the bottom of the ranking (out of all the words in the vocabulary) in the same order, would be considered perfect, even though it is clearly not the case.
As a possible solution for this problem, we suggest to use Discounted Cumulative Gain (DCG), which better captures also global rankings. As opposed to Spearman, this measure takes into account not only the order of the words, but also their actual scores:
$$
\mathrm {D C G} (\mathrm {M}) = \sum_ {w \in W} \frac {\text {G o l d S c o r e} (w)}{\log_ {2} \left(\operatorname {r a n k} _ {M} (w) + 1\right)} \tag {3}
$$
where  $W$  are the words in the gold dataset, and  $M$  is the model being evaluated.
We report the results in Table 3. We compute AlignCos results with the best parameters reported in Schlechtweg et al. (2019). Our method outperforms AlignCos on SUREl, both when measur
For our experiments, we follow the setup of Schlechtweg et al. (2019) and use 19/21 words for DUREl/ SURel respectively.
<sup>8</sup>We were unable to reproduce the exact results from the paper: spearman correlation of 0.866 and 0.851 on SUREl and DUREl, respectively.
(a) Female space
(b) Male space
Figure 2: t-SNE visualization of top-50 neighbors from each corpus for word 'clutch', Gender split, with cyan for female and violet for male.
(a) Young space
(b) Older space
Figure 3: t-SNE visualization of top-50 neighbors from each corpus for word 'dam', Age split, with cyan for older and violet for young.
Figures 2 and 3 show the visualizations for the
ing with spearman correlation<sup>9</sup> and with DCG. For DUREl, AlignCos gets better results when measuring with spearman, but both methods are on par when using DCG.
7.4 Interpretation and Visualization
We find that in many cases, it is not clear why the returned candidate words were chosen, and questions such as "why is the word 'dam' different across age groups?" often arise. The NN method lends itself to interpretation, by considering the top-10 neighbors, as shown in Table 2. We note that this interpretation approach is very reliable in our method, as we are guaranteed to gain insights about the usage change when looking at neighboring words, since most of the neighbors will be different for the identified words. While we can definitely attempt at looking at the NN also for the OP-based meth
ods, there we are not guaranteed at all to even spot a difference between the neighbors: it may absolutely be the case that the identified word moved in the embedding space "together" with most of its neighbors. In this case, looking at the neighbors will provide no insight on the nature of this change. We observed this phenomenon in practice. Nonetheless, comparing flat word lists is hard, and 10 words are often insufficient.
We present a visualization method that aids in understanding the model's suggestions. The visualization consists of projecting the word of interest and its top-50 neighbors from each corpus into two dimensions using t-SNE (Maaten and Hinton, 2008), and plotting the result while coloring the neighbors in the intersection in one color and the neighbors unique to each corpus in other colors. We expect the neighbors of a word of interest to have distinct neighbors across the corpora.
<sup>9</sup>Average Spearman score over model runs with different numbers of iterations, as done in (Schlechtweg et al., 2019).
word clutch in the Gender split, with cyan for female and violet for male, and the word dam in the Age split, with cyan for older and violet for young (in both cases they were no shared neighbours). We plot the projection of the words twice – one plot for each embedding space. We can see that, as expected, the neighboring words are distinct, and that the target word belongs to the respective neighborhood in each space. We conclude that this is a useful tool for interpreting the results of our model.
Extensive work has been done on detecting word usage change across corpora that predated the alignment-based methods (Mitra et al., 2014; Jatowt and Duh, 2014; Kenter et al., 2015; Ho et al., 2016; Frermann and Lapata, 2016).
In addition, two works are more closely related to our approach. In Azarbonyad et al. (2017), the authors also use the neighbors of a word in order to determine its stability (and therefore, the extent to which it changes). Their best model combines the traditional alignment-based approach with weighting the neighbors according to their rank and their stability. The algorithm is iterative, and they update the stability of all the words in the vocabulary in each update step. Our method uses the neighbors of the words directly, does not include an iterative process, and does not rely on cosine-distance in the aligned embeddings. In addition, their method requires computation for the whole vocabulary, while other methods, including ours, usually allow querying for a single word.
Another work that considers the neighbors of the word in order to determine the extent of change is that of Hamilton et al. (2016a), in which they suggest a measure that is based on the changes of similarities between the target word and its neighbors in both spaces. They find that this method is more suitable for identifying changes that are due to cultural factors, rather than linguistic shift. This may serve as another motivation to move from the global measures to a local one.
Recent works (Giullianelli, 2019; Martinc et al., 2019) explored the possibility of modeling diachronic and usage change using contextualized embeddings extracted from now ubiquitous Bert representations (Devlin et al., 2019). Focusing on the financial domain, Montariol and Allauzen (2020) use, on top of Bert embeddings, a clustering method that does not need to predefine the number
of clusters and which leads to interesting results on that domain. Another approach from Hu et al. (2019) relies on the inclusion of example-based word sense inventories over time from the Oxford dictionary to a Bert model. Doing so provides an efficient fine-grained word sense representation and enables a seemingly accurate way to monitor word sense change over time. Most of those approaches could be easily used with our method, the inclusion of contextualized embeddings would be for example straightforward, we leave it for future work.
Detecting words that are used differently in different corpora is an important use-case in corpus-based research. We present a simple and effective method for this task, demonstrating its applicability in multiple different settings. We show that the method is considerably more stable than the popular alignment-based method popularized by Hamilton et al. (2016b), and requires less tuning and word filtering. We suggest researchers to adopt this method, and provide an accompanying software toolkit.
We thank Marianna Apidianiaki for her insightful comments on an earlier version of this work. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme, grant agreement No. 802774 (iEXTRACT), and from the Israeli ministry of Science, Technology and Space through the Israeli-French Maimonide Cooperation programme. The second and third authors were partially funded by the French Research Agency projects ParSiTi (ANR-16-CE33-0021), SoSweet (ANR15-CE38-0011-01) and by the French Ministry of Industry and Ministry of Foreign Affairs via the PHC Maimonide France-Israel cooperation programme.
Jacob Levy Abitbol, Marton Karsai, Jean-Philippe Magué, Jean-Pierre Chevrot, and Eric Fleury. 2018. Socioeconomic dependencies of linguistic patterns in twitter: A multivariate analysis. In Proceedings of the 2018 World Wide Web Conference, WWW '18, pages 1125-1134.
Maria Antoniak and David Mimno. 2018. Evaluating the Stability of Embedding-based Word Similarities. Transactions of the Association for Computational Linguistics, 6:107-119.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789-798, Melbourne, Australia. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018b. Unsupervised Neural Machine Translation. In 6th International Conference on Learning Representations, ICLR.
Hosein Azaronyad, Mostafa Dehghani, Kaspar Beelen, Alexandra Arkut, Maarten Marx, and Jaap Kamps. 2017. Words Are Malleable: Computing Semantic Shifts in Political and Media Discourse. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM '17, pages 1509-1518.
Robert Bamler and Stephan Mandt. 2017. Dynamic Word Embeddings. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 380-389, International Convention Centre, Sydney, Australia. PMLR.
Mark Davies. 2015. Corpus of Historical American English (COHA).
Marco Del Tredici, Raquel Fernandez, and Gemma Boleda. 2019. Short-Term Meaning Shift: A Distributional Exploration. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2069-2075, Minneapolis, Minnesota. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Lea Frermann and Mirella Lapata. 2016. A Bayesian Model of Diachronic Meaning Change. Transactions of the Association for Computational Linguistics, 4(0).
Mario Giulianielli. 2019. Lexical semantic change analysis with contextualised word representations. Master's thesis, Institute for Logic, Language and Computation., University of Amsterdam, July.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016a. Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2116-2121, Austin, Texas. Association for Computational Linguistics.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016b. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1501, Berlin, Germany. Association for Computational Linguistics.
Anna Hatty, Dominik Schlechtweg, and Sabine Schulte im Walde. 2019. SURel: A Gold Standard for Incorporating Meaning Shifts into Term Extraction. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 1-8, Minneapolis, Minnesota. Association for Computational Linguistics.
Tin Kam Ho, Luis A. Lastras, and Oded Shmueli. 2016. Concept Evolution Modeling Using Semantic Vectors. In Proceedings of the 25th International Conference Companion on World Wide Web, WWW '16 Companion, pages 45-46. International World Wide Web Conferences Steering Committee.
Renfen Hu, Shen Li, and Shichen Liang. 2019. Diachronic sense modeling with deep contextualized word embeddings: An ecological view. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3899-3908, Florence, Italy. Association for Computational Linguistics.
Adam Jatowt and Kevin Duh. 2014. A Framework for Analyzing Semantic Change of Words Across Time. In Proceedings of the 14th ACM/IEEE-CS Joint Conference on Digital Libraries, JCDL '14, pages 229-238. IEEE Press.
Yova Kementchedjhieva, Sebastian Ruder, Ryan Coterell, and Anders Søgaard. 2018. Generalizing Procrustes Analysis for Better Bilingual Dictionary Induction. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 211-220, Brussels, Belgium. Association for Computational Linguistics.
Tom Kenter, Melvin Wevers, Pim Huijnen, and Maarten de Rijke. 2015. Ad Hoc Monitoring of Vocabulary Shifts over Time. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM '15, pages 1191-1200. ACM.
Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal Analysis of Language through Neural Language Models. In
Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 61-65, Baltimore, MD, USA. Association for Computational Linguistics.
Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detection of linguistic change. In Proceedings of the 24th International Conference on World Wide Web, pages 625-635. International World Wide Web Conferences Steering Committee.
Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic word embeddings and semantic shifts: a survey. In Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In Proceedings of ICLR.
Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018b. Word translation without parallel data. In 6th International Conference on Learning Representations, ICLR.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of machine learning research, 9(Nov):2579-2605.
Matej Martinc, Petra Kralj Novak, and Senja Pollak. 2019. Leveraging contextual embeddings for detecting diachronic semantic shift. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv:1301.3781.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems.
Sunny Mitra, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal. 2014. That's sick dude!! Automatic identification of word sense change across different timescales. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1020-1029, Baltimore, Maryland. Association for Computational Linguistics.
Syrielle Montariol and Alexandre Allauzen. 2020. Éfude des variations sémantiques à travers plusieurs dimensions. In Actes de la conférence conjointe JEP-TALN 2020, Nancy, France.
Maja Rudolph and David Blei. 2018. Dynamic Embeddings for Language Evolution. In Proceedings of the 2018 World Wide Web Conference, WWW '18, pages 1003-1011.
Maja Rudolph, Francisco Ruiz, Susan Athey, and David Blei. 2017. Structured embedding models for grouped data. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pages 250-260, USA. Curran Associates Inc.
Dominik Schlechtweg, Anna Hatty, Marco Del Tredici, and Sabine Schulte im Walde. 2019. A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 732-746, Florence, Italy. Association for Computational Linguistics.
Dominik Schlechtweg, Sabine Schulte im Walde, and Stefanie Eckmann. 2018. Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 169-174, New Orleans, Louisiana. Association for Computational Linguistics.
Peter H Schonemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1-10.
Milan Straka and Jana Straková. 2017. Tokenizing, pos tagging, lemmatizing and parsing ud 2.0 with udpipe. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 88-99, Vancouver, Canada. Association for Computational Linguistics.
Nina Tahmasebi, Lars Borin, and Adam Jatowt. 2018. Survey of Computational Approaches to Lexical Semantic Change. CoRR, abs/1811.06278.
Laura Wendlandt, Jonathan K. Kummerfeld, and Rada Mihalcea. 2018. Factors Influencing the Surprising Instability of Word Embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2092-2102, New Orleans, Louisiana. Association for Computational Linguistics.
Matti Wiegmann, Benno Stein, and Martin Potthast. 2019. Celebrity Profiling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2611-2618, Florence, Italy. Association for Computational Linguistics.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation. In Proceedings of the 2015 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006-1011, Denver, Colorado. Association for Computational Linguistics.
Jaewon Yang and Jure Leskovec. 2011. Patterns of Temporal Variation in Online Media. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, WSDM '11, pages 177-186, New York, NY, USA. ACM.
Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, and Hui Xiong. 2018. Dynamic Word Embeddings for Evolving Semantic Discovery. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM '18, pages 673-681. ACM.
Tokenization We tokenize the English, French and Hebrew tweets using ark-twokenize-py<sup>10</sup>, Moses tokenizer<sup>11</sup> and UDPipe (Straka and Strakova, 2017), respectively. We lowercase all the tweets and remove hashtags, mentions, retweets and URLs. We replace all the occurrences of numbers with a special token. We discard all words that do not contain one of the following: (1) a character from the respective language; (2) one of these punctuation: “-”, “,”, “;”; (3) emoji.
Word embeddings We construct the word representations by using the continuous skip-gram negative sampling model from Word2vec (Mikolov et al., 2013a,b). We use the Gensim $^{12}$  implementation. For all our experiments, we set vector dimension to 300, window size to 4, and minimum number of occurrences of a word to 20. The rest of the hyperparameters are set to their default value.
For the stability experiments we run the embedding algorithm twice, each time with a different random seed.
B Qualitative Evaluation: Detected Words
We show the top-10 words our method yields for each of the different splits, accompanied with the nearest neighbors in each corpus (excluding words in the intersection), to better understand the context. For comparison, we also show the top-10 words according to the AlignCos method. The splits are the following:
English: 1900 vs. 1990 The list of top-10 detected words from our method (NN) vs. AlignCos method, for corpus split according to the year of the English text is displayed in Table 4.