SlowGuess/NAACL_2022_OCR huggingface dataset:
To train our models, for each language we use a pretrained BERT (Devlin et al., 2019) model (bert-base-uncased, with 12 layers, 768 hidden dimensions, and 110M parameters) and post-pretrain it on the temporal corpus using our proposed temporal attention, as described in Section 3.2. For semantic change detection, we use the method described in Section 4. We use the Hugging Face's Transformers library for our implementation.
Before training, we add any missing target words to the model's vocabulary. Since a pretrained model's vocabulary may not contain all the target words in our evaluation dataset, this is necessary to avoid the tokenizer splitting any occurrences of the target words into subwords (which we found out to reduce performance). The added words are randomly initialized.
5https://github.com/huggingface/ transformers
We measure semantic change detection performance by the correlation between the semantic shift index (i.e., the ground truth) and the model's semantic shift assessment for each word in the evaluation set. We follow prior work (Rosin et al., 2022) and use both Pearson's correlation coefficient  $r$  and Spearman's rank correlation coefficient  $\rho$ . The difference between them is that Spearman's  $\rho$  considers only the ranking order, while Pearson's  $r$  considers the actual predicted values. In our evaluation, we make an effort to evaluate our methods and the baselines using both correlation coefficients, to make the evaluation as comprehensive as possible. There were some cases where we could not reproduce the original authors' results; in such cases, we opted to report only the original result.
5.5 Implementation Details
Due to limited computational resources, we follow Rosin et al. (2022) and train our models with a maximum input sequence length of 128 tokens. We perform all experiments on a single NVIDIA Quadro RTX 6000 GPU. We tune the following hyperparameters for each language: for training: learning rate in  $\{1\mathrm{e} - 8,1\mathrm{e} - 7,1\mathrm{e} - 6,1\mathrm{e} - 5,1\mathrm{e} - 4\}$  and number of epochs in  $\{1,2,3,4\}$ . For inference: number of last hidden layers to use for embedding extraction  $h\in \{1,2,4,12\}$ .
The chosen pretrained model and hyperparameters, along with the steps number and training time per language are as follows:
- For English: bert-base-uncased, with 1e-9 learning rate for 2 epochs (6.3K steps, took 70 minutes); all (12) hidden layers for inference.
- For Latin: latin-bert, $^{7}$  with 1e-5 learning rate for 1 epoch (3.5K steps, took 25 minutes); last hidden layer for inference.
- For German: bert-base-german-cased, with 1e-6 learning rate for 1 epoch (38.1K steps, took 10 hours); last hidden layer for inference.
In this section, we outline the results of our empirical evaluation. In all tables throughout the section, the best result in each column is boldfaced; performance is measured using Pearson's  $r$  and Spearman's  $\rho$  correlation coefficients.
6.1 Main Result
Table 2 shows the results for semantic change detection on the SemEval datasets. Our temporal attention model outperforms all the baselines for all datasets and metrics with significant correlations  $(p < 0.0005)$  and large margins  $(7\% -36\%)$ . We observe moderate to strong correlations (around 0.52-0.76) for all datasets. Even for the German dataset, on which recent BERT-based methods got relatively lower results (and were outperformed by word2vec-based methods such as Schlechtweg et al. (2019)), our model achieves strong correlations and state-of-the-art performance. In Section 6.2 and Section 6.3, we experiment with variations of our method and achieve even stronger performance on the English dataset.
Finally, looking at the three scaled attention baselines, they all perform similarly and are positioned between Martinc et al. (2020a) and our temporal attention model, as expected.
6.2 Temporal Attention with Temporal Prepend
Until now, we used temporal attention on BERT (Devlin et al., 2019) to create our model. In this section, in addition to using temporal attention, we also preprocess a time token to the input sequences, as done in Rosin et al. (2022). That is, we experiment with applying temporal attention on top of their model.
Table 3 shows the results of this combined model compared to each of its components. First, prepending time tokens is inferior to the other models. When comparing our proposed temporal attention and the combined model, we observe mixed results: temporal attention alone works better for the Latin and German datasets, but for the English dataset the combination of temporal attention and prepending time tokens performs better.
6.3 Impact of BERT Model Size on Temporal Attention
Our model is based on the most commonly used pretrained BERT model, called BERT-base, which
$^{6}$ https://huggingface.co/  
bert-base-uncased  
 $^{7}$ https://github.com/dbamman/latin-bert  
 $^{8}$ https://huggingface.co/  
bert-base-german-cased
Method	SemEval-Eng	SemEval-Lat	SemEval-Ger
r	ρ	r	ρ	r	ρ
Schlechtweg et al. (2019)	0.512	0.321	0.458	0.372	-	0.712
Gonen et al. (2020)	0.504	0.277	0.417	0.273	-	0.627
Martinc et al. (2020a)	-	0.315	-	0.496	-	0.565
Montariol et al. (2021)	0.566	0.456	-	0.488	0.618	0.583
Rosin et al. (2022)	0.538	0.467	0.485	0.512	0.592	0.582
Scaled Linear Attention	0.517	0.506	0.524	0.478	0.580	0.550
Scaled Exp. Attention	0.491	0.487	0.633	0.528	0.569	0.526
Scaled by Doc Attention	0.532	0.478	0.657	0.505	0.595	0.567
Temporal Attention	0.620	0.520	0.661	0.565	0.767	0.763
Table 2: Semantic change detection results on SemEval-English, SemEval-Latin, and SemEval-German, measured using Pearson's  $r$  and Spearman's  $\rho$  correlation coefficients.
Method	SE-Eng	SE-Lat	SE-Ger
r	ρ	r	ρ	r	ρ
Temp. Prep.	0.538	0.467	0.485	0.512	0.592	0.582
Temp. Att.	0.620	0.520	0.556	0.556	0.767	0.763
Both	0.655	0.548	0.541	0.508	0.645	0.682
Table 3: Semantic change detection results on the English, Latin, and German datasets, comparing time token preponding (Rosin et al., 2022) with our proposed temporal attention, and a combination of both.
contains 12 transformer layers and a hidden dimension size of 768. In this section, we train and evaluate models of different sizes, namely 'small' and 'tiny', that are based on much smaller pretrained variants of BERT: BERT-small $^{9}$  has  $26\%$  of the parameters of BERT-base, containing only 4 transformer layers while its hidden dimension is 512; BERT-tiny $^{10}$  has just  $4\%$  of the parameters of BERT-base, with 2 transformer layers and a hidden dimension of 128. We perform this evaluation only for the SemEval-English dataset, as smaller pretrained BERT models are currently publicly available only for the English language.
Table 4 shows the comparison results, where we compare the three variants of our temporal attention model, along with the two variants of Rosin et al. (2022). We also denote the number of trainable parameters for each model (see the theoretical analysis in Section 3.3). We observe a clear negative correlation between model size and performance (measured by both Pearson's  $r$  and Spearman's  $\rho$ );
Method	Params	r	ρ
Rosin et al. (2022) base	109.52M	0.538	0.467
Rosin et al. (2022) tiny	4.42M	0.534	0.427
Temp. Att. base	116.61M	0.620	0.520
Temp. Att. small	29.85M	0.660	0.584
Temp. Att. tiny	4.45M	0.703	0.627
Table 4: Results for semantic change detection for models of different sizes on SemEval-English.
the smaller the model, the better the performance. While this finding may sound counterintuitive, it is in line with Rosin et al. (2022), who hypothesized that to understand time there is no need to use extremely large models, and reported higher-than-expected performance for the tiny model. In their study, that model achieved a slightly lower performance compared to their standard (base) model, but still outperformed most baselines. Overall, this is an encouraging finding; smaller models mean faster training and inference times, as well as smaller memory footprints. This lowers the bar to enter the field.
Figure 3 shows the Spearman correlation between the ground truth ranks and our model's ranks for the SemEval-English dataset. The correlation is moderate (0.520), and we observe a similar number of false-positive words (top-left corner) and false negatives (bottom-right corner). Interestingly, we can see that the model performs better on the more changed words (right half, rank above 19, e.g., "plane", "tip", and "head"), while there are more errors on the static words (left half, e.g., "chair
$^{9}$ https://huggingface.co/prajjwall/ bert-small  
 $^{10}$ https://huggingface.co/prajjwall/ bert-tiny
Figure 3: Semantic change detection on the SemEval-English dataset: ground truth ranks vs. our model's ranks (Spearman's  $\rho = 0.520$ ).
man”, “risk”, and “quilt”). Most of the false positives seem to be either slang words or concerning word usages that are less likely to appear in our corpora which is mainly composed of newsletters and books (Section 5.1). For example, the verb “stab”, which traditionally means to push a knife into someone, has a newer meaning of attempting to do something. The noun “word” can be used to express agreement.
In this paper, we presented a time-aware self-attention mechanism as an extension of the original mechanism of the transformer. The proposed mechanism considers the time the text sequences were written when computing attention scores, thus allowing creating time-specific contextualized word representations. We conducted evaluations on the task of semantic change detection and reached state-of-the-art performance on three diverse datasets in terms of time, language, size, and genre. In addition, we experimented with small-sized pretrained models and found they outperform larger models on this task. We conduct an experiment evaluating the marginal addition of time token prepending along with temporal attention and conclude that on all but the English dataset it hurts performance. We wish to study how to best combine the two approaches in future work. Additionally, for future work, we plan to extend this work by applying temporal attention to other tasks, such as web search and sentence time prediction, as well as ex
perimenting with more time points and different granularities.
Oshin Agarwal and Ani Nenkova. 2021. Temporal effects on pre-trained models for language processing tasks. arXiv preprint arXiv:2111.12790.
Robert Bamler and Stephan Mandt. 2017. Dynamic word embeddings. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 380-389. PMLR.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Marco Del Tredici, Raquel Fernandez, and Gemma Boleda. 2019. Short-term meaning shift: A distributional exploration. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2069-2075, Minneapolis, Minnesota. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2022. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257-273.
Haim Dubossarsky, Simon Hengchen, Nina Tahmasebi, and Dominik Schlechtweg. 2019. Time-out: Temporal referencing for robust modeling of lexical semantic change. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 457-470, Florence, Italy. Association for Computational Linguistics.
Mario Giulianielli, Marco Del Tredici, and Raquel Fernandez. 2020. Analysing lexical semantic change with contextualised word representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3960-3973, Online. Association for Computational Linguistics.
Hila Gonen, Ganesh Jawahar, Djamé Seddah, and Yoav Goldberg. 2020. Simple, interpretable and stable method for detecting words with usage change across corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 538-555, Online. Association for Computational Linguistics.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1501, Berlin, Germany. Association for Computational Linguistics.
Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Dynamic contextualized word embeddings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6970-6984, Online. Association for Computational Linguistics.
Spurthi Amba Hombaiah, Tao Chen, Mingyang Zhang, Michael Bendersky, and Marc Najork. 2021. Dynamic language models for continuously evolving content. In KDD 2021.
Renfen Hu, Shen Li, and Shichen Liang. 2019. Diachronic sense modeling with deep contextualized word embeddings: An ecological view. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3899-3908, Florence, Italy. Association for Computational Linguistics.
Xiaolei Huang and Michael J. Paul. 2019. Neural temporality adaptation for document classification: Diachronic word embeddings and domain adaptation models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4113-4123, Florence, Italy. Association for Computational Linguistics.
Adam Jatowt and Kevin Duh. 2014. A framework for analyzing semantic change of words across time. In IEEE/ACM Joint Conference on Digital Libraries, pages 229-238. IEEE.
Nattiya Kanhabua and Avishek Anand. 2016. Temporal information retrieval. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, SIGIR 2016, Pisa, Italy, July 17-21, 2016, pages 1235-1238. ACM.
Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of language through neural language models. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 61-65, Baltimore, MD, USA. Association for Computational Linguistics.
Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detection of linguistic change. In Proceedings of the 24th International Conference on World Wide Web, WWW 2015, Florence, Italy, May 18-22, 2015, pages 625-635. ACM.
Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic word embeddings and semantic shifts: a survey. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1384-1397, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Severin Laicher, Sinan Kurtyigit, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. Explaining and improving BERT performance on lexical semantic change detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 192-202, Online. Association for Computational Linguistics.
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, et al. 2021. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems, 34.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. Timelms: Diachronic language models from twitter. arXiv preprint arXiv:2202.03829.
Matej Martinc, Petra Kralj Novak, and Senja Pollak. 2020a. Leveraging contextual embeddings for detecting diachronic semantic shift. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4811-4819, Marseille, France. European Language Resources Association.
Matej Martinc, Syrielle Montariol, Elaine Zosa, and Lidia Pivovarova. 2020b. Capturing evolution in word usage: Just add more clusters? In *Companion Proceedings of the Web Conference* 2020, pages 343-349.
Syrielle Montariol, Matej Martinc, and Lidia Pivovarova. 2021. Scalable and interpretable semantic change detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4642-4652, Online. Association for Computational Linguistics.
Alex Rosenfeld and Katrin Erk. 2018. Deep neural models of semantic shift. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 474-484, New Orleans, Louisiana. Association for Computational Linguistics.
Guy D. Rosin, Eytan Adar, and Kira Radinsky. 2017. Learning word relatedness over time. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1168-1178, Copenhagen, Denmark. Association for Computational Linguistics.
Guy D. Rosin, Ido Guy, and Kira Radinsky. 2022. Time masking for temporal language models. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 833-841.
Paul Röttger and Janet Pierrehumbert. 2021. Temporal adaptation of BERT and performance on downstream document classification: Insights from social media. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2400-2412, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Maja R. Rudolph and David M. Blei. 2018. Dynamic embeddings for language evolution. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018, pages 1003-1011. ACM.
Pavel Savov, Adam Jatowt, and Radoslaw Nielek. 2021. Predicting the age of scientific papers. In International Conference on Computational Science, pages 728-735. Springer.
Dominik Schlechtweg, Anna Hatty, Marco Del Tredici, and Sabine Schulte im Walde. 2019. A wind of change: Detecting and evaluating lexical semantic change across times and domains. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 732-746, Florence, Italy. Association for Computational Linguistics.
Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. SemEval-2020 task 1: Unsupervised lexical semantic change detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1-23, Barcelona (online). International Committee for Computational Linguistics.
Nina Tahmasebi, Lars Borin, and Adam Jatowt. 2018. Survey of computational approaches to lexical semantic change. arXiv preprint arXiv:1811.06278.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural
Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, and Hui Xiong. 2018. Dynamic word embeddings for evolving semantic discovery. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018, pages 673-681. ACM.