Methodological Insights in Detecting Subtle Semantic Shifts with Contextualized and Static Language Models
Authors: Sanne Hoeken
Özge Alacam
Antske Fokkens
Pia Sommerauer
Venue: EMNLP 2023 Findings (2023)
Keywords: semantic shift detection
contextualized embeddings
static embeddings
political communities
TL;DR: This paper investigates the detection of subtle semantic shifts between political communities using static and contextualized language models.

Abstract: In this paper, we investigate automatic detection of subtle semantic shifts between social communities of different political convictions in Dutch and English. We perform a methodological study comparing methods using static and contextualized language models. We investigate the impact of specializing contextualized models through fine-tuning on target corpora, word sense disambiguation and sentiment. We furthermore propose a new approach using masked token prediction, that relies on behavioral information, specifically the most probable substitutions, instead of geometrical comparison of representations. Our results show that methods using static models and our masked token prediction method can detect differences in connotation of politically loaded terms, whereas methods that rely on measuring the distance between contextualized representations are not providing clear signals, even in synthetic scenarios of extreme shifts.

Methodological Insights in Detecting Subtle Semantic Shifts with Contextualized and Static Language Models
Sanne Hoeken', Özge Alaçam 1.2 Antske Fokkens and Pia Sommerauer
Compputatinnll Linguistics, Dept. of Linguistics, Bielefeld University
CCnteer for Information and Language Processing, LMU Munich
3 CComputational Linguistics & Text Mining Lab, Vrije Universiteit Amsterdam
DDepartment of Mathematics and Computer Science, Eindhoven University of Technology de s. fokkens, pia. sommerauer n1
(sanne hoeken, oezge.
Abstract
and methods for detecting subtle shifts in conno- tation. The best-performing methods for LSCD predominantly rely on static embedding representa- tions. Several methods for contextualized models have been proposed but have generally been outper- formed by static methods on LSCD tasks (Kutuzov et al., 2022; Schlechtweg et al., 2020). Incidental results from Russian and Spanish shared task (Ku- tuzov and Pivovarova, 2021: Zamora-Reing et al.. 2022) show that contextualized models can outper- form static models when given the right informa- tion through fine-tuning on a task that foregrounds sense information. In this paper, we systematically investigate different methods using contextualized and static models on use case of subtle semantic shifts in political discourse. We test different ap- proaches to prioritize relevant information in the model and investigate how contextualized models react to information in the input data.
this paper, we investigate automatic detec- tion of subtle semantic shifts between social communities of different political convictions Dutch and English. We perform a method- ological study comparing methods using static and contextualized language models. We in- vestigate the impact of specializing contextual- ized models through fine-tuning on target cor- pora, word sense disambiguation and sentiment. We furthermore propose new approach us- ing masked token prediction which relies on behavioral information, specifically most probable substitutions, instead of geometrical comparison of representations. Our results show that methods using static models and our masked token prediction method can detect dif- ferences in connotation of politically loaded terms, whereas methods that rely measuring the distance between contextualized represen- tations are not providing clear signals, even in synthetic scenarios of extreme shifts.!
Our experiments yield the following insights: (1) Standard methods relying on contextualized rep- resentations fail detect subtle shifts. We use a data manipulation approach to simulate extreme shifts and demonstrate that none of the methods can detect them. Our experiments confirm the intu- ition from Schlechtweg et al. (2020) that informa- tion learned in pretraining provides such a strong signal that contexts reflecting target token do not impact its representation. (2) The gest-performing LSCD system of the Russian and Spanish shared task (Rachinskiy and Arefyev, 2022) did not perform best on our subtle shift task nor our evaluation on the English subtask of Se- mEval 2020. (3) We propose an alternative method for contextualized models relying on masked token prediction rather than representation comparison inspired by Arefyev and Zhikov (2020). In contrast to the representation-bast based contextualized methods, this method able to predict subtle shifts. Our sub- stitution method has the additional advantage that is more transparent, as the predicted substitutions
1 Introduction
With the increasing popularity and success of lan- guage models, Lexical Semantic Change Detection (LSCD) has developed into its own research field, The primary focus has been placed on the auto- matic detection of extreme semantic shifts across time periods (i.e. diachronic shifts). More subtle semantic shifts such as differences in connotations attributed to words by socially different commu- nities have received little attention. Such shifts can, however, provide highly relevant insights for social scientists, they may reflect fundamental differences in world views. As connotations and associations are likely to be reflected by the con- texts of words, static and contextualized language models have the potential to capture them.
sense of a
The current state of the field does not provide conclusive predictions of the most effective models
'The code used for the experiments can be found at https: //gi trub.huu..cc com/ Swweeeeeeeeeoeee/SS

---

give insights into the nature of the shifts. In ad- dition to our main insights, we confirm that word form has an impact on the performance of contex- tualized models (as suggested by Schlechtweg et al. (2020) and examined by Laicher et al. (2021)) raw test data is provided, instead of lemmatized (which is a de-facto input style for lexical change experiments), BERT outperforms static models on SemEval.
representations of target word. The representa- tions are then compared across the corpora (either by means of a pairwise comparison or by creat- ing aggregated word type or sense representations) (Giulianelli et al., 2020; Martinc et al., 2020, e.g.). Alternatively, pretrained language model can also be leveraged for masked token prediction. Arefyev and Zhikov (2020) extract the top k predicted can- didates for the target word induce clusters rep- resenting word senses. They then compare the occurrence frequency of these word senses across corpora. In both approaches, the pretrained lan- guage model can be fine- tinettneddee to the corpora under investigation by means of continued pretaining (Ku- tuzov and Giulianelli, 2020; Martine et al., 2020; Montariol et al., 2021, e.g.), which has been shown to lead to performance gains.
2 Background and Related Work
this section, we first provide an overview of ap- proaches for Lexical Semantic Change Detection (LSCD), the task of automatically detecting shifts in semantics. 2 This is followed by brief presenta- tion of proposed approaches for validating LSCD methods and a discussion of work that addresses social variation rather than diachronic change.
The pretrained (and domain-adapted) model are also further fine-tuned on a specific task to incor- porate information relevant to semantic change. So far, a method employing fine-tuning for Word Sense Disambiguation is the only method for con- textualized models that has been shown to out- perform static models (Rachinskiy and Arefyev, 2022) on two recent shared tasks on LSCD in Rus- sian (Kutuzov and Pivovarova, 2021) and Spanish (Zamora-Reine et al., 2022), respectively. The in- tuition behind this approach is that the fine-tuning step foregrounds information about word senses rather than word forms. This addresses one of the issues identified by Laicher et al. (2021), which pertains to the orthographic or grammatical bias observed in contextualized language models when performing LSCD.
2.1 Semantic shifts
Semantic shifts occur when words undergo a change in meaning. The most well-known scenario for this is diachronic semantic change, where words acquire new meanings over time, while old mean- ings may disappear. Another type of shift can occur due to social variation; different communities may use words in different senses or ascribe different connotations to them (Newman, 2015). Distribu- tional approaches hold great potential for study- ing both types of shifts, as shifts in word meaning should be reflected by differences in their contexts within community- or time-specific corpora.
Static Embeddings Count- and predict methods for static embeddings be used to detect seman- tic shifts by creating separate models for each cor- pus. Semantic shifts are detected by either compar- ing the distance between vector representations of the same word across the target corpora (requiring aligned model spaces) or by comparing the relative positions of a word in the semantic spaces (e.g. by comparing neighborhoods or distances to selected other words) (Hamilton al., 201 ba,b; Kim et al., 2014; Gulordava and Baroni, 201 1).
Static vs Contextualized Methods for both model types rely on the assumption that seman- tic changes in words are reflected in their contexts. However, they differ fundamentally in terms of how the models are created; while contextualized approaches for LSCD use already pretrained mod- els, static approaches create new models rep- resent each corpus. Apart from some diachronic approaches where the model for the previous time period is used as a starting point (Kim et al., 2014), they are largely created from scratch taking co- occurrences of the full vocabulary into account. Methods based on contextualized models extract instance representations from corpora using pre- trained models. The representations thus incorpo- rate information from their pretraining, which re- mains identical across the corpora being compared.
Contextualized Models Pretrained contextual- ized language models can be used to detect seman- tic shifts in various ways. One common approach is to use an existing pretrained model to extract
WWo focus on approuches closest to ours and refer the reader to Kutuzov et al. (2022) and Tahmaschi et al. (2021) or Montanelli and Periti (2023) for more extensive overviews.

---

Based our initial experiments, we observed that methods relying on contextualized representations did not reflect subtle shifts at all. Addressing the possible explanation of pretraining dominance, we simulate scenarios of extreme shift via replace- ment strategy.
the best of our knowledge, our study is the first investigate the detection of such shifts in political discourse using spectrum of methods based on static and contextualized models.
Task & Data
In this study, we investigate the use of language models for the detection of subtle semantic shifts between different social communities. We intro- duce the following objective to address this issue:
2.2 LSCD for Subtle Synchronic Shifts
While LSCD methods have been widely developed for historical purposes, only few studies exam- ined synchronic semantic shifts. Del Tredici and Fernández (2017) adapt the LSCD method of Du- bossarsky al. (2019) by replacing the temporal component with a community variable to compare language use across Reddit communities. Lucy and Bamman (2021) also dive into differences between Reddit communities, but explore the possibilities of using contextualized models following an approach similar to the Word Sense Induction technique of Giulianelli et al. (2020). These approaches focus clear denotation shifts.
Task: Given (i) a target word whose con- notative meaning is assumed to differ between two communities, and (ii) two subcorpora representing (one of) the two communities in which the target word occurs, determine whether these subcor- pora originate from the same community or different communities.
We address subtle semantic shifts through two use cases focusing political discourse in Dutch (use case 1) and English (use case 2). We lever- age user-generated web-data from Reddit, social media platform that enables the formation com- munities (subreddits) based on a wide range of topics and interests. We use political subreddits and examine how politically loaded terms undergo connotational semantic shifts across different com- munities. For example, the word climate carries the connotation of climate change being hoax in right-wing discourse, while it tends imply a genuine threat left-wing discourse.
More subtle shifts in connotation rather than de- notation have hardly been explored. Basile et al. (2022) propose system to automatically detect subtle shifts in connotation using static embed- dings, indicating the potential to extract them from distributional information. Their focus is on shifts along a single connotation 'dimension' such as po- larity. In contrast, we focus on shifts that are hard to capture by a singular dimension, encompassing variances in attitudes and worldviews within spe- cific political communities.
Use case We collect comments from two Dutch subreddits on opposing ends of the Dutch political spectrum: soldersocialisme (PS), left-wing community, and Forum_Democre wing community. We have collected all the existing comments from these two subreddits up until April 3, 2022, using the Pushshift API (Baumgartner et al., 2020).
The most widely used evaluation dataset came with the first SemEval shared task on LSCD in 2020 (Schlechtweg et al., 2020) and contains a relatively small set of clear diachronic shifts in denotation (e.g. plain, which gained the meaning of aircraft next to the meaning of surface) for four languages. We conduct experiments on the binary LSCD task for English, which involves determining whether the meanings of 37 target words have changed or not between the time spans of 1810-1860 and 1960- 2010. The results on this challenge indicate that overall, methods based on static representations outperform those relying on contextualized models. Nevertheless, given the state of the field, where incidentally contextualized representation methods also demonstrated superior performance (in another task as mentioned earlier), is unclear what ap- proach would perform best on our use case of sub- tle semantic shifts across political communities. To
(FD), right-
Use case We have collected all the comments from the English subreddits hillaryclinton (HC) and The The_Donald (TD) using a similar ap- proach. We specifically focus on the year 2016, the year of the presidential elections between Don- ald Trump and Hillary Clinton. The statistics of all collected Reddit data presented in Table and more details are described in Appendix A.
In each of the use cases, we took one of the datasets, FD and TD respectively, and split it ran- domly into two subcorpora. This resulted in three

---

Subreddit
Poldersocialisme
Forum Democratie
hillaryclinton
The Donald
May 2018
16 Dec. 2017
Jan. 2016
Jan. 2016
Perind
3 April 2022
April 2022
31 Dec. 2016
31 Dec. 2016 10 452 040
Comments
58 537
149 674
137 508
Tokens
3 077 786
526 494 57
48 902 320
341 901 610
Average length 53
43
33
Table Statistics of the collected Reddit data for the two use cases. Tokens are based on BERT subword tokenization and average comment length is given in tokens.
subcorpora for each language: two from the same community (FD1 and FD2 for the Dutch case, and TDI and TD2 for the English case) and one subcor- pus from the opposing community (PS for Dutch and HC for English). This division allowed us to compare the detection of semantic shifts within the same community and between different communi- ties.
concept of climate (particularly in reference to cli- mate change), disparities between the two Dutch commumities extend beyond evaluative connota- tions. FD members associate the concept with hys- teria while PS members view it as a problem. We refer to this latter type as associative connotation. Table 2 illustrates our division of target words into evaluative and associative connotation types, visually denoted by horizontal line. It is impor- tant to acknowledge that this categorization relies heavily on expertise in political discourse well as a certain degree of subjectivity.
Target words The target words for the experi- ments were carefully selected to test for hypothe- sized shift in connotation between the communities. Around a hundred random examples from each sub- reddit were taken to identify main topics that users discussed, and related keywords that go along with it. Comments mentioning these keywords were ex- tracted (rule-based script) and about 50 comments (randomly sampled) per keyword were further ana- lyzed by the first author with consultation from an expert political communication. The aim of the analysis was to grasp how the members of the op- posing communities conceptualized the keywords. Other relevant keywords that popped up frequently in the analysis were added to the list of keywords; words and topics that did not show contrastive con- ceptualizations after analysis were omitted. Table presents our final selection of target words, along with their frequency of occurrence in the three sub- corpora for each use case. We present summary of the specific hypotheses for each target word in Appendix B and a more detailed description of tar- get word selection and hypothesis generation for the Dutch use
Evaluation data and metries We conduct exper- iments on our use-cases by testing each method on the above outlined set of 12 target words for each language. For each target word, two test sce- narios are provided: a) two subcorpora originating from different communities b) two subcorpora orig- inating from the same community (control). This results in a total of 24 test instances, which we pro- vide with a label being either "same-community" "different-comee community"" For each of these instances a method outputs a distance value, the graded out- comes. We convert the graded outcomes to binary values by comparing the distance values of the two test scenarios for target word with each other, and assign "different community" to the largest value, and "same community" to the smallest. Sub- sequently, we evaluate the number of correct binary predictions as well as the correlation between the methods' graded distance values and the gold la- bels (converted to and 0). The latter is computed by calculating the Pearson Correlation Coefficient using the SciPy library. It should be considered that this distance comparison approach is limited as even very small differences between representa- tions can lead to different classifications. Just like the SemEval test set, our test set is relatively small, We employ additional validation strategies to gain deeper insights into what signals various methods of shift detection can pick up (see Section 4).
be found in Hoeken (2022).
Our analysis focuses on the connotational as- pects of a target word as perceived by different political communities. These differences can be observed with respect to distinct types of conno- tations. For example, the Dutch word socialist diverges solely in terms of evaluative connotation, with a negative evaluation among the right-wing community (FD) and a positive evaluation among the left-wing community (PS). In contrast, for the

---

Dutch targets
PS
FD1
FD2
English targets
HC
TD1
TD2
Klimaat climate
177
182
abortion
941
829
7911
vaccinatie vaccination
50
499
467
muslims
2 722
40 654
41 285
immigratie immigration
840
777
islam
910
666
404
vluchtcling refugee
251
393
382
immigrants
2 230
18 682
18 498
media media
95
250
237
citizenship
521
054
5
belasting taxes
590
607
599
guns
4 1.57
19 496
19 307
overheid government
1 355
847
850
6 6,362
18.888
17 040
income
3 986
7 523
689
links left
4 640
4 093
4 053
police
2.776
30 100
30 191
rechts right
2 382
2 968
3 019
socialist socialist
1 397
175
163
liberals
246
38 718
39 035
liberaal liberal
1 214
628
618
iran
299
7.978
692
kapitalisme capitalism
1 801
232
258
syria
515
9 9.329
9 319
Corpus size
2.3M
3.2M
3.2M
Corpus size
38.8M
133.6M
133.7M
Table 2: Target words and their subcorpus frequencies (based on lemmatized data). Middle horizontal line indicates boundary between connotation type that the target word mainly differs in, above associative and below evaluative.
4 Methods & Experimental Set-up
the model each comment in which a target word oc- curs² a community-specife subcorpus. For each context we extract the model's hidden layers, aver- age over them and select the target word's sentence position, resulting in contextualized embedding of the target word. The pairwise cosine distance computed between two sequences of contextual- ized embeddings from two different subeorpora, by calculating the pairwise cosine distances and tak- ing the average over them i.c. the Average Pairwise Distance (APD) (Giulianelli et al., 2020).* 4 Below, we provide description of all the different models we utilize in this procedure (see Appendix C for details).
In this section, we discuss the methods we employ to address the task of detecting subtle semantic shifts defined in the previous section. We evalu- ate all methods on our subtle connotative shift task. addition, we test the performance of all meth- ods on the SemEval 2020 task to examine their behavior on denotational shifts.
4.1 Models and methods
Static representations We consider two tradi- tional LSCD methods based on static word em- bedding models. First, we use Positive Point- wise Mutual Information (PPMI), transparent count-based model, where we align subcorpora via column intersection (Hamilton et al., 2016a; Du- bossarsky et al., 2017, 2019). Second, we use Skip- (SGNS) obtained
Vanilla Models First, we test the inherent ca- pability of a pretrained contextualized language model in detecting semantic shifts by using BERT models. We use the base version of BERT (De- vlin al., 2019) for English and BERTje (de Vries et al., 2019) for Dutch. We evaluate the impact of fine-tuning BERT further on the data from the same domain as the test data (D-BERT). For the Dutch use-case, we used the union of test corpora. For the English use-case, we fine-tuned on a ran- dom sample of 5.8 million Reddit comments span- ning from December 2005 to March 2023. We mainly follow Giulianelli et al. (2020) in finetuning BERT model with the Masked Language Mod- elling (MLM) objective. Our final language model XLM-R (Comneau et al., 2020), which yielded
gram with negative-sampling from Word2Vec (Mikolov et al., 2013), which per- formed best on SemEval-2020 (Schlechtweg et al., 2020). We apply the Orthogonal Procrustes tech- nique align two SGNS- SGNS-models trained on differ- ent subcorpora (Hamilton al., 2016b). Following common practices, we lemmatize all words and use Cosine Distance (CD) calculate the difference between target word representations. More details on both methods can be found in Appendix C.
Contextualized Representations We examine whether feeding contextualized model different types of information impacts its ability reflect subtle shifts when comparing internal representa- tions. For obtaining representations from contex- tualized model, either pre- pre-trained or fine-tuned on a specific task (to be elaborated hereafter), we feed
'Due the models' input limitations comments exceeding 512 tokens are excluded.
'lll models are implemented through Hugging Face's transformers lihrary and for APD calculation we use torch- metrics.

---

high performance in previous work (Rachinskiy and Arefyev, 2022).
4.2 Evaluation
Predicting real shifts We test whether the meth- ods can reflect the subtle semantic shifts using the test corpora and metries presented in Section 3. For each target word, we assess whether our methods can correctly reflect a higher difference between target word occurrences originating from different communities compared to target word occurrences originating from the same community.
Fine-tuned Models A second set of experiments explores the impact of task-specific fine-tuning. Following Rachinskiy and Arefyev (2022), we fine- tune XLMMR on word sense disambiguation (WSD XLM-R). This system is based on a Bi-Encoder Model (BEM) (Blevins and Zettlemoyer, 2020) that consists of a context encoder for generating target word representations and gloss encoder for gen- erating representations of WordNet glosses. These encoders jointly trained SemCor (Miller al., 1994), an English WSD-dataset, to classify a target word sense based on similarity of its rep- resentation with the glosses. As mentioned be- fore, the intuition behind their approach is that aspects on sense are emphasized in the represen- tations. As our task emphasizes connotation, we also investigate the impact of fine-tuning on senti- ment analysis, which should foreground evaluative connotative meaning. To this end, we utilize an XLM-R model fine-tuned on a set of unified senti- ment analysis Twitter datasets in eight languages (SENT XLM-R) (Barbieri et al., 2022).
We also test all methods on the English data for the SemEiall 2020 shared task on LSCD (Schlechtweg et al., 2020), comparing the outputs against both the gradual and binary gold values (using correlation and accuracy"). The task differs from ours, as it requires a binary classification for each word, distinguishing between those that have changed and those that have not. To determine this binary label, adopt the common practice of employing the mean distance values decision threshold. In contrast, our subtle shifts task as- sesses whether a method can distinguish whether word occurrences are taken from the same (no shift) or from different (shift) subcorpora. Thus, perfor- mance values cannot be directly compared across the two tasks.
Masked target prediction We propose an alter- native approach exploiting model behavior rather than internal representations. We leverage the masked token prediction task, which aligns with the natural pretraining process of BERT and D- BERT. We extract token substitutions in all given contexts of a target by masking it and predicting candidates. For each instance, we compile the top 10 most probable substitutions. Next, we determine the frequency of unique tokens predicted as sub- stitutes throughout contexts within a specific subcorpus. We compare substitutions for a target word in different subcorpora by analyzing the dif- ference in relative frequencies of token predictions that are part of the intersection of the two substitu- tion sets. We measure this difference by computing the Jensen-Shannon Divergence (ISD).
Predicting simulated extreme shifts Our ini- tial experiments showed that methods using con- textualized models underperform. We conduct an additional evaluation procedure 1) test if these methods could detect extreme semantic shifts, as a sanity check and 2) explore whether the under- performance of these methods could be caused by an overly strong pretraining signal overshadowing weaker signals from new contextual information.
We simulate a radical denotation shift in which words acquire entirely new senses and lose old senses (partially inspired by Schütze (1998) and Shoemark et al. (2019)). We duplicate a subcor- pus and replace all mentions of a donor word with recipient word. For English, we replace all oc- currences of guns with the recipient word taxes. The original mentions of the recipient word were replaced by [UNK] tokens. The recipient word has now 'acquired' the contexts of the donor word and 'lost' its original contexts in the duplicated subcorpus, i.e. we simulate that taxes now means guns. For Dutch, we replace all instances of the donor word klimaat (climate) by the recipient word liberaal (liberal) in the FD1 data. If the represen-
Our approach differs from the masked token pre- diction approach of Arefyev and Zhikov (2020) in two ways: 1) they do not mask the target word T, but replace T by the pattern 'T and [MASK]' and generate top k substitutions for the mask token, 2) they don't 't compare the prediction frequencies of the substitutes, but cluster the total collection of built Bag-of-substituue vectors and compare the frequencies of clusters across subcorpora.
AAccuracy is calculated using the scikit-leam library

---

tations of contextualized model are sensitive to context, the distance between the recipient word in its original context (laxes in original context)
Dutch
English
pears.
# correct
pears.r
correct
Static representations
and the recipient word in the donor context (taxes in the context of guns) should be similar to the orig- inal distance between the donor and recipient word in their original contexts. We assess whether the
PPMI
0,878
24
0,702
24
SGNS
0,726
24
0,890
24
Contextualized representations
BERT
0,091
0,012
16
D-BERT method can reflect the simulated shift.
D-BERT
0,004
0,081
22
XLM-R
0,084
0,026
22
5 Results
WSD XLM-R
0,057
16
0,006
18
SENT XLM-R
0,091
16
0,013
20
this section we first present the results on our main task of subtle semantic shift detection within our use-cases (5.1). In Section 5.2 we discuss the results on simulated extreme shifts, after which the insights generated by our mask prediction method are showcased (5.3). Next, we evaluate our meth- ods on the SemEval 2020 shared task (5.4). Finally, Section 5.5 provides an in-depth analysis of errors made by contextualized representation methods.
Masked targer prediction
BERT D-BERT
0,738
0,923
24
0,780
0,929
24
Table 3: Performance of the methods on detecting subtle semantic shifts in Dutch and English use-cases. ## correct' refers to the number of correct binary deci- sions 24 test instances.
rized in Table 4. As expected, in the original condi- tion, the pairwise distances of rexeetrepresstiations with each other are smaller than the pairwise dis- tances between taxes and guns. If model repre- sentations sensitive to context, the pairwise distances between the original and manipulated texes-representations should be comparable to the distances between the original occurrences of taxes and guns and larger than the distances between the original occurrences of taxes. However, we observe the opposite; the distance between the original and manipulated taxes- reeesseppressttaiions is comparable to the distance between the original occurrences of taxes and considerably lower than the distance between the original occurrences laxes and guns. This observation reveals the method's inability to detect extreme semantic shift, while simultaneously implying that minimal sensitivity of the model's 's representations to contextual variations could ac- count for this outcome. We observe the same pat- tern in experiments using the pretrained variant of BERT well conducted with Dutch data (see Appendix D). These results diverge from the find- ings by Ethayarajh (2019), who investigated the context specificity of word representations in vari- ous contextualized models. They suggest that, to varying degrees depending on the layer and model chosen, that on the whole contextualized models produce context-specifie representations.
5.1
Main task
The results of our methods on the detection of sub- tle semantic shifts in our Dutch and English use cases are presented in Table 3. The approaches relying on distance comparison between static rep- resentations (PPMI and SGNS) demonstrate strong performance in both Dutch and English, showing high correlations and perfect binary predictions. The contextualized representation methods display notably weak performance scores, except for a few methods that demonstrate better binary classifica- tion performance (i.e. D-BERT and XLM-R for English). These findings emphasize the efficacy of static representations in capturing subtle seman- tic shifts in both Dutch and English use cases, when
whereas contextualized representations, provided with various types of specialized fine- tuning information, fail achieve similar effective- ness.
The masked target prediction approach proved highly effective in detecting subtle semantic shifts. Specifically, it outperformed all previous methods for English, and for Dutch, it outperformed all methods except for the PPMI-method in terms of correlation performance. We dive deeper into the added transparency of this approach in Section 5.3.
5.2 Simulated extreme shifts
5.3 Deeper insights with mask prediction
We test our models' sensitivity to context by as- sessing whether they can reflect simulated extreme semantic shift (Section& 4.2). The results are summa-
We demonstrate the transparency the masked tar- get prediction method by analyzing the predicted

---

Condition
Target Rep.
APD
SemEval-2020
original (recipient donor)
taxes guns
0.715
pears.r
acc.
original (recipient- recipient)
0.242
Static representations
simulated shift
taxes taxes
0.251
(recipient recipient in donor context)
PPMI
0,137
0,568
SGNS
0,649
0.649
Table 4: APD between representations original and manipulated setting, where guns (donor) is replaced by taxes (recipient).
Contextualized representations
BERT
0,521
0,676
D-BERT
0,518
0.649
XLM-R
0,458
0,568
Target
FD
PS
WSD XLM-R
0,247
0,541
SENT XLM-R
0,413
0,541
immigratic
"veiligheid", "werkloosheid" "marktwerking". "veijheid"
"regime", "Westen" "mensen", "vrouwen"
kapitalisme
Masked target prediction
TD
HC
BERT
0,542
0,486
citizenship
"nationality". "status"
"equality", "immigratton" "social", "geader"
D-BERT
0,466
0,541
income
"moncy". "property"
Best SemEvaa (SGNS bused)
0,622
Table 5: Examples of predicted target word substitutions that demonstrate high disparity in relative frequency between two opposing communities
Table 6: Results on SemEval data
performance this task cannot be compared di- rectly to subtle shift task.
substitutions that demonstrated the highest dispar- ity in relative frequency between two opposing communities. Several examples are presented in Table 5, which illustrate how this analysis can pro- vide insights into the different (associative) conno- tations of a target word.
In terms correlation, our SGNS method out- performs all other methods. This is consistent with the current state of LSCD research. In terms of accuracy, however, the BERT method (pretrained Vanilla) demonstrates the best performance, which differs from the findings of the SemEval results. This discrepancy could be explained by the fact that we used raw versions of the SemEval test data for the contextualized methods, whereas only lemmatized data was available for the shared task. This discrepancy highlights the importance of word form for contextualized models, as has also been emphasized in previous work (Laicher et al., 2021). Moreover, we observe no noticeable improve- ment in performance on the SemEval test data when using the masked target prediction method, and fine-tuning XLM-R on downstream tasks did not yield improved results either. Regarding the WSD XLM-R method, which showed success in Spanish and Russian semantic shift tasks, our results im- ply that this method might not exhibit consistent performance across languages, least, across evaluation sets.
For instance, for the target word kapitalisme (capitalism) in contexts originating from the FD community, highly deviating associated words (compared to the PS community) are marktwerking (market forces) and vrijheid (freedom). Similarly, werkloosheid (unemployment) is a distinguishing association with the target word immigratie (immi- gration). In the English use-case social and gender emerge words that BERT strongly associates with the target word income in the HC community, when compared the TD community. The same goes for the association of equality with the target word citizenship. These observations align with our hypotheses and underscore the effectiveness of our approach relying on model behavior to detect connotative shifts.
5.4 Evaluation SemEval
This section presents the results of our methods on the SemEval 2020 shared task on LSCD for En- glish (Schlechtweg et al., 2020), as summarized in provide
5.5 Error analysis
To delve deeper into the failures of contextual- ized representation approaches dataset, we analyzed the specific error instances for each method (based on the binary pair classi- fication, see Section 3). The misclassified words, listed in Table Appendix E, show that high number of errors is shared by all methods. Overall,
Table 6. As basis for comparison, the score of the best-performing system of the chal- lenge (SGNS-based) in Table 6. The purpose of the SemFval evaluation is to compare the performance our methods on a widely used evaluation set for clear diachronic shifts. As explained in Section 4,
subtle shift

---

methods seem to struggle with highly subtle shifts, which manifest in different ways.
textualized representations. even when fine-tuned on specialized information such as domain or sen- timent, fail.
The majority of the errors occur for words with evaluative shifts (e.g. links (left), socialist (social- ist), kapitalisme (capitalism)), which are likely to be more subtle and thus more difficult to detect than associative shifts.
Through simulation experiments with extreme semantic shifts, we revealed that representations extracted from pretrained contextualized models lack sensitivity to context, which contributes their poor performance. We thus provide empirical evidence for speculations about the dominance of pretraining signals from previous work.
Some errors do, however, also occur for associa- tive shifts. The misclassified words overheid (gov- ernment), belasting (taxes) for Dutch and taxes for English could be argued to exhibit more nu- anced shifts with some associations being shared by both communities, while others fundamen- tally opposing. Both communities accept that the government provides support while sharing level of criticism for it. Within the FD community, over- heid is associated with a passive and limited role, whereas the PS community associates it with an active and larger role. In contrast, the associations of the concept of klimaat (climate) are much more distinct; it is associated with hysteria by the FD community and seen problem that needs to be solved by the PS community.
We explore the masked token prediction capabil- ities of contextualized models as alternative and transparent method for subtle shift detection. Our approach relies top k target word substitutions, and proved highly effective on our subtle semantic shift task. It allows for the analysis of predicted substitutions, which provided deeper insights into the different connotations attributed to the target word by different communities. Follow-up research could explore alternative values for k and use the prediction probabilities as additional signals. A threshold approach based substitute probability could yield more informative substitutes, tuned for the targeted semantic shift type, and thus better performance on the SemEval data.
We analyze the occurrences of the word islam, which was misclassified by all methods. The origi- nal expectation posited an association of Islam with religion by the HC community compared terror- ism illness in the TD community. We find that the corpus reflects more nuanced picture of the shift; members of the HC community distinguish between Islam and radical Islam, with the latter receiving heavy criticism and sharing more simi- larities with TD standpoints. Within the HC com- munity, there is extensive discussion about how the right perceives Islam, leading to similar language use as observed in the TD community. The shift can thus be characterized connotations within the HC community compared to a more consistent interpretation of islam within the TD community. This kind of distinction ap- pears challenging for contextualized representation methods to pick up.
Our evaluation on the SemEval task revealed that model performance differs substantially in re- lation to the types of shifts under investigation. Our results the SemEval task, which targets clear se- mantic changes in denotation over time, show that representations of pretrained BERT models can reflect such changes to a certain degree and under- score the importance of word form for contextual- ized models, as emphasized in prior research.
Our findings contribute to the understanding of semantic shifts language and highlight the chal- lenges and opportunities associated with the less- explored task of detecting subtle semantic shifts.
greater diversity of
Limitations
One of the prominent advantages that inherently come with BERT-like models is the ability to deal with out-of-vocabulary words using a subword to- kenization. However, how to re-build the sub- word representations that preserve the composi- tional meaning of the original word is still an open-diseussion. Representation-basee approaches, which perform poorly on this task, can use sim- plistic approaches for reconstruction of the word such as habword-pooling (as in Bommasani et al., 2020), but the effectiveness of its results compared
6 Discussion & Conclusion
We presented systematic study of different meth- ods using contextualized and static language mod- els for detecting subtle semantic shift in political discourse in Dutch and English. Based on limited but carefully selected test set, we found that meth- ods based on static representations show strong performance in capturing these shifts, while con-

---

single-subword meaning representations remains unclear. The implications of this issue is potentially more problematic for morphology-rich languages, To mitigate this, our target words are selected from the union of the vocabularies of all methods and consist of nre-subword-woordd
of Intelligent Technnil Systems" (Grant ID NW21-059A), which is funded by the program "Netzwerke 2021 of the Ministry of Culture and Science of the State of Northrhine Westphalia, Ger- many. Antske Fokkens was supported by the EU Horizon 2020 project VnTa In/Tangible Euro- pean Heritage- Visual Analysis, Curation and Com- munication (http://intavia.eu) under grant agree- ment No. 101004825.
We also acknowledge the importance of the num- ber and selection of target words in evaluating the performance of semantic shift detection methods. Our results exhibited clear distinctions among the methods, but for future research with less signifi- cant differences, a larger and more diverse set of target words is recommended.
References
Nikolay Arefyev and Vasily Zhikov. 2020. BOS at SemEval 2020 task Word sense induction via lex- ical substitution for lexical semantic change detec- tion. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 171-179, Barcelona (online). International Committee for Computational Linguisties.
Ethics Statement
An ethical concern of this type of research is that insights about perspectives arising from the corpus representative of group are applied to individuals associated with the group. Care should be taken when interpreting tendencies based on language used by a Reddit community. The fact that com- munity talks about a concept in a certain way does not mean that an individual part of or sympathetic to the group shares this view. Furthermore, given the nature of forum posted data, it is important to protect authors' identities. For this research, only the content of posts was used and author informa- tion is not needed. We provide the code used for scraping the data. If we share data directly our- selves (upon request), it will only be shared in compliance with GDPR.
Francesco Barbieri, Luis Espinosa Anke, and Jose 2022. XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 258-266, Marseille, France. European Language Re- sources Association.
Valerio Basile, Tommaso Caselli, Anna Koufakou, and Viviana Patti. 2022. Automatically computing con- notative shifts of lexical items. In Natural Language Processing and Information Systems: 27th Interna- tional Conference on Applications of Natural Lan- guage to Information Systems, NLDB 2022, Valencia, Spain, June 15-17, 2022, Proceedings, pages 425- 436. Springer.
The domain that we have chosen to investigate subtie semantic shift, namely political discourse, is also prone to have stereotypical tendencies, that might have effect our empirical results. More- over, it is also well-known to the NLP community that large pre-trained language models, which are trained on unstructured web-based text, inherently learn undesired stereotypical roles, therefore they might exhibit bias towards certain groups/ideas that can overlap with the content of our use-cases.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. Proceedings of the Interna- tional AAAI Conference on Web and Social Media. 14(1):830-
Terra Blevins and Luke Zettlemoyer. 2020. Moving down the long tail of word sense disambiguation with gloss informed bi-encoders. Proceedings of the 58th Association for Computational Linguistics.
Aeknowledgements
Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. Interpreting Pretrained Contextualized Representa- tions via Reductions to Static Embeddings. In Pro- ceedings of the 58th Anmual Meeting of the Asxo- ciation for Computational Linguistics, pages 4758- 4781, Online. Association for Computational Lin- guistics.
We like to thank Mariken van der Velden, an expert in political communication, and Alina Rullkötter, a student assistent with background in political science, for consultation the conducted man- ual analyses of the Dutch and English use cases respectively.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised
The first two authors acknowledge financial sup- port by the project "SAIL: Snste Life-cycle

---

cross- cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
on GEometrical Models of Natural Language Seman- rics, pages 67-71. Edinburgh, UK. Association for Computational Linguistics.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky, 2016a. Cultural shift linguistic drift? compar- ing two computational measures of semantic change. In Proceedings of the 2016 Conference on Empiri- cal Methods in Natural Language Processing, pages 21 16-212,, Austin, Texas, Association for Computa- tional Linguistics.
Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. Bertje: A dutch bert model.
ArXiv, abs/1912.09582.
Marco Del Tredici and Raquel Fernández. 2017. Se- mantic variation in online communities of practice. In IWCS 2017- 12th International Conference on Computational Semantics - Long papers.
William I. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016b. Diachronic word embeddings reveal statisti- cal laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1489- 1501, Berlin, Germany, Association for Com- putational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Sanne Hocken. 2022. Using language models for ana- lyzing semantic variation between dutch social com- munities. Master's thesis, Vrije Universiteit Amster- dam.
Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of lan- guage through neural language models. In Proceed- ings of the ACI. 2014 Workshop on Language Tech- nologies and Computational Social Science, pages 61 Baltimore, MD, USA. Association for Com- putational Linguistics.
Haim Dubossarsky, Simon Hengchen, Nina Tahmasebi, and Dominik Schlechtweg. 2019. Time-out: Tem- poral referencing for robust modeling of lexical se- mantic change. In Pruceedings of the 57th Annual Meeting of the Association for Computational Lin- guisties, pages 457-470 Florence, Italy, Association for Computational Linguistics.
Andrey Kutuzov and Mario Giulianelli. 2020. UiO- UvA at SemEval-2020 task 1: Contextualised em- beddings for lexical semantic change detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 126-134, Barcelona (online). Inter- national Committee for Computational Linguistics.
Haim Dubossarsky, Daphna Weinshall, and Eitan Gross- man. 2017. Outta control: Laws of semantic change and inherent biases in word representation models. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing. pages I 1136-1 145, Copenhagen, Denmark. Association for Computational Linguistics.
Andrey Kutuzov and Lidia Pivovarova. 2021. Rushifte- val: shared task semantic shift detection for russian. Computational Linguistics and Intellectual Technologies: Proceedings of the International Con- ference Dialogue 2021.
Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 55-65, Hong Kong, China. Association for Computational Linguisties.
Andrey Kutuzov, Erik Velldal, and Lilja Øvrelid. 2022. Contextualized embeddings for semantic change de- tection: Lessons learned. In Northern European Journal of Language Technology. Volume 8, Copen- hagen, Denmark. Northern European Association of Language Technology.
Severin Laicher, Sinan Kurtyigit, Dominik Schlechtweg Jonas Kuhn, and Sabine Schulte im Walde. 2021. Ex- plaining and improving BERT performance lex- ical semantic change detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 192-202, Online, Associ- ation for Computational Linguistics.
Mario Giulianelli, Marco Del Tredici, and Raquel Fer- nández. 2020. Analysing lexical semantic change with contextualised word representations. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3960- 3973, Online. Association for Computational Lin- guistics.
Kristina Gulordava and Marco Baroni. 2011. A dis- tributional similarity approach to the detection of semantic change the Google Books ngram cor- pus. In Proceedings of the GEMS 2011 Workshop
Li Lucy and David Bamman. 2021. Characterizing English Variation across Social Media Communities with BERT. Transactions of the Association for Com- putational Linguistics, 9:5385556.

---

Matej Martine, Petra Kralj Novak, and Senja Pollak. 2020. Leveraging contextual embeddings for detect- ing diachronic semantic shift. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 481 1-4819, Marseille, France. European Language Resources Association.
Nina Tahmasebi, Lars Borina, and Adam Jatowtb. 2021. Survey of computational approaches to lexical se- mantic change detection. Computational approaches to semantic change, 6:1.
Frank D. Zaimra-Reina, Felipe Bravo-Marquez, and Dominik Schlechtweg. 2022. LSCDiscovery: A shared task on semantic change discovery and de- tection in Spanish. In Proceedings of the 3rd Work- shop on Computational Approaches to Historical Language Change, pages 149-164, Dublin, Ireland. Association for Computational Linguisties.
Tomas Mikolov, llya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. In Advances in Neural Information Processing Sys- tems, volume 26.
George A. Miller, Martin Chodorow, Shari Landes, Claudia Leacock, and Robert G. Thomas. 1994. Us- ing semantie concordance for sense identification. Proceedings of the Workshop on Human Language Technology. HLT *94, page 240-243 USA. Associa- tion for Computational Linguistics.
Stefano Montanelli and Francesco Periti. 2023. A sur- vey contextualised semantic shift detection. arXiv preprint arXiv:2304.0166
Syrielle Montariol, Matej Martine, and Lidia Pivovarova. 2021. Scalable and interpretable semantic change detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4642-4652, Online. Association for Computational Linguistics.
John Newman. 2015. Semantic shift. In The Routledge handbook of semantics, pages 282-296. Routledge.
Maxim Rachinskiy and Nikolay Arefyev. 2022. Gloss- Reader at LSCDiscovery: Train to select proper gloss in English discover lexical semantic change in Spanish. Proceedings of the 3rd Workshop on Computational Approuches to Historical Language Change, pages 198-203, Dublin, Ireland. Association for Computational Linguistics.
Dominik Schlechtweg. Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. SemEval-2020 task 1: Unsupervised lexical semantic change detection. In Proceedings the Fourteenth Workshop on Semantic Evaluation, pages 1-23, Barcelona (online). International Committee for Computational Linguistics.
Hinrich Schütze. 1998. Automatic word sense discrimi- nation. Computational Linguistics, 24(1):97-123.
Philippa Shoemark, Farhana Ferdousi Liza, Dong Nguyen, Scott Hale, and Barbara McGillivray. 2019. Room to Glo: A systematic comparison of semantic change detection approaches with word embeddings. In Proceedings the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing Hong Kong, China. Association for Computational Linguistics.
pages 66-76,

---

A Data Collection
Targets
The Donald (TD)
hillaryclinton (HIC)
abortion muslims
women's right
murder
All subreddit comments posted within the specified time periods were collected using the Pushshift API, excluding: posts whose author is "AutoMod- erator" and post texts without any alphabetic char- acter or deleted/removed posts (which only contain the text '[deleted]' "[removed]'). For methods using contextualized models, we remove URLs and we replace some Reddit API inherent response body encodings with their characters (&, <, >). We lowercase text only if the pre-trained model used is trained on lowercased data (i.e. not BERTje). For methods using static models, we apply above preprocessing steps, lowercasing the text for all static models used, and additionally remove all non-alphanumeric characters and lemmatize the data using SpaCy.
terrorists
victims
islam
terrorism, illness criminals
religion
immigrants citizenship
pathway citizenship
privilege
right
guns
right to bear robbery
problem
laxes
to raise the wealthy
income
support current system
cause inequalities misconduct
police
victim violence negative
liberals
positive
iran
negative (policy)
positive (policy) positive (policy)
syria
negutive (policy)
Table English target words with hypothesized un-
shared connotative aspects
Method
Details
PPMI SGNS
window size 10, alpha 0.75, k
window size 5, dimensions 300, negative
5, minimum frequency
Hypotheses
Table 9: Static representation methods hyperparame- ters
Tables 7 and 8 summarize our hypotheses regarding the target words, which are derived from manual analysis of our datasets and expertise in political communication. A brief description is provided of the connotative aspect(s), either associative or evaluative (positive/nceattive, that the members the community attribute to a target word, that differs from the opposing community.
C Method details
Details of the static representation methods and contextualized representation methods given in Table and Table 10, respectively.
Method
Details
Targets
Forum Democratie (FD)
Puldersoclalisme (PS)
Alimaat climate
hysteria
problem
BERT (en)
'bert-buse-uncasedd
voccinatic voccination
issue freedom massive, problem problem, criminal misinformation
protecting the vulnerable
BERT (nl)
'GroNLP/bert-bass
immigratie monigration
responsibility. diversity
viuchteling refagee
viztim
D-BERT (all)
epochs = 3, batch size
media modia
information cource
XLM-R
'xlm-roberta--uss
belasting
theft pussivellmiied nole negative
capital active/bigger role positive
WSD XLM-R
epochs = 7, batch size =
overheid government
links inft
SENT XLM-R
'candiffulp/alm--ooetttttb
rechts right
selective pasitive negative
negative
ssentimentmuuliilingaae
socialler
positive
libersal liberaí
selective positive
negative
kapitalisme capitalism
positive
negative exploitation
Table 10: Contextualized models: Hugging Face's mod-
els used and hyperparameters for fine-tuning. All other hyperparameters were set to the default values
Table 7: Dutch target words with hypothesized unshared connotative aspects

---

D Manipulation Experiments
E Error Analysis
English Use cases. The following list displays APD values between D-BERT target word repre- sentations in original and manipulated version of TheDonald subset 1.
Table lists all error instances for each method based on contextualized representations.
Dutch
English
immigratie, links, rechts, socialist
taxes, muslims, islam, syria
gultSiiiinal
gunteciginal 0.214
BERT
taxeSerigiul
taXCSeriginal 0.242
links, rechts, socialist, liberaal, kapitalisme
taxeSorignal
gUlSSriginal 0.715
D-BERT
islam
gunSriginal
IaXeSmunipulane 0.692
taxeSeiginal
taxCSmaripulaee 0.251
belasting, overheid, links, rechts, socialist, kapitalisme
XLM-R
islam
taxeSumiipiied
taXCSmanipltued 0.228
The following list displays APD values between pretrained BERT target word representations in original and manipulated version of TheDonald subset
overheid, links, rechts, kapitalisme
WSD XLM-R
islam, iran, syria
helasting, rechts, socialist, kapitalisme
SENT XLM-R
islam, syria
gunSedigiall gunSeiiinal 0.203
Table 11: Error instances
taxeS.rigial taxeStriginal 0.224
taxeSerignal gunSSriginal 0.718
gunSrigital
taxCSmunipulattl 0.689
taxeS.iiguul taxeSmuriqutecd 0.221
taXCSquiipitee
taXCSmmnipulueed 0.188
Dutch Use cases. The following list displays APD values between pretrained BERT target word representations in original and manipulated version of Forum Democratie subset 1. Manipulation: lib- eraal in original contexts of klimaat.
klimaat,rignat rig 0.213
liberaslaigual liberaalaquai 0.222
liberaaleigioo klima 0.660
klimaatwiginal liberaalmanipuaee 0.631
liberaaluiginol liberaallaripuhuuc 0.254
liberaalmuriguunni liberaalmumiuuuni! 0.237
The following list shows APD values between D- BERT target word representations in original and manipulated version of Forum Democcatie subset 1.
klimaatarigial iig 0.175
liberaaleiginal liberaalxigua 0.175
liberaaluigioa klimant.rigiul 0.644
Alimaateigioa liberaal.uniiutues 0.643
liberaalaiginal liberaal manipised 0.213
liberaalmauiguuus liberaalmmuigauun 0.203


Reviews:
[{"review_id": "6zTwzutkPu", "reviewer": "Reviewer_ykvj", "summary": "", "questions": "A. Taking into account comparisons between different pairs of corpora (same vs. different community), could you provide more details on (i) the precise experimental procedure; (ii) the way in which evaluation scores were computed?\n\nB. Can you provide further details and examples regarding the selection of target items, especially when it comes to ensuring that the contexts in which they appear (i) are distributed as expected across the community-specific subreddits; (ii) are distinctive enough for the model to be reasonably expected to capture them?\n\nC. Regarding the extreme semantic shift simulation, have you considered replacing target words with [MASK] token and taking its embeddings in different contexts, rather than replacing one target word with another? This would presumably avoid the contribution specific to the target word and more clearly reflect the effect of context.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. To summarize our rebuttal: we agree with the issues raised and think that we can fix most issues for camera ready (in case of acceptance). We plan to use the extra page for the most vital information on data and experimental setup, provide full details in appendices and adjust the way we present results on the use case data. For readability, we provide short answers to the questions and reactions to critical points first and more details further below. \n\n### Answers to questions\n\n**Question A (Details of the experimental setup):** \n\nThe details can be found at the bottom of the rebuttal. We agree that they should be in the paper and will add main information in the running text and details in appendices. We will also ensure that our published code and documentation are in line with the information provided in the paper. \n\n**Question B (Dataset selection):**\n\nYes, we can provide more information and we checked the distribution of the examples over the community corpora (we provide details below). \n\n\n**Question C (semantic shift with [mask] token):**\n\nThe suggested experiment could serve as an extra validation of our observation; based on the outcome of our synthetic shift experiments, we would expect the mask token approach to fail terribly. We conducted this experiment on the Dutch use-case with the pre-trained Dutch BERT model. The results confirm the above. \n\n### Reactions to criticism\n\n**For point 1.** See our answers to question A and B and details below. \n\n**Point 2.** \n\nThe use case specific evaluation sets are indeed on the small side and we agree that an overall quantitative assessment risks being misinterpreted. The purpose of the 24 terms (12 per language) is to explore the validity of results given a realistic use-case scenario (as opposed to a large-scale evaluation dataset). As such, the  terms represent core concepts of a specific political discourse and their selection is informed by political expertise. We observe that standard methods using contextualized models cannot represent fundamental differences in perspectives for any of the 12 targeted concepts (visible in the extremely low Pearson correlations) neither in Dutch nor English. We realize that the way we represent the results is not the best way of showing a clear outcome based on a small dataset. We plan to rectify this by showing results broken down to the individual terms. \nWe provide more details on the process of finding the target terms below.\n\n**Point 3.** \n\nWe agree that showing the relation between Ethayarajh (2019) and our work would strengthen the paper. As far as we understand Ethayarajh’s paper, it shows that word representations extracted from contextualized models do carry context-specific information (in particular the experiment presented in Section 4.3 - Static v.s. contextualized).  We would greatly appreciate further clarification if we are misunderstanding Ethayarajh (2019). \n\nOur motivation for giving prominence to the outcome of the extreme semantic shift simulation is to foster awareness in the semantic change community. We would therefore like to keep it prominent, but will of course make sure that it is clear that this is in line with earlier findings, if the same tendency has indeed been found in earlier work. \n\nThank you for pointers to additional references and to p. 70 in Gulordava and Baroni (2011). They will improve the paper.\n\n### Details\n\n**Question A (details of the experimental setup and evaluation metrics):**\n\n**Embedding extraction**\n\nStatic embeddings: We create PPMI and SGNS embeddings using the hyperparameters stated in the appendix for each of the corpora. \n\nFor the extraction of contextualized embeddings and distance measurements we mainly follow the procedure outlined in Kutuzov & Giulianelli (2020). \n\nFor each target word, we feed the model, either the pre-trained model (BERT, BERTje or XLM-R) or the model fine-tuned on a specific task (to be elaborated hereafter), each comment in which the target word occurs (due to the models’ input limitations comments exceeding 512 tokens are excluded) in a community-specific subcorpus. For each context we extract the model’s hidden layers, average over them and select the target word’s sentence position, resulting in a contextualized embedding of the target word. \n\n**Distance measures**\n\nThe pairwise cosine distance is computed between two sequences of contextualized embeddings from two different subcorpora, by calculating the pairwise cosine distances and taking the average over them i.e. the average pairwise distance (APD). All models are implemented through Hugging Face’s transformers library and for APD calculation we use torchmetrics.\n\n**Evaluation scores** \n\nWe conduct experiments on our use-cases by testing each method on a set of 12 target words for each language. For each target word, two test scenarios are provided: a) two subcorpora originating from different communities b) two subcorpora originating from the same community (control). This results in a total of 24 instances, which we provide with a label being either \"same-community'' or \"different-community''. For each of these instances a method outputs a distance value, the graded outcomes. We convert the graded outcomes to binary values by comparing the distance values of the two test scenarios for a target word with each other, and assign “different community” to the largest value, and “same community” to the smallest. So, the outcomes for e.g. the PPMI-method on the English use-case look like: \n\n|    | target      | gold label          | method distance | method binary prediction |\n|----|-------------|---------------------|-----------------|--------------------------|\n| 1  | abortion    | different-community | 0,838           | different-community      |\n| 2  | abortion    | same-community      | 0,702           | same-community           |\n| 3  | citizenship | different-community | 0,816           | different-community      |\n| 4  | citizenship | same-community      | 0,725           | same-community           |\n| …  | …           | …                   | …               | …                        |\n| 23 | taxes       | different-community | 0,853           | different-community      |\n| 24 | taxes       | same-community      | 0,761           | same-community           |\n \n \nSubsequently, the correlation between the methods’ (graded) distance values and the gold labels (converted to 1 and 0) are computed by calculating the Pearson Correlation Coefficient using the SciPy library. For the classification performance, we calculate the accuracy score comparing the gold labels with the method’s binary predictions, using the scikit-learn library.\nAs mentioned above, we plan to adjust the presentation of the results to address your comment.\n\n**Fine-tuning the models:**\n\nD-BERT: we again mainly follow Kutuzov & Giulianelli (2020) in finetuning a BERT model with the Masked Language Modelling (MLM) objective on data from the same domain as the test data. For the Dutch use-case, we used the union of test corpora. For the English use-case, we fine-tuned on a random sample of 5.8 million Reddit comments spanning from December 2005 to March 2023. During fine-tuning, input instances were padded, i.e. supplemented with [PAD] tokens, to ensure identical instance lengths of 512 tokens. A model was fine-tuned for 3 epochs with a batch size of 8 instances. All other training hyperparameters were set to the default values, as also implemented by Kutuzov & Giulianelli (2020).\n\n\nWSD-XLMR: We exactly follow the methodology presented by Rachinskiy and Arefyev (2022). This system is based on a Bi-Encoder Model (BEM) (Blevins & Zettlemoyer 2020} that consists of a context encoder for generating target word representations and a gloss encoder for generating representations of WordNet glosses. These encoders are jointly trained on SemCor (Miller et al. 1994), an English WSD-dataset, to classify a target word sense based on similarity of its representation with the glosses. We finetune for 7 epochs with a batch size of 8. All other training hyperparameters were set to the values as also implemented by Blevins & Zettlemoyer (2020).\n\n**Question B (data selection details):**\n\n**Procedure of target word selection:**\n\nAround a hundred random examples from each subreddit were taken to identify main topics that users discussed, and related keywords that go along with it.\nComments mentioning these keywords were extracted (rule-based script) and about 50 comments (randomly sampled) per keyword were further analyzed.  \nThe aim of the analysis was to grasp how the members of the opposing communities conceptualized the keywords. We can for one or more keywords add several comments from each subreddit to illustrate their contrastive conceptualization.\nOther relevant keywords that popped up frequently in the analysis were added to the list of keywords and the same approach was taken; words and topics that did not show contrastive conceptualizations after analysis were omitted. \n(Only for the Dutch use-case) To validate the findings and gain additional insights, consultation was sought from an expert in political communication.\n\nTo answer the specific questions: \n\n(i) by taking a consistent approach that includes random sampling, we aim to ensure to capture a distribution of contexts that is representative for the subreddit, although we acknowledge that the limited sample sizes might not fully ensure this.\t\n\n(ii) Distinctiveness of contexts: It is difficult to measure distinctiveness in absolute terms. Beyond qualitative checks, we can see that the PPMI models can reflect the difference between the two corpora. As PPMI directly reflects contexts, a clear distinction by PPMI indicates that the contexts are distinctive. Full control over distribution and distinctiveness of contexts in the two corpora would require artificial data selection, creation or manipulation. Our substitution experiments are an example of this.\n\nWe provide more details on data statistics in our answer to reviewer j7jk (posted as the 2nd review).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper provides a novel point of view on LSCD, focusing on a fine-grained and theoretically motivated type of semantic shift, much closer in nature to issues targeted by descriptive linguistic work. This focus brings much needed depth of analysis to the field, including through a detailed qualitative and error analysis.\n* The proposed masking-based method is intuitive and performs well on the main dataset used in the study, warranting further experiments to optimize it on other datasets.\n* The experimental setup is overall well thought through, with adequate comparisons across models and datasets.\n* The analysis is conducted on two languages and hence not restricted to English.\n* The paper is overall well written and easy to follow.", "reasons_to_reject": "* The paper lacks important explanations on the precise experimental setup (extraction of embeddings and distance measurements across pairs of corpora) and especially the computation of evaluation metrics. This makes it difficult to reliably assess the reported results and limits reproducibility.\n* The size of the evaluation dataset (12 target words per language) is on the extreme low end of LSCD datasets. While this is adequate for a much appreciated qualitative analysis, the authors use it for a quantitative assessment leading to broad performance claims. However, the size of the dataset calls into question the generalizability of the results, which is also suggested by comparisons with SemEval data. \n* The extreme semantic shift simulation concludes that the contextualized embeddings of a target word in very different contexts remain more similar to one another than to those of a different word used in the same context. But this is not surprising, e.g. Ethayarajh (2019) reported the same general trend for BERT representations. Given established knowledge in the literature, I believe that these manipulations amount to a sanity check (as the authors also suggest) and should be presented less prominently.\n\n**Update after the rebuttal:** Thank you for your very thorough response, which has fully addressed my concerns. Regarding the work by Ethayarajh (2019), I was referencing findings from Section 4.1 (paragraph on BERT, p. 60). Having re-read the paper, I may have overstated their relevance, but I still think your work would benefit from a comparison with those findings and a clearer framing of the semantic shift simulation experiment. I am happy to increase the soundness score.", "questions_to_authors": "A. Taking into account comparisons between different pairs of corpora (same vs. different community), could you provide more details on (i) the precise experimental procedure; (ii) the way in which evaluation scores were computed?\n\nB. Can you provide further details and examples regarding the selection of target items, especially when it comes to ensuring that the contexts in which they appear (i) are distributed as expected across the community-specific subreddits; (ii) are distinctive enough for the model to be reasonably expected to capture them?\n\nC. Regarding the extreme semantic shift simulation, have you considered replacing target words with [MASK] token and taking its embeddings in different contexts, rather than replacing one target word with another? This would presumably avoid the contribution specific to the target word and more clearly reflect the effect of context.", "paper_topic_and_main_contributions": "The paper evaluates static and contextualized word embedding methods on  lexical semantic change detection (LSCD), distinctly focusing on differences in connotative meaning across speaker communities as a subtle type of semantic shift. The authors use English and Dutch subreddits of opposing political views, and evaluate their methods on a manually curated list of 12 target words per language, containing items for which they expect connotative differences. They furthermore evaluate on diachronic English data from SemEval 2020, and as a general control simulate an extreme change in denotative meaning for one target item per language. Static word embeddings perform strongest overall; out of the contextualized approaches, a masking-based method introduced here is the strongest, but does not generalize well to SemEval data.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "In addition to the reference cited above, you may be interested in the following:\n* Cassotti et al. (2023) for another recent BERT-based method which obtains good results\n* Miletic et al. (2021) as a further example of modeling subtle semantic shifts\n* Gulordava and Baroni (2011), for a discussion of the fact that differences in context do not necessarily equate to differences in meaning, which may be relevant to your view of subtle changes in meaning (see p. 70 in their paper; you already cite it but with a broad focus)\n\n\nCassotti, P., Siciliani, L., DeGemmis, M., Semeraro, G., & Basile, P. (2023). XL-LEXEME: WiC pretrained model for cross-lingual LEXical sEMantic changE. Proceedings of ACL. https://aclanthology.org/2023.acl-short.135\n\nEthayarajh, K. (2019). How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. Proceedings of the EMNLP-IJCNLP. https://aclanthology.org/D19-1006/ \n\nGulordava, K., & Baroni, M. (2011). A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. Proceedings of GEMS. https://www.aclweb.org/anthology/W11-2508\n\nMiletic, F., Przewozny-Desriaux, A., & Tanguy, L. (2021). Detecting contact-induced semantic shifts: What can embedding-based methods do in practice? Proceedings of EMNLP. https://aclanthology.org/2021.emnlp-main.847", "Typos_Grammar_Style_and_Presentation_Improvements": "Presentation suggestions:\n* It would be helpful to provide examples and definitions of connotative change from the very beginning of the paper (top of the introduction).\n* As indicated by the questions, I would suggest that you include further details on the experimental setup and recalibrate the importance of the section on semantic shift simulation.\n* You may wish to acknowledge the size of your dataset more directly (e.g. l. 327).\n* Table 1: consider normalizing the frequencies so they are comparable across the corpora.\n\nTypos etc.:\n* l. 11: that => which (or omit the comma)\n* l. 30: social different => socially different\n* fn 2, p. 2: Tahmasebia => Tahmasebi; also applies to the reference itself\n* l. 385: approachis => approach is\n* l. 446: extra \"values\" \n* l. 505: full stop missing"}}, {"review_id": "OE0Zjq9lU6", "reviewer": "Reviewer_j7jk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We agree with the criticism, which we think can easily be fixed with the extra page and time for camera ready, in case of acceptance. We provide the details of the dataset statistics below. The extra space should leave room to add a table on the statistics as well as room to move at least one of the tables with target terms and predictions from the appendix to the main text, which should make the story less abstract. Other details will be provided in the appendix. \n\n### Details:\n\n**General information on subreddits:**\n\n* Poldersocialisme: members supporting the Dutch left political movement that includes progressive parties such as the SP (Socialist Party) and GroenLinks (`GreenLeft’’).\n* Forum_Democratie: members supporting the Dutch right party Forum voor Democratie (Forum for Democracy).\nhillaryclinton: members supporting Hillary Clinton (Republicans).\n* The_Donald: members supporting Donald Trump (Democrats), banned since June 2020.\n\n**Dataset statistics** \n\n| Subreddit                   | Poldersocialisme            | Forum_Democratie            | hillaryclinton              | The_Donald                  |\n|-----------------------------|-----------------------------|-----------------------------|-----------------------------|-----------------------------|\n| Period                      | 30 May 2018 to 3 April 2022 | 16 Dec. 2017 to3 April 2022 | 1 Jan. 2016 to 31 Dec. 2016 | 1 Jan. 2016 to 31 Dec. 2016 |\n| Comments                    | 58 537                      | 149 674                     | 1 137 508                   | 10 452 040                  |\n| Tokens*                     | 3 077 786                   | 8 526 494                   | 48 902 320                  | 341 901 610                 |\n| Average length(in subwords) | 53                          | 57                          | 43                          | 33                          |\n\n\n*Tokens are based on BERT tokenization\n\n**Data scraping**\n\n* All subreddit comments posted within the specified time period were collected using the Pushshift API, excluding:\nposts whose author is “AutoModerator”\n* post texts without any alphabetic character or deleted/removed posts (which only contain the text ‘[deleted]’ or ‘[removed]’)\n\n**Data preprocessing**\n\n* For methods using contextualized models, we remove URLs and we replace some Reddit API inherent response body encodings (‘&amp;’, ‘&lt;’, ‘&gt;’) with their characters (&, <, >). We lowercase text only if the pre-trained model used is trained on lowercased data (i.e. not BERTje).\n* For methods using static models, we apply above preprocessing steps, lowercasing the text for all static models used, and additionally remove all non-alphanumeric characters and lemmatize the data using SpaCy.\n\nFor more information about the data selection process, see our reply to ykvj (posted as 1st review).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Very thorough evaluation comparing various models in a controlled fashion. What was refreshing is that the paper systematically studies both static as well as contextualized embeddings, motivated by prior observations that static embeddings are still more accurate on this task. The analysis on the synthetic task to highlight the failures of the contextualized models was also illuminating (if a little surprising) and hence is a major contribution of this work. \n* Overall, clearly written and well supported with meaningful analyses experiments.", "reasons_to_reject": "* There is a distinct lack of information about the data used and collected. What is the nature of the data? What are some key statistics (size, number of documents, distribution)? Without some of this information, the dataset is a bit abstract and hence I am unable to draw conclusions about the results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The goal of this paper is to study the differences between static and contextualized word representations in a controlled fashion in their ability to detect subtle connotation shifts in language. This paper studies the problem by identifying two communities and a set of words that have different connotations in the two communities. It then uses word representations from various models to classify whether a given document containing one of these words belongs to one community or the other. The data itself is collected from subreddits belonging to two ends of the political spectrum, while models used range from static (skip-gram/Word2Vec, PPMI), contextualized (BERT, XLM-R, etc), as well as contextualized representations fine-tuned on a related tasks (word sense disambiguation and sentiment analysis). A separate approach leverages predicting a masked word (rather than classifying the document) using masked language models. \n\nExperiments reveal that static representations are more accurate at detecting subtle semantic shifts and that contextualized models are unable to override the strong pre-training signals (also verified via a set of synthetic experiments). However, the masked target prediction approach is largely the most accurate one across the board. It is also observed that the fine-tuning on related tasks is not consistently helpful.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yImIzAMfNu", "reviewer": "Reviewer_af38", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We agree that the points raised need to be addressed and we are confident we can do so in a final version of the paper. \n\n**Point 1 (data)**\n\nWe’ll add the most prominent information in the main text and details in the appendix. See rebuttal to reviewer j7jk, 2nd review, (dataset statistics) and reviewer ykvj , posted as review 1, (data selection process) for details on what will be added.\n\n**Point 2**\n\nAgreed, we should have been explicit about this. We will include it in the ethics section of the paper. Specifically, we plan to include the following text in the ethics section: \n\n*An ethical concern of this type of research is that insights about perspectives arising from the corpus representative of a group are applied to individuals associated with the group. Care should be taken when interpreting tendencies based on language used by a reddit community. The fact that a community talks about a concept in a certain way does not mean that an individual part of or sympathetic to the group shares this view. \nFurthermore, given the nature of forum posted data, it is important to protect authors’ identities. For this research, only the content of posts was used and author information is not needed. We provide the code used for scraping the data. If we share data directly ourselves (upon request), it will only be shared in compliance with GDPR.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Subtle semantic change detection is a difficult task, and I enjoyed reading the various approaches tested by the authors. \n- Good analysis and discussion of the results. \n- Well written and clear paper.", "reasons_to_reject": "- Too little information about the dataset used.\n- No discussion about the ethical concerns related to gathering the data directly from Reddit (I'm thinking of GDPR and personal data).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores how to automatically detect subtle semantic shifts between social communities. The focus is on political discourse from Reddit in English and Dutch. The authors perform an analysis of pre-existing methods that uses static and contextualized embeddings. In addition, the authors propose an approach using masked token prediction. They also do experiments with fine-tuning models on domain detection and sentiment analysis.\n\nThe authors show that methods using contextualized representation are not able to detect subtle semantic shifts. They have also tested the best performing systems in a Russian and Spanish shared task, and showed that this model did not perform well for detecting subtle shifts. Finally, they show that the masked token prediction approach is able to detect subtle semantic shifts. \n\nThe paper is clear and well written. The experiments are clearly discussed and described. The results are also well discussed. However, I am not an expert in the semantic shift field. The main weakness of the paper is the lack of discussion of the used dataset. There are very little details about the Reddit data, how it was created, processed, and what kind of ethical issues might rise from using and collecting this dataset. \n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 2: Performance of the methods on the detecting subtle semantic shifts in Dutch and English use-cases --> ... on detecting subtle ... "}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree the paper provides a thorough evaluation of static and contextualized embeddings in lexical semantic change detection.
They greatly approve the depth of analysis and the experimental setup, especially on the synthetic task.

For the camera-ready version, the authors should:
1) Include detailed information about their datasets
2) Add discussion about ethical concerns of using Reddit data
3) Include explanation of the relation to experiments from Ethayarajh (2019).
Decision: Accept-Findings