Understanding Computational Models of Semantic Change: New Insights from the Speech Community
Authors: Filip Miletic
Anne PRZEWOZNY
Ludovic Tanguy
Venue: EMNLP 2023 Main (2023)
Keywords: semantic change detection
semantic shifts
word embeddings
BERT
language contact

Abstract: We investigate the descriptive relevance of widely used semantic change models in linguistic descriptions of present-day speech communities. We focus on the sociolinguistic issue of contact-induced semantic shifts in Quebec English, and analyze 40 target words using type-level and token-level word embeddings, empirical linguistic properties, and – crucially – acceptability ratings and qualitative remarks by 15 speakers from Montreal. Our results confirm the overall relevance of the computational approaches, but also highlight practical issues and the complementary nature of different semantic change estimates. To our knowledge, this is the first study to substantively engage with the speech community being described using semantic change models.

Understanding Computational Models of Semantic Change:
New Insights from the Speech Community
Filip Miletic Anne Przewozny-Desriaxx Ludovic Tanguy²
Institute for Natural Language Processing, University of Suutgart CLLE, CNRS & University of Toulouse felip.miletiien
uni uni-smssstttttttttttt.m
{anne przewozzy,,ozny, ludovic. Jenguyy@emivvttsse fr
Abstract
as well as range of empirical linguistic informa- tion. We use face-to-face sociolinguistic interviews with 15 speakers from Montreal to clicit accept- ability ratings and qualitative remarks for the target semantic shifts attested in tweets from our corpus.
We investigate the descriptive relevance of widely used semantic change models in lin- guistic descriptions of present-day speech com- munities. We focus the sociolinguistic is- sue of contact-induced semantic shifts in Que- hec English, and analyze 40 larget words using type-level and token-level word embeddings, empirical linguistic properties, and crucially acceptability ratings and qualitative remarks by 15 speakers from Montreal, Our results confirm the overall relevance of the computational ap- proaches, but also highlight practical issues and the complementary nature of different seman- tic change estimates. To our knowledge, this is the first study to substantively engage with the speech community being described using semantic change models.
We provide the following contributions. (1) A quantitative comparison of model-derived semantic change scores, human acceptability ratings, and em- pirical linguistic properties, confirming the overall relevance of standard approaches and highlighting their shortcomings. (2) A qualitative analysis of the social values linked to semantic shifts, clarifying the complementarity of semantic change estimates, (3) A set of 40 target words with model-derived semantic change scores, empirical linguistic prop- erties, as well ratings, synonyms, and qualitative remarks.' To the best of our knowledge, this is the first study to substantively engage with a speech community in the process of describing its language use with semantic change models.
opeaker-providee acceptability
1 Introduction
Research on computational analyses of semantic change has established a range of modeling ap- proaches, standard evaluation practices, and ev- idence of their practical utility as well short- comings in descriptive linguistic applications (Tah- masebi et al., 2021). However, most work date purpose diachronic corpora with little information on the underlying speakers, but the same methods are increasingly used to study finer-grained patterns in present-day speech com- munities. This raises the vital question of how the output of these models is perceived by the speakers whose linguistic behaviors they purport to describe.
Related work
Semantic change over time is mainly analyzed us- ing word embeddings (Kutuzov et al., 2018). Type- level models provide meaning representations that specific to time periods and can be compared across them (Dubossarsky al., 2019; Gulordava and Baroni, 2011: Hamilton et al., 2016; Kim al., 201 1). Token-level models provide contextualized embeddings that can be pooled into time-specific representations or used clustering (Martinc et al., 2020; Giulianelli et al., 2020). Other deep neural architectures use cross-lingual transfer (Rachinskiy and Arefyev, 2022) train on temporal data (Rosin and Radinsky, 2022). Similar methods used compare word meanings across online communi- ties (Del Tredici and Fernández, 2017), text types
has focused on general-purpose
We address this issue focusing the sociolin- guistic context of Quebec English, which exhibits cross-linguistic semantic change due to contact with French (Miletić et al., 2021). We analyze 40 semantic shifts in a regionally stratified corpus of tweets, obtaining semantic change estimates from type-level and token-level word embedding models,
'The resource is availahle at http://redac.uni univ-tlse2. fr/misc htcc//aaeeeetteeeeeeeee html
"Work done at the University of Toulouse.

---

(Fiser and Ljubešic, 2018), dialect regions (Kulka- mi et al., 2016), and languages (Uban al., 2019). The prevalent evaluation approach uses graded type-level semantic change scores (Schlechtweg et al., 2018) to assess models on ranking or binary classification. This has enabled improvements in model design, shown by shared tasks range of European languages (Basile et al., 2020; Pivo- varova and Kutuzov, 2021; Schlechtweg et al., 2020; Zamora-Reina et al., 2022). The models have been used in promising linguistic analyses (De Pascale, 2019; Rodda et al., 2019; Xu and Kemp. 2015), including of Quebee English (Miletić et al., 2021). but their descriptive value is yet to be fully determined (Boleda, 2020). Further valida- tion be provided by the targeted speakers, sim- ilarly collecting semantic change ratings from online communities under study (Del Tredici et al., 2019). However, we argue for much more extensive engagement based on variationist sociolinguistic practice (Labov, 1972; Tagliamonte, 2006).
posted by R 150k users, for total of 630m to- kens balanced across the regions.
3.2 Target semantic shifts
We use the only available dataset for semantic change detection Quebec English, which we previously introduced (Miletić et al., 2021); it is similar in size to those from recent shared tasks (e.g. Schlechtweg et al., 2020). We retain 40 items corresponding to semantic shifts, which were iden- tified in the sociolinguistic literature and through corpus exploration. The presence of contact mean- ings in our corpus was also manually validated. A typical semantic shift from the dataset is illustrated by the following tweets:
(1) This isn't the level of realistic exposition you'd d require in fiction.
(2) I really want to go an art museum or an art exposition.
In example (1), exposition is used with one of its conventional English meanings, 'introductory part of a narrative'. Example (2) illustrates the contact meaning 'art exhibition it is not generally used in English, but it is typical of the French homograph exposition. Its presence in Quebec English is likely explained by the locally widespread use of French. We further extend the set of semantic shifts with several empirical linguistic properties shown to af- feet computational estimates of semantic change (Del Tredici et al., 2019; Hamilton et al., 2016; Uban et al., 2019). (1) Frequency is taken from the Montreal subcorpus. (2) Polysemy is calculated as the number of synsets in WordNet (Fellbaum, 1998). (3) We compute target word's regional specificity the Montreal subcorpus compared to the whole corpus using the Sparse Additive Gener- ative model (SAGE) (Eisenstein et al., 2011). 1). It is based on the mikiiuum-likellhone criterion, with a regularization parameter ensuring that rare terms are not overemphasized. (4) We quantify a target word's 's contextual diversity by taking the mean pairwise cosine distance between the embeddings of its context words (from the type-level model described below). This is limited 1,000 (possi- bly non-unique) contexts in a symmetrical 10-word window; they are randomly subsampled if needed.
3 Data and method
We aim to gain a better understanding of previously proposed computational semantic change measures by examining how they relate to a standard soci- olinguistic estimate of the same phenomenon in this case, contact-induced semantic shifts in Que- bee English. We begin by presenting the data and methods used obtain these estimates.
3.1 Corpus of Canadian English tweets
We are unaware of any diachronic corpus of Que- bec English, and therefore rely on synchronic data to model regional differences in meaning which indirectly reflect semantic change over time. We use corpus of English-language tweets posted by users from Montreal, Toronto, and Vancouver (Miletić al., 2020). We aim to find effects of lan- guage contact by detecting word usage specific to Montreal, the only of the three cities where French is widely used. We rely on the two control regions to limit the impact of unrelated regional variation.
The original corpus is filtered for language, loca- tion, and near- -uar-dupl content. Like in previous work (Miletić et al., 2021), we only retain (i) tweets posted from 2016 onwards to limit spurious di- achronic effects; (ii) users with least 10 tweets in the corpus; (iii) maximum of 1,000 tweets per user, with a random subsample where that is ex- ceeded. The final corpus contains ss 35m tweets
3.3 Embedding- based models
We assess different types of semantic change esti- mates that can be derived across modeling strate-

---

gies. We use type-level models obtain broad estimates, and token-level models target indi- vidual contact-related occurrences. The setup is introduced below; see Appendix A for full details.
semantic shifts; each was presented in one tweet from the corpus, jointly chosen by three annotators, We asked the participants to (i) read the tweet out loud; (ii) rate the acceptability of the target word from I to 6; (iii) provide a contextual synonym for the target word; (iv) provide any other com- ments. Given our use of face-to-face interviews to assess online communication, stylistic differences between the two situations may have confounding effect. We aimed to limit it by prioritizing tweets which are not stylistically marked; asking the par- ticipants to rate the target word itself rather than properties of its context; and conducting the rating task at the end of the interview, when participants generally communicate more freely.
We learn type-level word embeddings using the best configuration from previous evaluation on this dataset (Miletić et al., 2021) word2vec models (Mikolov et al., 2013) trained separately for each subcorpus and aligned using Orthogonal Procrustes. For each word, we take the pairwise cosine distances for its vectors from models trained on the regional subcorpora, and quantify semantic change by averaging over the Montreal-Toronto and Connteal-Vancouvee distances (cosAvG). We expect this score to capture the amplitude of re- gional semantic differences on the type level, and to be higher for clear-cut than for fine-grained se- mantic distinctions.
Acceptability ratings widely used to quan- tify the perception of linguistic features in speech community (Dollinger, 2015). Here, they reflect the local diffusion contact-related uses: in direct terms, among our participants; indirectly, in the broader speech community, since more widespread general use of an item likely contributes to higher acceptability ratings. We compute mean acceptabil- ity ratings (humanAvG) by averaging over all par- ticipants who interpreted the tweet with the posited contact-related meaning. We include all words for which more than half of the participant ratings are retained (37 of 40); for these, 14.1 1 ratings are retained on average.
For each target word, we obtain token-level word embeddings of its occurrences using BERT-base-uncased (Devlin et al., 2019) and cluster them using affinity propagation, following Miletić et al. (2021). Up to 10 clusters with the highest proportion of tweets from Montreal are an- notated by domain experts. They manually tag each cluster with a binary label indicating if the preva- lent meaning in it is related to contact influence. For each target word, we calculate the proportion of tweets tagged as contact-related out of all an- notated tweets (clustcoNr); and the proportion of tweets from Montreal, out of all tweets tagged as contact-related (ClustREG). We expect clustCONT to capture the diffusion of contact-related uses, as implies a higher number of different usage contexts and users producing them. It should correlate with estimates of diffusion in face-to-fie sore-to-fcce communica- tion. By contrast, cluStREG measures the regional specificity of contact-related uses, which might vary for different reasons. For instance, an emer- gent semantic shift may be specific to Montreal, but an established one may spread elsewhere.
We transcribe the audio recordings of the seman- tic perception test so as to (i) extract the synonyms for the semantic shifts; (ii) annotate the associated social values: local specificity, for items described as typical of Quebec; French influence, for those de- scribed as typical of French speakers. At least one speaker expressed a social value for 26 semantic shifts. For the full interview protocol and speaker profiles, see Appendix B.
Results and discussion
4.1 Model-derived scores vs. speaker ratings
3.4 Speaker ratings and remarks
Speaker acceptability ratings show a weak nega- tive correlation with the type-level score coS (p - - -0.370, - 0.024) and a trend towards positive correlation with the token-level score clustCONT (p 0.238, 0.155). They are un- correlated with clUsTREG (p = 0.091, p 0.592). While the two stronger relationships obtain inverse
We conducted interviews with 15 Montrealers in early 2022.2 We used the PAC-LVTI sociolin- guistic interview protocol (Przewozny et al., 2020) to elicit detailed linguistic and sociodemographic background. This was followed by a semantic per- ception test in which they were shown our 40 target
'For items where all provided contextual synonyms cor- respond to the target interpretation, we also retain the ratings where no explicit synonym is given.
Futuree participants from Toronto and Vancouver may pro- vide further relevant information, over and above this analysis.

---

population
terrace
population
terrace
posss
aVaijability
boutique
hesitate
boutique
availability
Bigtant
exposition
mmiliann
expos
Pipllorratin exploration
mean acceptab lity rating
occasion
occasion
animatof deluiih
chalet
canindtiputtyy
permit
chalet
ambiiance
combiance
prudent
dirculationprudeti
man man festation
dossier formidablee
fofiiddatie
progosition intimmtioo exchange exc
merit
merit exchange
sproposition affirmation
entourage
entourage
reparation portable @ommgt00rdinatt
portable 0000 souvenir
rEpRA 10
deceives aoordinate
de coordinatee
remark
remark
re resume
definitively
resumme
definitively
grave
grave
0.2
0.4
0.0
0.2
0.4
0.6
1.0
CO5
clustooNi
Figure 1: Regression plots comparing mean human acceptability ratings with model-derived semantic change esti- mates. Left: type-level cosine distance (average of Montreal-Toromt and Montreal- Vaancuuurl.. right: proportion of tweets manually annotated as contact-related following token-level clustering.
correlations due their nature (semantic difference vs. diffusion), they confirm that model-derived esti- mates at least partly align with the target speakers' perception of semantic change. But the captured patterns are not the same, as shown in Figure 1.
ships with empirical linguistic properties. Spear- man's correlation coefficients are shown Table I;
statistically significant results are discussed below.
freq.
polys.
specif.
divers.
The left panel suggests that semantic change is more accepted with lower amplitude, as measured by type-level cosine distance. For example, pop- ulation 'general public' (cf. Fr. population) has a high acceptability rating, but this is a slight shift from 'inhabitants'. Acceptability is much lower for definitively 'for sure' (cf. QF définitivement), but this is a stark change from 'conclusively'. Other trends point to technical issues: exposition "exhibi- tion' (cf. Fr. exposition) has similarly high cosine distance but also higher acceptability. Its French homograph is sometimes used in codeswitched tweets from Montreal, inflating its cosine distance.
COSAYG
-0.764
-0.378
0.450
-0.439
clustCONT
-0,083
-0.371
0.146
-0.093
clustREG
-0.177
-0.063
0.533
-0.019
humanAvG
0.396
0.096
-0.091
0.254
Table I: Correlation (Spearman's p) of semantic change estimates and empirical properties. Shading: p< 0.05.
Frequency shows a strong negative correlation with cOSAVG. similarly to monolingual (Hamilton et al., 2016) and contrary to cross- Pross-lingual (Uban et al., 2019) diachronic models. The link between frequency and cosine may reflect model artifacts (Dubossarsky et al., 2017), but we also find a mod- erate correlation with human suggesting that frequency does play role. Polysemy has mod- erate negative correlation with both cosAvG and clustCONT- contrary to diachronic studies (Hamil- ton et al., 2016; Uban et al., 2019). This might be due to the effect of more complex semantic prop- erties, such as cultural relevance of lexical fields, on bilinguals (Zenner et al., 2012). Regional speci- ficity shows moderate positive correlation with coSAvG and clustREG- This is consistent with gional semantic changes being paralleled by higher frequency that region. Contextual diversity has moderate negative correlation with coSAvG. sug- gesting that type-level models may inadvertently
The teken-level score (right panel) measures the diffusion of the contact-related meaning in Mon- treal. parallels acceptability ratings in some cases, such as terrace 'patio' (cf. Fr. terrasse). But other highly accepted semantic shifts like trio "combo meal' (ef. QF trio) appear in few regionally- specific clusters, obfuscating their importance. By contrast, definitively 'for sure' would be described widespread in the community based on corpus information, contrary its low acceptability.
4.2 Effect empirical properties
We further explore the differences between seman- tie change estimates by looking at their relation-

---

capture topical differences. This has important implications for interpreting cosine-based change estimates, as also noted by Del Tredici et al. (2019),
5 Conclusion
We used different embedding- based models to an- alyze semantic shifts Quebec English, and as- sessed them based on acceptability ratings and qual- itative remarks by speakers from Montreal. We found that the models were overall relevant, but that (i) model-derived scores aligned differently, and never decisively, with human ratings; (ii) em- pirical linguistic properties affected all scores, but in different ways; (iii) semantic shifts with similar quantitative scores were often linked to divergent qualitative assessments. This both demonstrates the complementarity of different semantic change estimates and calls for caution in interpreting them, as they fail to capture the full range of information relevant to speakers from the target community.
4.3 Qualitative analysis
We now turn to two types of qualitative information: synonyms for the semantic shifts and spontaneous remarks on their use. As previously noted ($ 3.4), in nearly all cases large majority of participants provided synonyms in line with the posited inter- pretation semantic shifts. This is an important general validation of the pipeline used to identify target words and single contact-related examples. As for the spontaneous remarks, key trends for sam- ple items are shown in Table 2; see Appendix C for all semantic shifts and sample verbatim comments. Since the remarks were optional, their presence even in low absolute numbers indicative of salient sociolinguistic properties.
Limitations
In the computational experiments, relied monolingual models analyze cross-linguistic phenomena. Our high-level methodological aim identifying effects of language contact in monolin- gual data from demographically different speech communities - is grounded in the sociolinguistic literature (Tagliamonte, 2002), but the use of mul- tilingual models is an important alternative to be explored in future work. In the face-to-face in- terviews, we collected acceptability ratings for a single occurrence of target word, which limits direct comparability with computational type-level ratings. This was necessary to enable coverage of a relatively large number of target words, and we also took precautions to ensure that the cho- sen examples were representative (Appendix B). Moreover, due to public health restrictions at the time, we were only able recruit a participant sam- ple that is comparatively small and is not balanced across sociodemographie characteristics. However, the participants' diversity is reflective of the local sociolinguistic context, and is valuable from a qual- itative standpoint. Finally, our analysis focused on specific type of language variation in single target speech community. This is realistic use scenario for semantic change models which has highlighted important methodological issues, but we are unable to precisely estimate how much these issues would generalize to other applications.
sem. shift
humanAvg
synonym
loc.
Fr.
terrace
5.67 + 0,90
patio (7)
3
1
boutique
5.13 0.99
store (10)
3
3
entourage
3.20 +: 1.26
friends (5)
0
4
definitively
2.09 1.51
definitely (11)
0
0
Table 2: Examples with acceptability ratings (mean and standard deviation); top synonym with number of speak- ers who provided it; number of speakers describing the item locally specific (loc.) or linked to French (Fr.).
Semantic shifts with high acceptability are gen- erally described as regionally specific (e.g. terrace 'restaurant patio'), sometimes together with a per- ceived role of French (e.g. boutique 'shop'). By contrast, words with lower acceptability give rise to more varied comments. Some are more strongly related to a perceived impact of French, such as entourage 'close friends'. Most interviewees claim that they would not use it in English, but three indi- cate that they would in French. As for definitively 'for sure', most participants outright reject as unnatural without further qualification; only one suggests that they use it, and further two that they might hear it used. But both words are widely at- tested with contact-related meanings in the Twitter corpus, highlighting the need to nuance the conclu- sions drawn from this type of data. Moreover, these two words have similar type-level and token-level semantic change scores, but are perceived very dif- ferently. This highlights the need to complement quantitative scores with qualitative information.
Ethics Statement
The protocol used to collect and unalyze data was approved by the Research Ethics Boards of the Uni-

---

versité de Toulouse (approval number 2021-396) and of the Université de Sherbrooke (approval num- ber 2022-3289); the latter administered fieldwork- specific funding and facilitated on-site interviews. For further discussion of participant recruitment and informed consent, see Appendix B.
tered by the IRIT research laboratory and supported by the CNRS, the Région Occitanie, the French Government and the European Regional Develop- ment Fund (see https://osirim: irit.fr) fr).
References
Previously created datasets were used in line with their intended use and licenses. Specifically, the corpus of tweets ($ 3.1) created, and is distributed for research purposes in the form of tweet IDs,* 4 in accordance with Twitter's Developer Agreement and Policy.² The binary classification dataset from which we drew the target semantic shifts ($ 3.4) is distributed under the Creative Com- mons BY-NC-SA 4.0 license." 6 We are releasing the extended set of semantic shifts containing model-derivee semantic change scores, aggregate information from expert and interviewee annota- tions, and empirical linguistic properties - under the Creative Commons BY-NC-SA 4.0 license.
Pierpaolo Basile, Annalina Caputo, Tommaso Caselli, Pierluigi Cassotti, and Rossella Varvara. 2020. DIACR-IK
EVALITA2020: Overview of the EVALITA2020 Diachronic Lexical Semantics (DIACR-IC) Ita) Task. In Proceedings of the Seventh Evaluation Campaign of Natural Language Process- ing and Speech Tools for Italian. Final Workshop (EVALITA 2020).
Steven Bird, Edward Loper, and Ewan Klein. 2009. Natural Language Processing with Python, O' O'Reilly Media, Sebastopol, CA.
Gemma Boleda. 2020. Distributional semantics and lin- guistic theory. Annual Review of Linguistics, 6:213- 234.
The results of computational experiments pre- sented here are based on aggregate representations of word meaning; no personally identifiable or otherwise sensitive information is discussed. The individual corpus examples used in face-to-face interviews were manually checked (including for offensiveness) during protocol design, and any per- sonally identifiable information was redacted (Ap- pendix B). The interviewees' sociodemographic characteristics are anonymized and securely stored; we only report aggregate trends. As for more gen- eral risks, research on well-defined linguistic com- munities may be viewed relative to broader public debates, in particular language policy. We stress that our study is based on an empirical analysis of attested language use and not predetermined view of Quebec's linguistic communities.
K. Chambers and Troy Heisler. 1999. Dialect to- pography of Québec City English. Canadian Jour- nal of Linguistics/Revve canadienne de linguistique, 44(1):23-45, 48.
Stefano De Pascale. 2019. Models as Semantic Control in Lexical Lectometry. Ph.D. thesis, KU Leuven, Leuven.
Token-Based Vector Space
Marco Del Tredici and Raquel Fernández. 2017. Se- mantic variation in online communities of practice. In /WCS 2017 12th International Conference Computational Semantics Long Pupers.
Marco Del Tredici, Raquel Fernández, and Gemma Boleda. 2019. Short-term meaning shift: A distri- butional exploration. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2069-2075, Minncapolis, Minnesota. Association for Computational Linguisties.
Acknowledgements
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language under- standing, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- mologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
We are grateful to the anonymous reviewers for their remarkably helpful feedback. We are also indebted to our 15 volunteer participants. The re- search presented here was partly supported by the Fonds de recherche du Québec through a PBEEE short-term research scholarship awarded to the first author. The presented experiments were carried out using the OSIRIM computing platform, adminis-
Stefan Dollinger. 2015. The Written Questionnaire in Social Dialectology: History, Theory, Practice. John Benjamins, Amsterdam.
'http://redac.uuivv wtwp://ttsseee.fooooorss//eaaaa tttp:////pp//cennnnn html
httpps://deeellper.tttw// www.ter.com//e...
Haim Dubossarsky, Simon Hengchen, Nina Tahmasebi, and Dominik Schlechtweg. 2019. Time-Out: Tem- poral referencing for robust modeling of lexical se- mantic change. In Proceedings of the 57th Annual
"http://redac.un wnw.eeeddda...ui--illeee https//eeeee//iiicc/aannneeeeeee
html

---

Meeting of the Association for Computational Lin- guistics, pages 457-470, Florence, Italy. Association for Computational Linguistics.
61-65, Baltimore, MD, USA. Association for Com- putational Linguistics.
Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. 2016. Freshman or fresher? Quantifying the geo- graphic variation of language in online social media. Proceedings of the International AAAI Conference on Web and Social Media, 1066)665-6
Haim Dubossarsky, Daphna Weinshall, and Eitan Gross- man. 2017. Outta control: Laws of semantic change and inherent biases word representation models. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 1 1136-1145, Copenhagen, Denmark. Association for Computational Linguistics.
Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski. and Erik Velldal. 2018. Diachronic word embeddings and semantic shifts: A survey. In Proceedings of the 27th International Conference on Computational Lin- guistics, pages 1384-1397, Santa Fe, New Mcxico, USA. Association for Computational Linguistics.
Jacob Eisenstein, Amr Ahmed, and Tiic P. Xing. 2011. Sparse additive generative models of text. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML' pages 1041-1048, Madison, WI, USA. Omnipress.
William Labov. 1972. Sociolinguistic Patterns. Univer- sity of Pennsylvania Press, Philadelphia.
Christiane Fellbaum, editor. 1998. WordNet: An Elec- tronic Lexical Database. MIT Press, Cambridge, MA.
Matej Martine, Syrielle Montariol, Elaine Zosa, and Lidia Pivovarova. 2020. Capturing evolution in word usage: Just add more clusters? In Companion Pro- ceedings of the Web Conference 2020. WWW '20. pages 343-349, New York, NY, USA. Association for Computing Machinery.
Darja Fišer and Nikola jjjubSićć. 2018. Distributional modelling for semantic shift detection. International Journal of Lexicography, 32(2):163-183
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representa- tions vector space. In 1st International Conference on Learning Representations, ICLK 2013, Workshop Track Proceedings. Scottsdale, Arizona, USA.
Kevin Gimpel, Nathan Schneider, Brendan O'Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 201 Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 32-47, Portland, Oregon, USA. Association for Computational Linguisties.
Filip Miletić, Anne Przewozny-Desriauxx and Ludovic Tanguy. 2020. Collecting tweets to investigate re- gional variation in Canadian English. In Proceedings of the 12th Language Resources and Evaluation Con- ference, pages 6255-6264, Marseille, France. Euro- pean Language Resources Association.
Mario Giulianelli, Marco Del Tredici, and Raquel Fer- nández. 2020. Analysing lexical semantic change with contextualiedd word representations. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3960- 3973, Online. Association for Computational Lin- guistics.
Filip Miletić, Anne Przewozny-Destre
and Ludovic Tanguy. 2021. Detecting contact- contact-induced semantic shifts: What can embedding- based methods do in practice? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 10852-10865, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kristina Gulordava and Marco Baroni. 2011. A dis- tributional similarity approach to the detection of semantic change in the Google Books Ngram cor- pus. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Seman- tics, pages 67-71, Edinburgh, UK. Association for Computational Linguistics.
Olutobi Owoputi, Brendan O'Connor, Chris Dyer. Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In Proceed- ings of the 2013 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, pages 380- 390, Atlanta, Georgia. Association for Computational Linguistics.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statisti- cal laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1; Long Papers). pages 1489-1501, Berlin, Germany. Association for Com- putational Linguistics.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer. R. Weiss, V. Dubourg, J. Vanderplas, A. Passos. D. Cournapeau, M. Brucher, M. Perrot, and F. Duch- esnay. 2011. Seikit-leamn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal amalysis of lan- guage through neural language models. In Proceed- ings of the ACL 2014 Workshop on Language Tech- nologies and Computational Social Science, pages

---

Benedicte Pierrejean and Ludovic Tanguy, 2018, Pre- dicting Word Embeddings Variability. In Proceed- ings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 154-159, New Or- leans, Louisiana. Association for Computational Lin- guistics.
lexical semantic change. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 169-174, New Orleans, Louisiana. Associa- tion for Computational Linguistics.
Lidia Pivovarova and Andrey Kutuzov. 2021. RuShiftE- val: A shared task semantic shift detection for Russian. In Computational Linguistics and Intellec- tual Technologies, 20, pages 533-545.
Sali A. Tagliamonte. 2002. Comparative sociolinguis- tics. In J. K. Chambers, Peter Trudgill, and Natalie Schilling-Estee Estes, editors, The Handbook of Language Variation and Change, pages 729 229-763. Blackwell, Malden.
Anne Przewozny, Cécile Viollain, and Sylvain Navarro. 2020. The Corpus Phonology of English: Multifocal Analyses of Variation. Edinburgh University Press, Edinburgh.
Sali A. Tagliamonte. 2006. Analysing Sociolinguistic Variation. Cambridge University Press, Cambridge.
Maxim Rachinskiy and Nikolay Arefyev. 2022. Gloss- Reader LSCDiscovery: Train to select proper gloss in English discover lexical semantic change in Spanish. In Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change, pages 198-203, Dublin, Ireland. Association for Computational Linguistics.
Nina Tahmasebi, Lars Borin, and Adam Jatowt. 2021. Survey of computational approaches lexical seman- tic change. In Nina Tahmasebi, Lars Borin, Adam Jatowt, Yang Xu, and Simon Hengchen, editors, Com- putational Approaches Semantic Change, pages 1-91. Language Science Press, Berlin.
Radim Rehuřek and Petr Sojka, 2010. Software frame- work for topic modelling with large corpora. In Pro- ceedings of the LREC 2010 Workshop on New Chal- lengesfor for NLP Frameworks, pages 45-50, Valletta, Malta. ELRA.
Ana Uban, Alina Maria Ciobanu, and Liviu P. Dinu. 2019. Studying laws of semantic divergence across languages using cognate sets. In Proceedings of the 1st International Workshop Computational Ap- proaches Historical Language Change, pages 161- 166, Florence, Italy. Association for Computational Linguistics.
Martina Astrid Rodda, Philomen Probert, and Barbara McGillivray. 2019. Vector space models of Ancient Greek word meaning, and a case study on Homer. Traitement Automatique des Langues, 60(3):63-87.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz. Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-ther of-the--rttttttt natural language processing, In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Guy D. Rosin and Kira Radinsky. 2022. Temporal at- tention for language models. In Findings of the Asso- ciation for Computational Linguistics: NAACI. 2022, pages 1498-1508 Seattle, United States. Association for Computational Linguistics.
Julie Rouaud. 2019. Lexical and Phonological Integra- tion of French Loanwords into Varieties of Canadian English since the Seventeenth Century. Ph.D. thesis, Université Toulouse Jean Jaurès, Toulouse.
Dominik Schlechteeg Anna Hätty, Marco Del Tredici, and Sabine Schulte im Walde. 2019. A wind of change: Detecting and evaluating lexical semantic change across times and domains. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 732-746, Florence, Italy. Association for Computational Linguistics.
Yang Xu and Charles Kemp. 2015. A computational evaluation of two laws of semantic change. In Pro- ceedings of the 37th Annual Meeting of the Cognitive Science Society, pages 2703-2700 Austin, Texas. Cognitive Science Society.
Frank D. Zamora-Reina Felipe Bravo-Marquez, and Dominik Schlechtweg. 2022. LSCDiscovery: A shared task on semantic change discovery and de- tection in Spanish. In Proceedings of the 3rd Work- shop on Computational Approaches to Historical Language Change, pages 149- 149,164, Dublin, Ireland. Association for Computational Linguisties.
Dominik Schlechtweg. Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. SemEval-2020 task 1: Unsupervised lexical semantic change detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1-23. International Committee for Computational Linguistics.
Eline Zenner, Dirk Speelman, and Dirk Geeraerts. 2012. Cognitive Sociolinguisties meets loanword research: Measuring variation in the success of anglicisms in Dutch. Cognitive Linguistics, (3(4):749-792.
Dominik Schlechtweg, Sabine Schulte im Walde, and Stefanie Bekmann. 2018 Diachronic Usage Relat- edness (DURel): A framework for the annotation of

---

A Word embedding models
more than 1,000, and used a random sample for more frequent items. We fed each tweet single sequence into BERT, extracted the representations for the target word, and averaged over the last 4 hidden states to get single contextualized embed- ding. If target word was split into subwords by BERT's tokenizer, we averaged over them to get single embedding.
Preprocessing. Before being used in word em- bedding models, the corpus was tokenized and POS tagged using twokenize (Gimpel et al., 2011; Owoputi et al., 2013) and lemmatized using the NLTK WordNet lemmatizer (Bird et al., 2009).
Type-level embeddings. We used the best per- forming type-level configuration from an earlier standard evaluation (Miletié et al., 2021), which was formulated as binary classification task: word2vec skip-gram architecture with negative sampling, 100-dimensional vectors, window size of 5, and minimum word frequency of 100 in each subcorpus. The default values were used for other hyperparameters: negative sampling rate set to 5, subsampling rate set to 10-3, and number of itera- tions set to 5. The models were implemented using gensim (Rehüřek and Sojka, 2010).
We identified a word's similar uses by cluster- ing its contextualized embeddings using affinity propagation, in the scikit implementation (Pedregosa et al., 2011) with default parameters. Clusters containing at least 5 tweets, more than half of which were published in Montreal, were retained for annotation by domain experts. This was limited to a maximum of 10 clusters per item, starting with those with the highest proportion of tweets from Montreal.
Annotation consisted in cluster-level binary la- bels (presence vs. absence of contact-related use). A cluster was annotated as contact-related if, most occurrences in it, the target word's use was regionally specific to Montreal and potentially ex- plained by the influence of French. This was estab- lished based on the sociolinguistic literature and lexicographic evidence. We specifically did not consider that contact-related influence was con- stituted by locally specific proper names, French homographs, structural patterns (c.g. target word in tweet-initial position), or clusters where no reliable determination could be made. A 15-word sample was annotated by two annotators in order to test the reliability of the general procedure, obtaining a reasonably high Cohen's 's kappa coefficient of 0.55.
A model was trained independently for each of the three regional subcorpora. order for their vector spaces to be comparable, the models were aligned using Orthogonal Procrustes (Hamilton et al.. 2016); this corresponds to finding the optimal rotational alignment for each pair of matrices. The matrix columns were mean-centered before align- ment, as suggested by Schlechtweg et (2019).
We used the cosine distance (CD) to estimate se- mantic change across regional models. In order to control for word embedding instability (Pierrejean and Tanguy, 2018) 8), three models were trained for each subcorpus. We computed the CD for a word in subcorpora and b as follows:
CD(wa, mn) Σi CD( )
(I)
Computational infrastructure. The experi- ments were run on a CPU computational cluster, whose one node consists of 2 12 3GHz cores and 192GB RAM. We used 9 word2vec models (3 sub- corpora X 3 runs); each was trained on 12 cores in R 2 hours. We used pretrained BERT to model and cluster the occurrences of 40 target words; this completed in sz I hour on single node.
for n = runs of the model, where Wa, is the word's vector corresponding to the subcorpus a in the 7th run. We then averaged over the Montreal- Toronto and Montreal Vancouuee distances to ob- tain an estimate of semantic change.
Token-level embeddings. As in previous work (Miletić et al., 2021), we used teken-level embed- dings from the HuggingFace implementation (Wolf et al., 2020) of pert-bbas-unccased a 12-layer, 768-dimension version of BERT (110 million pa- rameters) pretrained on English data. No fine- tuning was performed under the assumption that the pretrained model should capture contextual dif- ferences necessary to distinguish word meanings.
B Interview protocol
Interview structure. We used the protocol devel- oped in the PAC research program (Phonology of Contemporary English: usage, varieties and struc- ture; Przewozny al., 2020). It is designed provide good coverage of key linguistic features and sociodemographic properties in different sur- vey locations. consists of the following tasks:
For each target word, we extracted its occur- rences from all three subcorpora; we retained no

---

(i) reading word lists targeting pronunciation of specific phonemes; (ii) reading a text targeting pronunciation in connected speech; (iii) a conver- sation with the fieldworker eliciting linguistic and sociodemographic background; (iv) a conversation with another interlocutor (optional in our case).
out of 15) were conducted remotely, using the Zoom video conferencing platform. For the se- mantic perception task, the participants used an online LimeSurvey interface, with the Zoom call running in the background; this enabled them to provide qualitative remarks on semantic shifts. The mean interview duration was I hour and 15 minutes (min= 56 min; max 1h 37min).
These tasks were followed by semantic per- ception test, which elicited range of information on 40 semantic shifts attested tweets from our corpus ($ 3.4). Key among those were acceptability ratings on a scale from to 6, for which the par- ticipants were instructed to (i) interpret I as "very unnatural, awkward, you would never say some- thing like that", and 6 as "completely natural, just like something you might say": (ii) rate the word used in the example, rather than the example as a whole; (iii) provide the rating with reference to their own use of the word rather than the way they think it should be used or they hear others use it.
The participants were asked to provide syn- onyms for the target word as an interpretation check, but not all of them did so systematically. We retained acceptability ratings lacking a synonym if all remaining speakers had provided the same interpretation. Otherwise, we only retained the rat- ings where participants had provided a synonym consistent with the target meaning.
Summary of speaker profiles. We recruited 10 female, 4 male, and non-binary participant. The distribution of other sociodemographic features is summarized in Table 3.
In choosing the tweets to be shown examples, we started from those tagged as contact-related in the cluster-level analysis (Appendix A). For each word, 3 annotators then chose potential examples each. Reconciliation was used to pick one, based on (i) idiomaticity and (ii) clarity of the context with respect to the target meaning. All personally identifiable information in the tweets, including user handles and hashtags, was redacted.
mean
std
min
max
age
37.5
18.4
19
70
RI
5.4
2.8
1
10
En.
0.83
0.19
0.43
1.00
Fr. 0.58
0.24
0.26
1.00
biling.
0.25
0.38
-0.57
0.70
Table 3: Summary of sociodemographic features
Participant recruitment. The interviews were conducted in Montreal between mid-January and mid-February 2022. The study was advertised through mailing lists of student associations, posters in public places etc. Recruitment mate- rials briefly explained the study and stated that it was open "all Montrealers aged 18 or over who are able to conduct a conversation in English". Potential participants were asked to contact the fieldworker by email, who in turn provided an in- formed consent form. It outlined the general aims of the study, its procedure, risks and benefits, right withdraw, and conditions of storing and using the collected data. The precise aims were disclosed after the interview. The consent form also explicitly stated that participation would not be remunerated; note that participation was entirely voluntary and required an active decision on the part of potential participants to contact the fieldworker. A total of 15 participants were recruited.
In the table, RI refers to the Regionality Index (Chambers and Heisler, 1999), which quantifies links with the local community. In our formulation, it ranges from 1, corresponding participants who were born, raised, and continue to live in the Mon- treal region; to 10, corresponding to those who live in Montreal, but were born and raised outside of Canada, as were their parents.
We also computed English and French language use scores (En. and Fr.) using a previously estab- lished procedure (Rouaud, 2019). The scores take into account proficiency, age and manner of acqui- sition, and frequency and domains of language use; they are normalized to a value range from We then subtracted the French score from the En- glish score to obtain a bilingualism score (biling.). It theoretically ranges from -1 (monolingual French speaker) to 1 (monolingual English speaker).
1.
Data recording and analysis. Due to public health restrictions at the time, most interviews (14

---

C Qualitative data
(2) would not say <entourage> in English speech.
I would say my close friends, would try to be more specific think, close friends or inner circle, yeah.
This section provides sample verbatim comments for the semantic shifts discussed in Section 4.3; note that italies denote metalinguistic reference and angled brackets denote pronunciation in French. We then provide summary of synonyms and qual- itative tags for semantic shifts in the test set.
(3) Yeah, this makes sense to me, but wouldn't t really use the word entourage. I think I'd just use the word group maybe, think the word group is all-encompassing in this context and would say that.
Sample comments: terrace
Rated example: The weather is still perfect for lunch on the terrace #iVny à Greenwich Village
Sample comments: definitively
Rated example: Thank you! My first try was tasty but kind of "liquidish". I will definitively try your recipes.
(I) Yeah, pretty normal. Sometimes I'll hear people say patio as an alternative, but usually terrace.
(2) That sounds super normal for and very fortable, and fully recognize that that's a Montreal thing, or Quebec thing. You would say, God I don't even remember what the, the non-Montreal version of that sent- that word is, it's not patio, it's something like patio.
(1) feel like somebody must have said it because it doesn't sound weird, don't it know, defi- I'll def- initely, what, definitively, yeah I think I'll leave it, I'll put it at three maybe because it does sound unnatural but it's not maybe that bad. Yeah guess would just say definitely instead but, yeah.
(3) Yeah, exactly what would use here. Terrace, yeah, outdoor, I can't even think of any other word for an outdoor space, it defimitely would be terrace, like I'm, I'm not sure, but terrace is hundred percent what would use here.
(2) Yes, people probably say it. Definitely but not definitively, that's not even what definitive means.
(3) almost, personally for me that's grammatically incorrect, would say definitely try your recipes.
Summary of collected information
Sample comments: boutique
Table outlines the information collected in the semantic perception task for the full set of 40 se- mantic shifts.
Rated example: The first H&M Home boutique will open shortly in Carrefour Laval.
(I) So this is, sounds very normal to me. I guess it would be like, you would say store, but boutique is just totally normal for me at this point, so that's very not awkward.
(2) Yeah, for sure, we use boutique all the time.
(3) I would replace it with store, again because I try to avoid French words in, in English, even though English is heavily influenced by French.
Sample comments: entourage
Rated example: believe this. I'm not there yet but my entourage is all approaching or in their early 30s and I can see/feel how they have much stronger sense of purpose, direction and self. Gives me something look forward and feel blessed to have people like this around me.
(1) would say it's bit unnatural, would say, for me, would say surroundings instead <en- tourage>. would say, would use <entourage> was speaking French, yeah.

---

acceptability
synonyms
social
item
top synonyms
type
tok.
loc.
Fr.
affirmation
3.27
1.79
statement (4); claim (2); argument (1)
11
0.64
ambiance
4.00
1.69
atmosphere (5); environment (2); vihe (2)
6
12
0.50
2
animator
4.11
1.54
advisor (2); coordinator (2); counsellor (2)
10
15
0.67
3
availability [pl.1
5.20
availability [sg.l (2); when you'l free (2); available time (1)
boutique
5.13
0.99
store (10); outlet (I); shop (1)
3
12
0.25
3
3
chalet
4.33
2.22
cottage (6); country (3); cabin (2)
5
14
0.36
3
1
circulation
3.80
1.82
traffic (11)
11
0.09
coordinate
2.70
1.64
contact information (5); information (3): phone number (2)
15
0.53
deceive
2.70
1.77
disappoint (6); let down (5)
2
11
0.18
deception
2.00
1.00
disappointmment (2); disappointed (1)
0.67
definitively
2.09
definitely (11); for sure (1)
12
0.17
deputy
4.20
1.94
elected official (4); MP (2); MNA (1)
8
12
0.67
dossier
3.62
1.77
issue (3); case (2); enjeu [Fr.] (1)
12
0.75
2
entourage
3.20
1.26
friend (5); surrounding (2); cirele (1)
17
0.71
exchange
3.27
1.98
talk (6); chat (3); interact (2)
9
18
0.50
exploration
4,60
1.76
investigation (2); search (2); examination (1)
7
9
0.78
exposition
4.87
1.68
exhibition (4); show (3); vernissage (2)
11
0.45
formation
2,80
1.78
training (8); formation [Irr.] (1): initiation (1)
13
0.46
5
formidable
3.47
1.96
great (6); amazing (2); awesome (1)
8
14
0.57
grave
1.87
1.12
serious (6): bad (4): grave [Fr.] (1)
3
11
0.27
3
hesitate
5.27
1.16
debate (3); be hesitant (I); can't decide
laureate
2.67
2.08
winner (2); accomplished people (I)
0.67
local
3.50
2.12
arcade (1); neighborhood establishment (1)
2
2
1.00
manifestation
3.67
1.88
protest (6); gathering (2); demonstration (1)
13
0.54
6
merit
3.33
1.54
deserve (11); earn (2)
13
militant
4.87
1.19
<omission> (2): activist (1); advocate (1)
7
8
0.88
1
2
nomination
4.53
1.96
appointment (2); <omission> (I); election (1)
4
5
0.80
occasion
4.47
1.55
opportunity (6); moment time (2)
11
0.36
1
pass
5.53
0.92
come by (4); stop by (4); drop by (3)
12
0.33
permit
4.27
1.87
licence (1))
1
II
0.09
3
population
5,60
0.74
people citizen (1); large group of people (1)
0.44
portable
2.87
1.92
phone (6); laptop (3): cell phone (2)
19
0.47
2
proposition
3.33
1.80
suggestion (5); idea (3); proposal (3)
6
14
0.43
2
prudent
3.80
1.74
careful (7); cautious (4); aware (I)
5
14
0.36
remark
2.47
1.81
notice (9); realize
1
reparation
2.87
1.85
repair (6); fixing (2); fix (1)
5
11
0.45
2
resume
2.07
1.16
show (6); sum up (2); summmarize (2)
8
15
0.53
4
souvenir
2.67
1.80
memory (9)
0.11
2
terrace
5,67
patio (7); outdoor place (1); outdoor space (1)
1
trio
4.80
1.74
combo (6); meal (1): triple order (1)
3
8 0.38
2
Table Summary of information collected the semantic perception test: acceptability ratings (mean and standard deviation); top 3 synonyms with number of speakers who provided them; quantitative characterization of the synonyms (number of types, number tokens, and type-token ratio); number of qualitative remarks expressing target social values: local specificity (loc.) and influence of French (Fr.).


Reviews:
[{"review_id": "b9PnLcAXn4", "reviewer": "Reviewer_3VVP", "summary": "", "questions": "1. Do you think that the fact that you used a model trained on \"generic\" English (not associated with a specific country/region) could have some effect on the results?\n2. It seems to me that it could be possible to obtain further insights into the contact-induced semantic shifts, by comparing the location-specific embeddings also with corresponding French word embeddings. Do you have any ideas for future work in this direction?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and for pointing out the novelty of our sociolinguistic approach.\n\n**Incomplete validation of the methods:** Our aim was not to optimize the validity of a specific method, but to assess existing approaches based on how their output is perceived by target speakers; we will clarify this. Low correlations between human judgments and some model-derived scores are informative in this respect.\n\n**Acceptability ratings in online communication vs. interviews:** We accept that there may be differences between the two settings, but this is inherent in our aim of assessing the outputs of Twitter-based models. We took care to explain the origin of the examples to the participants. We also adopted standard sociolinguistic practice to decrease the degree of formality, crucially by conducting the rating task at the end of the interview, when participants generally communicate more freely.\n\n**Question 1:** We deliberately chose BERT pretrained on generic English data so as to assess a widely used standard method. This may have some effect on a subset of our results, but we expect it to be minor. We assume that the model still captures contextual differences useful in disambiguating target word uses. We model and cluster the occurrences of one target word at a time, which should limit potential discrepancies in representations of different target words. A direct comparison with a fine-tuned model is a relevant direction of future work.\n\n**Question 2:** We believe that using French (or multilingual) word embedding models is a good idea which would enable us to directly compare the meanings of candidate semantic shifts in English and their French equivalents. As noted in Limitations, this is a promising methodological alternative, but it falls outside of the scope of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and reports on a thorough analyses both in the computational modeling and in the elicitation of judgements. The authors manage to report their work in detail, in spite of the 4-page limit. \n\nThe authors well describe the results of their study without simplifications and with attention to potential confounders. ", "reasons_to_reject": "Some of the results on the correlation between the computational scores and the human judgments remain inconclusive. Consequentially, the applicability of the methods is not fully validated. It is not clear if this is an issue of the general methodology often employed in the field, or of specifics of the way it is instantiated in the current study. \n\nAcceptability judgments elicited in interviews may not be as flexible as in the context of online speech. This may be a factor in the lack of a strong alignment with the semantic shift scores computed from Twitter data. ", "questions_to_authors": "1. Do you think that the fact that you used a model trained on \"generic\" English (not associated with a specific country/region) could have some effect on the results?\n2. It seems to me that it could be possible to obtain further insights into the contact-induced semantic shifts, by comparing the location-specific embeddings also with corresponding French word embeddings. Do you have any ideas for future work in this direction?", "paper_topic_and_main_contributions": "The paper employs computational methodologies to study semantic change to study contact-induced semantic shifts in Quebec English. Not only they aim to provide insights into the phenomenon, but they investigate the validity of these methods for a sociolinguistic study on speech, where they have not been extensively applied yet (the authors claim to be the first). The results indicate the applicability of the method overall but with some limits and caveats to consider. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YPBBzJbn8k", "reviewer": "Reviewer_a31m", "summary": "", "questions": "- Question A: could you please clarify in what way acceptability ratings reflect the local diffusion of contact-related uses? (Line 194)\n\n- Question B: If acceptability is assumed to be related to diffusion, what is the motivation behind testing whether it correlates with semantic distance (i.e. degree of shift)? I don't quite understand how acceptability ratings may help us to evaluate the cosine distance scores (the examples in 223-232 didn't help me unfortunately). Unlike in the study by Del Tredici et al. (2019), the speakers in this case did not evaluate degree of shift between two usages -- instead, they were asked to judge the acceptability of single usages (one single tweet per word). \n\n- Questions C: Are the correlations reported in sections 4.1 and 4.2 statistically significant? Please indicate this in the paper.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments and for underscoring the relevance of our synchronic analysis.\n\n**“A bit thin” amount of substance in the paper:** We respectfully disagree with this characterization. This short paper does not aim to propose a new computational method. Instead, we introduce a carefully designed and collected dataset, and use it to analyze existing computational methods. This analysis is strongly grounded in sociolinguistic practice and thereby provides a novel perspective on semantic change models, which we believe is a valuable contribution in its own right. \n\n**Small scale of the study:** We are aware of suboptimal aspects of our 15-participant sample (as discussed in Limitations), but we would not say that this is “a very small scale by any standards”. We deliberately collected in-depth qualitative information, for which we had to compensate by limiting the number of participants and rated instances; however, all participants rated all of the instances. We think this is a reasonable trade-off to obtain qualitative data, which is underrepresented in analyses of semantic change models. Regarding the robustness of claims, participants’ comments on local specificity and French influence were optional; a low absolute number may be relevant in relative terms, but we will clarify the strength of our claims.\n\n**Semantic shifts vs. code-switching in tweets:** As pointed out, we noted this issue for one example (l. 231-235), but it is overall rare in our data. We consider this as a minor tradeoff for access to orders of magnitude more data compared e.g. to interview recordings. We are unaware of alternatives which would allow us to model contact-induced semantic shifts at scale without this potential confound. \n\n**Question A:** We expect acceptability ratings to directly reflect the diffusion of semantic shifts among our participants, since they were instructed to provide the ratings based on their own language use.\nA more general assumption is that items which are more widespread in the broader speech community give rise to higher acceptability ratings. We accept that this link is indirect, and will clarify our assumptions.\n\n**Question B:** While we think that acceptability primarily reflects diffusion, it may also interact with other estimates, including the extent of change measured by cosine distance. We agree that this is not a targeted evaluation such as that of Del Tredici et al. (2019); our aim was to gain insights into how a standard sociolinguistic measure such as acceptability relates to a range of previously proposed computational estimates of semantic change. We will clarify this.\n\n**Question C:** In Section 4.1, the correlation with *cos-avg* is statistically significant, while those with *clust-cont* and *clust-reg* are not. Thank you for spotting this issue; we will report significance values. In Section 4.2, statistical significance is indicated through shading in Table 1, and the subsequent discussion is limited to statistically significant correlations (as noted l. 251).\n\n**Presentation:** Thank you for your suggestions; will provide examples and more detailed definitions in Section 3.2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting use of synchronic data from language contact regions to investigate semantic shift (rather than using diachronic data).\n- Qualitative analysis to complement and validate insights from authomatic methods for seamntic shift detection", "reasons_to_reject": "- While the use of qualitative analysis with community members is a strength, the current study has a very small scale: just 15 participants from only Montreal. This leads to not very robust claims (e.g., claiming at lines 287-288 that \"semantic shifts with high acceptability are generally described as regionally specific\" when only 3 out 15 participants have descrbed a word as locally specific, according to Table 2).  \n\n- Using written text (tweets) for this kind of study has the potential of confounding semantic shift with code-switching in cases where this would only be noticeable in spoken language. Lines 234-235 hint at this problem. While this may not be prominent in the current data, it seems a fundamental weakness of the approach.", "questions_to_authors": "- Question A: could you please clarify in what way acceptability ratings reflect the local diffusion of contact-related uses? (Line 194)\n\n- Question B: If acceptability is assumed to be related to diffusion, what is the motivation behind testing whether it correlates with semantic distance (i.e. degree of shift)? I don't quite understand how acceptability ratings may help us to evaluate the cosine distance scores (the examples in 223-232 didn't help me unfortunately). Unlike in the study by Del Tredici et al. (2019), the speakers in this case did not evaluate degree of shift between two usages -- instead, they were asked to judge the acceptability of single usages (one single tweet per word). \n\n- Questions C: Are the correlations reported in sections 4.1 and 4.2 statistically significant? Please indicate this in the paper.", "paper_topic_and_main_contributions": "This paper investigates how the output of lexical semantic change detection systems is perceived by the speakers of a given community. The focus is on Quebec English and the semantic shift that geographic contact with French may induce. The authors use an existing corpus of Canadian English tweets, focusing on language use specific to Montreal, where both English and French are widely used. They consider 40 words which previous work had identified as undergoing semantic shift due to language contact reasons. They derive type- and token-based word embeddings for these words and compute cosine distance across regions. The main contribution of the paper is to evaluate this computational estimates of semantic shift via face-to-face sociolinguistic interviews with 15 speakers from the community under study.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It would be useful to provide one or two examples of the 40 words considered in section 3.2 and explain more explicitely what is meant by \"contact meanings\" (line 126)."}, "extra_text": {}}, {"review_id": "pLhMNXe7QB", "reviewer": "Reviewer_TUJP", "summary": "", "questions": "What exactly are the assumptions behind the different computational measures? If the models give an accurate reflection of the semantic shifts, what should they show? How should the model scores interact with the human acceptability ratings?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and thought-provoking review, as well as for highlighting the relevance of our synchronic and sociolinguistic perspective. \n\n**Assumptions behind semantic change estimates:** In assessing several estimates, we aimed to account for different types of information that can be derived across modeling strategies. While the patterns they capture may partly overlap, our core assumptions are as follows.\n* *cos-avg* reflects the assumption that a contact-induced semantic shift should have a different static word embedding in the Montreal data compared to Toronto and Vancouver. This score should be higher for clear-cut usage differences (e.g. *definitively* ‘for sure’) than for those that are finer-grained (e.g. *population* ‘the people’).\n* *clust-cont* indicates the proportion of tweets in which a target word is used with a contact meaning. Since the vast majority of tweets containing a target word come from different users, we assume that this score indicates the diffusion of the contact-related use. It should correlate with estimates of diffusion measured in face-to-face communication.\n* *clust-reg* indicates the proportion of contact-related uses of a target word posted in Montreal, i.e. it measures their local specificity. We assume that this property varies for different reasons, e.g. an emergent semantic shift may be specific to Montreal, but an established one may spread elsewhere.\n\nThank you for pointing out the lack of clarity on this key issue. We will include more extensive explanations.\n\n**Establishing validity using human acceptability ratings:** In strict terms, we would consider that a computational method is validated by human acceptability ratings if it closely aligns with them. However, our motivation was broader: we wanted to gain insights into a range of previously proposed computational semantic change estimates by examining how they relate to a standard sociolinguistic estimate of the same phenomenon. Our main aim was therefore to understand how these estimates pattern with one another rather than establish a strict validity threshold. In retrospect, we realize that framing our analysis in terms of validity may have obscured this perspective. We will revise our phrasing in order to clarify this.\n\n**Confounding effect of genre (Twitter discourse):** We aimed to control for the potential effect of online-specific usage in several ways.\n* The rated examples were selected by three expert annotators, who prioritized tweets which did not read as stylistically marked. \n* We collected background information about the informants' perception of language use on social media.\n* We also instructed them to rate acceptability based on the usage of the target word rather than properties of its context. \n\nSome of these points are explained in Appendix B, but we will try to include more details in the body of the paper as well as acknowledge the remaining potential effect of genre variation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a (to the best of my knowledge) underrepresented application of language models to synchronic variation (rather than diachronic change. \n\nThe paper is well-grounded in work on regional variation and presents a carefully designed and collected dataset of speaker judgments. As such, it serves as a good example of computational work grounded in sociolinguistic research and data.\n\nThe study aims to control for several factors that may impact the reflection of shifts (e.g. frequency and polysemy).", "reasons_to_reject": "Core research questions and motivation of analysis: Ultimately, I found it very difficult to follow the core reasoning behind the experiments and observations. What should the two models show if they can reflect regional shifts accurately? What would a valid result look like? \n\nMethods for measuring change using language models: I was left wondering what exactly the two different ways of querying models should reflect and how they relate to each other. If I understand correctly, the method uses clustering (in combination with human labeling of clusters) to identify the tweets in which words are used in the language-contact sense (i.e. English words with high French influence used in English tweets). Two scores are calculated using this information: the proportion of contact-related tweets and the proportion of tweets from Montreal out of all contact-related tweets. The first score is argued to reflect the defusion of the contact-related word and the second score the regional specificity. The static method simply reflects the cosine difference between tweets from Montreal (high contact with French) to tweets from Vancouver and Toronto (low contact). To me, the intuition behind what the methods should show if they accurately reflect change is not completely clear. \n\nValidity analysis and interference between genre and region: The dataset only contains words that are affected by variation due to language contact. The scores obtained from the models are compared to human acceptability ratings of the target words. I am wondering whether factors not taken into account in the study could impact the comparison. Specifically, I could imagine that a twitter corpus may reflect word usages that are specific to online discourse (in particular Twitter discourse) that might not be seen as acceptable by all speakers. \n\nUpdate after rebuttal: Thank you for the clarifications! The explanations helped a lot and I am confident that the authors can make the necessary changes to their final version. I am thus increasing my soundness score.", "questions_to_authors": "What exactly are the assumptions behind the different computational measures? If the models give an accurate reflection of the semantic shifts, what should they show? How should the model scores interact with the human acceptability ratings?", "paper_topic_and_main_contributions": "The paper presents a study of regional semantic variation between English with high French influence in Montreal and English with less French influence (Vancouver and Toronto). The study compares insights obtained from static (type-based) and contextualized (token-based) models for measuring semantic shifts to insights obtained from sociolinguistic interviews with speakers of Montreal English. All experiments are based on 40 known semantic shifts and Twitter corpora from the three different regions. The models and interviews assess different aspects of semantic shifts: static embeddings are used to measure the distance between Montreal English (i.e. high French influence) to Vancouver and Toronto English; contextualized embeddings are used to measure the proportion of contact-related uses (tweets) and the degree of regional specificity via clustering approaches; the interviews provide acceptability ratings of the Montreal variants of the words (and other sociolinguistic information). The paper presents an analysis of how these different aspects interact. While some computational measures reflect human judgments, others do not. A qualitative analysis shows that similar computational scores can occur with highly different human assessments of a shift. \n\nI think the paper presents a valuable example of a sociolinguistically informed study of language models. However, I am not convinced by the argumentation. In particular, I find it very difficult to understand how the validity of the models is assessed (see reasons to reject). It is possible that I misunderstood something and I am willing to change my score if my questions can be clarified in the discussion period.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the paper is very interesting in that it uses diachronic semantic change models to investigate synchronic variation (in addition to presenting a new dataset).
Qualitative analysis is persuasive and the work is extremely well grounded in sociolinguistics. It is even more impressive for a short paper.

The questions posed by the reviewers got persuasive answers from the authors in the rebuttal phase. The authors should make sure to include the main points of these answers in the camera-ready version of the paper.
Decision: Accept-Main