Venue: NAACL 2019

# Short-Term Meaning Shift: A Distributional Exploration

Marco Del Tredici Raquel Fernandez\({}^{*}\) Gemma Boleda\({}^{\dagger}\)

\({}^{*}\)University of Amsterdam Universitat Pompeu Fabra

{m.deltredici|raquel.fernandez}@uva.nl

gemma.boleda@upf.edu

###### Abstract

We present the first exploration of meaning shift over short periods of time in online communities using distributional representations. We create a small annotated dataset and use it to assess the performance of a standard model for meaning shift detection on short-term meaning shift. We find that the model has problems distinguishing meaning shift from referential phenomena, and propose a measure of contextual variability to remedy this.

## 1 Introduction

Semantic change has received increasing attention in empirical Computational Linguistics / NLP in the last few years Tang (2018); Kutuzov et al. (2018). Almost all studies so far have focused on meaning shift in long periods of time--decades to centuries. However, the genesis of meaning shift and the mechanisms that produce it operate at much shorter time spans, ranging from the online agreement on words' meaning in dyadic interactions Brennan and Clark (1996) to the rapid spread of new meanings in relatively small communities of people in Wenger (1998); Eckert and McConnell-Ginet (1992). This paper is, to the best of our knowledge, the first exploration of the latter phenomenon--which we call **short-term meaning shift**--using distributional representations.

More concretely, we focus on meaning shift arising within a period of 8 years, and explore it on data from an online community of speakers, because there the adoption of new meanings happens at a fast pace Clark (1996); Hasan (2009). Indeed, short-term shift is usually hard to observe in standard language, such as the language of books or news, which has been the focus of long-term shift studies (e.g., Hamilton et al., 2016; Kulkarni et al., 2015), since it takes a long time for a new meaning to be widely accepted in the standard language.

Our contribution is twofold. First, we create a small dataset of short-term shift for analysis and evaluation, and qualitatively analyze the types of meaning shift we find.1 This is necessary because, unlike studies of long-term shift, we cannot rely on material previously gathered by linguists or lexicographers. Second, we test the behavior of a standard distributional model of semantic change when applied to short-term shift. Our results show that this model successfully detects most shifts in our data, but it overgeneralizes. Specifically, the model gets confused with contextual changes due to speakers in the community often talking about particular people and events, which are frequent on short time spans. We propose to use a measure of contextual variability to remedy this and showcase its potential to spot false positives of referential nature like these. We thus make progress in understanding the nature of semantic shift and towards improving computational models thereof.

Footnote 1: Data and code are available at: [https://github.com/marcoded13/Short-term-meaning-shift](https://github.com/marcoded13/Short-term-meaning-shift).

## 2 Related Work

Distributional models of semantic change are based on the hypothesis that a change in context of use mirrors a change in meaning. This in turn stems from the Distributional Hypothesis, that states that similarity in meaning results in similarity in context of use Harris (1954). Therefore, all models (including ours) spot semantic shift as a change in the word representation in different time periods. Among the most widely used techniques are Latent Semantic Analysis Sagi et al. (2011); Jatowt and Duh (2014), Topic Modeling Wijaya and Yeniterzi (2011), classic distributional representations based on co-occurence matrices of target words and context terms Gulordava and Baroni (2011). More recently, researchers have used

word embeddings computed using the skip-gram model by Mikolov et al. (2013). Since embeddings computed in different semantic spaces are not directly comparable, time related representations are usually made comparable either by aligning different semantic spaces through a transformation matrix Kulkarni et al. (2015); Azarbonyad et al. (2017); Hamilton et al. (2016) or by initializing the embeddings at \(t_{i+1}\) using those computed at \(t_{i}\)Kim et al. (2014); Del Tredici et al. (2016); Phillips et al. (2017); Szymanski (2017). We adopt the latter methodology (see Section 3.2).

Unlike most previous work, we focus on the language of online communities. Recent studies of this type of language have investigated the spread of new forms and meanings Del Tredici and Fernandez (2017, 2018); Stewart and Eisenstein (2018), competing lexical variants Rotabi et al. (2017), and the relation between conventions in a community and the social standing of its members Danescu-Niculescu-Mizil et al. (2013). None of these works has analyzed the ability of a distributional model to capture these phenomena, which is what we do in this paper for short-term meaning shift. Kulkarni et al. (2015) consider meaning shift in short time periods on Twitter data, but without providing an analysis of the observed shift nor systematically assessing the performance of the model, as we do here.

Evaluation of semantic shift is difficult, due to the lack of annotated datasets Frermann and Lapata (2016). For this reason, even for long-term shift, evaluation is usually performed by manually inspecting the \(n\) words whose representation changes the most according to the model under investigation Hamilton et al. (2016); Kim et al. (2014). Our dataset allows for a more systematic evaluation and analysis, and enables comparison in future studies.

## 3 Experimental Setup

### Data

We exploit user-generated language from an online forum of football fans, namely, the r/LiverpoolFC subreddit, one of the many communities hosted by the Reddit platform.2Del Tredici and Fernandez (2018) showed that this subreddit presents many characteristics that favour the creation and spread of linguistic innovations, such as a topic that reflects a strong external interest and high density of the connections among its users. This makes it a good candidate for our investigation. We focus on a short period of eight years, between 2011 and 2017. In order to enable a clearer observation of short-term meaning shift, we define two non-consecutive time bins: the first one (\(t_{1}\)) contains data from 2011-2013 and the second one (\(t_{2}\)) from 2017.3 We also use a large sample of community-independent language for the initialization of the word vectors, namely, a random crawl from Reddit in 2013. Table 1 shows the size of each sample.

Footnote 2: [https://www.reddit.com](https://www.reddit.com). We downloaded Reddit data using the Python package Praw: [https://pypi.python.org/pypi/python.org/pypi/praw/](https://pypi.python.org/pypi/python.org/pypi/praw/).

### Model

We adopt the model proposed by Kim et al. (2014), a representative method for computing diachronic meaning shift.4 While other methods might be equally suitable (see Section 2), we expect our results not to be method-specific, because they concern general properties of short-term shift, as we show in Sections 4 and 5. In the model by Kim et al. (2014), word embeddings for the first time bin \(t_{1}\) are initialized randomly; then, given a sequence of time-related samples, embeddings for \(t_{i}\) are initialized using the embeddings of \(t_{i-1}\) and further updated. If at \(t_{i}\) the word is used in the same contexts as in \(t_{i-1}\), its embedding will only be marginally updated, whereas a major change in the context of use will lead to a stronger update of the embedding. The model makes embeddings across time bins directly comparable.

Footnote 4: These choices ensure that the samples in these two time bins are approximately of the same size – see Table 1. The r/LiverpoolFC subreddit exists since 2009, but very little content was produced in 2009–2010.

Footnote 4: The model was implemented using the Python package Gensim: [https://pypi.python.org/pypi/gensim/](https://pypi.python.org/pypi/gensim/).

We implement the following steps: First, we create randomly initialized word embeddings with the large sample Reddit\({}_{13}\) to obtain meaning representations that are community-independent. We then use these embeddings to initialize those in LiverpoolFC\({}_{13}\), update the vectors on this sample, and thus obtain embeddings for time \(t_{1}\). This step adapts the general embeddings to the LiverpoolFC community. Finally, we initialize the word embeddings for LiverpoolFC\({}_{17}\) with those of \(t_{1}\), train on this sample, and get embeddings for \(t_{2}\).

The vocabulary is defined as the intersection of the vocabula 

(Reddit\({}_{13}\), LiverpoolFC\({}_{13}\), LiverpoolFC\({}_{17}\)), and includes 157k words. For Reddit\({}_{13}\), we include only words that occur at least 20 times in the sample, so as to ensure meaningful representations for each word, while for the other two samples we do not use any frequency threshold: Since the embeddings used for the initialization of LiverpoolFC\({}_{13}\) encode community-independent meanings, if a word doesn't occur in LiverpoolFC\({}_{13}\) its representation will simply be as in Reddit\({}_{13}\), which reflects the idea that if a word is not used in a community, then its meaning is not altered within that community. We train with standard skip-gram parameters Levy et al. (2015): window 5, learning rate 0.01, embedding dimension 200, hierarchical softmax.

### Evaluation dataset

Our dataset consists of 97 words from the r/LiverpoolFC subreddit with annotations by members of the subreddit --that is, community members with domain knowledge (needed for this task) but no linguistic background.

To ensure that we would get enough cases of semantic shift to enable a meaningful analysis, we started out from content words that increase their relative frequency between \(t_{1}\) and \(t_{2}\).5 A threshold of 2 standard deviations above the mean yielded \(\sim\)200 words. The first author manually identified 34 semantic shift candidates among these words by analyzing their contexts of use in the r/LiverpoolFC data. Semantic shift is defined here as a change in the ontological type that a word denotes, which takes place when the word starts to be used to denote an entity which is different from the one originally denoted and the new use spreads among the members of a community (see examples in Sec. 4). We added two types of confounders: 33 words with a significant frequency increase but not marked as meaning shift candidates, and 33 words with constant frequency between \(t_{1}\) and \(t_{2}\), included as a sanity check. All words have absolute frequency in range [50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60].

Footnote 5: Frequency increase has been shown to positively correlate with meaning change Wijaya and Yeniterzi (2011); Kulkarni et al. (2015); although it is not a necessary condition, it is a reasonable starting point, as a random selection of words would contain very few positive examples. Our dataset is thus biased towards precision over recall.

The participants were shown the 100 words (in randomized order) together with randomly chosen contexts of usage from each time period (\(\mu\)=4.7 contexts per word) and, for simplicity, were asked to make a binary decision about whether there was a change in meaning. In order to have the redditors familiarize with the concept of meaning change, we first provide them with an intuitive, non-technical definition, and then a set of cases that exemplify it. The instructions to participants can be found in the project's GitHub repository (see footnote 1).

Semantic shift is arguably a graded notion. In line with a suggestion by Kutuzov et al. (2018) to account for this fact, we aggregate the annotations into a graded _semantic shift index_, ranging from 0 (no shift) to 1 (shift) depending on how many subjects spotted semantic change. The shift index is exclusively based on the judgments of the redditors, and does not consider the preliminary candidate selection done by us. Overall, 26 members of r/LiverpoolFC participated in the survey, and each word received on average 8.8 judgements. Further details about the dataset are in Appendix A.

## 4 Types of Meaning Shift

We identify three main types of shift in our data via qualitative analysis of examples with a shift index \(>0.5\): metonymy, metaphor, and meme.

Metonymym.In metonymic shifts, a highly salient characteristic of an entity is used to refer to it. Among these cases are, for example, _'highlighter'_, which in \(t_{2}\) occurs in sentences like _'we are playing with the highlighter today'_, or _'what's up with the hate for this kit? This is great, ten times better than the highlighter'_, used to talk about a kit in a colour similar to that of a highlighter pen; or _'lean'_, in _'I hope a lean comes soon!'_, _'Somebody with speed...make a signing...Cuz I need a lean'_, used to talk about hiring players due to new hires typically leaning on a Liverpool symbol when posing for a photo right after signing for the club. Particularly illustrative is the 'F5' example shown in Table 2. While 'F5' is initially used with its common usage of shortcut for refreshing a page (1), it then starts to denote the act of refreshing in order to get the latest news about the possible transfer of a new

\begin{table}
\begin{tabular}{l c c} \hline
**sample** & **time bin** & **million tokens** \\ \hline Reddit\({}_{13}\) & 2013 & \(\sim\)900 \\ LiverpoolFC\({}_{13}\) & 2011–13 & 8.5 \\ LiverpoolFC\({}_{17}\) & 2017 & 11.9 \\ \hline \end{tabular}
\end{table}
Table 1: Time bin and size of the datasets.

 

player to LiverpoolFC (2). This use catches on and many redditors use it to express their tension while waiting for good news (3-5),6 though not all members are aware of the new meaning of the word (6). When the transfer is almost done, someone leaves the _'F5 squad'_[7], and after a while, another member recalls the period in which the word was used (8).

Footnote 6: Here the semantic change is accompanied by a change in the part of speech, and ‘F5’ becomes a denominal verb.

Metaphor.Metaphorical shifts lead to a broadening of the original meaning of a word through analogy. For example, in _\(t_{2}\) 'shovel'_ occurs in sentences such as _'welcome aboard, here is your shovel'_ or _'you boys know how to shovel coal'_: the team is seen as a train that is running through the season, and every supporter is asked to figuratively contribute by showing coal into the train boiler.

Meme.Finally, memes are another prominent source of meaning shift. In this case, fans use a word to make jokes and be sarcastic, and the new usage quickly spreads within the community. As an example, while Liverpool was about to sign a new player named Van Dijk, redditors started to play with the homography of the first part of the surname with the common noun 'van', its plural form 'vans', and the shoes brand 'Vans': _'Rumour has it Van Djik was wearing these vans in the van'_ or _'How many vans could Van Dijk vear if Van Dijk could year vans'_. Jokes of this kind are positively received in the community (_'Hahah I love it. Anything with vans is instant karma!"_) and quickly become frequent in it.

## 5 Modeling Results and Analysis

The positive correlation between cosine distance and semantic shift index (Pearson's \(r\)= 0.49, \(p<0.001\), see Figure 1) confirms the hypothesis that meaning shift is mirrored by a change in context of use. However, we also find systematic deviations.

### False negatives

A small, but consistent group is that of words that undergo semantic shift but are not captured by the model (blue vertical ellipsis Figure 1; shift index\(>\)0.5, cosine distance\(<\)0.25). These are all metaphorical shifts; in particular, cases of extended metaphor (Werth, 1994), where the metaphor is developed throughout the whole text. For instance, besides the _'shovel'_ example mentioned in Section 4, we find _'pharaoh'_, the nickname of an Egyptian player who joined Liverpool in 2017, used in sentences like _'approved by our new Pharaoh Tutankhamun'_, or _'our dear Egyptian pharaoh, let's hope he becomes a God'_. Despite the metaphoric usage, the local context of these words is similar to the literal one, and so the model does not spot the meaning shift. We expect this to happen in long-term shift models, too, but we are not aware of results confirming this.

### False positives

A larger group of problematic cases is that of words that do _not_ undergo semantic shift despite showing relatively large differences in context between \(t_{1}\) and \(t_{2}\) (red horizontal ellipsis in Figure 1; shift index=0, cosine distance\(>\)0.25). Manual in

Figure 1: Semantic shift index vs. cosine distance in the evaluation dataset (Pearson’s \(r\)= 0.49, \(p<0.001\)). Red horizontal ellipsis: false positives; blue vertical ellipsis: false negatives.

 

spection reveals that most of these "errors" are due to a referential effect: words are used almost exclusively to refer to a specific person or event in \(t_{2}\), and so the context of use is different from the contexts in \(t_{1}\). For instance, '_stubborn'_ is almost always used to talk about a coach who was not there in 2013 but only in 2017; '_entourage'_, for the entourage of a particular star of the team; '_independence'_ for the political events in Catalonia (Spain). In all these cases, the meaning of the word stays the same, despite the change in context. In line with the Distributional Hypothesis, the model spots the context change, but it is not sensitive to its nature. We expect long-term shift to not be as susceptible to referential effects like these because embeddings are aggregated over a larger and more varied number of occurrences.

We expect that in referential cases the contexts of use will be _narrower_ than for words with actual semantic shift, as they are specific to one person or event. Hence, a measure of _contextual variability_ should help spot false positives. To test this hypothesis, we define contextual variability as follows: For a target word, we create a vector for each of its contexts (5 words on both sides of the target) in \(t_{2}\) by averaging the embeddings of the words occurring in it, and define variability as the average pairwise cosine distance between context vectors.7 We find that contextual variability is indeed significantly correlated with semantic shift in our dataset (Pearson's \(r=0.55\), \(p<0.001\)), while it is independent from cosine distance (Pearson's \(r\)= 0.18, \(p>0.05\)). These two aspects are thus complementary. While both shift words and referential cases change context of use in \(t_{2}\), context variability captures the fact that only in referential cases words occur in a restricted set of contexts. Figure 2 shows this effect visually. This result can inform future models of short-term meaning shift.

Footnote 7: There are alternative ways of measuring contextual variability, but we expect them to yield the same picture. For instance, we experimented with a different window size and obtained the same pattern.

## 6 Conclusion

The goal of this initial study was to bring to the attention of the NLP community **short-term meaning shift**, an under-studied problem in the field. Our hope is that it will spark further research into a phenomenon which, besides being of theoretical interest, has potential practical implications for NLP downstream tasks concerned with user-generated language, as modeling how word meanings rapidly change in communities would allow a better understanding of what their members say. Future research should experiment with other datasets (reddits from other domains, other online communities) and also alternative models that address the challenges described here.

## Acknowledgements

The research carried out by the Amsterdam section of the team was partially funded by the Netherlands Organisation for Scientific Research (NWO) under VIDI grant no. 276-89-008, _Asymmetry in Conversation_. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 715154), and from the Spanish Ramon y Cajal programme (grant RYC-2015-18907). This paper reflects the authors' view only, and the EU is not responsible for any use that may be made of the information it contains.

## References

* Artstein and Poesio [2008] Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. _Computational Linguistics_, 34(4):555-596.
* Azarbonyad et al. [2017] Hosein Azarbonyad, Mostafa Dehghani, Kaspar Beelen, Alexandra Arkut, Maarten Marx, and Jaap Kamps. 2017. Words are malleable: Computing

Figure 2: Semantic shift index vs. context variability. Red horizontal ellipsis: referential cases which are assigned high cosine distance values by the model (false positives).

 

semantic shifts in political and media discourse. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_, pages 1509-1518. ACM.
* Brennan and Clark (1996) Susan E. Brennan and Herbert H. Clark. 1996. Conceptual pacts and lexical choice in conversation. _Journal of Experimental Psychology: Learning, Memory, and Cognition_, 22(6):1482.
* Clark (1996) Herbert H. Clark. 1996. _Using language_. Cambridge University Press.
* Danescu-Niculescu-Mizil et al. (2013) Cristian Danescu-Niculescu-Mizil, Robert West, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. No country for old members: User lifecycle and linguistic change in online communities. In _Proceedings of the 22nd international conference on World Wide Web_, pages 307-318. ACM.
* Tredici and Fernandez (2017) Marco Del Tredici and Raquel Fernandez. 2017. Semantic variation in online communities of practice. In _IWCS 2017-12th International Conference on Computational Semantics-Long papers_.
* Del Tredici and Fernandez (2018) Marco Del Tredici and Raquel Fernandez. 2018. The road to success: Assessing the fate of linguistic innovations in online communities. In _COLING 2018-27th International Conference on Computational Linguistics-Long papers_.
* Third Italian Conference on Computational Linguistics_.
* Eckert and McConnell-Ginet (1992) Penelope Eckert and Sally McConnell-Ginet. 1992. Communities of practice: Where language, gender, and power all live. In _Locating Power, Proceedings of the 1992 Berkeley Women and Language Conference_, pages 89-99.
* Frermann and Lapata (2016) Lea Frermann and Mirella Lapata. 2016. A bayesian model of diachronic meaning change. _Transactions of the Association for Computational Linguistics_, 4:31-45.
* Gulordava and Baroni (2011) Kristina Gulordava and Marco Baroni. 2011. A distributional similarity approach to the detection of semantic change in the google books ngram corpus. In _Proceedings of the GEMS 2011 Workshop on Geometrical Models of Natural Language Semantics_, pages 67-71. Association for Computational Linguistics.
* Hamilton et al. (2016) William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 1489-1501.
* Harris (1954) Zellig S Harris. 1954. Distributional structure. _Word_, 10(2-3):146-162.
* Hasan (2009) Ruqaiya Hasan. 2009. _Semantic variation: Meaning in society and in sociolinguistics_. Equinox London.
* Jatowt and Duh (2014) Adam Jatowt and Kevin Duh. 2014. A framework for analyzing semantic change of words across time. In _Proceedings of the 14th ACM/IEEE-CS Joint Conference on Digital Libraries_, pages 229-238. IEEE Press.
* Kim et al. (2014) Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of language through neural language models. In _Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science_, pages 61-65. Association for Computational Linguistics.
* Kulkarni et al. (2015) Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detection of linguistic change. In _Proceedings of the 24th International Conference on World Wide Web_, pages 625-635. ACM.
* Kutuzov et al. (2018) Andrey Kutuzov, Lilja Ovrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic word embeddings and semantic shifts: a survey. In _Proceedings of the 27th International Conference on Computational Linguistics (COLING)_, pages 1384-1397. Association for Computational Linguistics.
* Levy et al. (2015) Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. _Transactions of the Association for Computational Linguistics_, 3:211-225.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_, pages 3111-3119.
* Phillips et al. (2017) Lawrence Phillips, Kyle Shaffer, Dustin Arendt, Nathan Hodas, and Svitlana Volkova. 2017. Intrinsic and extrinsic evaluation of spatiotemporal text representations in twitter streams. In _Proceedings of the 2nd Workshop on Representation Learning for NLP_, pages 201-210.
* Rotabi et al. (2017) Rahmtin Rotabi, Cristian Danescu-Niculescu-Mizil, and Jon Kleinberg. 2017. Competition and selection among conventions. In _Proceedings of the 26th International Conference on World Wide Web_, pages 1361-1370. International World Wide Web Conferences Steering Committee.
* Sagi et al. (2011) Eyal Sagi, Stefan Kaufmann, and Brady Clark. 2011. Tracing semantic change with latent semantic analysis. _Current methods in historical semantics_, pages 161-183.
* Stewart and Eisenstein (2018) Ian Stewart and Jacob Eisenstein. 2018. Making" fetch" happen: The influence of social and linguistic context on the success of lexical innovations. _Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP)_ 

* Szymanski (2017) Terrence Szymanski. 2017. Temporal word analogies: Identifying lexical replacement with diachronic word embeddings. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, volume 2, pages 448-453.
* Tang (2018) Xuri Tang. 2018. A state-of-the-art of semantic change computation. _Natural Language Engineering_, 24(5):649-676.
* Wenger (1998) Etienne Wenger. 1998. _Communities of practice: Learning, meaning, and identity_. Cambridge University Press.
* Werth (1994) Paul Werth. 1994. Extended metaphor--a text-world account. _Language and literature_, 3(2):79-103.
* Vijaya and Yeniterzi (2011) Derry Tanti Vijaya and Reyyan Yeniterzi. 2011. Understanding semantic change of words over centuries. In _Proceedings of the 2011 international workshop on DETecting and Exploiting Cultural diversetTy on the social web_, pages 35-40. ACM.

## Appendix A Further Details on Evaluation Dataset

For our experiment, we considered content words only, which we identified by using the external list of common words available at [https://www.wordfrequency.info/free.asp](https://www.wordfrequency.info/free.asp).

Three words were discarded from the initial list after analysis of the redditor data: 'discord' and 'owls' due to the homonymy with proper names not detected during survey's implementation; 'tracking' because the chosen examples clearly mislead the judgements of the redditors.

As detailed in Section 3.3, 26 members of r/LiverpoolFC participated in the survey, and each word received on average 8.8 judgements. We computed inter-annotator agreement as Krippendorff's alpha, and obtained \(\alpha\) = 0.58, a relatively low value but common in semantic tasks [1].

The results of the annotation validate our initial word sampling procedure:

* the words that present a significant increase in frequency and were annotated as meaning shift by us received an average shift annotation of 0.72 (\(\pm\) 0.15);
* the words that present a significant increase in frequency but that were _not_ annotated as meaning shift by us received an average shift annotation of 0.15 (\(\pm\) 0.16);
* the words that keep a constant frequency between \(t_{1}\) and \(t_{2}\), and we don't consider examples of meaning shift, got 0.07 (\(\pm\) 0.12)