Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation
Year: 2023

Abstract:
Dynamic contextualised word embeddings (DCWEs) represent the temporal semantic variations of words. We propose a method for learning DCWEs by time-adapting a pretrained Masked Language Model (MLM) using timesensitive templates. Given two snapshots C1 and C2 of a corpus taken respectively at two distinct timestamps T1 and T2, we first propose an unsupervised method to select (a) pivot terms related to both C1 and C2, and (b) anchor terms that are associated with a specific pivot term in each individual snapshot. We then generate prompts by filling manually compiled templates using the extracted pivot and anchor terms. Moreover, we propose an automatic method to learn time-sensitive templates from C1 and C2, without requiring any human supervision. Next, we use the generated prompts to adapt a pretrained MLM to T2 by fine-tuning using those prompts. Multiple experiments show that our proposed method reduces the perplexity of test sentences in C2, outperforming the current state-of-the-art.

heading: 1 Introduction
text: Contextualised word embeddings produced by MLMs (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020; Yang et al., 2019) represent the meaning of a word with respect to the context in which it appears in and have reported substantial performance gains in various NLP tasks. The usage of words change over time and the same word might be associated with different words to mean different concepts over time (Koch, 2016; Baybee, 2015; Rama and Borin, 2015). For example, the word gay has gradually changed its meaning from happy to homosexual over the last five decades (Robinson, 2012; Campbell, 2004). However, MLMs are often trained using a static snapshot of a corpus taken at a specific timestamp, and are not updated afterwards. Because of this reason, existing pretrained MLMs do not capture the temporal semantic variations of
words. For example, Loureiro et al. (2022) showed that neither the original version of BERT (Devlin et al., 2019) nor RoBERTa (Liu et al., 2019) are upto-date with the information related to the current coronavirus pandemic.
To address the above-mentioned limitations, we propose a Dynamic Contextualised Word Embedding (DCWE) method that adapts a given pretrained MLM from one timestamp T1 to another T2 using two snapshots of a corpus C1 and C2, sampled respectively at times T1 and T2. We represent a word x by an embedding that depends both on the context c of x, as well as on time T . Our word embeddings are dynamic because they depend on the time, and contextualised because they also depend on the context.
We model the problem of adapting a given pretrained MLM to a specific timestamp T2 as an instance of prompt-based fine-tuning (Liu et al., 2021), which has been successfully used in prior work to adapt MLMs to various tasks such as relation representation (Ushio et al., 2021; Fichtel et al., 2021), domain adaptation (Ben-David et al., 2021), natural language inference (Utama et al., 2021) and question answering (Qin and Eisner, 2021). Compared to fine-tuning MLMs on manually labelled training instances, which might not be readily available or costly to manually annotate in sufficient quantities for a particular task to fine-tune a largescale MLM, prompt-based methods require only a small number of prompts (Le Scao and Rush, 2021). Luckily, in our case of temporal adaptation of MLMs (Agarwal and Nenkova, 2022), such prompts could be generated from a handful of manually designed templates (§3.1) or automatically extracted from unlabelled data (§3.3). This aspect of our proposed method is particularly attractive compared to prior work (see §2) on DWEs (Rudolph and Blei, 2018; Hofmann et al., 2021; Qiu and Xu, 2022; Loureiro et al., 2022) that require retraining of MLMs from scratch to incorporate the time-
9352
sensitive constraints into the embedding spaces. We first extract pivot words, w, that are common to both C1 and C2. Second, we extract anchor words u and v that are strongly associated with w in respectively C1 and C2. We then propose several methods to score tuples (w, u, v) such that the semantic variation of w from T1 to T2 is captured by its association with respectively u and v. Finally, we generate a large number of textual prompts using the top-scoring tuples (w, u, v) according to each method to fill the slots in manually written templates such as “⟨w⟩ is associated with ⟨u⟩ in ⟨T1⟩, whereas it is associated with ⟨v⟩ in ⟨T2⟩.” Here, the slots corresponding to T1 and T2 are filled by specific years when respectively C1 and C2 were sampled. We differentiate templates from prompts throughout the paper where the latter is formed by filling one or more slots in the former. We further propose a method to automatically generate templates from sentences selected from C1 and C2 using a text-to-text transformation model (Raffel et al., 2020), thereby obviating the need to manually create templates. Finally, the given MLM is adapted to T2 by fine-tuning it on the generated prompts.
Experimental results conducted on Reddit, Yelp, ArXiv and Ciao datasets show that the proposed prompt-based time-adapting of MLMs consistently outperforms previously proposed DCWEs (Hofmann et al., 2021) and temporal adaptation methods (Rosin et al., 2022) reporting better (lower) perplexity scores on unseen test sentences in C2. The source code for our proposed method is publicly available.1

heading: 2 Related Work
text: Methods that use part-of-speech (Mihalcea and Nastase, 2012), entropy (Tang et al., 2016), latent semantic analysis (Sagi et al., 2011) and temporal semantic indexing (Basile et al., 2014) have been proposed for detecting changes in word meanings. In SemEval-2020 Task 1 (Schlechtweg et al., 2020) two subtasks were proposed for detecting lexical semantic change: a binary classification task (for a given set of target words, decide which words had their meaning altered, and which ones not) and a ranking task (rank a set of target words according to their degree of lexical semantic change between the two corpora). Giulianelli et al. (2020) showed that contextualised embeddings obtained from an
1https://github.com/LivNLP/TimeAdapted-DCWE
MLM can be used to measure the change of word meaning. Rosin and Radinsky (2022) proposed a temporal attention mechanism by extending the self-attention mechanism in transformers, where time stamps of the documents are considered when computing the attention scores. Aida and Bollegala (2023) proposed a method to predict semantic change of words by comparing the distributions of contextualised embeddings of the word between two given corpora, sampled at different points in time. Our goal in this paper extends beyond the detection of a subset of words with a change in lexical semantics, and to adapt MLMs over time.
DWEs (Rudolph and Blei, 2018; Hofmann et al., 2021; Qiu and Xu, 2022; Loureiro et al., 2022) incorporate extralinguistic information such as time, demographic or social aspects of words with linguistic information. Welch et al. (2020) learnt demographic word embeddings, covering attributes such as age, gender, location and religion. Zeng et al. (2017) learnt socialised word embeddings considering both a social media user’s personal characteristics of language use and that user’s social relationships. However, Hofmann et al. (2021) showed that temporal factors have a stronger impact than socio-cultural factors when determining the semantic variations of words. Consequently, in this paper we focus on the temporal adaptation of DCWEs.
Diachronic Language Models that capture the meanings of words at a particular point in time have been trained using historical corpora (Qiu and Xu, 2022; Loureiro et al., 2022). These prior work learn independent word embedding models from different corpora. This is problematic because information related to a word is not shared across different models resulting in inefficient learning, especially when word occurrences within a single snapshot of a corpus are too sparse to learn accurate embeddings.
Rudolph and Blei (2018) proposed a dynamic Bernoulli embedding method based on exponential family embeddings, where each word is represented by a one-hot vector with dimensionality set to the vocabulary size. This model is extended to the temporal case by considering different timeslices where only the word embedding vector is time-specific and the context vectors are shared across the corpus and over time-slices. Because the joint distribution over time and context is intractable, they maximise the pseudo log-likelihood
of the conditional distribution for learning the parameters of their DWE model. Ben-David et al. (2021) proposed a domain adaptation method based on automatically learnt prompts. Given a test example, they generate a unique prompt and conditioned on it, then predict labels for test examples. Although their method uses prompts to adapt a model, they do not consider temporal adaptation of MLMs, which is our focus. Moreover, we do not require any labelled examples in our proposal.
Amba Hombaiah et al. (2021) proposed a model updating method using vocabulary composition and data sampling to adapt language models to continuously evolving web content. However, their work is specific to one dataset and two classification tasks, and focuses on incremental training. Jang et al. (2022) introduced a benchmark for everevolving language models, utilising the difference between consecutive snapshots of datasets, to track language models’ ability to retain existing knowledge while incorporating new knowledge at each time point. Jin et al. (2022) studied the lifelong language model pretraining problem, where the goal is to continually update pretrained language models using emerging data. Dhingra et al. (2022) introduced a diagnostic dataset to investigate language models for factual knowledge that changes over time and proposed an approach to jointly model texts with their timestamps. They also demonstrated that models trained with temporal context can be adapted to new data without retraining from scratch. Rosin et al. (2022) proposed TempoBERT, where they insert a special time-related token to each sentence and fine-tune BERT using a customised time masking. TempoBERT reports superior results in SemEval 2020 Task 1 semantic variation detection benchmark. As shown later in §4.1, our proposed method outperforms TempoBERT.
Hofmann et al. (2021) proposed DCWEs, which are computed in two stages. First, words are mapped to dynamic type-level representations considering temporal and social information. The typelevel representation of a word is formed by combining a non-dynamic embedding of a word and a dynamic offset that is specific to the social and temporal aspects of the word. Second, these dynamic embeddings are converted to context-dependent tokenlevel representations. To the best of our knowledge, this is the only word embedding method that produces both dynamic as well as contextualised representations, thus mostly relates to us. As
shown in § 4, our proposed method outperforms their DCWEs on four datasets.

heading: 3 Prompt-based Time Adaptation
text: Given two snapshots C1 and C2 of a corpus taken respectively at timestamps T1 and T2(> T1), we consider the problem of adapting a pretrained MLM M from T1 to T2. We refer to a word w that occurs in both C1 and C2 but has its meaning altered between the two snapshots as a pivot. We propose three methods for selecting tuples (w, u, v), where u is closely associated with the meaning of w in C1, whereas v is closely associated with the meaning of w in C2. We name u and v collectively as the anchors of w, representing its meaning at T1 and T2. If the meaning of w has changed from T1 to T2, it will be associated with different sets of anchors, otherwise by similar sets of anchors. We then fine-tune M on prompts generated by substituting (w, u, v) in templates created either manually (§3.1) or automatically (§3.3).

heading: 3.1 Prompts from Manual Templates
text: In order to capture temporal semantic variations of words, we create the template ⟨w⟩ is associated with ⟨u⟩ in ⟨T1⟩, whereas it is associated with ⟨v⟩ in ⟨T2⟩.2 We generate multiple prompts from this template by substituting tuples (w, u, v) extracted using three methods as described in § 3.2. For example, given a tuple (mask, hide, vaccine) and T1 = 2010 and T2 = 2020, the previous template produces the prompt: mask is associated with hide in 2010, whereas it is associated with vaccine in 2020. These prompts are used in §3.5 to fine-tune an MLM to adapt it to T2 for obtaining DCWEs.

heading: 3.2 Tuple Selection Methods
text: Given a template with slots corresponding to w, u and v, we propose three different criteria for selecting tuples to fill those slots.

heading: 3.2.1 Frequency-based Tuple Selection
text: Prior work on domain adaptation has shown that words highly co-occurring in both source and target domains are ideal candidates for adapting a model trained on the source domain to the target domain. Following prior work on cross-domain representation learning, we call such words as pivots (Bollegala et al., 2015). Specifically, we measure the
2We experimented with multiple manual templates as shown in the Supplementary but did not observe any clear improvements over this template.
suitability of a word w, score(w) as a pivot by (1).
score(w) = min(f(w,C1), f(w,C2)) (1)
Here, f(w,C1) and f(w,C2) denote the frequency of w respectively in C1 and C2, measured by the number of sentences in which w occurs in each corpus. We sort words in the descending order of the scores given by (1) and select the top k-ranked words as pivots.
Next, for each pivot w, we select its anchors x by the Pointwise Mutual Information, PMI(w, x;C), computed from the snapshot C as in (2).
PMI(w, x;C) = log
( p(w, x)
p(w)p(x)
) (2)
Here, p(x) is the marginal probability of x in C, estimated as f(x,C)/NC , where NC is the total number of sentences in C. Moreover, p(w, x) is the joint probability between w and x, estimated as cooc(w, x)/NC , where cooc is the total number of co-occurrences between w and x in C, considering sentences as the contextual window for the co-occurrences.
We select the set of words U(w) with high PMI(w, u;C1) values as the anchors of w in C1. Likewise, the set of words V(w) with the topPMI(w, v;C2) are selected as the anchors of w in C2. By construction, anchors are the words that are strongly associated with a pivot in each snapshot of the corpus, thus can be regarded as representing the meaning carried by the pivot in a snapshot according to the distributional hypothesis (Firth, 1957). Finally, for each w, we obtain a set of tuples, Sfreq = {(w, u, v)|u ∈ U(w), v ∈ V(w)}, by considering all pairwise combinations of anchors with a pivot for the purpose of filling the templates to generate prompts.

heading: 3.2.2 Diversity-based Tuple Selection
text: Recall that our objective is to select anchors u and v, respectively in C1 and C2 such that the change of meaning of a pivot w is captured by the tuple (w, u, v). Frequency-based tuple selection method described in §3.2.1 finds u and v, which are strongly associated with w in the two snapshots of the corpus. However, if U(w) and V(w) are highly similar, it could mean that the meaning of w might not have changed from T1 to T2. To address this issue, we define diversity of w as the dissimilarity between its sets of anchors as in (3).
diversity(w) = 1− |U(w) ∩ V(w)||U(w) ∪ V(w)| (3)
Here, |X | denotes the cardinality of the set X , and the term subtracted from 1 can be identified as the Jaccard coefficient between U(w) and V(w). We select the top scoring w according to (1) and rerank them by (3) to select top-k pivots. Finally, we generate a set of tuples, Sdiv(w) = {(w, u, v)|u ∈ U(w), v ∈ V(w)}, by pairing each selected pivot w with its anchors in C1 and C2 for the purpose of filling the templates to generate prompts.

heading: 3.2.3 Context-based Tuple Selection
text: The anchor words used in both frequency- and diversity-based tuple selection methods use PMI to measure the association between a pivot and an anchor. This approach has two important drawbacks.
First, the number of sentences in a snapshot of a corpus taken at a specific time point could be small. Therefore, the co-occurrences (measured at sentence level) between a pivot and a candidate anchor could be small, leading to data sparseness issues. PMI is known to overestimate the association between rare words.3 Second, PMI considers only the two words (i.e pivot and the anchor) and not the other words in their contexts.
We address the above-mentioned limitations of PMI by using contextualised word embeddings, M(x, d) obtained from an MLM M representing a word x in a context d. We use sentences as the contexts of words and represent a word x by an embedding x, computed as the average of M(x, d) over D(x), given by (4).
x = 1 |D(x)| ∑
d∈D(x) M(x, d) (4)
Using (4), for each word x we compute two embeddings x1 and x2 respectively in C1 and C2. If the word x is split into multiple subtokens, we use the average of those subtoken embeddings as x. If x does not exist in a particular snapshot, it will be represented by a zero vector in that snapshot.
Specifically, given w ∈ C1 ∩ C2, u ∈ C1 and v ∈ C2, we score a tuple (w, u, v) as in (5).
g(w1,u1) + g(w2,v2)− g(w2,u2)− g(w1,v1) (5)
Here, g(x,y) is the cosine similarity between the embeddings x and y. Note that (5) assigns higher scores to tuples (w, u, v) where w and u are highly
3For example, if p(w, x) ≈ p(w). Then, (2) reduces to − log p(x), which becomes larger for rare x (i.e. when p(x) → 0).
related in C1 and w and v in C2, whereas it discourages the associations of w and u in C2 and w and v in C1. This enforces the diversity requirement discussed in § 3.2.2 and makes the tuple scoring method asymmetric between C1 and C2, which is desirable. Finally, we rank tuples by the scores computed using (5) and select the set, Scont, of top-k ranked tuples to fill the templates to generate prompts.
This embedding-based tuple selection method overcomes the limitations of PMI discussed at the beginning of this section as follows. We can use contextualised embeddings from an MLM that is trained on a much larger corpus than two snapshots to obtain M(x, d), thereby computing non-zero cosine similarities even when a pivot and an anchor never co-occurs in any sentence in a snapshot. Moreover, contextualised embeddings are known to encode semantic information that is useful to determine the word senses (Zhou and Bollegala, 2021) and semantic variations (Giulianelli et al., 2020), thus enabling us to better estimate the suitability of tuples.

heading: 3.3 Prompts from Automatic Templates
text: Given two snapshots C1, C2 of a timestamped corpus and a set S of tuples (w, u, v) extracted from any one of the three methods described in §3.2, we propose a method to automatically learn a diverse set of templates. For this purpose, we can use any of the sets of tuples Sfreq, Sdiv or Scont extracted as S . We model template generation as an instance of text-to-text transformation. For example, given the context “mask is associated with hide in 2010 and associated with vaccine in 2020”, containing the tuple (mask, hide,vaccine), we would like to generate the sequences shown in red italics as a template. Given a tuple (w, u, v), we extract two sentences S1 ∈ C1 and S2 ∈ C2 containing the two anchors respectively u and v, and use a pretrained T5 (Raffel et al., 2020) model to generate the slots Z1, Z2, Z3, Z4 for the conversion rule Tg(u, v, T1, T2) shown in (6).
S1, S2 → S1 ⟨Z1⟩ u ⟨Z2⟩ T1 ⟨Z3⟩ v ⟨Z4⟩ T2 S2 (6)
The length of each slot to be generated is not required to be predefined, and we generate one token at a time until we encounter the next non-slot token (i.e. u, T1, v, T2).
The templates we generate must cover all tuples in S. Therefore, when decoding we prefer
templates that have high log-likelihood values according to (7).
|T |∑
i=1
∑
(w,u,v)∈S logPT5(ti|t1, . . . , ti−1; Tg(u, v, T1, T2)) (7)
where t1, . . . , t|T | are the template tokens belonging to the slots Z1, Z2, Z3 and Z4.4
Following Gao et al. (2021), we use beam search with a wide beam width (e.g. 100) to obtain a large set of diverse templates. We then select the templates with the highest log-likelihood scores according to (7) as auto templates. By substituting the tuples in S in auto templates, we generate a set of auto prompts.

heading: 3.4 Examples of Prompts
text: Table 1 shows the manually-written templates and the automatically learnt templates We see that prompts describing diverse linguistic patterns expressing how a word’s usage could have changed from one time stamp to another are learnt by the proposed method. Moreover, from Table 1, we see that automatically learnt templates tend to be shorter than the manually-written templates. Recall that the automatic template generation method prefers sequences with high likelihoods. On the other hand, longer sequences tend to be rare and have low likelihoods. Moreover, we require automatic templates to cover many tuples that are selected by a particular tuple selection method, thus producing more generalisable prompts. We believe the preference to generate shorter templates by the proposed method is due to those reasons.

heading: 3.5 Time-adaptation by Fine-Tuning
text: Given a set of prompts obtained by using the tuples in Sfreq,Sdiv, or Scont to fill the slots in either manually-written or automatically generated templates, we fine-tune a pretrained MLM M on those prompts such that M captures the semantic variation of a word w from T1 to T2. For this purpose, we add a language modelling head on top of M , randomly mask out one token at a time from each prompt, and require that M correctly predicts those masked out tokens from the remainder of the tokens in the context. We also experimented with a variant where we masked out only the anchor words from a prompt, but did not observe a notable difference in performance over random masking of all tokens.
4Each slot can contain zero or more template tokens.

heading: 4 Experiments and Results
text: Datasets: We use the following four datasets that were collected and used by Hofmann et al. (2021) for evaluating DCWEs: Yelp, Reddit, ArXiv, and Ciao (Tang et al., 2012). Details of these datasets and all pre-processing steps are detailed in Appendix C. We remove duplicates as well as texts with less than 10 words in each dataset. We then randomly split each snapshot of a dataset into training, development, and test sets, containing respectively 70%, 10% and 20% of the original dataset.
Evaluation Metric: If an MLM is correctly adapted to a timestamp T2, it should be able to assign higher probabilities to the masked out tokens in unseen texts in C2, sampled at T2. We follow prior work on DCWE (Hofmann et al., 2021) and use the masked language modelling perplexity as our evaluation metric on test texts in C2. If an MLM is well-adapted to T2, it will have a lower perplexity for test texts in C2.
Baselines: To put our results into perspective, we consider the following baselines: Original BERT: We use pretrained BERT-base-uncased5 as the MLM without any fine-tuning to be consistent with (Hofmann et al., 2021). Further evaluations on RoBERTa (Liu et al., 2019) are given in Appendix B. BERT(T1): We fine-tune the Original BERT model on the training data sampled at T1. BERT(T2): We fine-tune the Original BERT model on the training data sampled at T2. Note that this is the same training data that was used for selecting tuples in §3.2 FT: The BERT models fine-tuned by the proposed method. We use the notation FT(model, template) to denote the model obtained by fine-tuning a given MLM using a template, which is either manuallywritten (manual) or automatically-generated (auto) as described in §3.3.
5https://huggingface.co/bert-base-uncased
Hyperparameters: We use the held-out development data to tune all hyperparameters. We follow the recommendations of Mosbach et al. (2021) for fine-tuning BERT on small datasets and use weight decay (0.01) with bias correction in Adam optimiser (Kingma and Ba, 2015). We use a batch size of 4, learning rate of 3×10−8, the number of tuples used for prompt-based fine-tuning (k) is selected from ∈ {500, 1000, 2000, 5000, 10000}, and the number of epochs is set to 20. (further details on hyperparameters are given in Appendix D).
We used a single NVIDIA RTX A6000 and 64 GB RAM in our experiments. It takes approximately 6 hours to fine-tune, validate and test all methods reported in the paper for the four datasets. The run time varies depending on the number of tuples used in the proposed method. Tuple selection takes, on average, 0.5 hours.

heading: 4.1 Results
text: In Table 2, we compare the effect of fine-tuning BERT MLMs using the prompts generated by filling the selected tuples in either the manuallywritten (manual) or automatically learnt (auto) templates. We use the optimal tuples selection method
and the number of tuples for prompt-based finetuning, decided using the validation data in each datasets.
From Table 2 we see that the Original BERT model has the highest perplexity scores in all four datasets. This shows that the Original BERT model is not accurately capturing the meanings of words as used in C2. Although fine-tuning Original BERT using manual or auto prompts improves the perplexity scores, they still remain very high. BERT(T1), obtained by fine-tuning the Original BERT model on C1, immediately reduces the perplexity scores on both datasets. Recall that C1 is sampled from the same domain but at T1, which is different from T2. This indicates the importance of using in-domain data for fine-tuning MLMs eventhough it might be from a different timestamp.
On the other hand, BERT(T2), obtained by finetuning Original BERT on the training data from T2, further reduces perplexity over BERT(T2). Importantly, fine-tuning using both manual and auto prompts further reduces perplexity over BERT(T2). Overall, the best performances on Yelp and ArXiv are reported by fine-tuning BERT(T2) using auto prompts (i.e. FT(BERT(T2), Auto)), whereas the same on Reddit and Ciao are reported by manual prompts (i.e. FT(BERT(T2), Manual)). Applying auto or manual prompts for fine-tuning BERT(T1) improves perplexity in Yelp, ArXiv, and Ciao, but not on Reddit. This shows the importance of first fine-tuning BERT on C2 (in-domain and contemporary) before using prompts to further fine-tune models, because prompts are designed or learnt for the purpose for adapting to T2 and not to T1, reflecting the direction of the time adaptation (T1 → T2).
Although there is prior work on noncontextualised dynamic embeddings, we cannot perform language modelling with those as the
probability of predicting a word will be a constant independent of its context. Moreover, none of those work evaluate on the benchmarks proposed by Hofmann et al. (2021), which we also use. Therefore, we consider the current SoTA for DCWEs proposed by Hofmann et al. (2021) and the SoTA for time-adaptation, TempoBERT (Rosin et al., 2022), as our main comparison points in Table 3.
The DCWE method proposed by Hofmann et al. (2021) uses BERT fine-tuned on training data from T2 as the baseline MLM (i.e. CWE). Moreover, their method is available in two flavours: a version that uses both social and temporal information (denoted as DCWE [social + temp.]) and a version where social information is ablated (denoted as DCWE [temp.]). Given that we do not use social information in our proposed method, the direct comparison point for us would be DCWE [temp.]. We see that our proposed method with both manual and auto prompts consistently outperforms both flavours of the SoTA DCWEs proposed by Hofmann et al. (2021) in all datasets.
TempoBert inserts a special token indicating the time period at the beginning of each sentence in a training corpus, and fine-tunes BERT on the corpora available for each time period. We trained TempoBert on our datasets for the same number of epochs as and with an initial learning rate of 3e-6 and measured perplexity on the same test splits. As seen from Table 3, the proposed method using both manual and automatic templates outperforms TempoBert in all datasets.
The number of tuples selected (i.e. k) to generate prompts with manually-written or automatically generated templates determines the number of prompts used to fine-tune a given MLM. To study the effect of k, we use a particular tuple selection method and select the top-ranked k tuples according to that method with either a manually-written (Manual) or automatically learnt (Auto) template to generate prompts. This results in six different types of prompts. Next, we fine-tune a given MLM using the generated prompts and repeat this process for increasing k values. Figure 1 shows the results of fine-tuning BERT(T2) to T2. (Results for BERT(T1) are shown in Appendix A)
Overall, from Figure 1 we see that when the number of tuples used for fine-tuning increases, almost all methods reach some minimum perplexity score. However, we see that for each method, its
minimum perplexity scores on different datasets is obtained by using different k tuples. Recall that each tuple selection method ranks tuples by some goodness score. Therefore, when we increase k we are using less reliable noisy tuples to generate prompts, thus leading to reduced performance. Interestingly, we see that the best performances can be obtained with relatively a smaller number of tuples (< 5000) in all datasets.
A closer look reveals that on Yelp, all Freq+Auto and Context+Auto obtain similar best performances. However, Freq+Auto reaches its optimal point with 500 tuples, whereas Context+Auto requires 5000 tuples. Among the three tuple selection methods Context-based tuple selection is the best. Frequency-based tuple selection method works well with auto templates but not so much with manual ones. This shows that auto templates can be used to extract good performance from a simple tuple selection method such as Frequency-based tuple selection.
On Reddit, the overall best performance is obtained by Context+Manual with 5000 tuples, and its performance drops with higher k values, due to
the increasing noise in tuples as explained above. Likewise in Yelp, context-based tuple selection emerges as the overall best method in Reddit as well with 5000 tuples. However, context-based tuple selection has to be used with manual templates to obtain good performance on Reddit, whereas in Yelp using it with the auto templates was the best.
On ArXiv, both Freq+Manual and Diversity+Auto reach similar best performances. While Freq+Manual requires 500 tuples, it takes Diversity+Auto 10000 tuples to reach its best performance. Unlike Yelp and Reddit, the best tuple selection in ArXiv is Diversity-based tuple selection. The Frequency-based tuple selection also similar performance, but requires more tuples. For Context-based tuple selection, although it improves the perplexity scores over the baseline MLM, the improvements are smaller than other methods.
On Ciao, Context+Manual and Diversity + Auto obtain similar best performances, both with 1000 tuples. Similarly as Yelp and Reddit, the overall best tuple selection is Context-based tuple selection, which obtains the best perplexity scores. Diversity-based tuple selection also has good per-
formance, although it only occurs when it is used with auto templates.
Table 4 shows some examples of the top scoring pivots and their anchors retrieved by the contextbased tuple selection method from Yelp and Ciao. From Yelp, in year 2010 (T1), we see that dinein restaurants such as burgerville6 and joes7 are associated with place, whereas in 2020 takeaway and dominos8 are associated with place probably due to the COVID-19 imposed lockdowns restricting eating out. Moreover, we observe a shift in office-related job titles between these time periods where service is closely associated with doorman (T1: 108, T2: 48) and clerks (T1: 115, T2: 105), which are rarely used in 2020 and are replaced with more gender-neutral titles such as staff (T1: 28618, T2: 60421), personnel (T1: 85, T2: 319) and administration (T1: 37, T2: 109). From Ciao, in year 2001 (T1), we see that phone brands like nokia9 are closely connected with phone, while in year 2011 (T2), as companies such as apple10 and blackberry11 took a large part of the mobile phone market, iphone, ipod, and blackberry become more related with phone.

heading: 5 Conclusion
text: We propose an unsupervised method to learn DCWEs by time-adapting a pretrained MLM using prompts from manual and automatic templates. Experimental results on multiple datasets demonstrate that our proposed method can obtain better perplexity scores on unseen target texts compared to prior work. In the future, we plan to extend the proposed method to adapt multilingual word embeddings.
6www.burgerville.com 7www.joespizzanyc.com 8www.dominos.com 9www.nokia.com
10www.apple.com 11www.blackberry.com

heading: 6 Limitations
text: This paper takes into account the temporal semantic variations of words and proposes a method to learn dynamic contextualised word embeddings by timeadapting an MLM using prompt-based fine-tuning methods. In this section, we highlight some of the important limitations of this work. We hope this will be useful when extending our work in the future by addressing these limitations.
The learned dynamic contextualised word embeddings are limited to the English language, which is a morphologically limited language. Therefore, the findings reported in this work might not generalise to other languages. However, there are already numerous multilingual MLMs such as mBERT (Devlin et al., 2019), XLM (CONNEAU and Lample, 2019) and XLM-R (Conneau et al., 2020), to name a few. Extending our work to multilingual dynamic contextualised word embeddings will be a natural line of future work.
Dynamic contextualised word embeddings represent words as a function of extralinguistic context (Hofmann et al., 2021), which consider both time and social aspects of words. However, in this paper we focused solely on the temporal aspect and ignored the social aspect. Extending our work to take into account the social semantic variations of a word is an important future direction.
Due to the costs involved when fine-tuning largescale MLMs, we keep the number of manuallywritten and automatically learnt templates to a manageable small number as shown in Table 1 in §3.4. However, it remains to be evaluated the impact of increasing the number of templates on the performance of the proposed method.

heading: 7 Ethical Considerations
text: In this paper, we considered the problem of capturing temporal semantic variation of words by learning dynamic contextualised word embeddings. For this purpose, we proposed a method to adapt a masked language model from to a given time stamp. We did not collect, annotate or release new datasets during this process. However, we used pretrained MLMs and four datasets from the internet (Yelp, Reddit, Arxiv, and Ciao). It is known that pretrained MLMs contain unfair social biases (May et al., 2019; Nadeem et al., 2021; Kaneko and Bollegala, 2021; Kaneko et al., 2022). Such biases can be amplified by fine-tuning methods, especially when the fine-tuning prompts are extracted from
social data such as customer reviews in Yelp or discussions in Reddit. Therefore, we consider it is important to further evaluate (Nangia et al., 2020; Nadeem et al., 2021; Kaneko and Bollegala, 2022) the adapted MLMs for social biases, and if they do exist, then apply appropriate debiasing methods (Kaneko and Bollegala, 2021; Lauscher et al., 2021) before the MLMs are deployed in downstream NLP applications that are used by billions of users world-wide.

heading: Acknowledgements
text: Danushka Bollegala holds concurrent appointments as a Professor at University of Liverpool and as an Amazon Scholar. This paper describes work performed at the University of Liverpool and is not associated with Amazon.

heading: A Fine-tuning results on C1
text: Figure 2 shows the effect of the number of tuples (i.e. k) selected using different tuple selection methods, and the perplexity scores for the BERT(T1) models, fine-tuned using the prompts generated by filling the slots with those tuples in either manually-written (Manual) or automaticallygenerated (Auto) templates. Note that we have three methods to select tuples (i.e. Frequencybased tuple selection, Diversity-based tuple selection, and Context-based tuple selection). Combined with the two methods for obtaining tuples, we have six comparisons in Figure 2 on Yelp, Reddit, ArXiv, and Ciao datasets. Because a only a single template is used in each setting, the number of tuples (k) is equal to the number of prompts used to fine-tune an MLM in this experiment.
On Yelp, we see that Freq+Auto and Diversity+Auto both obtain the lowest (best) perplexity scores. In particular, we see that Freq+Auto reaches this optimal performance point with as less as 500 prompts, whereas Diversity+Auto requires 1000 prompts. However, when we increase the number of prompts beyond the optimal performance points for each method, we see that the perplexity increases due to the added noise when
using low-scoring tuples for generating prompts. Although for both of those methods the perplexity scores drop again when a large number of prompts are being used (i.e. more than 5000 prompts) only Diversity+Auto recovers to the best performance level it obtained with 1000 prompts. Therefore, we note that there is a trade-off here between the quality vs. quantity of using noisy prompts for fine-tuning. However, from a computational point of view it is desirable to use a small number of prompts if that suffice to obtain good performance. Therefore, we recommend using Freq+Auto in this case because it obtained good performance with only 500 prompts.
On Reddit we see that the perplexity increases with the number of prompts in almost all methods from the start. However, they reach a peak and then start decreasing again. However, among all methods we see that only Diversity+Auto recovers to its initial levels. In fact, with 10000 prompts it is able to report perplexity scores lower than that of its initial values, thus reporting the best performance on Reddit by any fine-tuning method. However, recall that auto templates were specifically learnt to obtain good performance when adapting from T1 to T2, and the perplexity scores obtained by finetuning BERT(T2) are much better than those obtained by fine-tuning BERT(T1) (which are shown in Figure 2) as explained in the main body of the paper.
On ArXiv we see that Freq+Auto obtain the best perplexity score. In almost all methods, the perplexity scores drop first and then increase. However, the increases are followed by drops and then increases. The trend of perplexity scores regarding the tuple numbers seems a wave. Unlike other mehtods, Context+Auto almost continues to improve its performances as the number of tuples increases. Freq+Auto is the overall best method as it reaches the best perplexity score with 2000 tuples. In addition, we see that the potential performances of Context+Auto would be high since its performances increase with the number of tuples.
On Ciao we see that Diversity+Auto obtains the best perplexity score and it is much better than other methods. Unlike other datasets, all methods reach their best perplexity scores with small numbers of tuples (< 1000). The trend of perplexity score changing regarding the numbers of tuples is almost the same in all methods: drop, increase, and drop.

heading: B Experiment on RoBERTa
text: To explore the proposed method’s potential on other MLMs than BERT, we conduct a small-scale experiment on RoBERTa. The baselines and evaluation metric setting are similar to the experiment in the main body except that the MLM is changed to RoBERTa-base12 and we only use the Reddit datasets.
In Table 5 we compare the effect of fine-tuning RoBERTa MLMs using the prompts from both automatic and manual templates. From Table 5 we see that the Original RoBERTa has the highest perplexity score in Reddit dataset, and fine-tuning Original RoBERTa with manual or auto prompts improves the perplexity. While applying manual prompts does not improve the perplexity score over RoBERTa(T1), fine-tuning with auto prompts makes some improvements. Likewise the results of the main experiment on BERT, fine-tuning using both manual and auto prompts further reduces perplexity over RoBERTa(T2).
Figure 3 shows the results of fine-tuning RoBERTa(T1) and RoBERTa(T2) to T2.
For RoBERTa(T1), Context+Auto has the best perplexity score with 1000 tuples. However, the context-based tuple selection method only improve the perplexity score over the baseline when it is used with auto templates. Moreover, Context+Auto is the only method that improves the perplexity against the baseline MLM.
For RoBERTa(T2), similar as RoBERTa(T1), Context+Auto obtain the lowest (best) perplexity score with 1000 tuples. Freq+Auto also reaches a similar perplexity score with 2000 tuples. As tuple numbers increase, almost all methods first reach optimal points, and then their perplexity scores increase as the tuple numbers increase. Context+Auto is the overall best method because its best performance and the smallest tuple number.

heading: C Datasets
text: Yelp: Yelp is a platform which provides crowdsourced reviews on businesses. We select publicly available reviews13 covering the years 2010 (=T1) and 2020 (=T2).
Reddit: Reddit is a social media platform covering a wide range of topics arranged into communities called subreddits. Following Hofmann et al.
12https://huggingface.co/roberta-base 13https://www.yelp.com/dataset
(2021), from the publicly released Reddit posts,14 we take all comments from September 2019 (=T1) and April 2020 (=T2), which reflect the effects of the COVID-19 pandemic. We remove subreddits with fewer than 60 comments and randomly sample 60 comments per subreddit.
ArXiv: ArXiv is an open-access repository of scientific articles. We obtain abstracts of papers published at years 2001 (=T1) and 2020 (=T2) on ArXiv from a publicly available dataset15. Following Hofmann et al. (2021), we drop those data under ArXiv’s subjects (e.g., CS.CL) that has less than 100 publications between 2001 and 2020.
Ciao: Ciao is a product review site. We select reviews from years 2000 (=T1) and 2011 (=T2) from a publicly released dataset (Tang et al., 2012)16.

heading: D Hyperparameters
text: Table 6 shows the hyperparameter values for fine-tuning BERT(T2) and RoBERTa(T2) using prompts on T2. We used T5-base17 to generate automatic prompts. The batch size of generating process is 32, and the width of the beam search is set to 100.
To examine the influence of the random seeds, we firstly perform FT(BERT(T2),Auto) on ArXiv with different numbers of tuples and three random seeds. Then we calculated the mean values and the standard deviations of perplexities with different tuple numbers regarding the random seeds. As we average the mean values and the standard deviation, we see that the average standard deviation (i.e. 0.0066) is much smaller than the average mean (i.e. 3.5079), which is nearly 1/1000. Thus, we only use 123 as the random seed for the formal experiments.
14https://files.pushshift.io/reddit/comments/ 15https://www.kaggle.com/datasets/Cornell-University/
arxiv 16https://www.cse.msu.edu/~tangjili/trust.html 17https://huggingface.co/t5-base
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
section 6
3 A2. Did you discuss any potential risks of your work? section 7
3 A3. Do the abstract and introduction summarize the paper’s main claims? abstract and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? sections 3 and 4
3 B1. Did you cite the creators of artifacts you used? sections 3 and 4
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? sections 3 and 4
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. sections 3, 4 and Appendix
C 3 Did you run computational experiments? sections 3, 4, and Appendix
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? section 4 and Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? section 3, section 4 and Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 4 and Appendix
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 4 and Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We hope this will be useful when extending our work in the future by addressing these limitations. The learned dynamic contextualised word embeddings are limited to the English language, which is a morphologically limited language. Therefore, the findings reported in this work might not generalise to other languages. However, there are already numerous multilingual MLMs such as mBERT (Devlin et al. , 2019), XLM (CONNEAU and Lample, 2019) and XLM-R (Conneau et al. , 2020), to name a few. Extending our work to multilingual dynamic contextualised word embeddings will be a natural line of future work. Dynamic contextualised word embeddings represent words as a function of extralinguistic context (Hofmann et al. , 2021), which consider both time and social aspects of words. However, in this paper we focused solely on the temporal aspect and ignored the social aspect. Extending our work to take into account the social semantic variations of a word is an important future direction. Due to the costs involved when fine-tuning largescale MLMs, we keep the number of manuallywritten and automatically learnt templates to a manageable small number as shown in Table 1 in §3. 4. However, it remains to be evaluated the impact of increasing the number of templates on the performance of the proposed method