Word Sense Extension
Year: 2023

Abstract:
Humans often make creative use of words to express novel senses. A long-standing effort in natural language processing has been focusing on word sense disambiguation (WSD), but little has been explored about how the sense inventory of a word may be extended toward novel meanings. We present a paradigm of word sense extension (WSE) that enables words to spawn new senses toward novel context. We develop a framework that simulates novel word sense extension by first partitioning a polysemous word type into two pseudo-tokens that mark its different senses, and then inferring whether the meaning of a pseudo-token can be extended to convey the sense denoted by the token partitioned from the same word type. Our framework combines cognitive models of chaining with a learning scheme that transforms a language model embedding space to support various types of word sense extension. We evaluate our framework against several competitive baselines and show that it is superior in predicting plausible novel senses for over 7,500 English words. Furthermore, we show that our WSE framework improves performance over a range of transformer-based WSD models in predicting rare word senses with few or zero mentions in the training data.

heading: 1 Introduction
text: Humans make creative reuse of words to express novel senses. For example, the English verb arrive extended from its original sense “to come to locations (e.g., to arrive at the gate)” toward new senses such as “to come to an event (e.g., to arrive at a concert)” and “to achieve a goal or cognitive state (e.g., to arrive at a conclusion)” (see Figure 1). The extension of word meaning toward new context may draw on different cognitive processes such as metonymy and metaphor, and here we develop a general framework that infers how words extend to plausible new senses.
A long-standing effort in natural language processing (NLP) is to build systems that support automatic word sense disambiguation (WSD) from linguistic context. This line of work typically takes a discriminative approach toward word meaning and has developed models relying on both traditional machine learning (Gale et al., 1992; Kilgarriff and Rosenzweig, 2000; Zhong and Ng, 2010; Iacobacci et al., 2016) and modern neural language models (Huang et al., 2019; Wiedemann et al., 2019; Loureiro and Jorge, 2019; Bevilacqua and Navigli, 2020). However, existing WSD models often struggle with recognizing rare word senses with few or no mentions in training (Blevins et al., 2021). Here we show that by modelling the generative extensional processes of word meaning, WSD models can become better at recognizing infrequent word senses in natural context and without relying on external lexical resources.
Work in computational and cognitive linguistics shows that word senses do not extend arbitrarily (Nunberg, 1979; Lehrer, 1990; Rumshisky and Batiukova, 2008). Lexical semanticists have suggested that a number of cognitive devices may be
3281
applied to generate creative word usages, such as logical metonymy (Copestake and Briscoe, 1995; Pustejovsky, 1998) and metaphor (Lakoff and Johnson, 2008; Pustejovsky and Rumshisky, 2010). Cognitive linguists have also suggested that systematic mappings between conceptual domains underlie the metaphorization of word meaning (Brugman and Lakoff, 1988; Lakoff and Johnson, 2008; Gentner, 1983). However, the reliance on hand-crafted rules of semantic productivity makes it difficult to implement systems that support flexible and scalable extension to new word senses.
We present a paradigm that considers the problem of word sense extension (WSE) illustrated in Figure 1. Given a novel context and an intended meaning, a speaker wishes to choose an existing word in the lexicon to express that meaning which the word has never been used to convey. To operationalize a speaker model without prior knowledge about pairings between the novel meaning and existing word forms, we replace each candidate word type with a pair of “pseudo-tokens” that signify one of its existing senses (called the target sense) the other senses (called the source senses) respectively, a method related to previous work in polysemy induction (Pilehvar and Navigli, 2014; Dubossarsky et al., 2018). We then infer whether a partitioned pseudo-token denoting the source sense may be extended to express the target sense denoted by its sibling token partitioned from the same word type. We propose a family of cognitively-inspired probabilistic models for this inference problem. We show that our WSE models can reliably predict plausible novel senses on a large usage-based dataset with approximately 34,000 senses for over 7,500 English word types.1

heading: 2 Related work

heading: 2.1 Models of word meaning extension
text: Researchers in lexical semantics and cognitive linguistics have both proposed theories to account for the malleable nature of lexical meaning. The Generative Lexicon theory by Pustejovsky (1998) argues that a fixed set of generative devices, such as type-coercion and co-composition, can operate on the lexical structure a word to produce various related meaning interpretations. Copestake and Briscoe (1995) also illustrates how formal lexical
1We release the code and data for our work here: https://github.com/jadeleiyu/word_sense_ extension.
rules such as grinding and portioning can be applied to produce novel word usages such as logical metonymy. In cognitive linguistics, Lakoff (1987) argues that word meanings grow relying on processes of chaining, whereby novel meanings link to existing ones that are close in semantic space. Similar processes are also relevant to the construction of metaphorical usages in natural language drawing on image schemas (Brugman and Lakoff, 1988; Dewell, 1994; Gibbs Jr and Colston, 2008) and analogy or structural alignment between domains (Gentner, 1983; Falkenhainer et al., 1989).
Our work builds on the cognitive theory and recent computational work on chaining (Lakoff, 1987; Malt et al., 1999; Ramiro et al., 2018; Habibi et al., 2020; Grewal and Xu, 2020; Yu and Xu, 2021), and we show that a chaining-based framework learns systematic patterns of word sense extension discussed in the tradition of generative lexical semantics. Related work has taken a similar approach for modelling sense extension in slang usages (Sun et al., 2021), but here we consider the more general problem of word sense extension.

heading: 2.2 Models of word sense disambiguation
text: A large community in NLP has been working on the problem of word sense disambiguation (WSD). Early WSD systems adopt a knowledge-based approach by comparing the neighborhood context of a target word with its gloss or definition in lexicographic databases such as WordNet (Miller, 1995; Gale et al., 1992; Kilgarriff and Rosenzweig, 2000). Later work develops feature-based classification models to predict sense labels for a word based on its linguistic features (Zhong and Ng, 2010; Iacobacci et al., 2016; Raganato et al., 2017). Recent progress in deep learning also motivates the development of WSD systems based on deep contextualized language models (CLM) or its combination with external lexical knowledge base (Huang et al., 2019; Hadiwinoto et al., 2019; Bevilacqua and Navigli, 2020). Despite these impressive advances, many CLM-based WSD systems still suffer from the data sparsity that stems from the Zipfian distribution of word senses (Kilgarriff, 2004) – i.e. the most frequent sense of a polysemous word often accounts for a dominant portion of its mentions, while other senses have much less or even zero frequency in training data. Recent work has proposed to mitigate this sense sparsity problem by resorting to gloss information (Luo et al., 2018; Kumar et al.,
2019; Huang et al., 2019; Blevins and Zettlemoyer, 2020) or non-parametric few-shot learning (Holla et al., 2020; Chen et al., 2021). We shall demonstrate that learning word sense extensions offers an alternative approach to improve WSD system performance on infrequent word senses by leveraging the systematic semantic relational patterns between conventional and novel word senses.

heading: 2.3 Contextualized semantic representations
text: Existing work has proposed to apply contextualized language models to lexical semantic tasks that involve polysemy. Diachronic studies show that contextualized representations of word usage and sense definitions can be used to detect lexical semantic shifts (Giulianelli et al., 2020; Hu et al., 2019). Probing studies also suggest that pretrained contextualized language models encode rich lexical semantic information that may help decide the levels of word polysemy (Garí Soler and Apidianaki, 2021) and infer semantic relations between word senses (Vulić et al., 2020). The WSE paradigm we propose is related to lexical substitution, where a model is used to replace a target word in a sentence with a substitute word without changing the sentence meaning (McCarthy and Navigli, 2007; Melamud et al., 2016; Zhou et al., 2019). However, our framework goes beyond this research by asking whether a word can extend its sense inventory to express novel intended meanings in natural context.

heading: 3 Computational framework
text: Our framework of word sense extension involves three interrelated components: 1) A procedure for partitioning polysemous words in the lexicon into new pseudo-tokens that signify their different senses; 2) a probabilistic, chaining-based formulation of word sense extension for lexical choice making under novel linguistic context; and 3) a learning algorithm for a transformed semantic space to learn flexible extensions of word senses.

heading: 3.1 Sense-based word type partitioning
text: Let W = {w1, ..., w|V |} be our vocabulary of polysemous (English) word types, where each w has a set of n senses Sw = {s1, ..., sn}. Assume that for each w there is also a collection of its sense-annotated sample usage contexts Cw = {(c1, y1), ..., (cm, ym)}, where each contextual sequence c ∈ Cw is labeled with a sense y ∈ Sw instantiating the meaning of w in that usage con-
text. We want to simulate the scenario where a speaker, without knowing a priori that a word w has a sense s∗ ∈ Sw, is able to extend the meaning of w to expressing s under novel context.
To operationalize this idea of word sense extension, we first partition each w into two hypothetical tokens: a source token t0 that denotes the set of existing source senses S0 = S \ {s} of w, and a target token t∗ that denotes the novel target sense s∗ to which w extends beyond its existing senses. We then replace w with t0 in all usage contexts that reflect one of its source senses (i.e., (ci, yi) where yi ∈ S0), and replace w with t∗ in all usage contexts where w signifies the target sense (i.e. (ci, yi) where yi = s∗).
To guard against information smuggling in predicting novel word sense extension, we learn a contextualized language model from scratch using the set of replaced usage instances. Specifically, the language model is trained on the task of masked language modeling (MLM), where it takes batches of sampled usage instances with some randomly chosen tokens masked out, and updates its parameter weights to maximize the probability of infilling the correct missing tokens. Through this procedure, we obtain a language model that can compute meaningful contextualized representations for the usages of w that instantiate the target sense s∗ without knowledge that s can be expressed by w.

heading: 3.2 Probabilistic formulation of WSE
text: Let C0, C∗ be the two sets of usage instances with w replaced by t∗ and t0 respectively. We consider an inference scenario where the language model learned using the procedure from the previous section is presented with a novel usage c∗ ∈ C∗ of target token t∗, and is queried to choose among a set of candidate source tokens to convey the same (and new) intended meaning as that of t∗.
Concretely, suppose the target token t∗ partitioned from the verb w = arrive denotes its metaphorical sense s∗ = “to achieve a goal”, and the source partitioned token t0 of arrive is comprised of its existing source senses (that exclude the metaphorical sense in question). We then use the model to infer whether t0 can be used to convey the new meaning t∗ in novel metaphorical usages such as c = “They finally t∗ at a conclusion after a long debate” (note here the original verb arrive is replaced by the target token t∗ through word type partitioning). We assess the success of our model
by analyzing how it ranks the ground-truth source token (i.e., t0 of arrive) among the space of alternative candidate source tokens partitioned from other polysemous words in the lexicon. For example, one source token might signify the literal senses of the verb leave which differs from the ground-truth verb arrive. Formally, we cast WSE as finding a source token t that maximizes the following probability:
argmaxtP (t|m(t∗|c∗)) (1)
Here m(t∗|c∗) is the representation of target token t∗ under context c∗ to which t is extended.

heading: 3.3 Chaining-based models of WSE
text: We present a family of probabilistic models for Eq.1 that draw inspirations from the cognitive theory of chaining (Lakoff, 1987; Habibi et al., 2020). Our chaining-based WSE models assume that a source token t0 can be extended to express a novel meaning if the new intended meaning is overall similar to t0’s existing senses. We operationalize m(t∗|c∗) as the contextualized word embedding of target token t∗ under context c∗ computed by the speaker language model, denoted as h(t∗|c∗). We represent the existing senses of source token t as the collection of all of its contextualized embeddings H(t0) = {h(t0|c)|c ∈ C0}. The chainingbased WSE models take the general form:
P (t0|m(t∗|c∗)) ∝ sim(H(t0),h(t∗|c∗)) (2)
We consider two common types of chaining model that specify the similarity function sim().
WSE-Prototype model. The prototype model takes inspiration from prototypical network for fewshot learning (Snell et al., 2017; Holla et al., 2020) and follows the prototype theory of categorization (Rosch, 1975) in cognitive psychology. It assumes that the existing senses of a source token t0 can be summarized by a global average (i.e., prototype) of its contextualized embeddings in H(t0), so that the probability of t0 being a good candidate to convey the intended meaning of the target token is proportional to the semantic similarity between the contextualized embedding h(t∗|c∗) of the target token and the prototype of its sibling source token:
P (t0|m(t∗|c∗)) ∝ exp[−d(h(t∗|c∗), z(t0))] (3)
z(t0) = 1 |C0| ∑
c∈C0 h(t0|c) (4)
Here z(t0) is the global mean contextualized embedding of t0, and we compute dot product as the similarity function d(·, ·) between two vectors.2
WSE-Exemplar model. The exemplar model resembles the memory-augmented matching network in deep few-shot learning (Vinyals et al., 2016), and formalizes the exemplar theory of categorization (Nosofsky, 1986). This model postulates that the meaning of t0 is represented by the collection of its individual usages c ∈ C0. The probability that t0 can be extended to the meaning m(t∗|c∗) is proportional to the mean similarity score between h(t∗|c∗) and each contextualized embedding of t0:
P (t0|m(t∗|c∗)) ∝ 1|C0| ∑ c∈C0 exp[−d(h(t∗|c∗),h(t0|c))]
(5)

heading: 3.4 Learning sense-extensional semantic space
text: Chaining relies on identifying close semantic relations between existing senses and generalizing the recognized relations to generate new senses. For instance, if a WSE model has observed how the English verb grasp relates its literal sense “to hold an item firmly" to the extended metaphorical sense “to understand an idea", the model should also predict similar but novel non-literal sense extensions for other verbs that involve such metaphorical mappings (e.g., the meaning extension of the verb get from “to get a car" to “to get someone’s idea", which also reflects the conceptual metaphor IDEAS ARE OBJECTS) (Lakoff and Johnson, 2008).
Following work in deep few-shot learning, we propose an episodic learning algorithm to transform the language model embedding space of the WSE model into a semantic space that better captures the regular, systematic patterns in sense extension. At each episode, we sample a mini-batch of N source-target token pairs {(t0i , t∗i )}Ni=1 partitioned from N distinct polysemous word types, and sample a usage context c∗i for each target token t ∗ i . The WSE model then chooses the most appropriate source token to convey the contextualized meaning of each target token. The parameter weights in the language model are optimized to minimize the negative log-likelihood of the ground-truth source
2We experimented with negative squared Euclidean distance suggested in Snell et al. (2017) as an alternative similarity function but found it to yield worse performance on both WSE and downstream WSD tasks compared to dot product.
token t0i for each target token t ∗ i :
J = N∑
i=1
− log sim(H(t 0 i ),h(t∗i |c∗i ))
N∑ j=1
sim(H(t0j ),h(t∗i |c∗i )) (6)
Here sim(·, ·) can be either a prototype-based similarity function in Eq.3, or its exemplar-based counterpart specified in Eq.5.

heading: 4 Data

heading: 4.1 Dataset of polysemous word usages
text: We construct our WSE dataset by collecting naturalistic usage instances of English polysemous words from the Wikitext-103 linguistic corpus (Merity et al., 2016) that is commonly used as a language modeling benchmark. We first extract the sentences and lemmatize the corpus using SpaCy. We then apply a state-of-the-art word disambiguation algorithm by Bevilacqua and Navigli (2020) on each sentence to annotate each of its token with one of its associated WordNet synset IDs as the sense label (Miller, 1995). We construct a polysemous English word vocabulary by taking word lemma types that satisfy the following conditions: 1) the word type has least 2 different senses detected in the corpus; 2) each mention of the word type has one of the four part-of-speech categories as detected by SpaCy: noun, verb, adjective, or adverb; 3) each sense of the word type has at least 10 mentions in the corpus. This process yields a large repertoire of 7,599 polysemous word types with a total number of 1,470,211 usage sentences, and an average number of 4.27 senses per word type.

heading: 4.2 Partioning polysemous word types
text: To construct and evaluate our WSE framework, we partition each polysemous word types into multiple source-target pseudo-token pairs. In particular, for each word type w with n senses, we randomly choose one sense as the target sense s∗, and the remaining n−1 senses as the source senses. A sourcetarget token pair is then created, which replace w in usage sentences based on their sense labels following the procedures described in Section 3.1. We repeat this partitioning process 5 times so that each word type with at least 5 senses will have 5 distinct senses chosen as target, and for words with less than 5 senses, the 5 target senses will be sampled with replacement from its sense inventory. Each partition will therefore create 2× 7, 599 = 15, 198 pseudo-tokens.

heading: 5 Evaluation and results

heading: 5.1 Experimental setup
text: We use a transformer model with the same architecture as BERT-base-uncased (Devlin et al., 2019) as the main language model in our WSE framework. The parameter weights of our language models are randomly initialized to prevent any information smuggling (i.e., the models are trained from scratch). In the masked language modeling training stage on replaced usage sentences, we increase the vocabulary size of each model by replacing all polysemous word types in our WSE dataset vocabulary with their partitioned pseudo-tokens, and add rows to embedding layer and final classification layer of the BERT model accordingly. Five language models are trained independently, one for each set of partitioned tokens as described in section 4.2. During sense-extensional semantic space learning, we randomly choose 70% of the original polysemous word types and take usage sentences containing their partitioned tokens as the training set. Sentences containing partitioned tokens spawned by the remaining 30% word types will be taken as the test set, so that there is no overlap in the vocabulary of partitioned tokens or their parent word types between training and testing.3

heading: 5.2 Baseline models
text: We also compare the performance of our WSE models against a set of baseline models without chaining-based inference mechanisms: 1) a BERTMLM baseline ignores the intended meaning information and predicts P (t0|m(t∗|c∗)) as the infilling probability of t0 under context c∗ with t∗ replaced by a masking placeholder; 2) a BERTSTS baseline computes the contextualized representation h(t0|c∗) of each candidate source token t0 under c∗, and calculates P (t0|m(t∗|c∗)) as proportional to the cosine similarity between h(t0|c∗) and the contextualized embedding h(t∗|c∗) of the target token under the same context (i.e. based on the semantic textual similarity between contextualized meanings of t0 and t∗). Both baselines are built on the same BERT encoder just as the two chaining-based WSE models. We also consider a random baseline that randomly draws a source token from the set of alternative candidate tokens.
3See Appendix A for more implementation details.

heading: 5.3 Evaluation on WSE
text: We first evaluate our models on the task of predicting source partitioned tokens formulated in Eq.1. At each trial, for each target token t∗w partitioned from w, we present the model with the groundtruth source token t0w partitioned from the same word w, and 99 negative candidate source tokens t0w′ spawned from different polysemous word types w′. Both the ground-truth source token and the negative candidates are sampled from the evaluation set for sense-extensional semantic space learning. We assess each model in two settings: an unsupervised version of a model that does not learn from the training set of WSE, and a supervised version that is trained on the training set of sense extensional space learning. The BERT encoders of the supervised versions of two BERT baselines are trained using the same objective function and data as defined in Section 3.4.
We quantify model performance with two metrics: 1) the mean precision is the percentage of cases where a model correctly predicts the groundtruth source token as the most likely candidate, and 2) the mean reciprocal rank (MRR-100) is the av-
eraged multiplicative inverse of the ranks of the ground-truth source tokens in all evaluation examples. Table 1 summarizes the overall results in the five sets of independently partitioned tokens. We make several observations: 1) all BERT-based models perform substantially better than chance even without explicit training on WSE. This can be explained by the fact that many polysemous word types in our dataset have very fine-grained WordNet senses, so that the target senses chosen from its sense inventory are often highly similar or even hardly distinguishable from the some source senses of the same word; 2) all BERT-based models benefit from learning a sense-extensional semantic space, suggesting the presence of regularity shared among examples of sense extension across word types; 3) both chaining-based WSE models consistently outperform other baselines in both the unsupervised and supervised settings. The exemplarbased WSE models generally outperform than their prototype-based counterparts, suggesting that word sense extension depends on the speaker’s sensitivity to the semantic similarity between the intended meaning and the individual (exemplar) usages.
Table 2 shows example predictions on sam-
ple polysemous words made by the supervised exemplar-based WSE model and the supervised BERT-MLM baseline. The WSE model successfully predicts many types of sense extension, such as metaphorical senses for both the verb cover example and the noun cell. In contrast, the BERTMLM baseline shows a greater tendency to predict a literal paraphrase for a partitioned token. Still, both WSE and baseline models struggle with predicting some usages that involve strong non-literal sense extension (e.g., the grasp example).

heading: 5.4 Sense relatedness and model predictability
text: Prior work in psycholinguistics suggests that both adults and children often find it easy to infer a new intended meaning of a word if they can access a highly related conventional sense of that word to constrain their interpretation (Clark and Gerrig, 1983; Klepousniotou et al., 2008; Rodd et al., 2012). We examine whether our WSE models exhibit human-like sensitivity to the conceptual relatedness between existing and novel word senses. For each source-target partitioned token pair (t0, t∗), we quantify their degree of conceptual relatedness as the mean Wu-Palmer semantic distance (Wu and Palmer, 1994) between the WordNet synset of the target sense denoted by t∗ and the synset of each existing source sense of t0. Figure 2 shows the performance of 4 WSE model variants on predicting sense pairs binned with respect to their degree of conceptual similarity. We observe that the WSE models generally make better predictions on source-target token pairs that are semantically more related (e.g., metonymy), and perform less well on examples where the target sense is concep-
tually very different to the existing source senses (e.g., strong metaphor or homonymy).

heading: 5.5 Application of WSE to WSD
text: As a final step, we show that state-of-the-art word sense disambiguation models can benefit from the word sense extension framework. We evaluate WSD models on the standard WSD evaluation framework proposed by (Raganato et al., 2017), where in each trial, the model is given an input sentence and is asked to assign WordNet sense labels for a subset of tokens within the sentence. We consider two BERT-based WSD models: 1) a BERT-linear model that learns a linear classifier for WSD on top of a frozen BERT encoder. This model does not incorporate gloss information, and cannot predict novel senses that do not appear in training; 2) a bi-encoder model (BEM) by (Blevins and Zettlemoyer, 2020) independently encodes input sentences with target words and sense glosses via two encoders, each of which are initialized with BERT-base. The contextualized embedding of the target word then takes dot product with the gloss embedding of each candidate sense, and the model predicts the sense with highest dot product score with the embedded target word. This model has been shown to yield impressive results on WSD examples with rare senses.
To integrate WSE into WSD, we fine-tune the BERT encoder of each WSD model on the WSE training set of Wikitext-103 usage sentences via the objective in Eq. 6, which can be formulated as either a prototype model or an exemplar model. Unlike the case of WSE evaluation, here we use pretrained BERT-base-uncased encoders and keep the original word form of each polysemous word without partitioning it into source-target token pairs. The resulting BERT encoder is then taken to learn one of the two WSD models described above, and evaluated on WSD tasks. For BEM, both encoders are initialized as the BERT-base fine-tuned on WSE. Since the sense labels of usage sentences in the WSE dataset are not fed to BERT during training, none of the models has access to any usage examples of target senses in the WSD test set.
Table 3 reports overall results on the WSD datasets under the standard F1-score. We also include the performance of two simple baselines: 1) WordNet S1 always predicts the first sense, and 2) MFS always predicts the most frequent sense in the training data. We found that chaining-based WSE
models improve the performance of the two BERTbased WSD models on almost every test subset, as well as on all POS categories except for the adverb class. These results show that WSE may serve as useful pretraining for improving WSD models both with and without access to gloss information.
Rare word-sense pairs. We hypothesize that WSE improves WSD because learning word sense extension helps the model to better interpret rare senses that bear systematic semantic relations with more conventional senses. Table 5 shows the performance of WSD models grouped by the frequency of the target word sense in the WSD training set. We define zero-shot test cases as target senses that never appear during WSD training, and few-shot
test cases as those with 1 to 10 mentions, and highfrequency senses as those with more than 10 training mentions. The BERT-linear model resort to a most frequent sense heuristic for zero-shot examples, since it cannot learn a classification layer embedding for previously unattested senses. We observe that all WSD models trained on WSE yield substantially greater improvement for few-shot and zero-shot test cases, while maintaining high performance on the more frequent cases. Table 4 shows test examples where incorrect predictions of BEM are improved with WSE integration. These examples often exhibit regular semantic relations between target and conventional senses of a word (e.g., the relation between physical size and amount that underlies the two attested senses of full).

heading: 6 Conclusion
text: We have presented a framework for word sense extension that supports lexical items to extend to new senses in novel context. Our results show that chaining provides a general mechanism for automated novel sense extension in natural context, and learning a transformed sense-extensional space enables systematic generalization to a certain degree. We also show that word sense extension improves
the performance of transformer-based WSD models particularly on rare word senses. Future work may extend our framework in several ways, such as how to better model systematic word sense extension, and do so over time and in different languages.

heading: 7 Ethical considerations
text: We discuss the limitations and potential risks of our work.

heading: 7.1 Limitations
text: Our current framework does not explicitly consider the temporal order via which word senses have emerged. In particular, in the data collection step, we construct source-target token pairs for each word type by randomly sampling a target sense from its sense inventory. An alternative and more realistic approach would be to sort all senses of a word chronologically by their times of emergence in history, and use the model to incrementally predict each sense of a word based on usages of its older senses. However, we found that it is infeasible to find accurate timestamps of senses in natural corpora at a comprehensive scale. Another approach is to have human annotators evaluate the plausibility of each ground-truth source-target token pairs that are automatically created in our data collection pipeline, which is a potential area for future consideration.

heading: 7.2 Potential risks
text: All scientific artifacts in this study have been made publicly available and are consistent with their intended use and access conditions. We acknowledge that our focus on English might introduce linguistically or culturally specific biases in modelgenerated outputs. For instance, we observe that the WSE models trained on English sentences learn to generate a metaphorical expression “to spend some time” for the English verb spend, which is common in English but differ in other languages (e.g., Hungarian speakers instead tend to say “to fill some time” as in Kövecses et al. 2010). We believe that by training WSE models cross-linguistically to cover various innovative lexical uses should help mitigate this issue.

heading: 8 Acknowledgements
text: This work was supported by a NSERC Discovery Grant RGPIN-2018-05872.

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: Another approach is to have human annotators evaluate the plausibility of each ground-truth source-target token pairs that are automatically created in our data collection pipeline, which is a potential area for future consideration