Systematic word meta-sense extension
Authors: Lei Yu
Venue: EMNLP 2023 Main (2023)
Keywords: lexical creativity
regular polysemy
systematicity
contextualized language model
analogical inference
figurative language processing
TL;DR: A computational framework of systematic word meaning extension and its application to figurative natural language inference

Abstract: The meaning of polysemous words often varies in a highly productive yet predictable way. Generalizing the regularity between conventional senses to derive novel word meaning is crucial for automated processing of non-literal language uses such as figurative expressions. We introduce a novel task called systematic word meta-sense extension (SWORME) to test and improve language models' ability to extend word meaning to denote new semantic domains (also called meta-senses) that bear regular semantic relations with existing senses. We found that language models prefer incremental lexical semantic change toward conceptually similar meta-senses such as logical metonymy, and are much worse at predicting highly non-literal meaning extensions such as metaphors. We propose a novel analogy-based method of word meaning extension, and show that it effectively improves language model systematicity in making both gradual and radical types of meta-sense extension. We further demonstrate that learning systematic meta-sense extensions benefits language models on multiple benchmarks of figurative language understanding.

Systematic word meta-sense extension
Lei Yu
Department of Computer Science, University of Toronto jadeleiyuêcs, toronto.. edu
Abstract
LEXICON
novel figurative use
related domains
literal use
The meaning of polysemous words often varies in a highly productive yet predictable way. Gen- eralizing the regularity between conventional senses derive novel word meaning is enu- cial for automated processing of non-literal language uses such figurative expressions. We introduce novel task called systematic word meta-sense extension (SWORME) to test and improve language models' ability ex- tend word meaning to denote new semantic do- mains (also called meta-senses) that bear regu- lar semantic relations with existing senses. We found that language models prefer incremental lexical semantic change toward conceptually similar meta-senses such logical metonymy. and are much worse at predicting highly non- literal meaning extensions such as metaphors. We propose a novel analogy-base -based method of word meaning extension, and show that effec- tively improves language model systematicity in making both gradual and radical types of meta-sense extension. We further demonstrate that learning systematic meta-sense extensions benefits language models on multiple bench- marks of figurative language understanding.
ITEM
INFORMATION
pencil
concept
bag purse wallet
to steal idea
book
theory
ball
argumant
getigrasp/sell..
steal
Figure 1: Illustration of systematic word meta-sense extension. Given two conceptually related semantic domains (e.g. ITEM and INFORMATION) and usages of polysemous words describing both domains (e.g. the verbs get, grasp, sell that can take both ITEM class and INFORMATION class nouns objects), we wish to extend the meaning of another word (e.g. steal with its literal sense only) from denoting one of the two domains to denoting both.
to disambiguate related word senses (Reisinger and Mooney, 2010; Mikolov al., 2013; Wiedemann et al., 2019; Reif et al., 2019) and recognize regu- lar relations between lexical items (Boleda et al., 2012a; Vulić et al., 2020; Garí Soler and Apid- ianaki, 2021), few has investigated whether ma- chines can also productively leverage the detected regularity to generate and understand novel lan- guage use in human-like way.
1 Introduction
Linguists and cognitive scientists have suggested that the extensional processes of many polysemous words from conventional novel senses gov- erned by the same set of generative lexical rules (Copestake and Briscoe, 1995; Pustejovsky, 1998; Gentner, 1983; Gentner et al., 2001: Pustejovsky and Rumshisky, 2010) and are therefore intrinsi- cally related to each other that is, word mean- ing extensions exhibit systematicity, as suggested by both theoretical studies of human cognition (Gentner and Toupin, 1986; Fodor and Pylyshyn, 1988) and empirical investigations of word mean- ing change (Xu and Kemp, 2015; Xu al., 2017; Fugikawa et al., 2023). Here we show that neural language models often fail to generate plausible novel word meaning that bears predictable system-
Many words in the lexicon are polysemous in that the same word form can express multiple distinct yet related senses: for instance, some English verbs describing our interactions with physical ob- jects such as get, grasp can also denote the acqui- sition or distribution of abstract knowledge (e.g. to graspiget someone's idea); as a result, human speakers are able to extend the meaning of other interaction verbs like steal to form metaphorical expressions such "to steal information". On the other hand, although recent work suggests that dis- tributed semantic models such as word embeddings and contextualized language models can be applied
We release the code and data for our work here: https:
/gi thub. com/ jadeleyyu/ aodeleiy/swwore. sworme.

---

atic relations with existing senses, a pattern that
2 Related work
is consistent with their poor systematicity in NLP (Ettinger et al., 2018; Goodwin al., 2020; Key- sers et al., 2020; Yanaka et al., 2020) and simi- lar failures observed other domains of machine learning (Bentivogli et al., 2016; Lake and Baroni, 2018; Bahdanau et al., 2018). The lack of system- aticity in word meaning extension also explains recent findings that language models tend to strug- gle at processing under-repen redre-rreeeeentee figurative ex- pressions including metaphor (Stowe al., 2022), simile (Chakrabarty ct al., 2022) and slang (Ni and Wang, 2017; Sun et al., 2022).
2.1 Regular polysemy and meaning extension
Several lexical semantics and cognitive linguistic theories have been proposed explain word mean- ing extension using symbolic rules operating on the semantic structures of lexical entries, including the Generative Lexicon theory by Pustejovsky (1998), the semi-productive sense extension framework by Copestake and Briscoe (1995), and the conceptual metaphor theory by Lakoff and Johnson (2008). Inspired by the ontological view of word meaning variation in Generative Lexicon, some pioneering studies on regular polysemy grouped word senses into broader classes of semantic categories based on WordNet (Buitelaar, 1998; Tomuro, 2001) or linguistic corpus statistics (Boleda et al., 2012b), so that regular polysemy can be defined as a set of words showing the same variation between two (or more) categories (Utt and Padó, 2011). Our framework adopts similar definition of regular polysemy but instead tackles the problem from a generative perspective.
A recent line of work has proposed to predict word meaning extension based on the cognitive theory of chaining (Lakoff, 1987; Malt al., 1999), where novel meaning is linked to existing ones due to their proximity in semantic space (Habibi al., 2020; Yu and Xu, 2021: Grewal and Xu, 2021; Sun et al., 2021; Yu and Xu, 2023). However, existing chaining models prefer extensions across literally similar domains with high overlapping in semantic features, while ignoring the relational similarity between word senses that is essential to understanding conceptual and linguistic metaphors (Gentner et al., 2001; Gentner and Bowdle, 2008). As result, chaining models often fail to predict many figurative word senses that share few similar semantic features with literal meaning.
2.2 Systematicity in NLP
It has been argued for a long time that neural net- works are not cognitively feasible models of nat- ural language because they fail to make system- atic generalizations (Fodor and Pylyshyn, 1988; Marcus, 1998), and there has been extensive line of empirical work evaluate and improve the systematicity of neural networks (Bentivogli et al., 2016; Lake and Baroni, 2015 8; Bahdanau et al., 2018). Existing NLP studics on systematicity mostly focus on investigating whether words have consistent contributions to the meaning representa- tions of their composed expressions (Ettinger et al., 2018; Goodwin et al., 2020; Keysers et al., 2020; Yanaka et al., 2020). However, there also exists wide range of non-compositional, idiosyneratic expressions that can still confuse sfate-of-the-eee art large language models like GPT-3 (Li et al., 2022). We shall demonstrate that while many figurative expressions are non-compositional at word-level, their meaning can be modeled as the composition of literal word senses and regular types of semantic relation.
We propose a novel task called systematic word meta-sense extension (SWORME) to evaluate a language model's ability to predict regular types of word meaning extension in naturalistic context. As illustrated in Figure 1, given two semantic domains that are conceptually related via general cognitive processes such analogy, we wish simulate the scenario where a person, after learning usages of polysemous words describing both domains, can leverage the regular relation between them to ex- tend the meaning of a new target word from one domain to the other. Inspired by research in ana- logical inference (Falkenhainer et al., 1989; Tur- ney, 2006; Levy et al., 2015), introduce new model that infers novel word meta-sense based on the relational similarity between systematically al- ternating word meta-senses, which predicts both incrementally and radically novel usages for over 7,300 polysemous English words.
2.3 Figurative language processing
Most previous work on figurative language focuses on constructing datasets and training models of identifying metaphors in text (Stowe and Palmer,

---

2018: Leong et al., 2018; Aghazadeh al., 2022). Several studies built metaphor interpretation sys- tems by first identifying metaphorical usages and then translating them into its literal word sense recorded in WordNet (Su et al., 2017; Bizzoni and Lappin, 2018; Mao et al., 2018). Other work has focused on interpreting figurative language in narra- tives in context (Chakrabarty et al., 2022; Jhamtani al., 2021) and observed that many models show very large drops in performance compared to con- texts without figurative language.
meta-aliernation with metonymic lexical instanti- ations such as chicken and lamb that denote both animal names and their meat.
We use the CoreLex ontology made by Buite- laar (1998) as our meta-sense inventory for English words. Corelex builds on WordNet (Miller, 1995) and defines layer of abstraction above WordNet synsets consisting of 39 basic meta-senses, with each meta-sense having namesake anchor synset in WordNet. We follow the method introduced in Boleda et al. (2012a) to map each WordNet synset 8 to a meta-sense whose anchor synset is closest to s on the taxonomy tree, and we can therefore assign meta-sense label for each usage of word in a sense-annotated corpus. Since CoreLex only covers noun synsets, extend meta-sense catego- rization to verbs and adjectives by assigning each usage of a verb or adjective the same meta-sense label as its syntactic noun object - for instance, the both verb grasp and the adjective big can then have two meta-senses ITEM and INFORMATION, with the former meta-sense being signified in phrases like "to grasp an item" and "a big item", and the latter being reflected by expressions such as "to grasp idea" and "a hig idea".
3 Computational framework
In this section, we first introduce the concept of word meta-sense, and formulate regular polysemy as systematic types of meta-sense alternation. Next, we introduce the process of partitioning polyse- mous word type into multiple hypothetical tokens signifying its different meta-senses to operational- ize the scenario of meaning extension toward novel domains. We then define SWORME as task of inferring partitioned token pairs denoting systemati- cally related meta-senses to substitute each other in naturalistic context. We finally introduce methods of learning systematicity in meta-sense extension.
3.2 Meaning-based word type partitioning
3.1 Meta-sense and systematic alternation
We wish to investigate whether language models can flexibly extend word meaning across system- atic meta-alternation (m, m'). We operationalize this idea by training a language model from scratch on a text corpus in which some lexical instantia- tions w of (m, m²) are partitioned into two new hypothetical tokens: a token t(w, m) replacing all mentions of w in cense-annotated corpus that ex- hibit the meta-sense m, and another token ((w, m') replaces ue for sentences in the corpus signifying the meta-sense m', illustrated in Figure 2(a)- (c). The resulting language model can therefore compute valid meaning representations for usages of w with meta-sense m'' using the partitioned to- ken t(w, m') without knowing that w can actually express m'.
It has been suggested that regular polysemy can be indicated by multiple words sharing the same distribution over denoted semantic domains (Apres- jan, 1974; Nunberg, 1979). Wc define meta-sense group of word senses that share certain high- level semantic features, and pair of meta-senses is called metta-alternation if there exists word form that has senses from both meta-sense categories, and we call such word a lexical instantiation of the meta-alternnatio. Following the frequency-basee definition of systematic polysemy in Utt and Padó (2011), we consider a meta-alternation as system- atic if there is large set of words instantiating the same meta-alternation and a systematic word meta-sense extension (SWORME)) is the case where a word to with existing senses only under meta- sense m used to express new sense from m' which together with m forms systematic alterna- tion (m, mr'). For example, the two meta-senses ANIMAL and FOOD together form a systematic
3.3 SWORME as token substitution
Let (m,m') be a systematic meta-altermation with a lexical instantiation w, and let U((ww, m)). ([(w, m²)) be two sets of us- age sentences with replaced by its partitioned tokens t(w, m), (we m²) respectively. As illus-
'In particular, we define a meta-alternation to be systematic if its amount of observed lexical instantiations a reference corpus is greater than a threshold 6, whose value will he specified in the Data section.
"See Appendix B for full list of CoreLex meta-senses.

---

to
gallumph
school
word chortee
in "They finally
to arrive school
PSY
to arrive at conclusion to steal purse to steal knowledge
to chartea conclusion
Language Model
chorle
an agreement can be
to jabber purse
substitued by
ITEM INFO
wocky knowledge
gelumpi
jabber
wocky
(a) sense-annotated corpus
systematic mota-altornaiton
(c) word partition
(d) pretrain
(e) token substitution
Figure 2: Illustration of the SWORME framework. Given a aense-annotated text corpus, we first decide a set of systematic meta-allernations (e.g. the INFORMATION/IEN and the LOCATTONPSYCHOLOOICCAL...
alternations in (b)) with sufficient lexical instantiations denoting both meta-senss (e.g. arrive at with both m 1.OCATION type objects such as school and m' WECCOLLOGICAL-SSS
type objects such as conclusion). We then partition each lexical instantiation by replacing it with hypothetical tokens e.g. the nonce words t(w, m) = galumph and t(w, m') chortle in (c) replace mentions of arrive at exhibiting the LOCATION and the PSYCHOLOGICAL-STAAE
meta-senses respectively, and their systematic relation is indicated by their matching background shape figures. A language model is then pretrained from scratch on the replaced corpus and is then evaluated on the token substitution task, where the model is asked to choose the correct partitioned token galumph in (e) paraphrase its "sibling" token chortle.
trated in Figure 2(e), given a usage sentence u E Ut((w, m²)). we say that a model extends the meaning of (w, m) to mí' under context if the model infers that t(w, m) is good substitution to paraphrase (ww, m²) in In particular, let T be a list of candidate paraphrase tokens containing (w,, m), we would ask the language model to first compute the contextualized embedding htt,, u) of each E in context u (with t(w, m) replaced by t), and choose the best paraphrase token t" that maximizes the semantic similarity between the contextualized embeddings of and (wwm') in
ing meta-sense m in a reference corpus, and 2(s, m. m²) = h(w, m) - h(w, m') be the off- set between the prototypes of w's two meta-senses. Let W (m, m') be the whole set of lexical instan- tiations of meta-alternation (m,,m), the analogi- cal chaining model draws inspirations from par- allelogram models of human and machine ana- logical inference (Gentner, 1983; Turney, 2006; Mikolov et al., 2013; Peterson et al., 2020) and assumes that the relational representations between the meta-sense prototypes of any two (w1, (w1, c W (m, m²),, operationalized dings 2 2(W1, m. m') h h(w,, m) h(w,, m') and =ww2, m, m²) h(w2, m) h(w2, m'), should be similar. We could therefore train a language model to align z(w1, m, m²), =(w2, m, m²) for subset of lexical instantiations of each meta-alternation, and then test whether the model can generalize the learned relational regularity to unseen lexical items in the same meta-alternation category. In particular, at cach trial, we sample systematic alternation (m, m') and pair of its lexical instan- tiations (wi, w2). and train the language model minimize the following loss function:
the offset embed-
t" argmin,gr/lh((t u) h(t(w, m'), w)ll (I)
the meaning extension of t(w, m) to m' is success- ful if and only if t t(w, m).
3.4 Learning systematic meta-sense extensions
We hypothesize that the language model embed- ding space optimized standard pretraining ob- jectives such as masked language modeling may not well capture the regularity underlying meta- alternations, and we next propose two methods to incorporate knowledge of systematic meta-sense extension into language models. Our methods are based on the cognitive theory of chaining (Lakoff, 1987) which states that word meaning extends to novel yet semantically similar meta-senses, and we consider two chaining models with different operationalizations of semantic similarity.
Canalog
Σ
d(wi, w2, m, m,m') (2)
(m.m' (01,)02)
d(w), u2 m. m') ||a(w], m,) =(wy, (3)
Associative chaining. The associative model follows recent computational implementations of semantic chaining (Ramiro et al., 2018; Habibi et al., 2020; Pinto Jr and Xu, 2021) and predicts that the token f(w, m) with an existing meta-sense
Analogical chaining. We define a word meta- sense prototype h(w, m) as the mean contextu- alized embedding of all mentions of w exhibit-

---

Systematic
Word
POS
Usage
CoreLex mieta-sense
meta-sense atternation
The Scots had a uradition of deep frying chicken in fat.
chicken
FOOD
ANIMAL FOOD
unlike their English counterparts who baked or boiled chicken.
then a rising and expanding parcel of will arrive at the new altitude at a temperature than the surrounding
LOCATION
arrive (at)
verb
DEFINITE QUANTITY
DEFINITE QUANTITY
Although shows cold attitude, she realizes she can't help but love him.
SUBSTANCE-
cold
adjective
PSYCH.FEATURE
PSYCH.FEATURE
Table Sample entries of the SWORME dataset. Target words (lexical instantiations of meta-alternntions) in usage sentences are shown in bold italic, and noun objects that decide meta-sense labels of verb and adjective lexical instantiations are underlined.
m can be extended to express a new meta-sense m' if they share similar semantic feature values i.e. the semantic distance between their prototypes zww, m, m') h(w, m) h(w, m²) is small. We use the formulation of prototype-based chaining (Sun et al.. 2021; Yu and Xu, 2023) and train language models on a contrastive learning objec- tive: in each step, we sample a meta-sense triplet Muri = (m,m* m) so that (m, m²) together is not
types of regular meaning alternation including logi- cal metonymy, weak metaphor and strong metaphor For each systematic aeta-allernation, the top-100 lexical instantiations with highest num- bers of usage examples the corpus. This pipeline finally yields approximately 880,000 usage sen- tences for 7,346 English words (3,155 nouns and 2576 verbs and 1,615 adjectives). See Table for sample entries of the resulting dataset.
take
form meta-alternation while systematic alternation. We then sample lexical instantiation to of (n) and another word w with meta-sense m , and train the language model to minimize the following loss function:
5 Results on SWORME
5.1 Experimental setup
We split the collection of lexical instantions W (m, (m,m') of cach meta-alternation (m, m') into two subsets Wirain (aaiimmmm))) m'), Weea(m, (m),m') and eval- uate transformer-based language models on the task of SWORME via three steps: 1) in the pretrain- ing step, the model is trained from scratch via the masked language modeling (MLM) objective on usage sentences of each w E W(m, m'), where the model takes batches of sampled usage sentences with 15% of randomly chosen tokens masked out, and updates its parameter weights to maximize the probability of infilling the correct missing tokens. We replace each Wrestmm, m²) with its par- titioned tokens, and increase the vocabulary size of the language model by adding rows to its first embedding layer and its language model head layer accordingly. For words with multiple tokens, we would replace all of its constituent tokens with a single new token added into the tokenizer vocab- ulary. We keep the original word form for each w Wirann (1,, m') so that the model learns that (m, can be expressed together by some word forms suggesting systematic relations. 2) in the SWORME learning step, the language model is further fine-tuned on one of the two chaining objec- tives Lanalog or Lasso over usage sentences of cach
Cassoe ΣΣ
Uw, w')
(4)
Mup un,1o
(wrww)) = m)) Mw., m)
1)1 [[h(w, mi) MMwl. II
(5)
4 Data
We construct our SWORME usage dataset based on the sense-annotated text corpus made by (Yu and Xu, 2023), which consists of 1.47M sentences taken from the Wikitext-103 corpus (Merity et al., 2016) and contains usages of over 7,500 English polysemous words labeled with their associated WordNet synset IDs. We obtain the CoreLex meta- sense label for each polysemous word usage via the mapping method introduced in section 3.1. For each word, we only keep usages of its top-2 most frequent meta-senses in the corpus, so that there is no overlap between the lexical instantiation sets of any two meta alternattentno classes. To decide a set of systematic meta- alttrnaltesttionss meta-sense pairs (m, m²) with least 50 lexical in- stantiations of more than 10 usage examples under each meta-sense (i.e. with at least 20 mentions in total). This gives us a total of 50 meta-sense alter- nation pairs that covers a variety of widely studied
then take all
"See Appendix B for the full list of systematic meta- alternations in our datasct.

---

Wirain (m, m') in its original word form; in the evaluation step, we test the language model on the lexical substitution task over usage sentences of w E Wica(m, m')) with uo replaced by its par- titioned tokens. In particular, at each evaluation trial, we present the model with usage sentence of hypothetical token (w, m²'), and list of 100 candidate tokens consisting of a ground-truth sub- stitution t(w, m) and 99 negative alternatives ran- domly sampled from the set of hypothetical tokens partitioned from other words w' E Wes (m, m²) We use mcan precision to measure model perfor- mance, which is the percentage of cases where the model predicts (wc, m) as the most likely substitu- tion among 100 candidates, so random baseline would yield 1 % predictive accuracy.
BERT-MLM
SWORME SWORME associate
SWORME-ana analogy
SWORME-FUl
Model precision
9.05
20
40
60
80
% of training word per alternation (a 100)
Figure Average model precision on SWORME with increasing amount of of training evidence for each meta- sense alternation. Error bars show the standard devia- tions over five independent runs.
Meta alternation
Example usage
Meta-wenst
Model
We expect a systematic model of SWORME to generalize the meaning express a new meta-sense m' after learning from a small set of examples indicating the regularity between (m, m'). We therefore change the pro-
similurity
accuracy
token (ww, m) to
ARTIFACT
light box
0.276 (35/50)
Assoc:: 0.087
ATTRIBUTE
light colar
Analog: 0.361
SUBSTANCE
food
Assoc.: 0.1
0.158 (44/50)
TIME
waste time
Analog: 0.357
LOCATION
reach destination
Assoc:: 0.133
0.213 (40/50)
CONSEQUENCE
gool
Analog:
portion of unpartitioned training words per meta- WWain (m.m')
Table 2: Top-3 meta-alternation classes with most im- proved model accuracy by analogical chaining (Analog.) over associative chaining (Assoc.).
alternation a = Wiiai (m, m// (m.""" from to 0.8 with a step size of 0.2, and learn 5 inde- pendent SWORME models to examine how their performance change as the linguistic evidence of systematic meta-sense alternation increases. Fur- ther details of experimental setups can be found in Appendix A.
5.3 Results
Figure 3 shows model precision with various val- ues of a over 5 independent runs. We observe that all BERT-based models achieve significantly above chance accuracy and perform better as be- ing exposed to more lexical instantiations per meta- alternation during pretraining. In particular, even in the case where pair of systematically related meta- senses are never expressed together by any word form in training data (i.e. a 0), BERT can still predict that words denoting one of the two semantic categories can be extended express the other, sug- gesting that the language model has captured some intrinsic conceptual relatedness between semantic domains during MLM pretraining. Moreover, the superior performance of the analogical chaining models over their associative chaining counterparts suggest that the analogical or relational similar- ity between semantic domains more useful than their overall featural proximity for systematic word meaning extensions.
5.2 Models of SWORME
We take randomly initialized transformer coder with the same architecture as BERT-base- uncased by Devlin et al. (2019) as our main lan- guage model, based on which we implement three models of SWORME: 1) a -ooloE-analogyy model pretrained on MLM and fine-tuned on SWORME using the analogical chaining objec- tive, 2) SWORMB-associatt model pretrained on MLM and fine-tun tuned using the associative chaining objective, and 3) a SWORME-full model that is fine-tuned on both chaining objectives after being pretrained via MLM. We also include baseline model BERT-MLM bascline that is only pretrained om MLM but is not fine-tuned on chaining.
We further examine model sensitivity to the con- ceptual relatedness between existing and extended meta-senses. We quantify the degree of conceptual relatedness as the mean Wu-Palmer similarity (Wu
We experimented with several alternative sumpling meth- ads of negutive source tokens, such as taking the top- 1pp-100 parti- tioned tokens with most similar static emheddings to the target token, but did not observe significant performance change.

---

Ip-
0.52)
Associative chaining (p- 0.80]
Assecative chaining 0.70;
Analogical chaining (p= 0.442)
Aralegical chaining (p= 0.40)
0.5
0.5
0.5
a=0
0.2
a 0.8
Model precision
0.4
0.3
0.3
0.2
0.1
0.1
0.1
02 0.6
1.0
0.0
0.8 Meta similarity
0.2 0.6 1.0
Meta-sense similarity
Meta-sense similarity
Figure 4: Meta-sense semantic similarity vs. mean predictive accuracy of models trained on SWORME via associative and analogical chaining objectives under zero-shot (a = 0), few-shot (a = 0.2) and many-shot (a = 0.8) setups. When a all models are equivalent BERT-MLM so only set of data points plotted.
Pearson correlations between accuracy and semantic similarity are shown in legends (p < 10 35535 for all cases).
and Palmer, 1994) between the anchored Word- Net synsets of two meta-senses, and we then com- pute the mean model precision of predicting sub- stituted partitioned tokens from each meta-sense alternation pair (averaged over both extensional directions), as shown in Figure 4 for three exper- iment setups with increasing amount of training words per meta-alternation (a = [0, 0.2, 0.8). We found that all models generally make better pre- dictions on meta-alcernations that are conceptu- ally more contiguous (e.g., metonymy), and per- form less well on examples where the novel meta- sense is conceptually very different to the exist- ing one (e.g., strong metaphor). Moreover, ana- logical chaining model exhibits less sensitivity to semantic proximity and generally does better at predicting radical meta-sense extensions than its associative chaining counterpart. Table 2 shows the top-3 meta-alternation classes on which analog- ical chaining improves model performance most significantly over associative chaining. We found that all these meta-alternations are typical examples of "metaphorical" extensions consisting of a con- crete meta-sense and semantically very different abstract meta-sense. These results again suggest that the literal similarity between conventional and novel meaning is insufficient account for various types of lexical creativity.
Data. We evaluate models on two publicly avail- able datasets of natural language inference (NLI) with figurative expressions: the IMPLI dataset by Stowe et al. (2022) contains 25,860 figurative- literal expression pairs, where each literal expres- sion can be either entailed or non-entailed by its paired figurative expression that comes from one of the two classes: metaphors or idioms. The Fig- QA dataset by Liu et al. (2022) consists of 10,256 Winograd-styll questions (Levesque al., 2012). where model is asked to identify literal entail- ment among two candidates for a pair of superfi- cially similar figurative expressions with opposite meaning. The questions in Fig-QA can be catego- rized into four classes based on the type of knowl- edge required to answer them: objective knowledge (Obj), visual metaphors (Vis), social understanding (Soc), and cultural metaphors (Cul).
Models. We test three off-the-sheif pretrained transformer language models on FL.U: 1) BERT- base-uncased (with 0.1 parameters, pretrained 40 GB of text) implemented by HuggingFace (Wolf et al., 2019), 2) GPT2-XL (with 1.5B param- eters, pretrained on 800GB of text) implemented also by HuggingFacce, and 3) LLaMA (with 7B pa- rameters, pretrained on 1TB of text) implemented by Meta (Touvron et al., 2023). Before FLU eval- uation, each language model is fine-tuned on the the training set of SWORME with a = 0.8 us- ing either associative analogical chaining ob- jective (usage sentences containing the other 20% word types are left out as the validation set to de- cide model convergence). For auto-regressive mod- cls (GPT2-XL and LLaMA), the contextualized embeddings of a target word is computed only
6 Application to figurative language understanding
We finally demonstrate that learning SWORME can benefit transformer language models on the task of figurative language understanding (FLU).

---

Model
IMPLI
Fig-QA
Metaphors
Idioms
All
Obj
Vis
Soc
All
BERT-base
80.15
69.72
71.18
86.50
89.49
82.11
86.32
86.05
+ assoc.chaining
78.60
72.33
73.29
86.41
90.19
80.87
79.19
85.51
+ analog.chan gaalong
85.04
74.98
76.52
86.70
96.24
80.08
86.76
87.84
GPT2-XL
77.56
61.45
61.99
73.72
72.97
72.23
76.10
73.90
+ assoc assoc.chaining
77.31
64.72
65.05
72.18
74.01
71.16
75.34
73.82
+ amalog.chaining
79.96
66.20
68.48
73.55
78.96
71.12
80.60
77.03
LLaMA-7B
87.85
84.93
85.21
86.99
90.94
$7.02
85.17
89.10
+ assoc sesoc.chaining 88.95
80.01
80.97
83.51
83.27
85,50
80,44
83.39
+
91.62
87.90
88.11
89.73
93.29
86.64
84.08
89.74
Table 3: Model classification accuracy two figurative language understanding datasets.
Model predicted
Dataset
Premise
Hypothesis
True Label
entailment probability
How you weathered the storm?
How have you calmed the storm?
BERT: 0.76 (X)
IMPLI
non-cutailmnen
BERT chain.: 0.30
Time come out from under
Time to come out from under a roof and enjoy yourself.
GPT2: (X)
IMPLI
non-cntailment
cloud and enjoy yourself.
GPT2+amalog chain.; 0.41 (V)
His imagination is as broad us the sky.
LLaMA: 0.39 (X)
Fig-QA
He has vivid imagination.
entailment
aM A+ A+taallggcchhinn........ 0.53 (V)
The place was joy/ul as funeral.
I.LMMA: 0.57 (X)
QA
place was joyful.
non-entailment
chain.. 0.55 (K)
Table 4; Example FLU questions and model outputs. Entailment labels and model predicted entailment probabilities are marked blue, and non-entailment labels/probabilities are marked red.
using its prefix context in each sentence. After SWORME training, each model is fine-tuned on the official training sets of the two FLU datasets, where add linear classification layers on top of each language model that takes contextualized embeddings of the last [CLS] token of each con- catenated gremise-hypoteeste sentence pair and outputs a binary entailmend/oon-eetiimmn label. The classification layers and the underlying en- coders are then trained together to minimize on the standard cross entropy loss between model pre- dicted and true entai Imen labels. We perform full model fine-tuning for BERR-bbas-uncaseed and ap- ply parameter-efficienn fine-tuning via LoRA (Hu et al., 2021) for GPT2-XL and LLaMA. We also include a baseline version for each language model that is not fine-tuned on SWORME.
that associative chaining pushes usage embeddings of related meta-senses too close to each other, so that some important sentence-level semantic fea- tures in the sentence embedding become degener- ated. These results together suggest that learning relational similarity between systematic word meta- senses can serve as a simple yet effective method to drive language models toward human-level un- derstanding of figurative language.
Table shows model predictions on sample FLU questions. We found that many idiomatic expres- sions in IMPLI can also be interpreted as system- atic meaning extensions from more "literal" meta- senses of common polysemous words (c.g. "storm" referring to "difficult situation", which signifies a systematic extension from (hostile) NATURAL PHENOMENON to (poor) COGNITIVE STATE), so learning analogical chaining helps model bet- ter distinguish such usages against the adversarial hypothesis with high lexical overlap. We also ob- serve that even the largest LLaMA- 7B model still makes errors metaphorical expressions whose interpretations are obvious humans (e.g. broad imagination), while learning SWORME through analogical chaining helps correct many of these mistakes. Meanwhile, analogical chaining helps lit-
Results. Table 3 summarizes model classifica-
tion accuracy on the official evaluation sets of the two FLU datasets. We found that language models fine-tuned on SWORME through analogical chain- ing yield best overall classification accuracy, as well as on most sub-categories of figurative lan- guage use. Fine-tuning via associative chaining, on the other hand, is much less helpful or can some- times even be harmful for FLU. We hypothesize

---

de on understanding ironic expressions such as "as joyful as funeral", which can also be considered as a systematic semantic extension toward the oppo- site word meaning. Future work can explore how antonymic meaning change can be incorporated into the SWORME framework.
References
Ehsan Aghazadeh, Mohsen Fayyaz, and Yadollah Yaghoobzadeh. 2022. Metaphors in pre-trained lan- guage models: Probing and generalization across datasets and languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2037- 2050.
7 Conclusion
Ju D Apresjan. 1974. Regular polysemy. Linguistics.
We have presented a framework of systematic word meta-sense extension (SWORME) that supports lexical items to express new semantic domains in a productive yet predictable way. Our results show that the feature associative similarity only predicts incrementally novel meaning, while analogical sim- ilarity provides a general account for both grad- ual and radical types of word meaning extension. We also show that learning analogical chaining- based meta-sense extension improves transformer language model performance on figurative natural language inference.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. 2018. Systematic generaliza- tion: What is required and can it be learned? arXiv preprimt arXiv: 181 12889.
Luisa Bentivogli, Raffalla Bernardi, Marco Marelli, Stefano Menini, Marco Baroni, and Roberto Zampar- elli. 2016. Sick through the semeval glasses. lesson learned from the evaluation of compositional distri- butional semantic models on full sentences through semantic relatedness and textual entailment. Lan- guage Resources and Evaluation, 50:95-124.
Yuri Bizzoni and Shalom Lappin. 2018. Predicting human metaphor paraphrase judgments with deep neural networks. In Proceedings of the Workshop on Figurative Language Processing, pages 45-55, New Orleans, Louisiana. Association for Computational Linguistics.
8 Limitations
Our work has some limitations. For instance, in the current SWORME framework we train models to predict extensions across systematically alternating meta-sense pairs in both directions, while research in leixcal semantic change suggests that such ex- tension sometimes only happens uni-directionally (Xu et al., 2017: Winter and Srinivasan, 2022) - for example, it is quite natural to extend word meaning from the ANIMAL domain to the MEAT domain (e-g. to raise chicken -> grilled chicken) but much less plausible for the opposite direction (e.g. grilled beef- + to raise beef). A more realistic approach would be to sort all meta-senses of a word chrono- logically by their historical time of emergence, and only ask the model to predict the newer meta-sense based on the older one. However, we found it in- feasible to determine accurate timestamps of the meta-senses their associated WordNet senses at a comprehensive scale, and we believe that learn- ing to make some unattested types of meta-sense extension would be beneficial for language mod- els to understand idiosyncratic word uses that are usually under-representted in training corpora.
Gemma Boleda, Sebastian Padó, and Jason Utt. 2012a. Regular polysemy: distributional model. In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics- Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop Semantic Evaluation (SemEval 2012), pages 151- 160.
Semantics-Volume
Gemma Boleda, Sabine Schulte im Walde, and Toni Badia. 2012b. Modeling regular polysemy: A study the semantic classification of catalan adjectives. Computational Linguistics. 38(3):775 616.
Paul Buitelaar. 1998. Corelex: An ontology of system- atic polysemous classes.
Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. Flute: Figurative lan- guage understanding through textual explanations. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 7139-7159.
Ann Copestake and Ted Briscoe. 1995. Semi-produetive polysemy and sense extension. Journal of semantics, 12(1):15-67.
9 Acknowledgements
The author would like to thank Yang Xu, Gemma Boleda and anonymous OpenReview reviewers for their helpful suggestions on the manuscript.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for

---

Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Harsh Jhamtani. Varun Gangal, Eduard Hovy, and Tay- lor Berg- Kirkpatrikk.k 2021. Investigating robustness of dialog models to popular figurative language con- structs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 7476-7485, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Allyson Ettinger, Ahmed Elgohary, Colin Phillips. and Philiip Resnik. 2018. Assessing composition in sen- tence vector representations. In Proceedings of the 27th International Conference Computational Lin- guistics, pages 1790-1801.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. 2020. Measuring com- positional generalization: A comprehensive method realistic data. In International Conference on Learning Representations.
Brian Falkenhainer, Kenneth D Forbus, and Dedre Gen- tner. 1989. The structure- mappiig engine: Algo- rithm and examples. Artificial intelligence, 41(1):1- 63.
Jemry A Fodor and Zenon Pylyshyn. 1988. Connec- tionism and cognitive architecture: A critical analysis. Cognition, 28(1-))33-7
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR (Poster),
Olivia Fugikawa, Oliver Hayman, Raymond Liu, Lei Yu, Thomas Brochhagen, and Yang Xu. 2023. A computational analysis of crosslinguistic regularity in semantic change. Frontiers in Communication, 8:1136338.
Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-seqqenee recurrent networks. In- ternational conference on machine learning, pages 2873-2882 PMLR.
Aina Garí Soler and Marianna Apidianaki. 2021. Let's play mono-poly: Bert can reveal words' polysemy level and partitionability into senses. Transactions the Association for Computational Linguistics, 9:825- 844.
George Lakoff. 1987.
Women, fire, and dangerous things: What categories reveal about the mind. Uni- versity of Chicago press.
Dedre Gentner. 1983. Structuremappingg A theoretical framework for analogy. Cognitive science, 7(2):155- 170.
George Lakoff and Mark Johnson. 2008. Metaphors we live by. University of Chicago press.
Dedre Gentner and Brian Bowdle. 2008 Metaphor structure-mapping. The Cambridge handbook of metaphor and thought, 109:128.
Chee Wee (Ben) Leong, Beata Beigman Klebanov, and Ekaterina Shutova. 2018. A report on the 2018 VUA metaphor detection shared task. In Proceedings of the Workshop on Figurative Language Processing, pages 56-66, New Orleans, Louisiana. Association for Computational Linguistics.
Dedre Gentner, Brian Bowdle, Phillip Wolff, Consuclo Boronat, et al. 2001. Metaphor like analogy. The analogical mind: Perspectives from cognitive science, pages 199-253.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thir- leenth international conference the principles of knowledge representation and reasoning.
Dedre Gentner and Cecile Toupin. 1986. Systematicity and surface similarity in the development of analogy. Cognitive science, 10(3):277-300.
Emily Goodwin, Koustuv Sinha, and Timothy O' Donnell. 2020. Probing linguistic systematicity, In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 1958- 1969.
Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Im- proving distributional similarity with lessons learned from word embeddings. Transactions of the associa- tion for computational linguistics, 3:211-225.
Siyan Li, Riley Carlson, and Christopher Potts. 2022. Systematicity in gpt-3's interpretation of novel english noun compounds. arXiv preprint arXiv:2210.09492.
Karan Grewal and Yang Xu. 2021. Chaining algorithms and historieal adjective extension. Computational approaches to semantic change, 6:189.
Amir Ahmad Habibi, Charles Kemp, and Yang Xu. 2020. Chaining and the growth of linguistic cate- gories. Cognition, 202: 104323.
Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022. Testing the ability of language models to interpret figurative language. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4437-4452, Seattle, United States. Association for Computational Linguistics.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhi Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. 2021. Lora: Low-rank adaptation of large lan- guage models. International Conference on Learn- ing Representations.

---

Barbara C Malt, Steven Sloman, Silvia Gennari, Meiyi Shi, and Yuan Wang. 1999. Knowing ver- namiig: Similarity and the linguistic categoriza- tion of artifacts. Journal of Memory and Language, 4002)2230-26..
Joseph Reisinger and Raymond Mooney. 2010. Multi- prototype vector- space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the As- sociation for Computational Linguistics, pages 109- 117, Los Angeles, California. Association for Com- putational Linguistics.
Rui Mao, Chenghua L Lin, and Frank Guerin. 2018. Word embedding and WordNet based metaphor identifica- tion and interpretation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics Vooume 1: Long Papers), pages 1222- 1231, Melbourne, Australia. Association for Compu- tational L Linguistics.
Kevin Stowe and Martha Palmer. 2018. Leveraging syn- tactic constructions for metaphor identification. In Proceedings of the Workshop on Figurative Language Processing, pages 17-26, New Orleans, Louisiana. Association for Computational Linguisties.
Gary F Marcus. 1998. Rethinking climinative connec- tionism. Cognitive psychology, 37(3):243-282.
Kevin Stowe, Prasetya Utama, and Iryna Gurevych. 2022. Impli: Investigating nli models' performance on figurative language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5375- 5388.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els. arXiv preprint arXiv: 07.44349.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jef- frey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.371.
Chang Su, Shuman Huang, and Yijiang Chen. 2017. Automatic detection and interpretation of nominal metaphor based on the theory of meaning. Neuro- computing, 219:300-311.
Zhewei Sun, Richard Zemel, and Yang Xu. 2021. A computational framework for slang generation. Transactions of the Association for Computational Linguistics, 9:462-47 478.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(1 38)1444994
Ke Ni and William Yang Wang. 2017. Learning to ex- plain non-standard english words and phrases. In Proceedings of the Eighth International Joint Con- ference on Natural Language Processing ( (Volume 2: Short Papers), pages 413-417.
Zhewei Sun, Richard Zemel, and Yang Xu. 2022. Se- mantically informed slang interpretation. In Pro- ceedings the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 5213-5231.
Geoffrey Nunberg. 1979. The non- con-unqqueeess of seman- tic solutions: Polysemy. Linguistics and philosophy. pages 143-184.
Noriko Tomuro. 2001. Tree-cut and a lexicon based systematic polysemy. In Second Meeting of the North American Chapter of the Association for Com- putational Linguistics.
Joshua C Peterson, Dawn Chen, and Thomas L Griffiths. 2020. Parallelograms revisited: Exploring the limi- tations of vector space models for simple analogies. Cognition, 205:104440.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397
Renato Ferreira Pinto Jr and Yang Xu. 2021. A com- putational theory of child overextension. Cognition, 206: 104472.
James Pustejovsky, 1998. The generative lexicon. MIT press.
Peter D Turney. 2006. Similarity of semantic relations. Computational Linguistics, 32(3):379-44
James Pustejovsky and Anna Rumshisky. 2010. Mecha- nisms of sense extension in verbs.
Jason Uut and Sebastian Padó. 2011. Ontology-based distinction between polysemy and homonymy. In Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011).
Christian Ramiro, Mahesh Srinivasan, Barbara C Malt, and Yang Xu. 2018. Algorithms in the historical emergence of word senses. Proceedings of the Na- tional Academy of Sciences, 113(10):2333-2328
Ivan Vulié, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš. and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222-72222222 -240, Online, Association for Computa- tional Linguistics.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. Advances in Neural Information Processing Systems, 32.

---

Gregor Wiedemann, Steffen Remus. Avi Chawla, and Chris Biemann. 2019. Does bert make any sense? interpretable word sense disambiguation with contextualized embeddings. arXiv preprint arXiv:/909.100 10430.
Bodo Winter and Mahesh Srinivasan. 2022. Why is se- mantic change asymmetric? the role of concreteness and word frequency and metaphor and metonymy. Metaphor and Symbol, 37(1):39-54.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz. et al. 2019. Huggingface's 's transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.03771.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac. Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2020. Transformers:
natural language processing. In Proceedings of the 2020 con- ference empirical methods in natural language processing: system demonstrutions, pages 38-45,
Zhibiao Wu and Martha Palmer. 1994. Verbs semanties and lexical selection. In Proceedings of the 32nd annual meeting Association for Computational Linguistics, pages 133-138.
Yang Xu and Charles Kemp. 2015. A computational evaluation of two laws of semantic change. In CogSci.
Yang Xu, Barbara C Malt, and Mahesh Srinivasan. 2017. Evolution of word meanings through metaphorical mapping: Systematicity over the past millennium. Cognitive psychology, 96:41-53.
Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, and Kentaro Inui. 2020. Do neural models learn system- aticity of monotonicity inference in natural language? In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 6105- 6117.
Lei Yu and Yang Xu. 2021. Predicting emergent lin- guistic compositions through time: Syntactic frame extension via multimodal chaining. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 920-931.
Lei Yu and Yang Xu. 2023, Word sense extension. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 3281-3294, Toronto, Canada. Association for Computational Linguistics.

---

A Details of SWORME experiments
We use the BERT-base-uncased configuration pro- vided by Huggingl'ace (Wolf et al., 2020) to initial- ize all BERT-bascd SWORME models (the BERT- MLM baseline and two chaining-bann baaeidd SWORME models).
During MLM pretraining, randomly mask 15% of tokens in each sentence, and train each model on predicting the masked tokens. We add all partitioned tokens as special tokens into the vocab- ulary of the BERT tokenizer, so each pseudo-token will be encoded as whole in the input sequence. Learning is performed using the Adam optimizer (Kingma and Ba, 2015), with a learning rate of 5e-5 and batch size of 128, for 50 epochs (after which all models achieved highest evaluation accuracy). During SWORME training. we kept 10% of us- age sentences in SWORME training set for valida- tion, and fine-tune the associative and analogical chaining models on the rest 90% sentences via their corresponding objective functions in Eq.3.4 and E..3.4 using Adam, with batch size of 32 and a learning rate of 2c-5. The associative chaining model is trained for 8 epochs and the analogical chaining model is trained for 24 epochs. All exper- iments are run on machines with an NVIDIA Tesla A100 GPU.
B CoreLex meta-sense and systematic meta-alternations
See Table and Table 6.

---

abs
ABSTRACTION
COl
ENTITY
loc
LOCATION
pet
PART
ACT
log
NEC.LLOCTTONN
psy
PSYCHOL
agt
AGENT
FOOD
MEASURE
DEFINITE QUANTITY
ANIMAL
fr
FORM
MICROORGANISM
INDEFINITE QUANTITY
ARTIFACT
grb
BIOUOG.GROUP
nat
NATURAL BODY
RELATION rel
B1T
ATTRIBUTE
GROUPING
phm
PHENOMENON
spe SPACE
cel
CELL
grs
SOCIAL GROUP
pho
PHYSICAL OBJECT
sta
STATE
CHEMICAL
pls
SUBSTANCE
COMMUNICATION
Ifr
LIVING BEING
pos
POSSESSION
TIME
CONSEQUENCE
Ine
LINEAR MEASURE
pro
PROCESS
Table 5: CoreLex's 's meta-senses (names in lowercase) with their corresponding WordNet anchor synsets (names in
uppercase).
grs-psp uss-psy
com-evt
art-com
atr-com
art-fim
pro-sta
art-grs
act
atr-sta
act-hum
fod-plt
hum-psy
phm-sta
act-phm
anm-art
psy-sta
hum-nat
atr-psy
act-grp
act-pro
hum-prt
anm-hum
fod-hum
art-atr
art-log
art-loc
com-psy
plt-sub
sub-psy
anm-fod
grs-log
act-grs
act-com
sub-tme
com-hum
act-evt
atr-rel
grp-grs
art-evt
loc-con
evt-psy
art-qui
art-psy
atr-evt
art-sub
act-sta
art-prt
art-sta
evt-sta
Table 6: Top-50 systematic CoreLex meta alternations with highest corpus frequency.


Reviews:
[{"review_id": "CX7ZXhHHjD", "reviewer": "Reviewer_91N3", "summary": "", "questions": "A)\tConsidering the challenges in dataset creation, could you elaborate on any strategies or methodologies employed to ensure the quality and reliability of the SWORME dataset, mitigate bias, and how these efforts might influence the framework's reliability and potential limitations in real-world applications?\nB)\tGiven the emphasis on strong supervision and carefully annotated SWORME datasets, how do you envision the adaptability of the proposed framework to languages and domains with limited resources or access to extensive lexical databases?\nC)\tGiven the framework's focus on systematic word meta-sense extension, do you foresee potential applications beyond figurative language understanding, such as potential impacts on creative language generation or other NLP tasks that involve nuanced word usages?\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and helpful comments, and here we would like to address the main concerns raised by the reviewer in the “Questions For The Authors” section:\n\nQuestion (a): we would like to emphasize that both the meta-sense inventory and the sense annotated text corpus in our paper are taken from existing published work (the paper by (Yu and Xu, 2023) that introduced the Wikitext sense-annotated text corpus has recently been published at ACL 2023). Our frequency-based approach of deciding systematic meta-sense alternations also follows the definition of systematic polysemy in (Utt and Pado, 2012). Moreover, the previous work by (Boleda et al., 2012) independently found a set of 60 most frequent meta-sense alternations using a different referential corpus (British National Corpus), and our manual inspection showed that our 45 out of 50 of the meta alternations in our SWORME dataset overlap with theirs, which further confirms the reliability of our data collection methods.  \n\nQuestion (b): Although our current framework relies on a rich lexical ontology, our analysis showed that for each polysemous word in the SWORME dataset, the contextualized embeddings of its usages often form separatable clusters in the BERT semantic space, where in each cluster most usages tend to share the same meta-sense label (we did not include this line of results in the paper due to space limit). Therefore, we believe that for any language with a reliable sense-annotated usage corpus, one could automatically construct a meta-sense ontology similar to the CoreLex database in an unsupervised way, which can then be taken as the training corpus to learn language models with higher flexibility in making word meaning extensions. We shall add our analysis results of meta-sense usage clustering and discuss possible future extensions toward low-resource languages in the revised paper.\n\nQuestion (c): Our evaluation results in section 6 suggests that even large language models such as LLaMA could still benefit from learning SWORME to improve its understanding on metaphorical language uses. As (Liu et al., 2022) suggests, LMs that perform better on metaphorical NLI also tend to generate metaphorical expressions of higher quality as judged by human annotators. We therefore hypothesize that learning SWORME may also encourage large LMs to generate more creative language uses, and we will consider performing human evaluations on expressions generated by LMs tuned on the two chaining-based SWORME objectives in future extensions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The introduction of the SWORME framework presents a new approach for systematically extending word meanings, enhancing language models' understanding of language nuances. The paper provides valuable methods of analogical and associative chaining, contributing to the toolbox of NLP researchers. They empirical demonstrate the framework's effectiveness, particularly in improving language models’ performance on figurative language tasks. The introduced SWORME usage dataset enriches resources for studying word sense extension.  Overall, the paper's innovation, empirical insights, and practical benefits make it a valuable contribution to the NLP field.", "reasons_to_reject": "The paper exhibits several commendable strengths; however, it is imperative to acknowledge the potential vulnerabilities and associated risks. Notably, the strong reliance on robust supervision, hinging upon a meticulously annotated dataset for the framework's training, raises the pertinent concern of potential constraints on its applicability and transference to diverse domains and linguistic contexts. It is also paramount to recognize that the intricate process of crafting the dataset for the SWORME framework could potentially engender challenges during both implementation and broader adoption.", "questions_to_authors": "A)\tConsidering the challenges in dataset creation, could you elaborate on any strategies or methodologies employed to ensure the quality and reliability of the SWORME dataset, mitigate bias, and how these efforts might influence the framework's reliability and potential limitations in real-world applications?\nB)\tGiven the emphasis on strong supervision and carefully annotated SWORME datasets, how do you envision the adaptability of the proposed framework to languages and domains with limited resources or access to extensive lexical databases?\nC)\tGiven the framework's focus on systematic word meta-sense extension, do you foresee potential applications beyond figurative language understanding, such as potential impacts on creative language generation or other NLP tasks that involve nuanced word usages?\n", "paper_topic_and_main_contributions": "This paper introduces the \"Systematic Word Meta-Sense Extension\" (SWORME) framework, which aims to to systematically expand word meanings to convey new semantic domains in a predictable manner. The framework uses analogical chaining and associate chaining to predict novel meta-senses for words based on their existing ones. The authors create a SWORME usage dataset from sense-annotated text corpus and empirically show that models trained using analogical chaining on this dataset within the proposed framework outperform those using associative chaining in capturing both gradual and radical word meaning extensions. The paper applies the SWORME framework to enhance language models' understanding of figurative language recognition tasks resulting in improved performance. Overall, the SWORME framework contributes a method for coherent and predictable word meaning extension, with implications for enhancing language models' capabilities in various NLP applications.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 067: “systematically” should be corrected to “systematicity”."}, "extra_text": {}}, {"review_id": "kCZKOiJvab", "reviewer": "Reviewer_zLV7", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and the constructive comments. We will consider human evaluations of the generation outputs of LMs trained on SWORME in the future. We will follow the reviewer’s suggestion to bring up the token substitution procedure earlier in the introduction. The main goal of this paper is to show that transformer-based language models could acquire the ability of making systematic word meaning generalization, and such linguistic creativity can further benefit downstreaming non-literal language processing tasks. We shall revise the introduction section accordingly to highlight our main contributions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiments are set up in a clever and convincing way. \nThere is a clear connection with establish theories of lexical semantics and in particular to Lakoff's theory of chaining.\nThe graphical representations are nice and help a lot.", "reasons_to_reject": "I have some doubts on the fact that the task looks at cases of polysemy and figurative language that are documented. The work does not look at non-documented but possible meaning alternations which could for example be evaluated by human annotators. \nThe overall goal of the paper is a bit unclear, in particular in the introduction - does it aim at testing the properties of BERT language models or rather at using language models as a model to learn something about sense alternations and productivity?\nThe qualitative analysis is rather brief.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work present a new task aimed at testing language models on their ability to understand regular polysemy and specifically to \"discover\" new possible uses of words to a new meaning domain (given some semantic relation with the existing domain). For example, if the model has never seen steal + idea but know that ideas can be grabbed and both ideas and purses can be grabbed and stolen, can the model understand steal + idea?\n\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The token substitution and the rationale behind it could be anticipated at least intuitively much sooner, for example in the introduction. Until then, the reader is a bit puzzled about what constitutes \"novel\" figurative uses.\nThe paper is conceptually very dense, a few things could be made clearer, such as\n- overall goal\n- what is meant by feature proximity\n- different sense alternations (please provide more examples)"}}, {"review_id": "dFJKG17bPT", "reviewer": "Reviewer_rNR6", "summary": "", "questions": "Question A: How do you handle words that require multiple tokens?\nQuestion B: When you label verbs or adjectives you mention the label comes from its \"syntactic noun object\"---I assume that for adjectives this means the noun it modifies, and for verbs its direct object? Do you have a different way of handling intransitive verbs? \nQuestion C: While your precision on the evaluation is much higher after SWORME training than before, it seems far too low to be qualified as systematic. On the other hand, the evaluation task is quite hard since it requires not only learning a) the meta-sense analogy but b) the underlying relationship between the two novel, partitioned tokens. I suspect b) is quite hard, particularly for more exotic sense-pairs than \"animal-food\". Do you have any idea how these two problems could be dissociated or otherwise how to determine how well a truly systematic model would perform?\nQuestion D: Could the failure of the associative-chaining loss just be a result of the fact you use a random unrelated word as the negative sense example? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and helpful comments, and here we would like to address the main concerns raised by the reviewer in the “Questions For The Authors” section:\n\nQuestion (a): for words with multiple tokens, we would replace all of its constituent tokens with a single new token added into the tokenizer vocabulary. \n\nQuestion (b): yes for adjectives, the noun arguments are their modifying nouns, and for transitive verbs the arguments are their direct objects. We did not consider intransitive verb usages as they were not included in the corpus by (Yu and Xu, 2023), but we believe that one can still distinguish the meta-sense of an intransitive verb by looking at the semantic category of its syntactic object (e.g. a literal usage “**the person** arrives” vs. a metaphorical usage “**the pandemic** arrives”). \n\nQuestion (c): we believe that in our usage-based framework, the two tasks of learning systematic meta-sense analogy and relating partitioned novel tokens are inseparable, as both the source and the target tokens are required to represent the meaning of the two systematically related meta-senses, while merging them into one token would result in information smuggling. We think that the performance of human speakers on SWORME may serve as a “truly systematic” standard to compare with, but unfortunately we could not find a straightforward way to test whether human speakers can “come up with” those non-literal yet mostly established expressions in our dataset. \n\nQuestion (d): we tried several alternative sampling methods of candidate negative words, such as 1) taking the top-100 partitioned tokens with most similar static embeddings compared to the target token, and 2) sampling partitioned tokens sharing at least one meta-sense with the target token. We found that in both cases the overall results do not change, and we will add these details in the final revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The approach has an interesting cognitive basis that is well-explained and is relatively straightforward to implement and understand. \n- The paper is well-written and with several different robust baselines showing the effectiveness of the approach.\n- The contrast between both analogical and associative chaining in both their evaluation task and downstream NLI tasks is interesting and helps put the successes of the analogical approach in context.\n- The increase of performance even for larger language models like LLaMA-7B show that the approach is not limited to smaller models, even if the gains are more modest. ", "reasons_to_reject": "- It's somewhat unclear whether this approach is scalable to larger models trained on more data as the kinds of sense-tagged datasets that the model is trained on are not particularly common and are relatively expensive to make.\n- The paper could potentially be improved by seeing if there were performance regressions on non-figurative language tasks compared to the original language models.", "questions_to_authors": "Question A: How do you handle words that require multiple tokens?\nQuestion B: When you label verbs or adjectives you mention the label comes from its \"syntactic noun object\"---I assume that for adjectives this means the noun it modifies, and for verbs its direct object? Do you have a different way of handling intransitive verbs? \nQuestion C: While your precision on the evaluation is much higher after SWORME training than before, it seems far too low to be qualified as systematic. On the other hand, the evaluation task is quite hard since it requires not only learning a) the meta-sense analogy but b) the underlying relationship between the two novel, partitioned tokens. I suspect b) is quite hard, particularly for more exotic sense-pairs than \"animal-food\". Do you have any idea how these two problems could be dissociated or otherwise how to determine how well a truly systematic model would perform?\nQuestion D: Could the failure of the associative-chaining loss just be a result of the fact you use a random unrelated word as the negative sense example? ", "paper_topic_and_main_contributions": "The paper introduces two new learning objectives which can serve as fine-tuning for pretrained language models which aims to help the models learn the systematic metaphorical relations between different uses of the same word, taking inspiration from both linguistic and cognitive models. For example, \"chicken\" can either be used to refer to an animal, or the food that is produced using that animal. Conversely, words like beef and cow do not have the same word form, but have an analogous relationship to chicken_animal and chicken_food. \n\nThe fine-tuning method consists of finding some polysemous words like \"chicken\" and splitting all references into new tokens for each sense of the word, e.g. all uses of chicken are replaced by chicken_animal or chicken_food. The model is trained with a mixture of partitioned (test words) and unpartitioned words (train words).  The ratio between partitioned and unpartitioned is a hyperparameter and they train several models with different ratios. The novel losses are used to ensure that systematic metaphoric modifications are learnt between the *unpartitioned* words---partitioned words only are used with typical MLM loss. To evaluate whether the learnt approach is systematic, they take the partitioned words and compare substitutions of alternative tokens. The model is successful if \"chicken_animal\" has the closest contextualised embedding when replacing \"chicken_food\" out of 100 candidate tokens. \n\nThe two losses are analogical and associative chaining.\n\nAnalogical chaining consists of ensuring that the offset between  the mean contextual embedding for chicken_animal and the mean contextual embedding for chicken_food should be similar to the offset between lamb_animal and lamb_food, and trying to minimize a simple squared L2 norm between the different sense pairs. \n\nAssociative chaining uses a contrastive learning approach where triplets of senses $(m, m+, m-)$ are used where $m$ and $m+$ are valid alternations e.g (animal to food) and $m$ to $m-$ is not (animal to location). While there are examples of the same word, $w$, with both $m$ and $m+$, there is not by definition a use of $w$ with sense $m-$, so a different word, $w'$ is used for $m-$. This loss minimises the distance between $w_m$ and $w_{m+}$ while maximising the distance between $w_m$ and $w'_{m-}$.\n\nTheir results show that they can obtain decent precision (0.35) on the evaluation tasks once they have a high ratio of training words to test words, and that in all cases, their new loss improves the evaluation performance considerably over a normal BERT MLM loss on the same corpus data (e.g. with the novel partitioned tokens). However, the analogical loss performs much better than the associative loss, even if both are better than simple MLM. \n\nThey also show that this approach is useful to improve performance on training NLI models for figurative languages across a range of base models (BERT-base, GPT2-XL, LLaMa-7B). Crucially however, only the analogical chaining loss improves performance. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "-page 8, line 571: \"sarcasms\", Sarcasm cannot normally be pluralised and I don't think the quote in question is really sarcastic rather just humorous since the statement is not meant to be false, it merely says that the place was as joyful as a funeral, which is of course, not particularly joyful.  \n\n"}}]

Author rebuttal:

Meta-review: The reviewers unanimously agree that the paper is both sound and exciting. The SWORME framework it introduces can improve large language models' understanding of polysemy and sense extension processes, while being easy to understand and clearly explained. The approach is well based in linguistic and cognitive science, making it even more interesting.
Decision: Accept-Main