IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions
Authors: Ziheng Zeng
Kellen Tan Cheng
Srihari Venkat Nanniyur
Jianing Zhou
Suma Bhat
Venue: EMNLP 2023 Main (2023)
Keywords: idiomatic expression
figurative semantics
commonsense knowledge
idiomatic expression comprehension
natural language understanding
TL;DR: We proposed and constructed a commonsense knowledge graph with a focus on the figurative meanings of idiomatic expressions and we show that such a graph is useful for enabling language the ability of idiomatic expression comprehension.

Abstract: Idiomatic expression (IE) processing and comprehension have challenged pre-trained language models (PTLMs) because their meanings are non-compositional. Unlike prior works that enable IE comprehension through fine-tuning PTLMs with sentences containing IEs, in this work, we construct IEKG, a commonsense knowledge graph for figurative interpretations of IEs. This extends the established ${ATOMIC}_{20}^{20}$ converting PTLMs into knowledge models (KMs) that encode and infer commonsense knowledge related to IE use. Experiments show that various PTLMs can be converted into KMs with IEKG. We verify the quality of IEKG and the ability of the trained KMs with automatic and human evaluation. Through applications in natural language understanding, we show that a PTLM injected with knowledge from IEKG exhibits improved IE comprehension ability and can generalize to IEs unseen during training.

IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions
Ziheng Zeng' Kellen Tan Cheng² Srihari Venkat Nanniyur' Jianing Zhou' Suma Bhat'
'University of Illinois Urbana-Champaign PPrinceton University
WWasiingoon University in St. Louis
[zzeng13, 3, 2 spbhat2)@111inois. edu
ke weeleentneerriii wnc.aneeeppinnttt..oom inceton. edu, Psnamiyur@@gail.coom com
Abstract
ings of its components (Baldwin and Kim, 2010). For example, consider the case of sentiment clas- sifier that incorrectly recognizes the negative senti- ment in the statement They have stirred up a hor- ner's nest". by failing to account for the figura- tive interpretation of the IE (Balahur et al., 2010). This study focuses on injecting IE-related knowl- edge into small-frame PTLMs known for their wide use, such as BERT (Devlin et al., 2019) and BART (Lewis et al., 2020), considering their struggle understand the figurative meanings of IEs (Bhar- gava and Ng, 2022: Zeng and Bhat, 2022). We dis- cuss the corresponding capabilities of large PTLMs, such as GPT-3.5, in the limitation section.
Idiomatic expression (IE) processing and com- prehension have challenged pre-trained lan- guage models (PT).Ms) because their mean- ings are non co-mossiiiioal.. Unlike prior works that enable IE comprehension through fine- tuning PTLMs with sentences containing IEs, this work, we construct IEKG, common- sense knowledge graph for figurative interpre- tations of TEs. This extends the established ATOMIC28
20 (Hwang et al., 2021) graph, con- verting PTLMs into knowledge models (KMs) that encode and infer commonsense knowledge related to IE use. Experiments show that var- ious PTLMs can be converted into KMs with IEKG. We verify the quality of IEKG and the ability of the trained KMs with automatic and human evaluation. Through applications in nat- ural language understanding, we show that PTLM injected with knowledge from IEKG ex- hibits improved IE comprehension ability and can generalize to IEs unseen during training.
Towards enabling IE comprehension, prior ef- forts range from automatic idiom-detection meth- ods (Liu and Hwa, 2019; Škvore et al., 2022; Zhou et al., 2021) to learning IE representations, all us- ing sentences with IEs (i.e., idiomatic sentences), where IE-specific information would be learned im- plicitly from limited samples and sparse contextual information (Zeng and Bhat, 2022). For instance. MAGPIE (Haagsma et al., 2020), the largest-to- date corpus for IEs, spans 1,755 distinct IEs, where staggering 81% of them each have less than 50 idiomatic sentences (IEs are individually rare in natural language), and certain idiomatic sentences therein contain little to no contextual information to allow models to learn the lEss meanings (e.g.. The thing was on the blink again.) ). As such, prior work on IE embedding learning relied on the auxiliary aid of IE dictionary definitions to augment the infor- mation available in idiomatic sentences (Zeng and Bhat, 2022). An alternative approach of explicitly learning commonsense knowledge about IEs with- out significantly scaling the data/parameters consti- tutes the crux of this study. We rely on psycholin- guistic findings about the impact of IE-related as- pects, such as mental states, emotions, and likely
1 Introduction
Idiomatic expressions (IEs) are frequently used in natural language and include a variety of versa- tile figures of speech that improve language flu- ency and conciseness in multiple genres (Moon et al.. 1998; Baldwin and Kim, 2010: Haagsma et al., 2020). Prior NLP research in IE process- ing has focused on detecting idiomaticity ((Liu, 2019: Zeng and Bhat, 2021) among others), in- cluding tasks and applications that require IE com- prehension, e.g., sentiment classification (Biddle et al., 2020), machine translation (Fadaee et al., 2018), natural language inference (Stowe et al., 2022; Chakrabarty et al., 2022b), and dialog sys- tems (Jhamtani et al., 2021). IE comprchension poses challenges to NLP systems (Sag al., 2002; Tayyar Madabushi al., 2021; Stowe et al., 2022) mainly owing to their failure to account for TEs' characteristic non compositionality, i.c., the mean- ing of an expression is not derivable from the mean-
'The idiom "Stir a horeet's nest" is used to convey the idca of causing trouble or inciting a commation.

---

fects, causes, and attributes for both the subject and the object in an IE (see Section 3). Working closely with human annotators that created 56,315 instances for 1,229 idiomatic events and 11 rela- tion types, I'KG's quality and diversity were then ascertained via human and automatic evaluation.
reclbrocate
do whatever
X pothetic
o2sesaive
a way convince easily
try to provi layality
desperate
X insinrere
a rESSR.
As resuit
licks Y's boots
We exhibit its wide utility via (1) neural knowl- edge models (KMs) trained on IEKG to show IEKG equips PTLMs with IE knowledge that is generalizable to out-of-graph instances; and (2) IE knowledge injection to showcase the usefulness of TEKG for IE comprehension tasks, such as natural language inference (NLI) with IEs and continua- tion classification in which the model decides if a continuation is appropriate for given context with an IE.
Y focks
wanta
bocouss
Y will
to gain fevar with Y
distance from
someone ne else to rollow other
help them
Blucka
Figure I:
Example IE lick someone's 's boot in IEKG.
actions, on human IE comprehension (Rohani et al., and Mashal, 2019), to explore the use of commonsense knowledge about IEs to- wards their comprehension.
The contributions of this work are as follows. (1) We propose IEKG, a commonsense knowledge graph focusing on idiomatically interpreted IEs; (2) We show that IEKG transforms various PTLMs into IE Iw-rwere KMs to infer IE knowledge; com- pared to the ATOMIC20 trained KM, IB-aware KMs can generalize better to unseen IEs (+22% METEOR (Lavie and Denkowski, 2009)) and unseen relations for seen IEs (+30% METEOR) on IE knowledge tuple completion.
2012;
Specifically, we build on the findings that commonsense knowledge graphs (KGs), c.g., ATOMIC (Hwang et al., 2021), organized as if-then relations for inferential knowledge enable linguistic and social reasoning abilities for PTLMs (Bhargava and Ng, 2022). Indeed, models rely- ing on their applications have benefited figura- tive language processing, such as their interpre- tation (Chakrabarty et al., 2022a) and generation (Chakrabarty et al., 2021b). However, the sparse IE coverage in these large-scale KGs limits their wider applicability for IE-related reasoning; as illustra- tion, ATOMIC covers only 347 out of the 1,755 IEs (0%% in MAGPIE, with some contextually semantically ambiguous IEs only annotated with their literal sense while their figurative meanings are either missing or inaccurate. For example, for the IE out sight, the only instance in ATOMIC20 is <AtLocation, arctic>.
(3) We show IEKG endows PTLMs with improved IE comprehension ability; after injecting IEKG, PTLM achieves SOTA results on the IE NLI bench- mark IMPLI (+12% accuracy), and exhibits a sig- nificant increase on the continuation classification task the Figurative Narrative Benchmark com- pared to the baseline PTLM (+25%. accuracy)2
2 Related Work
Datasets with Idiomatic Expressions. Classic datasets for idiomatic sentences, such as VNC (Cook et al., 2008), SemEval5b (Korkontzelos et al., 2013), and MAGPIE (Haagsma et al., 2020), have been the primary source of implicit idiomatic knowledge in tasks such as IE identification (Zeng and Bhat, 2021; Škvore et al., 2022), IE type dis- ambiguation (Feldman and Peng. 2013; Rajani et al., 2014; Peng and Feldman, 2016; Salton et al., 2016; Liu and Hwa, 2017: Taslimipoor et al., 2018; Peng et al., 2014; Liu and Hwa, 2019), and IE representation learning (Zeng and Bhat. 2022). However, the contextual information avail- able in these collections has been sparse and a
Hence, we construct the Idiomatic Expression Knowledge Graph (IEKG), an IE-centered ex- tension to the ATOMIC2) KG, that serves as an instance-efficient and explicit IE-related source of knowledge (compared to the implicit knowledge available in large number of idiomatic sentences) for IE processing. This permits a novel exploration of how TE-related knowledge from a KG can be used for IE-related comprchension in parameter- efficient and sample-limited scenarios.
IEKG G follows the schema of ATOMIC (see Figure 1) and organizes the idiomatic interpreta- tions for IEs into reasoning types covering the ef-
IIKK data and the related code can be found at https: //github. ctto com/zzengl 3/IEKG.

---

non-uniform number of instances per IE for LMs learn their meanings from (Zeng and Bhat, 2022). Datasets of idiomatic sentences are avail- able for specific tasks, such as machine translation (Fadaee et al., 2018), paraphrase generation (Zhou et al., 2021), natural language inference (Stowe et al., 2022; Chakrabarty et al.. 2022b), and lan- guage generation (Chakrabarty et al., 2022a). This study presents an alternative general-purpose IE- related dataset as knowledge graph, aiming to pro- vide more comprehensive, explicit, and instance- efficient way to represent IE usage.
guage generation (Chakrabarty al., 2021b, 2022a; Sabour et al.. 2021). To inject explicit knowledge while preserving the PTLMs' ability as general- purpose LMs, prior works have continued LM pre- training on sentences converted from knowledge tuples (Chang et al., 2020; Agarwal et al., 2021). These broad schemes of KG-PTLMs combinations inspire work to explore linguistic reasoning us- ing the explicit knowledge in the IE-centric IEKG that we construct. Comparing various KG/PTLMs incorporation methods (Hu et al., 2022) constitutes a concrete direction for future work. Finally, others (Phang et al., 2018; Wang et al., 2020; Chang and Lu, 2021) use task-specific knowledge injection before or to replace fine-tuning when the task data is limited. We explore combining task-specific and IEKG G knowledge injection (see Section 5) to able IE comprehension on tasks with limited data.
Commonsense Knowledge Graphs. Knowledge graphs organize information into an ontology as a multi-relational graph of edges and entity nodes (Zou, 2020; Ji et al., 2022). Commonsense KGs, such as ATOMIC (Sap et al., 2019), ConceptNet (Speer et al., 2017), CSKG (llievski et al., 2021), ASER (Zhang et al., 2022), and TRANSOMCS (Zhang al., 2020a), are subset of KGs that focus on generic world commonsense knowledge. Specif- ically, ATOMIC (and its successor ATOMIC20 (Hwang et al., 2021)) utilizes if-then relations to de- scribe rich causal and inferential knowledge span- ning several dimensions, including causes vs. ef- fects, agents vs. themes, and voluntary vs. in- voluntary events. ATOMIC has also been used in figurative language-related applications, such as interpretation and generation of figurative language (e.g., simile, idioms, and metaphor) (Chakrabarty et al., 2021b, 2022a). However, as mentioned in Section 1, ATOMIC 200' coverage of IEs is lim- ited. Through the current work, IEKG extends ATOMIC20
Idiomatic Expression Knowledge Graph
By design, IEKG aligns with ATOMIC 20 gen- eral structure and relation types to describe the linguistic knowledge of TE uses that it enables commonsense inference on figurative language.
3.1 Expressions and Relation Types Selection
Idiomatic event creation. Given a collection of IEs, we first convert them into idiomatic events 3. For example, the IE, add fuel to the fire, is con- verted to the idiomatic event, PersonX adds fuel to the fire. We use the 1,755 IEs from MAGPIE as the initial set and apply a heuristic-based algorithm that automatically converts verbs and pronouns into appropriate and grammatically correct forms while adding subject/object placeholders, such as Per- sonX, to convert all the IEs to their corresponding idiomatic events. For the current scope of IEKG, we focus on IEs with persons as subjects and/or objects. Thus, in the second phase, two members of the research team went through the entire list of idiomatic events (1) to correct any grammatical errors in the idiomatic event and (2) to filter out IEs that cannot be used with persons as subject/object (conflicts were resolved via mutual agreements be- tween the evaluators). Finally, 1,229 IEs (70%)
20 's representation to idiomatic events to further IE-related applications.
Explicit Knowledge Injection using KGs. Tradi- tionally, entities in general KGs are converted into embeddings before being used for downstream ap- plications (Wang et al., 2017: Dai al., 2020). For commonsense KGs, most prior works fol- low the COMET paradigm (Bosselut et al., 2019), where the knowledge model (KM) is trained via the knowledge tuple completion task, where given a tuple <head, relation, tail>, the KM uses the head and relation to predict the tail. The KM, thus trained, becomes on-demand commonsense knowledge query machine equipped with knowl- edge about any head entity. Additionally, it can generalize previously unseen head entities or unseen relation types for a seen head entity, infer- ring additional information to aid tasks, e.g., lan-
"IE" "idiomatic event" are equivalent hereafter due to the one-to-one mapping between them.
"Note that the IEs from MAGPIE are sourced from the British National Corpus (BNC Consortium, 2007), and, as such, predominantly reflect their British usage.

---

are preserved in the IE collection with their corre- sponding idiomatic event created.
tors think of possible contexts and write up to four free-form phrases as tails, given the head (IE) and relations, to complete the annotations.
Relation type selection. To allow IEKG to be- come natural extension to ATOMIC we sc- lect a subset of relation types from ATOMIC20 for IEKG. Several of ATOMIC relation types are not suitable when the subject/object of given event are persons, c.g., the relation type is made of. Hence, we exclude such relation types to se- lect 11 relation types (See Table 5 in Appendix A) that are appropriate for the idiomatic events, cover- ing, for instance, the intent, reaction, effect, causes, and attributes for the subject/object of the events, providing a multifaceted view of IE usage and in- terpretation (See Appendix D for case study).
Quality assurance. We implemented various mea- sures to ascertain annotation quality. During re- cruitment, we conducted annotator interviews to confirm their interest, background, and English pro- ficiency native speakers. Before the annotation phase, we organized info sessions the research background and tutorials on the annotation proce- dure. We prepared detailed instructions with exam- ples completed annotations for ten events and four examples per relation type to demonstrate the definitions and differences among relation types, During the annotation phase, we held weekly office hours to provide clarification as needed.
3.2 Knowledge Graph Construction
3.3 IEKG Statisties
Human annotators. Prior efforts (Hwang al., 2021; Haagsma et al., 2020) construct KGs and idiomatic sentence corpora via crowd-sourcing tools, such as Mechanical Turk, and control qual- ity via numerous performance testing and ongo- ing monitoring schemes. For the construction of IEKG, three accountable, knowledgeable, and na- tive English-speaking college students interested in an undergraduate research experience, volunteered their time. We communicated with the annotators throughout the annotation process, ensuring they understood the expectations, talking to them about research in NLP/ML. with our research projects examples, and, after our study, sharing our ex- perimental results with them. Each annotator per- formed their annotation individually their own pace with no time monetization pressure rush the annotation process. It may appear that our an- notation would lack the diversity of typical crowd- sourced annotation; diversity analysis (see Sec- tion 3.4) indicates that our resulting annotation was reasonably diverse. Thus, we believe that the gain in annotation quality in our con-crowd-sourced ap- proach outweighs the loss in diversity.
IEKG comprises 56,315 knowledge tuples, cover- ing 1,229 idiomatic events and 11 relation types, with a mean annotation (tail) length of 2.98 words and a mean number of knowledge tuples per IE of 45.82 (standard deviation 13.67). On average. each IE has annotations for 7.41 applicable relation types (standard deviation 1.69).
Knowledge tuples are not evenly distributed over the 11 relation types. The most frequent relation type, xEffect, attributes to 18.63% of knowledge tuples, while the least frequent relation type, oWant, appears in only 2.63% of tuples. We account for this uneven distribution later our sampling strat- egy for evaluations.
3.4 Annotation Quality Assessment
As shown in Table 1, IEKG provides more accurate idiomatic interpretations compared to ATOMIC We perform human evaluation and diversity anal- ysis to rate the annotation quality.
Human evaluation. Members of the research team rated a sample of 500 knowledge tuples, with each sample evaluated by three members. To ensure the samples represent the overall quality, we mix anno- tations from all annotators and stratify sample tu- ples, preserving the original tuple distribution over the relation types. We use a 4-point Likert scale as our scoring metric (Hwang et al., 2021), reflecting how reasonable each knowledge tuple is and rang- ing from Invalid (score 4) Always/Often (score 1), with a lower score indicating higher quality. We average the metric scores across human evaluators on each sample and then average across all samples. The final score is 1.51 (inter-annotator agreement
Annotation procedures. To construct TEKG, for each idiomatic event, the annotators: (1) Confirm the understanding of the given IE and the validity of its idiomatic event by first looking up the IE and recording its dictionary definition and then verify- ing the idiomatic event is appropriate for the IE meaning and is grammatical. (2) Select the relation types, as applicable, to avoid forcing unreasonable annotations for unsuitable relation types. (3) Write annotations for each selected relation type; annota-

---

Idiomatic Event
Relation
2
IEKG
xNeed
save up money
a tragic event
PersonX buys the farm
xintent
to grow crop and earn a living.
To pass away peacefully
HinderedByy
The farm cost too much money.
a lack of immediate dangers
Table I: A comparison between ATOMIC and IEKG for the PersonX buys the farm which means die, Unlike
ATOMIC21 IEKG provides accurate idiomatic interpretations.
is 50%), i.e., most annotations are either always or sometimes reasonable, indicating the annotations are of good quality.
sense idiomatic knowledge after training on IEKG. We include the following baseline models (more details in Appendix B):
Diversity analysis. Having established annotation quality, we show annotations have high semantic and lexical diversity across annotators as follows. First, for given event and relation type annotated by all three annotators, we connect the tails from each annotator with "and" into three respective sentences. Then, with these triplcts of annotation sentences, we compute the semantic diversity using average pairwise BERT score (Zhang et al., 2020b) and embedding cosine similarity across all triplets of annotations. The resulting mean/std. BERT score after baseline scaling is 0.42/0.13, and the mean/std. embedding cosine similarity is 1,5 both indicating low semantic similarity among the annotations from different annotators. For lexical diversity, we concatenate each triplet annotation sentences into a single sentence and compute the mean Google-BLEU BLEU score (i.e., mBLEU (Zhang et al., 2020b)) that captures the distinctiveness of the n-grams across annotations and has range between (no match) and (all matches). Our dataset shows an average score of 0.16, indicating a low lexical overlap across annotators.
BART-Comee the baseline 12-layer BART-large model trained on ATOMIC and we use the trained checkpoint released by Hwang et al. (2021).
BART/GPT2TT--EKG are the 12-layer BART- large, GPT2, and T5 models fine-tuned on IEKG for the knowledge tuple completion task. We record the checkpoints with the best Rouge-L score on the test set during training.
4.2 Knowledge Model Experiments
Data. To evaluate KM's ability to generalize to unseen relation types for seen idiomatic events and to idiomatic events, create types of train/test splits of IEKG: (1) relation-type split and (2) IE-rype split. For the relation -ype split, we separate the relation types per idiomatic event and randomly assign a fraction of the relation types (and associated annotations) to a test set, retaining the rest in the train set, thereby ensuring that the two sets do not have overlapping relation types for the same events. This generalization ability is sential because IEKG's annotations for existing idiomatic events could be incomplete, as evidenced by the uneven distribution over the relation types (see 3.2). For the IE-type split, we ensure no over- lap between the TE types in the train and test sets and test a KMs' ability to generalize to IEs outside of the KG. This uses the explicit information from the KG and the implicit language knowledge en- coded in the PTLM and accommodates the need to grow the KG to new IE types. We maintain an 80/20 train/test ratio: for the relation -ype split, we have 45, 10/0/111,12 training/testing tuples; for the IE-type split, we have 45,317/10,999 (983/246 IE types) training/testing tuples.
4 Neural Knowledge Model for IEs
We experiment with KG/PTLM incorporation by training KMs on IEKG and comparing their ca- pability in processing idiomatic events to the KM trained on ATOMIC to showcase the additional IE knowledge that can be learned from IEKG.
4.1 Learning to Infer Knowledge
Our neural KM is a knowledge completion model that performs the knowledge tuple completion task. Various PTLMs that have conditional generation ability (c.g., BART (Lewis et al., 2020), GPT-2 (Radford et al., 2019), and T5 (Raffel et al., 2020)) and previously used for commonsense knowledge learning can converted into KMs with common-
Automatic Evaluation. We used three widely used metrics for language generation evaluate the tail generation quality, i.e., ROUGE (Lin, 2004), ME- TEOR, and BERTscore. For a given test event and relation, we took the l Ee-gennrated tail as the
*Wo used sentence sentence-transforees transformer's best encoder (all-mpnet-basee base-v2) v2) to generate sentence embedding.

---

Model
BERT
BERT
BERT
ROUGE-1
ROUGE-L
METEOR
BART-Comet
0.8737
0.8742
0.1576
0.1532
0.1189
T5-IEKG
0.9274
0.9342
0.9293
0.4415
0.4365
0.3488
GPT2-IEKG
0.9313
0.9285
0.9287
0.4420
0,4397
0.3069
BART-IEKG
0.9484
0.9465
0.9463
0,5688
0,5670
0.4161
(a)
Performance on the relation-type split.
Model
BERT P
BERT
BERT Fl
ROUGE-1
ROUGE-L
METEOR
BART-Comet
0.8745
0.8749
0.8729
0.1664
0.1640
0.1262
T5-IFKG
0.9202
0.9277
0.9222
0.3800
0.3761
0.2914
GPT2-IEKG 0.9254
0.9218
0.9222
0,3902
0.3893
0.2711
BART-IEKG
0.9401
0.9377
0.9377
0.4937
0.4905
0,3474
(h) Performance on the IE-type split.
Table Knowledge models performances by automatic metrics, including BERT score precision (BERT P), recall (BERT R), F1 (BERT FI), ROUGE-I, ROUGE-L, and METEOR score. Best performances are boldfaced.
candidate and used all corresponding annotations as references to compute the scores.
ples from the relation-type split, the average human evaluation score for BART-Comes is 2. 12, while the score for BART-IEKG is 1.65, gaining around 0.47 points (lower better). To put this gain into per- spective, we also check the human evaluators' pref- erences between the outputs from BART-Comet and BART-IEKG. Specifically, given a pair of out- puts from the same test instance, say BART- IEKG's output is preferred over BART-Comet's output if the BART-IEKG's score strictly smaller or equals (the best score) by two or more anno- tators out of three. The evaluators prefer BART- TEKG's outputs on 74% of the test samples. The average evaluation score for BART-IEKG's out- puts on the IE-type split is 1.64, suggesting that BART-IEKG can generalize to unseen IEs and pro- duce reasonable outputs (based the score is less than according human judgment. See Appendix for more example generations.
Human Evaluation. Three annotators of IEKG manually evaluated the tail generation quality. We utilize the same 4-point Likert scoring met- ric for annotation quality assessment as detailed Section 3.4. Finally, we average the manual scores per sample and then average over the in- stances. We sample 100 generated instances from the relation-type test split of BART-Comel (indiffer- ent to IE/relation-type split) and BART-IEKG; and 100 generated instances from the IE-type test split of BART-IEKG. Comparing BART-Comet's gen- assess
erated instances against BART-IEKG's, the quality gain from IEKG. Evaluating BART- IEKG's generated instances on the IE-type split, we assess its ability to generalize to unseen IEs.
4.3 Results and Discussion
As shown in Table 2, and unsurprisingly, all PTLMs trained on IEKG substantially outperform the baseline BART-Comet in the tuple completion task across all metrics and both splits. Compared to BART-Comer, BART-JEKG attains absolute gains of 7.4% in BERT F1, 41% (260% relative) in ROUGE-L, and 27% (225% relative) in METEOR in relation-type split. Considering IE-type split, it achieves comparable gain over BART- Comet with gain of 6.5% in BERT FI, 33% (201% relative) in ROUGE-L, and 22% (174%) in METEOR score. Besides, BART-IEKG outper- forms GPT-2 and b5-based models (which is con- sistent with the results from Hwang et al. (2021)) achieving the best performances across both splits.
Applications
We show how IEKG enables comprehension through two applications that require model to un- derstand IEs' semantics to perform correctly with their largest and most recent benchmark datasets.
5.1 IEKG Injection
Many prior works (e.g., (Chakrabarty et al., l0) combine input sentences and additional common- sense information queried from KM to perform downstream tasks. However, this incurs significant computational overhead and does not endow the PTLM with an innate IE comprehension ability. In- stead, we fine-tune PTLMs using the mask-in infilling objective to imbue a pretrained BART model with
Then, we compare the best-performing BART- IEKG model and the baseline BART-Comet via human evaluation. Quantitatively, for the test sam-

---

explicit 1E-related knowledge from the IEKGS similar to Agarwal et al. (2021). First, we trans- form each knowledge tuple into a two-mask tem- plate. For example, the knowledge tuple <PersonX wins by a hair's 's breadth, xAur; Ambitious>> is con- verted into In PersonX wins by hair's breadth, hair's 's breadth means <MASK>, PersonX is seen <MASK>. The model must infill the first mask with the IE definition and the second mask with the uppropriate inference given the relation type (xAttr in this example). Unlike KMs, IEKG injec- tion allows the PTLM to learn the IE semantics and related commonsense knowledge while preserving its ability to be further fine-tuned for other tasks.
injection followed by fine-tuning on MNLI. Context continuation involves selecting the con- tinuation sentence that makes the most sense with the provided context with an IE. Prior work (Chakrabarty et al., 2021a) illustrated this task's 's difficulty even for very large models, c.g., GPT-3. We aim to determine whether IEKG injec- tion helps identify correct continuations. Specifi- cally,
the Figurative Narrative Benchmark (FNB) dataset (Chakrabarty et al., 2021a), where continuation selection requires correct IE interpre- tation. Each instance consists of a given context with an IE, followed by one correct and one in- correct continuation. We transformed this dataset into binary classification task by generating two examples (an incorrect and correct continuation) from each instance in the dataset, formulated <context, continuation>>. The model must then classify whether the example is correct. We ana- lyze the performance of the idiomatic samples' test split, where the labels are graciously provided by the authors of FNB (Chakrabarty et al., 202
5.2 Tasks in the presence of IEs
Natural Language Inference (NLI) involves de- termining if a given hypothesis agrees with a given premise (entailment, denoted as E) or not (non- entailment, denoted as NE). Prior work (Stowe et al., 2022) shows that the presence of IEs de- grades NLI performance. Thus, our first objective is examine whether IEKG injection benefits NLI in the presence of IEs. Specifically. we use the Id- iomatic and Metaphoric Paired Language Inference (IMPLI) (Stowe et al., 2022) dataset. For training, we use IMPLI's silver split, where samples are taken from MAGPIE, PIE, and SemEval, with la- bels created automatically, to harness their relative abundance compared to the test set (see below). We randomly select an equal number of entailment and non-entailment samples to balance the classes. For testing, we use IMPLI's gold split; these are manually generated samples and are hence more reflective true model performance.
Note that in the above tasks, we have included the two largest and most well-defined publicly available datasets for figurative language compre- hension the moment. In Appendix H, we also describe experiments conducted on two additional datasets for IE comprehension. In these experi- ments, we observe results similar to those presented in the main paper: models equipped with IEKG knowledge injection exhibit enhanced figurative language comprehension abilities. We did not in- clude generic NLU datasets like SNLI, MNLI, or SST because these datasets are not specifically de- signed to assess NLU in the presence of idiomatic expressions.
As second objective, we study the individual effects and interactions of task-specific knowledge injection and task-specific fine-tuning of PTLMs. This process quantifies how much of the perfor- mance gain, if any, can be explained by only task- specific knowledge injection, and how much can be attributed to our IEKG injection. Specifically, some baselines may also be fine-tuned on the MNLI dataset (Williams et al., 2018). with balanced en- tailment and non-entailment samples. Overall, consider 5 possible settings: vanilla PTLM (BART- large). PTLM fine-tuned on MNLI, PTLM with IEKG injection, PTLM fine-tuned on MNLI fol- lowed by IEKG injection, and PTLM with IEKG
5.3 Experimental Setup
Fine-tuning uses a pretrained BART-large model, with all experiments utilizing the same hyperparam- eter settings to maintain consistency across our find- ings. Further details can be found in Appendix E.
5.4 Application Results
Our first objective is to quantify the benefits, if any, of TEKG injection on both the NLI and continua- tion tasks in the presence of IEs. On the NLI task, Table 3 illustrates that IEKG injection results in sig- nificant gains in performance. BART-IEKG outper- forms the pretrained BART by 2.33%, while BART- IEKG-MNLI achieves significant gains of 18.01% 01%. Most importantly, combining IEKG injection with
"Wo also found IEKG-trained KMs are useful for down- stream IE comprehension tasks. See Appendix H.

---

task-specific fine-tuning, which is used to help the model understand the NLI task itself, results in state-of-the-art performance. BART-MNLI-IEKG scores 53.75% on the IMPLI gold samples, which represents a 23.68% increase in accuracy compared to a pretrained BART mode and is still 5.79% bet- ter than BART-MNLI. These performance gains may be partly explained if the MNLI dataset is similar the IMPLI gold samples, which are out- of-distribution (OOD) to the silver samples (as the vanilla BART-large model performs poorly during testing). Fine-tuning on external datasets similar to the OOD dataset has been shown to increase model robustness (Liu et al., 2022). However, MNLI alone cannot fully explain our overall performance, as IEKG injection brings additional improvements. As the IEKG step utilizes the mask-infilling objec- tive, it cannot be the case that this step is similar how we perform inference on the IMPLI gold samples (the IEKG step is generation, not clas- sification). Thus, we conclude that task-specific fine-tuning by itself is inadequate while additional IE knowledge introduced by IEKG injection is nec- essary to achieve the best performance.
much more involved, as it includes both genera- tion and classification, and is followed by what is essentially a 12 inference ensemble for each sam- ple (Chakrabarty et al., 2021a). Our results still demonstrate IEKG utility, as BART-IEKG can per- form quite well without making unnecessary ar- chitecture adjustments or creating large ensembles. Additionally, across both datasets, IEKG injection has shown to be widely applicable any tasks with IEs, and only requires a single fine-tuning step to imbue models with this knowledge.
Interestingly, IEKG injection provides a per- formance gain across IÉs covered and not cov- ered by IEKG. There are marginal gains in the performance of IEKG-covered samples on IM- PLI (84.65% v. 82.39%) and comparable perfor- mance in accuracy on the FNB dataset (77.77% v. 77.81%). The performance on the uncovered idioms was particularly impressive compared to models without IEKG injection. Examining this phenomenon more closely, we had BART-IEKG generate definitions for large set of uncovered IEs to understand how well the model could com- prehend unseen IEs, Additionally, we wanted to see if the unseen IEs were very similar to the IEs present in IEKG. Similarity for these unseen id- ioms was computed in token-wise and semantic- wise similarity with idioms from IEKG. We found little correlation, if any, between the definition qual- ity (as evaluated using the BERTScore) and either token-wise or semantic- wise similarity. Uncovered idioms similar to covered idioms exhibited higher quality definitions, but the converse was not neces- sarily true. See Appendix G for detailed analysis.
Notably, our best-performing model, BART- MNLI-IEKG, achieves a non-entailment accuracy of 68.11%, which is almost double the previous SOTA (34.80%) published with IMPLI (Stowe et al., 2022), outperforming both BART (+15.35%) and BART-MNLI (+33.07%) in accuracy.
Non-entailment performance is crucial barom- eter of true comprehension, as entailment is often easily classified for all models (this may be ex- plained by high token overlap or predicting entail- ment as the majority class). That is why bascline models perform well on entailment samples but struggle greatly on the non-entailment and antonym non-entailment samples (see Appendix F for exam- ples). While poor non-entailment performance was identified as major challenge by the authors of the IMPLI dataset (Stowe et al., 2022), our results demonstrate that IEKG can significantly mitigate this challenge. BART-MNLI-IEKG exhibits strong performance across the board, and its high non- entailment performance implics that our model is truly comprehending the samples instead of apply- ing rudimentary heuristics such as token overlap.
The trends described in this section also hold for the BART-base models (See Appendix F).
Conclusion
We propose TEKG, a knowledge graph that de- scribes the commonsensical, figurative interpreta- tion of IEs. We show various PTLMs can be con- verted into KMs via the tuple completion task by training on IEKG and TEKG imparts useful utility for tasks that require IE comprehension.
On the FNB test split, Table 4 shows that IEKG injection results in a gain of 25.16% in overall performance compared to the baseline. Note that while state-of- arr is around 83%, its method is
Future work should extend IEKG to disam- biguate between literal and figurative IE interpreta- tions, and investigate factors that affect the models' generalizability to IEs not covered by IEKG.

---

Model
Overall Ace.
NE Ace.
ANT Acc.
BART
60.07%
95.83%
52.76%
14.67%
BART-IEKG
62.40%
96.97%
48.82%
22.93%
BART-MNII
77.96%
96.21%
35.04%
81.33%
BART-MNLI-IEKG
83.75%
96.02%
68.11%
77.07%
BARI-IEKG-MNLI
78.08% %
97.53%
32.81%
81.28%
Table 3: A comparison of different BART-large models the IMPLI dataset. E denotes entailment, NE denotes non-entailment, and ANT denotes antonym con-entailment samples. The best performing scores are bolded.
Model
CC Acc.
IC Acc.
tasks such as natural language generation. It would also be interesting to know whether larger models could also exhibit performance gains on OOD id- ioms, as we saw in our experiments. Specifically, it would be useful to examine if larger models could better explain this increased robustness that occurs with IEKG injection. However, due to their inher- ent complexity and costs, we did not make use of these larger models for testing IEKG injection in our experiments.
BART 52.63%
52.40%
52.92%
BART-IEKG
77.79%
78.47%
77.11%
Table 4: A comparison of the effect of IEKG injec- tion with BART-large model on the FigurativeNarra- tiveBenchmark test dataset. Performances for correct (CC) and incorrect (IC) classes are shown. The best performing scores are bolded.
Limitations
Finally, hyperparameter testing was not opti- mized for each individual step in model. It is our belief that while optimizing hyperparameters should yield higher performance, the trends be- tween different pipelines should remain the same (i.e., IEKG injection should still be beneficial). Nonetheless, may also be possible in any ex- perimental setting to achieve better performance with better-tuned hyperparameters.
The main limitation of the current IEKG is that it only provides figurative interpretations of TEs. However, IEs are often contextually ambiguous, i.e., certain IEs can be interpreted literally in spe- cific contexts. Hence, one future direction would be to find ways to include literal IE interpretations and distinguishing them from the figurative seman- tics. Second, this work focuses enabling smaller frame PTLMs (with only millions of parameters) targeting the specific ability of comprehension using a data-efficient method. We are aware of and have experimented with larger PTLMs, e.g.. GPT-3.5 and GPT-4, and noted their IE compre- hension ability. For example, given appropriate prompts, GPT-3.5 can retrieve the definition for given idiomatic event and generate reasonable, commonsense knowledge for a given relation type. However, given the extensive data and computing resources required by models like GPT-3.5, we be- lieve that studying the ability of larger PTLMs is beyond the scope of the present work. Instead, GPT-3.5 could be used to collect and annotate ef- ficiently IEKG-like knowledge graphs for more JEs, reducing the workload and the requirement of human annotators. Due to its unavailability during the time of data annotation for IEKG, we did not use GPT-3.5 produce an even larger KG.
Ethics Statement
In this work, we proposed and constructed a knowl- edge graph dataset, IEKG. During data collection (Section 3, trustworthy undergraduate annotators interested participating in a research experience in the field of NLP, were fully aware of the mo- tivation and intent of the dataset and volunteered their time in creating the dataset while also learning about SOTA NLP models and their challenges. As the annotators are the sole creators of the dataset, there is no concern about infringing existing intel- lectual property or violating privacy rights. The annotators were instructed not to create explicit or toxic data. During our evaluation of the an- notation and case studies, we found no data in- stances that contain toxic language. To further ensure data integrity, combine annotations for the same event and relation type into single sen- tences and used hate speech detector (Vidgen et al., 2021) that covers various types of hate evaluate the toxicity. As a result, only 0.066% (103 out of 15,684 combined annotation sentences)
Additionally, our paper used the NLI and contin- uation classification tasks to understand the effect IEKG injection on IE comprehension. may be relevant for future endeavors to explore other

---

the annotations contain potential hate speech. Man- ual inspection reveals that these samples contain annotations for idioms that describe PersonX nega- tively in nature, such as being "ignorant" and "fool- ish," and no actual targeted hate speech or toxic expressions were found. Hence, we believe that our dataset is safe for release. Additionally, the an- notators are native speakers of standard American English and would not have accounted for differ- ences in background/common knowledge of idiom- related attributes from other Anglish-speaking cul- tures (Acharya et al., 2021).
References
Anurag Acharya, Kartik Talamadupula, and Mark A Fin- layson. 2021. An atlas of cultural commonsense for machine reasoning, In AAAI Conference on Artificial Intelligence.
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 3554-3565, Online. As- sociation for Computational Linguistics.
For our applications in IE comprehension tasks, the intended use case is for the semantic anal- ysis in the presence of IEs, which traditionally would cause misunderstanding for smaller-frame language models. In our experiments, only peer- reviewed and publicly available datasets are used, and no data contains sensitive personal/private in- formation that could potentially breach privacy rights. We did not study biases in our experiments, but we do not anticipate there be additional bi- ases outside of those that pervasively exist in all facets of the English language (our experiments only focused on English datascts). The failure of our model would cause misinterpretation of IE's figurative semantics and thus may result in incor- rect classifications in IE-related natural language understanding tasks. We do not anticipate these models to be deployed in high-risk environments (e.g., financial, medical and culturally-sensitee set- tings) as yet, but if they are, users should beware that model failure could cause semantic misinter- pretation which could be detrimental depending on the actual use case. Finally, we ran moderately sized models that are orders of magnitude smaller than models such as GPT-3.5 or GPT-4. Thus, our environmental impact throughout the process is minimal, and run times were not overly excessive.
Alexandra Balahur, Ralf Steinberger, Mijail Kabadjov, Vanni Zavarella, Frik van der Goot, Matina Halkia, Bruno Pouliquen, and Jenya Belyaeva. 2010. Sen- timent analysis in the news. In Proceedings of the Seventh International Conference on Language Re- sources and Evaluation (LREC' 10). Valletta, Malta. European Language Resources Association (ELRA).
Timothy Baldwin and Su Nam Kim. 2010. Multiword expressions. In Nitin Indurkhya and Fred J. Dam- erau, editors, Handbook of Natural Language Pro- cessing. Second Edition, pages 267-292. Chapman and Hall/CRC.
Prajjwal Bhargava and Vincent Ng. 2022. Common- sense knowledge reasoning and generation with pre-trained language models: survey. CoRR, abs/2201.124336
Rhys Biddle Aditya Joshi, Shaowu Liu, Cecile Paris, and Guandong Xu. 2020. Leveraging sentiment dis- tributions to distinguish figurative from literal health reports on Twitter. In Proceedings of The Web Con- ference 2020, pages 1217-1227.
BNC Consortium. 2007. British national corpus, XML edition. Oxford Text Archive.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Flo- rence, Italy. Association for Computational Linguis- tics.
Acknowledgements
Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. 202 It's not rocket science: Interpreting fig- urative language in narratives. arXiv preprint x1Xiv:2199.00887.
We acknowledge and thank the assistance of Razi Ahmed Khan, Andrew Zhang, and Zachary Kuo in creating and curating the knowledge graph. This research was supported in part by the National Sci- ence Foundation under Grant No. IIS 2230817 and by a U.S. National Science Foundation and Institute of Education Sciences grant (2229612).
Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. 2022a. It's not Rocket Science: Interpreting Fig- urative Language in Narratives. Transactions of the Association for Computational Linguistics, 10:589- 606.
Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022b. FLUTE: Figurative

---

language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (FMNLP), pages 7139-7159. Association for Computational Linguistics.
Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 279- 287.
Tuhin Chakrabarty, Xurui Zhang, Smaranda Muresan, and Nanyun Peng. 2021b. MERMAID: Metaphor generation with symbolism and discriminative de- coding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4250-4261, Online. Association for Computational Linguistics.
Linmei Hu, Zeyi Liu, Ziwang Zhao, Lei Hou, Liqiang Nie, and Juanzi Li. 2022. A survey of knowledge- enhanced pre-trained language models. ArXiv, abs/2212.1 13428.
Jena D. Hwang, Chandra Bhagavatula. Ronan Le Bras, Jeff Da. Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. Comet-atomic 2020: On sym- bolic and neural commonsense knowledge graphs. AAAI
Ting-Yun Yun Chang, Yang Liu, Karthik Gopalakrishnan, Behnam Hedayatmia, Pei Zhou, and Dilek Hakkani- Tur. 2020. Incorporating commonsense knowledge graph in pretrained models for social commonsense tasks. In Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 74-79, Online. Association for Computational Linguistics.
Filip Ilievski, Pedro Szekely, and Bin Zhang. 2021. Cskg: The commonsense knowledge graph. Ex- tended Semantic Web Conference (ESWC).
Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Tay- lor Berg-Kirkpatre 2021. Investigating robustness of dialog models to popular figurative language con- structs. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7485, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Ting-Yur Yun Chang and Chi-Jen Lu. 2021. Rethinking why intermediate-task fine-tuning works. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 706-713, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Shaoxiong Ji. Shirui Pan, Erik Cambria, Pekka Martti- nen, and Philip S Yu. 2022. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33(2):494-51 514.
Paul Cook, Afsaneh Fazly, and Suzanne Stevenson. 2008. The VNC-tokens dataset. In Proceedings of the LREC Workshop Towards a Shared Task for Multiword Expressions (MWE 2008), pages 19-22.
Joannis Korkontzelos, Torsten Zesch, Fabio Massimo Zanzotto, and Chris Biemann. 2013. Semeval-2013 task 5: Evaluating phrasal semantics. In Second Joint Conference Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh Inter- national Workshop on Semantic Evaluation (SemEval 2013), pages 39-47.
Yuanfei Dai, Shiping Wang, Neal N. Xiong, and Wen- zhong Guo. 2020. A survey on knowledge graph em- bedding: Approaches, applications and benchmarks. Electronics, 9(5).
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Alon Lavie and Michael J. Denkowski. 2009, The me- teor metric for automatic evaluation of machine trans- lation. Machine Translation, 23(2-3):105-115.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Matt Errey. 2018 Common English Idioms: ebook by Mat Errey. EnglishClub.
Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2018. Examining the tip of the iceberg: A data set for idiom translation. In Proceedings of the Eleventh In- lernational Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Chin-Yer Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Anna Feldman and Jing Peng. 2013. Automatic detec- tion of idiomatic clauses. In International Confer- ence on Intelligent Text Processing and Computa- tional Linguistics, pages 435-446. Springer.
Changsheng Liu. 2019. Toward Robust and Efficient Interpretations of Idiomatic Expressions in Context. Ph.D. thesis, University of Pittsburgh.

---

Changsheng Liu and Rebecca Hwa. 2017. Represen- tations of context in recognizing the figurative and literal usages of idioms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Sahand Sabour, Chujie Zheng, and Minlie Huang. 2021. Cem: Commonsense-aween empathetic response gen- eration. In AAAI Conference on Artificial Intelli- gence.
Changsheng Liu and Rebecca Hwa. 2019. A general- ized idiom usage recognition model based on seman- tic compatibility. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pages 6738-6745
Ivan Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions: A pain in the neck for NLP. In Interna- tiomal conference on intelligent text processing and computational linguistics, pages 1- 15. Springer.
Nelson F. Liu, Ananya Kumar, Percy Liang, and Robin Jia. 2022. Are sample-efficient nip models more robust?
Giancarlo Salton, Rohert Ross, and John Kelleher. 2016. Idiom token classification using sentential distributed semantics. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 194-204.
Rosamund Moon al. 1998. Fixed Expressions and Id- ioms in English: A Corpus-Bases Approach. Oxford University Press.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan- dra Bhagavatula, Nicholas Lourie. Hannah Rashkin. Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. ATOMIC: an atlas of machine commonsense for if-then reasoming. In The Thirty-Third AAAI Con- serence Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial In- telligence Conference, IAAI 2019, The Ninth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pages 3027-3035. AAAI Press.
Jing Peng and Anna Feldman. 2016. Automatic id- iom recognition with word embeddings. In Infor- mation Management and Big Data- - Second Annual International Symposium, SIMBig 2015, Cusco, Peru, September 2-4, 2015, and Third Annual International Symposium, SIMBig 2016, Cusco, Peru, September 1-3, 2016, Revised Selected Pupers, volume 656 of Communications in Computer and Information Sci- ence, pages 17-29. Springer.
Jing Peng, Anna Feldman, and Ekaterina Vylomova. 2014. Classifying idiomatic and literal expressions using topic models and intensity of emotions. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 2019-2027, Association for Computational Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI17, page 4444 AAAI Press.
Kevin Stowe, Prasetya Utama, and Iryna Gurevych. 2022. IMPLI: Investigating NLI models' perfor- mance on figurative language. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers), pages 5375-5388, Dublin, Ireland. Association for Compu- tational Linguistics.
Jason Phang. Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Shiva Taslimipoor, Omid Rohanian, Ruslan Mitkov, and Afsanch Fazly, 2018 Identification of multiword expressions: A fresh look at modelling and evalua- tion. In Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop, vol- ume 2, page 299. Language Science Press.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the lim- its of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research,
21(140):1-67.
Harish Tayyar Madabushi, Edward Gow-Smith, Car- olina Scarton, and Aline Villavicencio. 2021. "StitchinLanguageeMMe
Nazneen Fatema Rajani, Edaena Salinas, and Raymond Mooney, 2014. Using abstract context to detect figu- rative language.
Dataset and methods for the exploration of idiomaticity in pre- pre-trained lan- guage models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3464-3477, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Gholamreza Rohani, Saced Ketabi, and Mansoor Tavakoli. 2012. The effect of context the efl leam- ers' idiom processing strategies. English Language Teaching (Toronto, Canada), 5(9): 104 114.
Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. Learning from the worst: Dy- namically generated datasets to improve online hate detection. Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics
Ronit Saban-Bezalel and Nira Mashal. 2019. Different factors predict idiom comprehension in children and adolescents with ASD and typical development. J Autism Dev Disord, 49 49(12):740-44

---

and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 1667-1682, Online. Association for Computa- tional Linguisties.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Quan Wang, Zhendong Mao, Bin Wang. and Li Guo. 2017. Knowledge graph embedding: survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12):7724
Jianing Zhou, Ziheng Zeng, Hongyu Gong, and Suma Bhat. 2021. Idiomatic expression paraphrasing with- out strong supervision.
Xiaohan Zou. 2020. A survey application of knowl- edge graph. Journal of Physics: Conference Series. 1487(1):012066.
2743.
Rui Wang, Shijing Si. Guoyin Wang, Lei Zhang. Lawrence Carin, and Ricardo Henao. 2020. Inte- grating task specific information into pretrained lan- guage models for low resource fine tuning. In Find- ings of the A Association for Computational Linguisties: EMNLP 2020, pages 3181-3186, Online. Association for Computational Linguistics.
Tadej Škvore, Polona Gantar, and Marko Robnik- Sikonja. 2022. Mice: Mining idioms with con- textual embeddings. Knowledge- BBssed Systems, 235:107606.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison. Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 48, Online. Association for Computational Linguistics.
Ziheng Zeng and Suma Bhat. 2021. Idiomatic ex- pression identification using semantic compatibility. Transactions of the Association for Computational Linguistics, 9:1546-5562.
Ziheng Zeng and Suma Bhat. 2022. Getting BART
Ride the Idiomatic Train: Learning to Represent Id- iomatic Expressions. Transactions of the Association for Compulational Linguistics, 10:1 120011177
Hongming Zhang, Daniel Khashabi, Yangqiu Song, and Dan Roth. 2020a. Transomes: From linguistic graphs to commonsense knowledge. Proceedings of the Twenty-Ninte International Joint Conference on Arti- ficial Intelligence, IJCAI-20, pages 4004 4010. Inter- national Joint Conferences on Artificial Intelligence Organization. Main track.
Hongming Zhang, Xin Liu, Haojie Pan, Haowen Ke, Jiefu Ou, Tianqing Fang, and Yangqiu Song. 2022. Aser: Towards large-scale commonsense knowledge acquisition via higher-order selectional preference over eventualities. Artif. Intell., 309(C).

---

Relation Type
Meaning
for models with BART as the backbone and 64 for other models. For all training, we used the AdamW optimizer with a learning rate of le-5. All the other hyper-parree parameters are set to their default values. We record the checkpoints with the best Rouge-l. score on the test set during training.
Causes
causes
HinderedBy
Can he hindcred by
oEffect
As result, Y or others will
oReact
As a result, Y or others feels
oWant
As result, Y or others want
XAttr
X is a result, PersonX will Because PersonX wanted But before, PersonX needed
xFfffect
xIntent
xNeed
C Examples of Generated tuples from BART-Comet and BART-IEKG
xReact
As result, PersonX feels As a result, PersonX wants
XWnt
In this section, we show a sample of generated knowledge tuples from BART-Comet and BART- IEKG for the relation-type split (Table 6) and IE- type split (Table 7). As shown in Table 6, observing example outputs from the relation-type test split, we find that BART-Comet often considers IEs liter- ally when the figurative meaning not understood while BART-IEKG generates correct figurative in- terpretations for the idiomatic events. Moreover, BART-IEKG can generalize to certain IEs unseen during training, as shown by the first three rows in Table 7; though when the BART-IEKG fails to un- derstand the IE semantic, it tends also produce literal interpretations as shown by the last three rows in Table
Table 5: Relation types in IEKG and meanings.
A
Relation Types in IEKG
IEKG is constructed to align with the ATOMIC such that the relation types in IEKG is a subset of that in ATOMIC Please refer to Table 5 for the 11 relation types in IEKG and their meanings.
B Knowledge Model Experiment Details
In this section, provide details the base- line knowledge models and experiment settings discussed in Section 4.
BART-Comet is the pre-trained BART-large lan- guage model with 12 Transformer encoder-decoder layers trained on the ATOMIC KG. BART's encoder-decoder structure and denoising pre- training objective make it suitable for language generation tasks and has been shown to achieve the best performance among the KG-adapted PTLMs. We use the trained checkpoint released by Hwang et al. (2021). This model serves as a baseline for idiomatic knowledge inference.
D IE Grouping with IEKG
Grouping semantically related expressions is fre- quently encountered in NLP and naturally ap- plicable to IEs, but remains under-explored ow- ing to the challenges in processing their non- compositionality. Recently, Zeng and Bhat (2022) have used IE definition embeddings to group IES based on their semantic relatedness. We propose an alternative harnessing the IE dimensions conveyed by the relations in IEKG, useful for downstream applications such as paraphrasing. e.g., by finding IEs with similar intents (the xIntent relation). Grouping methods. We use Definition Grouping as a bascline, which computes the definition em- bedding for each IE using a sentence embedding model". Then, for each IE, we find the closest related IEs from the lexicon by computing the pair- wise cosine similarity of all IE definition embed- ding pairs and identifying the IEs with the highest cosine similarities. For IEKG Grouping, we iden- tify related TEs by each relation type. First, we generate embeddings for all knowledge tuples IEKG. To better utilize the sentence embedding model, we convert a given knowledge tuple into a
BART-IEKG is a pre-trained BART-large lan- guage model that is fine-tuned on IEKG using the knowledge tuple completion.
GPT2-IEKG is an auto-regressive language model with 12 Transformer decoder layers, fine-tuned into KM using IEKG with knowledge tuple comple- tion task.
T5-IEKG another 12-layer encoder-decom transformer-base LM that is pre-trained with mul- tiple NLP tasks; we fine-tuned the T5 model similar to BART-IEKG.
Experiment Setup. For all the PTLMs in our ex- periments, we use the HuggingFacce implementa- tions (Wolf et al., 2020). We train the models (ex- cept for BART-Comet, which is already fine-tuned) on IEKG for 50 epochs with a batch size of 16
"We use the same sentence encoder from our diversity analysis.

---

Idiomatic Event
Relation
BART-Comee
BART-IEKG
PersonX tells tales out school
Causes
PersonX not allowed to tell tales
others dislike them
school.
PersonX gets down to brass tacks
HinderedBy
PersonX doesn't have the time to get distractions down to brass lacks.
PersonX turns blind eye
oReact
angry at personx for ignoring them.
grateful
PersonX is out of the running
xNeed
to not want to be in the running
to be in competition
PersonX picks holes
HinderedBy
PersonX doesn't i know how to pick holes.
having strong work ethic
PersonX gets their feet
xintent
play in the water
get better at something
PersouX is treading water xNeed
to get the water
lack interest scared
PersonX sails close to the
oReact
like they have a good friend to sail with.
wind
PersonX rolls with the xlntent
to be good person
to prove themselves
punches
PersonX is on the level xintent
to be on the same level as everyone else
to be fair
Table 6: Knowledge tuples completed by BART-Comet and BART-IEKG on relation-type split.
great guns
Grasp the nettle
in cahoots
joined at the hip
Get Cracking
Kiss and made up same page
Hand glove Right up your street Thick as thieves
Defn
xEffect:
Talk to Persony more
One the same wavelength
Hit the ground running
HinderedBy: Lack agreement
In tandem
xNeed:
Be Friendly Persony
Hindereday:
xNeed:
Lack motivation
Desire
Defn
Home and dry
+
Roll with the punches Burn the candle at both ends
Have it both ways
Strain at the leash
Kill two birds with one stone
Stop at nothing
Burn the midnight
Hand glove
IEs grouped by their definition emheddings (Defin) and relation types from IEKG. IEs from IEKG grouping may not be semantically interchangeable but share similar attributes by relations.
Figure 2:
Idiomatic Event
Relation
Output
ding for the tuple. Next, for any two given IlBs from IEKG, J and I2, and each relation type r, we gather their sets of knowledge tuple embeddings for r as T7, and T7, then, we compute the mean pairwise cosine similarity for pairs from T7, and T as the similarity measure for Jı and Tn w.r.t In the end, for any given IE, we can find a list of the closest related IEs for each relation type in IEKG.
PersonX drops ball xAttr
unreliable
PersonX reaches for the
HinderedBy
laziness
stars
PersonX is on thin ice xWatt
somcone to help them strange
PersonX rub shoulders
xAttr
with PersonY
PersonX falls aslcep at the xReact wheel
sleepy
PersonX is tilting at wind- xEffect consult
mills
experts
Case study. In Figure 2, we show the most closely related IES identified via Definition Grouping and TEKG Grouping for various relation types. For example, for the IE in tandem, the most similar IEs found via Definition Grouping are have it both ways and hand in glove, which are mostly inap- propriate. From IEKG Grouping, however, kiss and make up and on the same page are identified via the relation HinderedBy because a lack of mu- tual goals or agreements or dislike of each other will hinder these IEs; in cahoots and joined at the hip are identified via the relation xNeed because the
Table Knowledge tuples completed by BART-IEKG the IE-type splits. The first three rows show cases where the KM generalizes to unseen IEs while the last three rows show cases where the IEs are not correctly comprehended.
natural sentence by replacing its relation type with its corresponding text as listed in Table and then concatenate the idiomatic event, translated relation type, and its tail into a sentence; then, we use the sentence embedding model to generate an embed-

---

subject needs have a fondness for the object in all three IEs. These results show that IEKG enables the IEs to be related to each other beyond seman- ties from their definitions but also by the shared attributes along each relation type, thus allowing for more comprchensive IE understanding.
gold is thus 60.24%.
The FNB step uses 6, 408 examples as the fine- tuning dataset, taken from an original amount of 3, 204 samples in the FNB training split. The FNB test split consists of 3, 084 examples, taken from an original amount of 1, 542 samples in the FNB test split. Regarding coverage, 902 of the 1,5 1,512 test samples contain IEs that are present in the IEKG knowledge base. The coverage of IEKG on the FNB test split is thus 58.50%.
E Settings for Application Experiments
All fine-tuning steps are conducted at 5 epochs, with learning rate of 2e 5, and weight decay of 0.01. The batch size is 32. All experiments use a random seed of 42.
F BART-Base Model for IMPLI and FNB
For the IEKG injection step, we use beam-search with 4 beams, and maximum new token count of 256, and minimum new token count of 10. The knowledge tuple templates themselves are set at maximum token length of 128, with padding set to the maximum length, and truncation enabled. The dataset consists of 26, 020 templates in the fine- tuning step. The samples are organised such that the model sees all possible idioms in the IEKG dur- ing training, but we withhold some of the relation types for each idiom. For example, if a particular idiom had 7 possible relation types, the model may only see 2 or 3 of these relation types during the IEKG injection step. Note that different relation types are withheld for each idiom to ensure that all relation types are seen across the entirety of the fine-tuning dataset. This ensures that the model both learns the idiom definition and how to apply the idiom semantics towards IE comprehension.
We observe that on the IMPLI gold dataset, the BART model with IEKG injection preceded by task-level MNLI fine-tuning still outperforms its contemporaries, with gain of 2.15% compared the BART model that was only fine-tuned on MNLI, as evidenced by Table There is still a significant gain compared to models without IEKG injection and MNLI fine-tuning, as it outperforms a vanilla BART model by 15.12%. Notably, even in the BART-base scenario, we still demonstrate a signifi- cant gain the non-entailment performance, do- ing approximately 25% better than the original IM- PLI state-of-the-arl, and outperforms its other con- temporaries by at least 7.48% here. This demon- strates that despite model size difference, IEKG injection represents a significant advancement in model comprehension towards understanding the -on-entailment examples that hitherto tfate-ffthhe- art models struggled on. These trends are exactly what we with our results on the BART-large models in Section 5.4.
The MNLI fine-tuning step uses 258, 511 sam- ples to imbue the model with task-level knowl- edge about NL.I These samples are split evenly between 129, 257 samples each for entailment and non-entailment, to ensure against classifier variabil- ity due to class imbalance. The samples are set at a maximum token length of 256, with padding set to the maximum length, and truncation enabled.
Additionally, Table 8 shows that the vanilla pre- trained BART-base model also struggles on the non-entailment and antonym non-entailment sam- ples. As mentioned previously in Section 5.4, en- tailment samples are easy classify, as the pre- trained BART model sees the high token overlap and will predominantly predict entailment. To give one such example, consider the IMPLI entailment sample with the premise and hypothesis as follows: <Th Book of Proverbs makes it clear that hap- piness and discipline go hand in hand from the beginning of our lives 'He who spares the rod hates his son, but he who loves him is careful to dis- cipline him., The Book of Proverbs makes it clear that happiness and discipline are associated with the beginning of our lives : 'He who spares the rod hates his son, but he who loves him careful to discipline him'>. With so many overlapping
The IMPLI step uses 13, 650 samples as the sil- ver fine-tuning dataset. There are 6, 825 entailment samples, and 6, 825 non-entailment plus antonym non-entailment samples. In the IMPLI gold dataset, there are a total of 1, 157 samples, with 528 being entailment, 254 being non-entailment, and 375 be- ing antonym non-entailment samples. On quick note regarding coverage, 697 of the gold samples contain IEs that are present in the IEKG knowledge base Of these 697 samples, 317 are entailment, 147 are non- non-entailment, and 233 are antonym non- entailment. The coverage of IEKG on the IMPLI

---

tokens, is easy for the pretrained BART model to employ a rudimentary heuristic like token overlap to predict entailment. However, this strategy fails on the non-entailment and antonym non non-entailment samples (antonym non-entailment specifically uses the opposite meaning of the IE as opposed to just a random meaning), where the premise and hy- pothesis may still have high token overlap, but cor- rect classification is now dependent on true model comprehension of the IE. For the non-entailment sample: <It was pretty interesting, like one of the guys goes native and one of the guys doesn't tt. It was pretty interesting, like one of the guys goes wild and one of the guys doesn's i,>, there is still high token overlap, but the two sentences are actu- ally not in entailment. However, as the pretrained BART model still predicts entailment, the model fails at true IE comprehension. Similarly, this is also the case for the antonym non-entailment sam- ples, with one such sample as follows: <He prefers acting with other countries to going it alone., He prefers acting with other countries to doing it with all the assistance of others.>. The model's overall poor performance on non-entailment and antonym non-entailment samples, at both the BART-bas and BART-large cases, indicates that pretrained BART models are not actually comprehending the IE meaning in the samples, which why our IEKG injection step is so necessary.
ing factor in IE comprehension. Our results still show a significant gap on the BART-large models however, and still demonstrate some separability, at least for the IMPLI results.
G Analysis of IEKG Injection Robustness
In both the BART-base and BART-large cases, Ta- ble 11 and Table 10 indicate that performance on uncovered idiom samples appears to be pretty good.
To examine why IEKG injection increases model generalizability, we tested the definition generation quality of our IEKG-injected BART-large model. We used a dataset of 2, 166 idioms, 937 of which are not found in the IEKG knowledge base. As ground truth definitions are automatically pulled from the web (Wikitionary and Google Dictionary). they inherently carry some noise. After removing invalid ground truth definitions (e.g.. definitions of the format "alternative spelling altteeate form of"), the result is a total of 899 valid, uncovered idioms.
We then define two notions of similarity that can be used to compare uncovered versus cov- ered idioms. First, token-wise similarity computes the idiom embeddings by passing the idiom to- kens through a sentence embedding model (all- mppet- mpnet-base-v2 from centence-transformee pack- age) model, which generates a (2166, 768) idiom embedding matrix. We then compute the pairwise similarity matrix, of shape (2166, 2166), using the cosine distance the metric of similarity. From here, each idiom is assigned a similarity score, com- puted as the mean of the top-3 highest values for that idiom's 's entry in the pairwise similarity ma- trix. For token-wiss wiiee similarity, we aim to test if the model better generalizes to the unseen idioms that share token-level similarity with the seen idioms. Note that scores are bounded between O and 1, with larger scores indicating higher similarity and vice versa. The same procedure is followed in semantic- wise similarity, except that the idiom embedding is now computed from the idiom's 's ground truth definition, as opposed to the tokens. For semantic- wise similarity, we test if the model generalizes to unseen idioms with seen figurative semantics better.
Similarly to the BART-large case, we see that for the FNB test split, the IEKG injected model still outperforms the vanilla BART model, with a performance gain of 1.66% as seen in Table 9. It is interesting to note that the BART-base models here tend to perform better than the BART-large models, at least in the vanilla case. We hypothesize that at this model size, the models aren't 't able truly sepa- rate since the size constrains the model's 's ability for IE comprehension, and why at larger sizes we see a much more significant gap. Regardless, we see that IEKG injected models still outperform their counterparts across the board on the FNB test split, which follows the trends we saw in Section 5.4.
Finally, we see that, similarly to our results on the BART-larre models, that IEKG injection seems to benefit both samples covered and not covered by IEKG, which adds a level of robustness and makes the model more generaliseable. Once again, we suspect that such small model complexities, sep- arability between models with and without IEKG is more difficult, as the model size itself is a constrain-
As we explained briefly in Section 5.4, there is little correlation between generated definition qual- ity and either token-wise similarity or semantic- wise similarity. From Table 12, we see a marginal correlation between the definition quality and

---

Model
Overall Acc.
E Acc.
NE Acc.
ANT Acc.
BART
56.27%
89.20%
51.97%
12.80%
BART-IEKG
56.87%
93.37%
48.82%
10.93%
BART-MNLI
69.24%
86.91%
30.83%
70.32%
BART-MNLI-EEKG 71.39%
92.05%
59.45%
50.40%
BART-IEKG-MNLI
70.80%
88.61%
33.99%
70.59%
Table 8: A comparison of different BART-base models on the IMPLI dataset. E denotes entailment, NE denotes non-entailment, and ANT denotes antonym entailmil samples. The best performing scores bolded.
Model
Acc.
CC Acc.
Acc.
H IE Comprehension Tasks with IEKG-trained KMMs
BART
62.48%
61.87%
63.10%
BART-IEKG
64.14%
65.24%
63.10%
This section demonstrates that IEKG-trained KMs can also improve the performances for downstream IE comprehension tasks. We show how our KMs enhances PTLM's IE-related knowledge compre- hension ability via concrete tasks related Ilss and figurative language, i.e., the IE comprehension test and the figurative language inference
Table 9: A comparison of the effect of IEKG injection test dataset. Each
on the sample has been split into corresponding correct (CC) and incorrect sample (IC). The best performing scores are bolded.
H.1 IE Comprehension Test
semantic-wise similarity, but less so with token- wise similarity. Thus, similarity plays little role in the generated definition quality, but similar- meaning idioms are slightly more likely to re- ceive better performance. This slight correlation is also eration BERTScores and semantic-wise similarity, shown in Figure Note that correlation here denotes the Pearson correlation coefficient.
In English language learners, the ability to under- stand and use idioms is tested using idiom compre- hension tests, which we adapt here to test PTLMs' idiom-related related reasoning using IEKG.
the scatter plots between the gen-
H.1.1 Task and Data
Here, the comprehension ability is tested by asking learner to select the correct continuation of the first half of sentence containing idiom. As shown in Table 13, given the first half of a sen- tence with an idiom, three possible continuations are presented, and the learner is asked select the option that coheres with the meaning of the idiom. Note that the wrong options misunderstand the id- iom's 's meaning in the context sentence, not neces- sarily confusing the literal and figurative interpre- tations. The idiom comprehension test dataset collected from an idiom study resource by (Errey, 201 8) consisting of 587 unique idiom instances, of which 131 (22.3%) are covered by IEKG. Hence, even with the additional knowledge from IEKG, models need to generalize to out-of-coverage id- ioms to perform well in this task.
Finally, note that in many cases, we observed that low scores were not necessarily an indicator low generalizability. Given the automatic na- ture of the ground truth idiom definitions, in many instances, definitions were either too short, had much different length compared to the generated definition, contained synonyms with the gen- erated definition but exhibited little no token overlap. Thus, many samples with a good defini- tion may have ended up with poor BERTScores. As an example, the idiom "in the raw", with a ground truth definition of "in its true state", had a generated definition of "to be umsophisticated or unrefined.. Despite this definition, this sample's ' BERTScore (precision) was -0.2669. This was just of many samples we examined have good definition, but poor overall score. There- fore, we cannot claim that low similarity (either token-wise or semantic-wist wise) necessarily proves low generalization performance.
H.1.2 Models
We consider a PTLM and two trained KMSs mod- els for performance comparison.
LM-BART the pretrained BART-large language model in its off-the-shelf mode; we consider this

---

Base Model
Overall Ace.
E Acc.
Acc.
ANT
IMPLI TEKG Samples
72.60%
93.06%
61.90%
51.50%
IMPLI Unseen Samples
69.57%
90.52%
56.07%
48.59%
FNB IEKG Samples
62.25%
65.41%
63. 63.08%
N/A
FNB Unseen Samples
63.98%
65.00%
63.13%
N/A
Table 10: A comparison of BART-base performance on samples with IEs covered and uncovered by IEKG. Note denotes entailment, NE denotes non-entailment, and ANT denotes antonym non-entailment, For FNB dataset, E denotes the correct continuation samples, and NE denotes the incorrect continuation samples. The best performing scores are bolded.
Large Model
Overall Acc.
E Acc.
NE Acc.
ANT Acc.
IMPLI IEKG Samples
84.65%
96.21%
69.40%
78.54%
IMPLI Unseen Samples
82.39%
95.73%
66.36%
74.65%
FNB IEKG Samples
77.77%
78.49%
77.05%
N/A
FNB Unseen Samples
77.81%
78.44%
77.19%
N/A
Table comparison BART-large performance on samples with IEs covered and uncovered by IEKG. Note that E denotes entailment, NE denotes non-entailment, and ANT denotes antonym con-entailment. For the FNB dataset, E. denotes the correct continuation samples, and NE denotes the incorrect continuation samples. The best performing scores are bolded.
Similarity
Precision
Recall
F1
H.1.3 Methods
TW
0.0243
0.0487
0.0380
Our experiments are performed to gain insights into linguistic comprehension related to idioms and their ability to generalize knowledge from IEKG to idioms that are not in IEKG. In this experiment, we perform the comprehension test in a zero-shot clas- sification setting, where the LM directly works on the comprehension task without being fine-tuned on any IE comprehension data. Using the following methods, we adapt LM and KM differently to perform zero-shot classification.
SS
0.1324
0.1860
0.0380
Table 12: A comparison of BERTScore correlation with both token-wise (TW) and semantic-wise similarity (SS). The best performing scores bolded.
trained KM
Context Sentence
Options
gave him bonus (V)
Angelo saw it vote of confidence when his hoss
told him to work harder
dismissed
bites people (V)
Tell your kids to steer clear of that dog. It
loves children
LM Method. For LM-BART, we concatenate a given context sentence and each of its three poten- tial continuations and compute the log-loss for each continuation. The continuation with the lowest log loss is taken as the classification result. This zero- shot method has been used in (Chakrabarty et al., 2022a) for other figurative language interpretation tasks.
wags tail
Table 13: Examples of context sentences and their con- tinuation options from the Idiom comprehension test dataset (correct answers marked by v).
model compare the ability of language model that of the knowledge models.
KM Method. For both the KMs, turn the tail prediction task into a classification task to choose the best continuation as follows. First, we observe that most continuations in the dataset pertain to the subject of the given context sentence. Based on this, we select set of six subject-related relation types in the experiment, i.c., we select R [xAttr, XEffect, xIntent, xNeed, xReact, xWant]. Next, given R. set of relation types, and context sen- tence with each of three options of continua- tions O,, E 1, 3, we use the KM to consider the likelihood (correctness) O to be the likelihood
KM-ATOMIC is the BART-large model trained on the ATOMIC20 KG with the officially released checkpoint after one epoch of training.
KM-Fuull is pretrained BART-large model trained both ATOMIC20 and IEKG tuples with the tu- ple completion task for epochs with the same hyper-parameter settings as the KM-ATOMIC model. Comparing KM-Full to KM-ATOMIC al- lows us to assess the utility of the additional IEKG tuples.

---

Table 14: Knowledge models' performance on the com- prehension test in accuracy (%) for IEs. The results are broken down by IEs covered (Cov.) and uncovered (Uneov.) by IEKG. The best performances are bold- faced. (El and E2 refer to training in and 2 epochs, respectively)
Model
Overall
Cov.
Uncov.
LM-BART
37.65
39.69
37.06
KM-ATOMIC
45.32
44.27
45.61
KM-Full EI
50.60
50.38
50.65
KM-Full E2
50.60
51.15
50,43
the correct continuation by the most relations.
H.1.4 Results
As shown Table 14, LM-BART cannot perform the IE comprehension test under the zero-shot setting, achicving an accuracy of 37.6%, which is barely above the random baseline accuracy of 33.3% (choosing one of three choices at random). LM's poor performance in the zero-shot setting is expected and aligns with prior results in sim- ilar classification settings (Hwang et al., 2021; Chakrabarty al., 2022a), showing that PTLM lacks the ability for commonsense and figurative language reasoning without further fine-tuning.
On the other hand, KMs are more competent in this task. Despite learning from KG with a relatively smaller IE coverage and idiomatic amno- tations, KM-ATOMIC curacy, which is a 7.67% accuracy gain over LM- BART. After training with the knowledge tuples from both ATOMIC and IEKG for only one epuch, KM-Full attains an accuracy of 50.6%, with an ad- ditional gain of 5.28% over KM-ATOMIC and an overall gain of 12.95% over the LM-BART. Also, we note that KM-Full's accuracy for IEs that IEKG does not cover slightly above 50%, which is similar the accuracy for the covered IEs. This result shows that KM-Full's comprehension abil- ity is not restricted only to LEs that are seen from TEKG. Though the KM-full's accuracy is relatively low in number, we stress that this performance is achieved without any supervised training, i.e., KM- Full passes the comprehension test for half of the IEs in the test data in zero-shot fashion. We ex- pect the KMs achieve much higher performance with larger dataset for the comprehension test and supervised fine-tuning. Taken together, these re- sults demonstrate (1) KM-Full's ability to carry out the IE comprehension test for certain IEs without
achieve a 45.32%
Figure 3: Scatter plots of generated definition BERTScores (precision, recall, F1) with idiom semantic- wise similarity.
of the knowledge tuple formed by S, O, >,
such that rj R and l.....k. Specifically, for each relation type rj. we use the KM compute three log-losses for the three tuples < C, rj. : O > where E 1, 2, 3 and the O, with the smallest log- loss is taken as the classification result for the rela- tion type rj. Following this process, we produce a classification result for each of the k relation types and then take majority vote produce the final classification result for the context sentence S. In other words, for each relation type rj. we apply the LM method on the three continuation-convente tuples < C, C, O, where E 1,2, and in the end, the classification result is the one chosen as

---

being fine-tuned to the task and (2) the KM's abil- ity to generalize its comprehension IEs unseen during KM training.
inferences for the same relation type with the word
'and'; we format each instance as
" "Premise: [P] 
 Hypothesis: [H] 
 Infer-
One limitation of the current zero-shot method is that every relation type is weighted equally; how- ever, some relation types could be more contextu- ally appropriate than the other relation types and thus should be considered with higher weight. Observing the errors from KM-Full, we also found that the subject-focused relation types are some- times inappropriate since the continuation is re- garding the actions of the object of the sentence. We leave the exploration of potentially more capa- ble zero-shot methods to future research as it is not the main focus of this study.
ence: [I]"
where [I] is the sentence inferred by the KM, Once
we convert all input instances into the above format, we fine-tune base version BART model using the input instances with the inferences and their corre- sponding output classes.
(3) BART-IEKG: In this method, we test the use- fulness of the additional TE-related tuples in TEKG. We first convert a BART-large model into KM by training the model with the knowledge tuple completion task on tuples from both ATOMIC and IEKG; the model is trained until convergence. Then, we use this new KM to replace the KM used in the BART-ATOMIC method to produce infer- ence sentences and reformat the input sentences. Finally, we fine-tune a base version BART model with the inputs with new inference sentences.
H.2 Figurative Language Inference
H.2.1 Task and Data
We use the FLUTE dataset (Chakrabarty et al., 2022b), with 7,534 NLI instances covering four types of figures of speech, i.e., idiom, metaphor, sarcasm, and simile. In partieular, the data contains 479 idioms with a ~588 (278) overlap with IEKG. Hence, to perform well, the model needs to general- ize to idioms not covered by IEKG. Each instance consists of a premise sentence and a hypothesis sentence that includes a figure of speech: the task is to predict if the premise entails or contradicts the hypothesis. We reserve 500 random instances as the test set.
All three models are trained for 25 epochs with a batch size of 32, a learning rate of le-5, an A optimizer, and other hyperparameters in default.
H.2.3 Results and Discussion.
Table 15 presents the results from the best- performing checkpoints. BART-IEKG outperforms BART-BASE by 6.20% in overall accuracy; specif- ically, there is an accuracy gain of 5.61 % for id- ioms, 8.97% for metaphors, and 20.73% for sim- iles. Because the official test set is not released by Chakrabarty et al. (2022b), we report that our results have comparable performance trends with theirs, c.g., models perform the best on sarcasm. Furthermore, we find that BART-IEKG performs well for both the "contradiction" (0.93 F1 score) and the "entailment" (0.90 F1 score) classes, gain- ing 6% and 7% in F1 respectively over to BART- ATOMIC. We identify that BART-IEKG correctly classifies 9% of test instances (9.8% of idiom in- stances) that are misclassified by BART-ATOMIC, while BART-ATOMIC only outperforms BART- TEKG on 2.8% of instances (4.5% of idioms in- stances). A qualitative analysis shows that BART- IEKG produces more accurate inferences when it outperforms BART-ATOMIC. For example, for the hypothesis on "cross examine the witnesses", BART-IEKG infers the narrator is attentive and dutiful and intended to know the truth while BART- ATOMIC infers the narrator is assertive and in- tended to get their point across. The improvement in similes is also significant. We attribute this to the
H.2.2 Method and Models
To demonstrate the usefulness of the inferential knowledge afforded by IEKG, we fine-tune and compare three base version BART models to di- rectly output "Entailment" or 'Contradiction' to perform this task:
(I) BART-Base: We format the input instance
"Premise: [P] 
 Hypothesis: [H]"
where [P] and [H] are the premise and hypothesis sentence, and 
 is a separator token;
(2) BART-ATOMIC: We test the performance of the KM that is trained with the original ATOMIC We take the ATOMIC2 KG with the officially released checkpoint as the inference KM. Then, we use the KM to produce two infer- ences on each of the six relation types, including R [xAttr, XEffect, xIntent, xNeed, xReact, xWant], for the hypothesis sentence in the FLUTE instance, resulting in twelve inferences total. We then convert the inferences into a single sentence using the template shown in Table 5 connecting

---

Table 15: Performances for NLI task with figurative speech terms of overall accuracy (%) and accuracy by each figurative speech type. The best performances bold-faced.
Model
Idiom
Metaphor
Sarcasm
Simile
Overall
BART-BASE
81.31
73.08
98.28
70.73
86.20
ATOMIC
80.37
74.36
98.28
64.63
85.20
BART-IEKG
85,98
83,33
98.71
85.37
91.40
syntactic similarity between idioms and similes- many idioms are conventionalized similes that fol- low the same structure of using "like" or "as", c.g., drink like a fish and drunk as lord; 48 IEs in IEKG follow this structure. In short, KMs trained with the additional IEKG tuples are more capable of comprehending and producing useful inferen- tial knowledge figurative language compared when trained on ATOMIC2 Additionally, the ability generalizes to only unseen idioms but also to figures of speech beyond idioms.
Finally, we also note that our KM makes literal interpretations and sometimes misses the figure of speech, especially in longer sentences, where the KM infers over other events in sentences. We attribute this to the KM's training iomatic events containing single IEs. Future re- search should explore other ways of combining KG and PTLMS improve on natural (non-event) sentences.
short id-


Reviews:
[{"review_id": "yeMLtHZIl5", "reviewer": "Reviewer_Ch1Z", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Question**: Could you please explain the example from Balahur et al. (2010) in more detail for non-native readers? What is the idiomatic reading of “stirred up a hornest’s nest”? Is it supposed to have a positive or a negative sentiment?*\n\nThank you for your review and insightful suggestions! The idiom \"Stir up a hornet's nest\" is used to convey the idea of causing trouble or inciting a commotion. If the figurative meaning of this idiom is not accurately understood, the model may fail to see the negative sentiment of the sentence. In our revised version, we will ensure that readers are provided with clear definitions of the idioms used in our examples. \n\nAdditionally, we will clarify at the outset that our idioms have been sourced from the British National Corpus (BNC), and, as such, predominantly reflect British usage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Introduction of IEKG, a useful resource for English NLP that captures non-compositional readings of (multi-word) idiomatic expressions which are not captured well in \"small frame\" LLMs like BERT and BART due to their non-compositional semantics\n* A thorough description of the annotation process and the resulting dataset including quality control measures that evaluated the linguistic diversity of the annotated \"tails\" and assurance of non-toxicity.\n* The usefulness of the resource is demonstrated by the downstream evaluation of three \"small frame\" LLMs that are fine-tuned on IEKG in a knowledge tuple completion task and by experiments on the performance of LLMs injected with IEKG knowledge and fine-tuned for an NLI classification task.\n* Adaptation of a masking task as an injection method \n* The paper is written in a very clear way and provides thorough discussions of all evaluation results and also the limitations of the dataset and the evaluations (so far not tested to distinguish between idiomatic and non-idiomatic readings).", "reasons_to_reject": "* None", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces the (American-English) Idiomatic Expression Knowledge Graph (IEKG) which extends the existing knowledge graph  ATOMIC 20/20 (Hwang et al. 2021) by 1,229 idiomatic events that are derived from idiomatic expressions in MAGPIE (Haagsma et al. 2020), annotated with up to 11 ATOMIC relations types and up to 4 relation \"tails\" by 3 annotators, resulting in 56,315 instances of <event, relation, tail>. The idiomatic events are restricted to events that contain a human referent as subject and/or object, which also limits the relations types to 11 out of the original ATOMIC relations.\n\nThe dataset is evaluated for its linguistic diversity in terms of ROUGE scores and claim that gain in quality compensates for the loss in diversity.\n\nDownstream evaluations by BART, GPT2, T5 models fine-tuned on IEKG in a knowledge tuple completion task, evaluated by 3 human annotators show that the best model (BART) learned to generalize even to unseen idiomatic expressions\n\nThe paper adapts a masking task as an injection method for \"small frame\" LLMs like BERT and BART such that they can be further fine-tuned for other tasks. Downstream evaluation of different injected/fine-tuned models for natural inference (that is operationalized as a binary classification task) on the IMPLI dataset (Stowe et al. 2022) produce SOTA results.\n\nThe limitation that the models are not explicitly tested on their handling of non-idiomatic readings of idiomatic expressions is addressed by the authors in the future work and limitation sections. \n\n\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Please state early in the paper that the resource is for (American) English (in addition to your discussion in the Ethics section that I appreciate). \n* 043ff.: Could you please explain the example from Balahur et al. (2010) in more detail for non-native readers? What is the idiomatic reading of “stirred up a hornest’s nest”? Is it supposed to have a positive or a negative sentiment?\n\nTypos:\n\n* 098 “IEs ( 20%)”  => delete space after “(“\n* Faulty reference: Moon et al., 1998\"  => is single-authored Moon, 1998: \"Rosamund Moon. 1998. Fixed Expressions and Idioms in English: A Corpus-Based Approach. Oxford => also in the bibliography.\n* 163 “However ...” => There is something grammatically wrong with this sentence /incomplete?\n* 242 “events 2.” => delete space before footnote index\n* 370 and elsewhere “mean/std.” => mean/sd\n\nBibliography:\n\n* There are some incomplete references ,e.g., Radford et al 2019, Rajani et al. 2014, Zhang et al. 2020b, Zhou et al. 2021 – also check for upper case in titles.\n"}}, {"review_id": "4fdH67d9pu", "reviewer": "Reviewer_vAg2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Question**: This paper only compared large models trained on proposed IEKG, and compared the models with/without IEKG on one dataset for natural language inference, and context continuation tasks How about the performance of other models and on other datasets, such as SNLI and MNLI.*\n\nThank you for your review and suggestions. In Section 4, we evaluated various pretrained language models capable of conditional generation, such as GPT-2, T5, and BART, for our experiments. BART demonstrated the best performance, which is consistent with the results from the original ATOMIC paper. Consequently, we chose BART as our backbone model for further experimentation. Our primary objective with the application experiments is to demonstrate the advantages of incorporating IEKG to enhance language models' comprehension abilities for idiomatic expressions. We believe that the benefits and usefulness of IEKG are evident even with only BART in our experiments.\n\nRegarding publicly available benchmark datasets for idiom comprehension tasks, we included two datasets and tasks in the main paper, and the reason more are not included is that there are very limited options. The datasets we utilized in this paper, namely the IMPLI dataset for NLI and the Chakrabarty et al. dataset for context continuation, are currently the largest and most well-defined datasets for figurative language comprehension. In Appendix H, we also described experiments conducted on two additional datasets: (1) an IE comprehension test where, given the first half of a sentence containing an idiom, models are asked to select the continuation that aligns with the idiom's meaning from three possible options in a zero-shot fashion, and (2) the FLUTE NLI dataset, which encompasses a broader range of figurative languages, such as metaphor, simile, sarcasm, and idiom. In these experiments, we observed similar results to those presented in the main paper: models equipped with IEKG knowledge injection exhibit enhanced figurative language comprehension abilities. Due to the limited availability or smaller size of these datasets, we excluded them from the main paper to comply with the page limit.\n\nAs for the absence of generic NLU datasets like SNLI, MNLI, or SST in the paper, these datasets are not specifically designed to assess a model's ability to comprehend idiomatic languages. For instance, SNLI contains a limited number of sentences containing idiomatic expressions. From these sentences, we would need to manually annotate (1) whether the idiomatic expressions are used in their figurative sense instead of their literal sense and (2) if understanding the idiomatic expressions is crucial for classifying the given instance. Unfortunately, we do not currently possess the resources to perform such manual filtering of these datasets, and creating these NLU datasets is beyond the scope of this paper. Nevertheless, we believe that our existing experiments and tasks effectively demonstrate the benefits of IEKG in enhancing IE comprehension.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper describes the process of knowledge graph constraction, including idiomatic event creation, relation type selection, and the assessment of the annotation. Experimental results on IEKG injection, natural language inference, and context continuation show that neural model with this knowledge graph perform better than the model with ATOMIC.", "reasons_to_reject": "This paper only compared large models trained on proposed IEKG, and compared the models with/without IEKG on one dataset for natural language inference, and context continuation tasks . How about the performance of other models and on other datasets, such as SNLI and MNLI.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper constructs a commonsense knowledge graph for idiomatic expressions (IEKG) based on ATOMIC KG.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RXayYAWGbU", "reviewer": "Reviewer_qKwD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Question**: It is batter to utilize a greater number of downstream tasks to evaluate the effectiveness of the dataset.*\n\nThank you for your review and suggestions. Our primary objective with the application experiments is to demonstrate the benefits of incorporating IEKG to enhance language models' comprehension abilities for idiomatic expressions. As a proof of concept, we applied our IEKG injection with BART to four different tasks (two in the main paper and two in the Appendix) specifically designed to assess models' IE comprehension abilities.\n\nThe main reason we did not include more downstream tasks is the limited availability of publicly accessible benchmark datasets for idiom comprehension tasks. The datasets we used in this paper, namely the IMPLI dataset for NLI and the Chakrabarty et al. dataset for context continuation, are currently the largest and most well-defined datasets for figurative language comprehension. In Appendix H, we also described experiments conducted on two additional datasets: (1) an IE comprehension test where, given the first half of a sentence containing an idiom, models are asked to select the continuation that aligns with the idiom's meaning from three possible options in a zero-shot fashion, and (2) the FLUTE NLI dataset, which encompasses a broader range of figurative languages, such as metaphor, simile, sarcasm, and idiom. In these experiments, we observed similar results to those presented in the main paper: models equipped with IEKG knowledge injection exhibit enhanced figurative language comprehension abilities. Due to the limited availability or smaller size of these datasets, we excluded them from the main paper to comply with the page limit.\n\nWe did not include generic NLU datasets like SNLI, MNLI, or SST because these datasets are not specifically designed to assess a model's ability to comprehend idiomatic languages. For instance, SNLI contains a small number of sentences featuring idiomatic expressions. From these sentences, we would need to manually annotate (1) whether the idiomatic expressions are used in their figurative sense instead of their literal sense, and (2) if understanding the idiomatic expressions is crucial for classifying the given instance. Unfortunately, we do not currently possess the resources to perform such manual filtering of these datasets, and creating datasets to test model comprehension abilities is beyond the scope of this paper. Nevertheless, we believe that our existing experiments and tasks effectively demonstrate the benefits of IEKG in enhancing IE comprehension.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper addresses the issue of Idiomatic expression (IE) comprehension by constructing an IEKG (Idiomatic Expression Knowledge Graph). The design of IEKG is extensively described in the article.\n2. Using this dataset to enhance various Pre-trained Language Models (PLMs) shows significant improvements in performance on downstream tasks.", "reasons_to_reject": "It is batter to utilize a greater number of downstream tasks to evaluate the effectiveness of the dataset.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the issue of Idiomatic expression (IE) comprehension by constructing an IEKG (Idiomatic Expression Knowledge Graph). The design of IEKG is extensively described in the article.\n\nUsing this dataset to enhance various Pre-trained Language Models (PLMs) shows significant improvements in performance on downstream tasks.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes the introduction of IEKG, a commonsense knowledge graph extending ATOMIC for the figurative meaning of idiomatic expressions. After describing the annotation process, the authors present some experiments on the benefits on training LMs on IEKG with a knowledge tuple completion task, showing that all LMs fine-tuned with IEKG manage to outperform a BART Comet baseline. Additionally, they present an extrinsic evaluation of the resource, showing that IEKG injection leads to better performances both in a natural language inference and in a context continuation task.

All the reviewers agree on the usefulness of the proposed resource and appreciate the careful description of the annotation process and of the steps taken for quality control. The experimental results seem solid, with the incorporation of IEKG leading to consistent improvements in tuple completion and in the two downstream tasks.
A possible weakness that has been mentioned (R2 and R3) is that the authors could have evaluated their model on a larger number of tasks/datasets (although I must say that I am already positively impressed by the experiments included in the current version of the paper, which also includes additional model evaluation in the Appendix).
Decision: Accept-Main