Enhancing Language Models with Idiomatic Reasoning
Authors: Jianing Zhou
Ziheng Zeng
Hongyu Gong
Suma Bhat
Venue: COLM (2024)
Keywords: idiomatic expressions, multi-view data augmentation, parameter-efficient fine-tuning, meta-pretraining
TL;DR: We propose a parameter-efficient idiomatic knowledge injection framework that imparts models generic knowledge on fundamental idioms properties.

Abstract: Advancements in Large Language Models (LLMs) have significantly propelled the field of Natural Language Processing (NLP); however, nuanced reasoning in the presence of non-canonical language forms, such as figurative language, remains an intricate challenge. These language forms, integral to human communication, elude standard LLM comprehension due to their inherent non-compositionality, contextual ambiguity, and sparse representation in text corpora. Addressing these challenges, this paper introduces an innovative approach to seamlessly incorporate idiomatic knowledge into pre-trained language models (PTLMs). Our methodology first employs a multi-view data augmentation strategy that uses idiomatic instances representing one property to generate training data for various idiom-related tasks. When combined with a novel parameter-efficient tuning mechanism that accounts for the unique attributes of idiomatic language, we embed task-specific and idiomaticity-aware inductive biases within a PTLM.  Integrating a meta-pretraining protocol based on meta-learning principles, further equips the model with enhanced adaptability to diverse downstream idiom-aware tasks. Empirical validation on diverse benchmarks centered around idiom comprehension and reasoning, demonstrates the efficacy of our approach. Notably, our model surpasses various parameter-efficient fine-tuning baselines outperforming the conventional full fine-tuning paradigms, thereby creating more contextually aware and linguistically robust language models.

Published as a conference paper at COLM 2024
Enhancing Language Models with Idiomatic Reasoning
Jianing Zhou
Ziheng Zeng
University of Illinois Urbana-Champaign Urbana, IL 61801, USA 3jn1746eillinos.. edu
University of Illinois Urbana-Champaig Urbana, IL 61801, USA zzeng1 7zeng13eillinois 1eillinoisseeeeeemeeeei
Hongyu Gong Facebook AI Seattle, WA 98109, USA hygongefb. com
Suma Bhat
University of Illinois Chaama-Cha Urbana, IL 61801, USA spbhat2@illinois.com edu
com
Abstract
Advancements in Large Language Models (LLMs) have significantly pro- pelled the field of Natural Language Processing (NLP); however, nuanced reasoning in the presence of non-canonical language forms, such figura- tive language, remains an intricate challenge. These language forms, inte- gral to human communication, elude standard LLM comprehension due their inherent ton-compositionalliy, contextual ambiguity, and sparse representation in text corpora. Addressing these challenges, this paper introduces an innovative approach to seamlessly incorporate idiomatic knowledge into pre-trained language models (PTLMs). Our methodol- ogy first employs novel multi-view data augmentation strategy that uses idiomatic instances representing one property to generate training data for various idiom-related tasks. When combined with a novel parameter- efficient tuning mechanism that accounts for the unique attributes of id- iomatic language, we embed task-specific and idiomaticity-aware inductive biases within a PTLM. Integrating meta-learning principles, further equips the model with enhanced adapt- ability to diverse downstream idiom-aware tasks. Empirical validation on diverse benchmarks centered around idiom comprehension and reasoning, demonstrates the efficacy of our approach. Notably, our model surpasses various pfrfmeter-efficient fine-tuning baselines outperforming the con- ventional full fine-tuning paradigms, thereby creating more contextually aware and linguistically robust language models.
pretrainnng protocol based on
Introduction
Idiomatic expressions (IEs), commonly found in natural language, encompass broad range of dynamic figures of speech that enhance fluency and concision across various genres and play key role in human communication (Moon et al., 1998; Baldwin & Kim, 2010a). However, pre-trained language models (PTLMs), such as BART (Lewis et al., 2020), T5-large (Raffel et al., 2020), and TO, struggle to reason in the presence of IEs (Chakrabarty et al., 2021b), failing to understand their figurative semantics (Bhargava & Ng, 2022; Zeng & Bhat, 2022). Despite their impressive scale of pre-training, even large language models (LLMs) such as GPT-4 and LlaMa2 struggle to reason effectively in the presence of figurative expressions (Liu et al., 2023; Kabra et al., 2023). This in part due to LEs' long tail distribution in natural language, their non--comositioonaltty i.e., the overall meaning of an IE is not directly derived from its individual components (Sag et al., 2002), and their contextual ambiguity, i.e., an IE can be interpreted idiomatically literally depending on the context (Baldwin & Kim, 2010b). Beyond this, it is known that comprehension of idioms requires a broad linguistic awareness of the relationship between meaning and surface form, apart from commonsense
1

---

Published as a conference paper at COLM 2024
Continued Pre-training
Downstresm Fine-tuning
IMPLI
Idiom Identification
FNB
Parameter-EMiee Efficient Meta: Protraining
Dataset
capturing some
Model with Idapter
Injected Model with Idapter
Idiom Generation
charasteristic (eg. MAGPIE)
Idiom Definition Generation
Entelimen/Cnntraditionn
Data Augmentation
Coreež/Wroog
Figure 1: The overview of our framework.
associated with their use (Cacciari & Levorato, 1989). Therefore, it is necessary to provide idiomatic knowledge to the PTLMs for better reasoning in the presence of idioms.
Recent works (Škvorc et al., 2022; Zeng & Bhat, 2021) have enhanced PTLMs for specific IE tasks, such as IE identification and usage recognition, without aiming for generalizable IE comprehension ability. Others (Zeng & Bhat, 2022; 2023) adapt PTLMM to learn semantically meaningful TE representation, thus rendering PTLMS suitable for multiple downstream IE tasks with limited reasoning abilities. A recurring obstacle encountered by previous studies is incorporating 1E-related knowledge when IE-related data is scarce: MAGPIE (Haagsma et al., 2020), the largest and the most popular IE corpus, contains 56k sentences covering 1755 IEs. As we will show in this study, scaling up pre-traines models with more parameters and larger pre-training data than ever before fails to automatically resolve this deficiency. While open-source LLMs also lack IE reasoning ability, in this study, we show how state-of-the-an IE comprehension and reasoning abilities are achievable by explicitly tackling the data scarcity challenge. Remarkably, we can accomplish this without relying on astronomically expensive pretraining data or parameter up-scaling; instead we adapt smaller-frame L'S to various downstream tasks through meta-learning via novel data augmentation and parameter-effiiient tuning methods.
Figure summarizes our approach. Given a dataset of sentences with IEs, we propose multi-view data augmentation where single dataset is transformed into an array of instances for multiple tasks, each representing an IE-related characteristic, e.g. identification task for learning contextual ambiguity, definition generation task for learning non-compositionality and generation cloze task for learning the usages. Then, we continued pre-train PTLM on these tasks to learn generic IE knowledge. Unlike traditional data augmentation methods that generate more data for the same task generate different data for different tasks that are irrelevant, we use the same data but transform them into different tasks. Sharing the same IEs and the different forms of sentences (as different tasks), permits the model to learn IE-related properties introduced by each task and the shared IE knowledge across tasks.
To address the training efficiency challenge, we utilize parameter-efffciiet fine-tuning (PEFT) during continued pre-training and fine-tuning for downstream tasks. Traditional PEFT methods (Rebuffi et al., 2017; Guo et al., 2021; Hu et al.; Pfeiffer et al., 2020a) are primarily designed to learn from single-task data while ignoring linguistic processes (e.g., non-composition language). In contrast, our data augmentation leads to mixed-task train- ing batches containing exclusively non-compositional language. Therefore, inspired by Adapters (Pfeiffer et al., 2020b), we propose the Idapter. It enables appropriate token repre- sentation by generating a token-level hon-compositionallty scalar to linearly combine the compositional and non-coopoiitional representations in accordance with the its contextual semantics. Further, it enables mixed-task batch training by generating a task-specific scalar to alter the intermediate representations for each task. Finally, towards orienting the model to different downstream tasks, we also propose meta- meta-learning based meta-pree meta-pretraining mechanism that optimally initializes the model parameters for effective downstream use.
To showcase the efficacy of our method, we first continue pre-training the Idapter attached to a frozen backbone PTLM using our multi-view augmented data and meta- meta-pretraining mechanism. Then, we deploy our TE-Knowledge-enhance model on different downstream
2

---

Published as a conference paper at COLM 2024
IE-related tasks (see Figure 1). Empirical results show our method outperforms not only other LMs that are fine-tuned fully or in a parameter-efficiet manner but also open-source LLMs such LlaMa2 and Mistral that are 200 times larger than the models we study.
The contributions of this work are as follows.
(1) We propose framework that equips PTLMs with idiomatic reasoning, which could then be used to solve different downstream IE-aware tasks;
(2) We address IE data scarcity via multi-view data augmentation strategy that transforms
a given dataset into a set of datasets for continued pre-training;
(3) To inject idiomatic intelligence efficiently, we propose Idapter-an idiom-aware PEFT
method designed for continued pre-training with multi-view data augmentation;
(4) To adapt the model different TE-aware tasks effectively, we propose our meta- pretraining mechanism based on meta-learning:
(5) We show the efficacy of our method on diverse set of downstream IE-related tasks. Our Il--nowleegge enhanced model achieves new state-of-the-art results on IE-related tasks.
2 Related Prior Work
Knowledge Injection. Recent studies have shown the promise of integrating explicit external knowledge for transformative impacts on downstream NLP tasks by continued pre-training (Zhang et al., 2019b; Beltagy et al., 2019; Lee et al., 2020; Zhang et al., 2022; Yu et al., 2022). Among them, some works explore to inject idiomatic knowledge into PTLMs via task-specific data augmentation (e.g,, generating sentence pairs for IE paraphrasing (Zhou et al., 2021c)), leveraging external knowledge (Zeng & Bhat, 2022), and learning with parameter efficient methods (Zeng & Bhat, 2023) (e.g., Adapters (Pfeiffer et al., 2020a)). However, prior works have limited utility in that they only focus on a single task (e.g., identifying IEs (Flor& & Klebanov, 2018; Amin et al., 2021; Zeng& & Bhat, 2021), disambiguat- ing between their figurative/ /auuraliveliterall literal use (Liu & Hwa, 2017; 2018) or paraphrasing (Zhou et al., 20)1a;b)). Although Zeng & Bhat (2023) show their model's performance downstream tasks, their method is still limited since they utilize adapters without accom- modating IE-specific characteristics. Besides, their training method does not mitigate the discrepancy between the training objectives and downstream tasks, rendering their method less adaptable to tackle practical downstream tasks. Additionally, none of the prior works show performance improvements in natural language understanding tasks in the presence of [Es; their performance improvements focus solely on processing tasks.
different
Data Augmentation. Data augmentation, generating synthetic task-specific data, is common, well-studied approach in the NLP community to mitigate data scarcity issues (Feng et 2021). The augmented data can be derived from labeled data and directly utilized in supervised learning (Wei & Zou, 2019) or in supiriupe learning for unlabeled data (Xie et al., 2020). While several approaches have been proposed to address learning with limited labeled data, such as unsupervised pre-training (Devlim lin et al., 2019; Raffel et al., 2020), multi-task learning (Liu et al., 2017; Augenstein et al., 2018) and semi-supervised learning (Miyato et al.; Xie et al., 2020), these methods have limitations in IEs' domain due to IEs' non-compositionality which complicates the generation of new texts. Consequently, we introduce a novel multi-view data augmentation technique that circumvents the need to generate new texts with IEs by transforming the original texts into different tasks.
Fine-tuning. While conventional fine-tuning updates all parameters of PTLMs, recent research has shown that updating or incorporating a minute fraction (e.g., 0,01%) of the parameters can achieve performance comparable to that of full fine-tuning. Beginning with Adapters (Rebuffi et al., 2017), a variety of advanced Parameter Efficient Fine-Tuning (PEFT) strategies have been developed, including selecting a sparse subset of parameters for training (Guo et al., 2021), low-rank updates (Hu et al.), implementing optimization in lesser-dimensional subspace (Aghajanyan et al., 2021), incorporate low- rank Adapters using hypercomplex multiplication (Karimi Mahabadi et al., 2021). However, all prior PEFT strategies are designed for the more generic, compositional expressions, without any special consideration for non-compositional figures of speech such as IEs. More critically, we hypothesize that considering the figurative meanings and degrees of
Parameter-Efficient
3

---

Published as a conference paper at COLM 2024
Idiom Identification
Input: What is the idiomatic expression in the sentence: It looks like the temperature is going to drop tonight, , so be careful not catch a cold.
Output: catch a cold
Idiom Generation
Imput: Fill in the mask with an idiomatic expression in the sentence: It looks like the temperature going drop tonight, so be careful not to [MASK].
Output: catch a cold
Idiom Definition Generation
Impul: What is the definition of the idiom 'catch cold' in the sentence: It looks like the temperature going to drop tonight, so be careful not to catch a cold.
Output: become infected with cold
Table 1: Examples of input sentences and output sentences in our tasks. Non-compositional expressions are highlighted in bold red
non-compositionality of IEs to be unique to each IE, learning each IE can be considered a separate subtask. This renders prior PEFT methods impractical for mixed-task training batches produced by our multi-view data augmentation since these methods require isolated modules for distinct subtasks. Thus, we propose Idapter with non-compooitionaaliy and task-specific scaling to aid idiomatic knowledge injection.
Meta-learning. The concept of meta-learning, also known as "learning to learn," has garnered substantial attention, resulting in a vast body of literature. Among the various meta-learning frameworks, Model-Agnostic Meta-Learning (MAML) stands out Finn et al. (2017). Notably, MAML-related techniques have found applications across a range of natural language processing (NLP) tasks (Dou et al., 2019; Kang et al., 2020; Qian et al., 2021; Huang et al., 2023). Despite these advancements, a critical gap remains: the application of meta-learning to pre-training of language models. While some NLP' researchers have explored learning (Ke et al., 2021; Wu et al., 2022), few have delved into its potential for domain-specific knowledge injection. Our work aims to address this gap by investigating how meta- meta-learning can enhance the pre-training of domain-specifie language models, filling niche that has yet to be fully explored in the literature.
3 Multi-View Data Augmentation
Traditionally, PTLMs rely on large-scale corpora as implicit sources of domain-specific knowledge (Lee et al., 2020). However, the long tail nature of idioms and the unavailability of comparable sources focusing on idiomatic expressions- the largest-to-date collection of sentences with IEs, MAGPIE, is made of only 56k sentences of non-uniform quality of context spanning 1,755 IEs wisth sufficient contexts from which to learn their meanings, calls for novel approaches to idiomatic knowledge injection.
We propose multi-view data augmentation technique in which available dataset of idiomatic sentences, such as MAGPIE, is augmented via different tasks derived from this dataset, to provide diverse range of idiomatic knowledge. Specifically, given the MAGPIE sentences as input, we create datasets for different tasks by tailoring the corresponding outputs to these ends. To inform the model of IE non-composiiionnnng non-compositionality and the associated figurative meaning, we create the definition generation task, in which the model is asked to produce the dictionary definition of the IE from the input. To impart TEs' contextual ambiguity, we introduce the idiom identification task in which the model is expected to extract the IÈs that are used in their idiomatic sense (e.g., People talk about Alice behind her back.), while ignoring IEs that are used in their literal sense (e.g., Alice put her bag behind her back.), outputting an empty string. Finally, to allow the model to recognize proper IE uses in given context, we add the idiom generation cloze task in which, given the contextual cues, the model is expected to fill in a mask using the IE with an appropriate figurative meaning. We resort to these tasks because, despite the pretraining objective of using the context to
4

---

Published as a conference paper at COLM 2024
000000
00o000
OOoo0o
Sealing
Foodforward cutiut
Hypernsworkk
Oooo0o
Feedforwart-
intermedatte
Feedfonward ip-pinject
Linoar Layer
00
Resetonnant Non byn-coopaattooodly Scaling
Muff--Head
Mean
down-prject
000000
000000
000000
SASA A
tokani
1000000
1000000
of acthattio sentence
Figure? The overview Idapter. For each token, non-compositioonaliy scaling is used for figurative semantics and task-specific scaling is used for training with mixed-tast batches.
predict given Laken/words the emerging understanding is that this is insufficient to learn to represent idioms and much less to reason in their presence (Miletić & Walde, 2024).
As a result, for each sentence in the dataset that uses the IE idiomatically, we create three expected outputs corresponding to these three tasks. A sentence where the IE is used literally gets used only for the idiom identification task.
As an illustration, the input/ / output pairs of these three tasks for an input sentence with the IE catch cold are shown in Table 1. Since all outputs are generated from the same original IE sentence, we hypothesize that by learning these tasks jointly, the model effectively learns comprehensive knowledge about the (common) IE, catch cold, shared across tasks. In contrast to the traditional data augmentation methods that aim to generate more data for a single task and the traditional multitask learning methods that utilize different data for different tasks, our method creates different tasks from the same data, thus allowing more effective data utilization in a low-resuurce scenario.
4 Parameter-efficient Training for Idiomatic Knowledge
Advancements in language models have significantly scaled up the number of parameters and the pretraining corpus. However, injecting new knowledge into large PTLMs using PEFT methods, rather than full fine-tuning, makes for a more efficient model. In this paper, we present an improved PEFT method that, instead of being fask/rooesst agnostic, incorporates considerations for non-compositional language, thus enabling the training of multiple tasks simultaneously. Specifically, building upon the Adapter framework, we propose an Idapter, novel PEFT method designed to inject idiomatic knowledge into PTLMs using the multi-view augmented data.
Adapters. An Adapter is lightweight neural network placed between inside the intermediate transformer layers of a PTLM. During training, only the Adapter weights are trainable, while the PTLM's parameters frozen to improve efficiency and reduce training costs. Similarly, we use the paradigm where the Adapters are placed inside the intermediate layers. With our focus idiomatic knowledge injection, the Adapter is expected to learn different representation he for each token to incorporate the potential non-compositiinall meaning given the original representation h; E I2 from the frozen PTTLM:
h Adapter(h;)
coaacompositionalityy Scaling. The Idapter takes into account the non-compositionality in its architecture. It has non-compositionalliy scaling that generates non-compositionality

---

Published as a conference paper at COLM 2024
score s/, for token i:
s/1 Wi,h; bn
where Wn E Rlxd and bn the trainable parameters of the non-compositionaliy scaling layer (ı stands for "ton-compositionalty") Then, for each token, the final representation is the linear combination of the Adapter representation, h,, scaled by the non-compositionalliy score, Sn, and the original PTLM representation scaled by 1 - Sp:
h (1 s h s/
where refers to token i.
We include the idiom identification task in our continued pre-training to teach the model about idioms' contextual ambiguity and thus disambiguate each token as part of com- positional or non-compositional phrase. Therefore, learning the idiom identification task optimizes the parameters of the non-compositionality scaling layer (referred to as NC layer).
Task-specific Scaling. Because our data augmentation method formats data into different tasks, mrixed-task batches used during training, which is second point of departure from traditional PEFT methods. The Idapter aids mixed-task batch training by learning scalar vector for each task to help the model distinguish and optimize each task separately. Each task-specific vector is generated by dedicated, task-specific hypernetwork used only for the inputs from its corresponding task. We consider simple, efficient linear layers as hypernetworks that are functions of the original PTLM's representations, h, E Id, where f denotes specific task in our knowledge injection phase. We implement these hypernetworks after each transformer layer from the PTLM. The hypernetwork H(-)) that generates the task-specific vector I, E R is defined as:
4 H(;, W
where WH Rexd are the trainable parameters of the hypernetwork. The task-specific vectors are then multiplied by the sequence representations obtained after the non- compositionality scaling as:
h = 4 h hh
where O represents element-wise multiplication.
In effect, inputs from different tasks will be represented differently by Idapter using task- specific scaling (referred as TS layer), making learning mixed-task batches possible.
Meta-Pretraining
We argue that most PTLMs' objective is to maximize their performance on pretraining tasks without considering any aspect of the downstream tasks, which results in discrepancy between pre-trained models and downstream tasks. To alleviate this, we leverage a meta- learning algorithm for pre-training optimization of the Idapter. With different augmented tasks as meta-tasks, we utilize meta-learning to update the initial parameters of our model, which could anticipate to improve downstream performance on tasks involving idiomatic expressions. Our meta-pretran mmeta-prtg can be divided into two stages:
meta-train stage, which updates the model's parameters using the train set of pre-train data:
0: 0 10-1-Ve
1--VVoo LV (0A
1: Dirain train
ii) meta-test stage, which updates the model's initial parameters dp by one gradient descent step over the test set of pre-train data:
0 00 -BV 0g (0z; D

---

Published as a conference paper at COLM 2024
IMPLI
I2L
L21
FNB
Method
E
NE
ANT
Acc
B
R
B
R
CC
ICC
Acc
Full
85.21
59.71
89.96
81.16
82.46
77. 15
82.89
76.69
82.82
78.03
80.42
Adapter
81.4
66.01
87.43
79.98
81.21
75,36
82.72
76.66
83.94
74.93
79.44
Lora
91.08
71.15
70.59
80.07
81.12
75.28
81.85
75.74
72.11
77.75
74.93
(1A)3
86.72
38.74
80.75
74.26
76.44
69.82
76.38
70.19
76.06
77.46
76.76
PIER+
89.39
53.54
13.07
56.78
66.80
63.36
65.08
Llama2-1ssott
53.32
84.19
19.84
71.92
61.09
56.36
65.38
60.63
62.82
81.41
72.11
Mistral-1Oshot
36.05
84.58
90.11
64.21
77.12
69.24
77.88
70.55
74.08
59.72
66.90
Ours
87.86
84.19
88.77
87.34
86.11
81.32
85,78
80,79
83.10
84.23
83,66
-w/o Augmentation
87.02
66.32
88.47
82.95
83.31
77.69
83.05
77.16
76.62
83.10
79.86
w/o Idapter
94.31
77.87
75.94
84.75
83.68
78.22
83.13
77.94
76.62
83.10
80.14
-w/o NC layer
93.55
78.66
78.88
85.53
84.76
79.53
84.32
79.14
82.29
79.19
80.74
w/w TS layer
94.29
79.37
79.84
86.34
85,92
80.06
85,36
79.53
81.13
81.13
81.13
wwo Meta-Pretrain
85.23
82.21
87.61
85.34
84.45
78.69
84.33
79.21
82.65
78.99
80.82
Table 2: Performance of different methods for different downstream tasks. E, NE and ANT refers to the accuracy on the entailment examples, non-entailment examples and antonym non-entailment examples for IMPLI. B and R refers to the BLEU score and Rouge-2 score respectively. CC and ICC refers to the accuracy on the correct labels and wrong labels for FNB. Ace denotes the overall accuracy averaged based on all the examples. Bolded results represent the best performance and underlined results represent second best. The differences on overall accuracies, BLEU scores and Rouge-2 scores between our model and the other baselines are statistically significant at level 0.05.
where a refers to the learning rate during the meta-train stage and f refers to the learning rate during the meta-test stage. D represents the kth batch in train set of pre-train data from task T. Dfrst represents the test set of pre-train data from task T.
The returned meta pettamaasssters in are used as the pre-trained model parameters for different downstream tasks. We utilize Reptile (Nichol et al., 2018), variant of MAML (Finn et al., 2017) in our meta-pretraining stage for better efficiency.
Experiments
Continued Pre-training Tasks. The continued pre-training tasks aim to inject fundational IE knowledge into the models. We use the MAGPIE (Haagsma et al., 2020) for our multi- view data augmentation. The original dataset was intended for idiom identification, where the task is to decide an IE is used figuratively or literally in a given context. Thus, it serves as the data foundation for the first task of idiom identification. For the task of idiom definition generation, we retrieve the definitions from online resources, such as Wiktionary and Google Dictionary. For the task of idiom generation, we replace the whole IE with single [MASK] token. The model is expected to reconstruct the original IE to replace the [MASK] token. By construction and the nature of the tasks, these tasks are learned via self-supervision (idiom generation) and provided labels (definition generation and idiom identification).
Downstream Tasks. We further verify the effectiveness of our method on diverse set of downstream tasks including IE-related natural language inference (IMPLI) (Stowe et al., 2022), Idiomatic Expression Paraphrasing (I2L) (Zhou et al., 2021c), Literal Expression Stylizing (I.2I) (Zhou et al., 2021c) and Figurative Narrative Benchmark (FNB) (Chakrabarty al., 2021a). The tasks differ from the continued pre-training tasks described above and involve relatively deeper interpretation of idioms. Details are provided in the appendix.
Experimental Procedure. Our experiments are performed as follows: We use the T5-large model (Raffel et al., 2020) as the backbone PTLM since its multi-task pre-training aligns with our continued pre-training and thus provides better and faster adaptation. We first add an Idapter to the backbone model and then update only the Idapter during the meta- pretraining stage. After meta-pretr,,ning, we deploy the model on different downstream tasks for fine-tuning.
7

---

Published as a conference paper at COLM 2024
IMPLI
I2L
L21
FNB
Dataset
E
NE
ANT
Acc
B
R
B
R
CC
ICC
Acc
O-task
87.02
66.32
88.47
82.95
83.31
77.6 67.69
83.05
77.16
76.62
83.10
79.86
No GEN
94.12
77.08
81.28
86.22
85.06
81.21
85.69
80,44
82.28
81.15
81.72
No IDT
94.12
76.68
81.55
86.22
85.92
81.06
85.48
80.39
82.82
81.41
82.11
No DEF
93.74
78.26
79.14
85.62
84.79
80.88
85.02
80.09
81.41
80.85
81.13
All
87.86
84.19
88.77
87.34
86.11
81.32
85.78
80.79
83.10
84.23
83.66
Table 3: comparison of model performance while varying the number of injection tasks
used during pre-training
Baselines. To validate our method, compare it to full fine-tuning and three SOTA PEFT methods including Adapters (Rebuffi et al., 2017), Lora (Hu et al.) and (IA)² (Liu et al., 2022). We also compare our model with PIER+ (Zeng & Bhat, 2023). To study the extent to which scaling and strategic fine-tuning methods enable idiomatic reasoning, compare our proposed method with open-source LLMs, including Llama2-13b (Touvron et al., 2023) and Mistral-7b (Jiang et al., 2023) with 10 in-context examples (the number 10 was empirically found to yield reasonable results beyond which the performance did not improve significantly).
also
Results
IMPLI. Prior works (Sag et al., 2002; Tayyar Madabushi et al., 2021; Stowe et al., 2022) have demonstrated that the presence of lE's negatively impacts NLI performance. We use this as a case study to demonstrate how our proposed IE-related knowledge injection increases robustness in the presence of IEs. As shown in Table 2, our method efficiently injects idiomatic knowledge into the model. For the models without a PEFT module (Full), our proposed model outperforms the original model by 6.18% in accuracy. Notably, there is gain of 24.48% accuracy in the non-entailment (NE) class. We point out that good non- entailment performance is much harder to attain compared entailment since the model resorts to predicting all instances ascribe this performance gain to the model's understanding of the IE semantics rather than relying on spurious features towards the entailment predictions, which is further verified by our ablation study in Section 8.
entailment as discussed in (Stowe et al., 2022). We
Paraphrasing & Stylizing. Here present the results of two related tasks: I2L and L2I. As presented in Tables 2, our proposed approach substantially benefits these two tasks by incorporating idiomatic knowledge. Similarly, our model achieves the highest performance. It surpasses the model trained solely on the 12L dataset by a significant margin of 3.65 points in BLEU score (82.46 vs. 86.11) and outperforms the model trained solely on the L2I dataset by margin of 2.89 points in BLEU score (82.89 vs. 85.78). For both tasks, this improvement holds consistently across models with different PEFT modules, showcasing the effectiveness of our method. Considering that BLEU and ROUGE scores only focus on the n-gram overlapping, we also utilize Bertscore to evaluate our proposed method. is shown that our model also achieves the best performance on Bertscore. The details are provided in Table 6 in the appendix.
FNB. As shown in Table 2, our method efficiently equips the model with the ability of T reasoning in the presence of idioms. For the models without a PEFT module (Full), the model trained on our multi-view augmented data (+PT) outperforms the original model by 3.24% in accuracy.
We note that the tasks study (IMPLI, L2I, 12L, and FNB) contain IEs not covered by idiomatic knowledge injection, further demonstrating our model's generalizability to IEs unseen before fine-tuning. Details are provided in the Section ?? in the appendix.
8

---

Published as a conference paper at COLM 2024
IMPLI
I2L
L.I
FNB
Method
E
NE
ANT
Acc
B
R
B
R
CC
ICC
Acc
Full
78.75
68.38
86.1
78.86
78.9
73.31
81.22
74.21
13.80
86.48
50.14
Adapter
75.33
60.24
82.01
74.19
79.25
72.74
80.03
73.36
19.15
82.82
50.99
Lora
70.21
79.05
92.78
79.46
78.11
71.36
79.05
72.4
40.56
62.25
51.41
(IA)3
80.22
62.38
78.13
75.63
72.95
68.66
73.04
68.6
19.44
81.41
50.42
Ours
93.55
77.47
81.82
86.22
82.82
76.92
83.01
77.81
82.54
74.65
78.59
Table 4:
Performance of different methods for different downstream tasks when only 20% of data is used for downstream fine-tuning.
Analysis
We gain more insights about our multi-view data augmentation method and Idapter via further analyses.
Ablation Study. In our ablation studies, we analyze the impact of excluding different components from our proposed model. Table 2 demonstrates that omitting any part of our model results in decreased performance, highlighting the effectiveness of our approach. Specifically, when the multi-view data augmentation is excluded, performance is signif- icantly worse, emphasizing the importance of this augmentation method. Additionally, excluding either the non-compositionaaliy scaling layer or the task-specific scaling layer leads to performance degradation, reinforcing the effectiveness of the various inductive biases incorporated in our proposed Idapter.
Injection Task Influence. To study the influence of different continued pre-training tasks, we pre-train the Idapter with the absence of different tasks. For the OHaak case, we directly fine-tune the Idapter with the backbone PTLMM on the downstream tasks with no pre-training injection tasks. Then we exclude different injection tasks from the pre-training data and pre-train the Idapter with the remaining two tasks. Finally, for the 3-task case, we first pre-train the Idapter on all three augmented tasks, adding IE appropriateness compared to the 2-task case. After the pre-training with augmented tasks, we further fine-tuned the Idapter on different downstream tasks. As shown by the results in Table 3, all instances of excluding different injection tasks degrade the performance across all four downstream tasks, while pre-training with all three tasks provides the best performance, It should be noted that excluding the definition generation task impacts the performance the most, presumably because this task directly injects the meaning of idioms into the model.
Data efficiency. Another question concerns whether a model equipped with generalized idiomatic knowledge is able to learn different downstream tasks in data-efficient way. To this end, we fine-tune the continued pre-trained Idapter on the downstream tasks with fewer data. We use only 20% of the original data in different downstream tasks for fine-tuning and summarize the results in the Table 4. As is shown, using only 20% of the data, our proposed model achieve the best performance. In fact, the model augmented with the Idapter achieves a competitive performance that is only slightly worse than the model fully fine-tuned on the entire data for IMPLI; this confirms the data efficiency of our method.
Zero-Shot Setting For IMPLI. Our multi-view data augmentation method shows an improvement in a zero-shot, out-of-distributio setting. We observe this by fine-tuning the full model only on the MNLI dataset (Williams et al., 2018), a generic NLI dataset with negligible number of IE instances, and test on IMPLI; models trained with the multi-view augmentation data show an improved overall performance on the IMPLI dataset (+8.04% gain in accuracy). Our method's 's notable improvement comes from a 1 1.24% accuracy increase in NE class, 2.1% in ANT class and 7. 7.43% in the class. Details are provided in Table 7 in the appendix.
Qualitative Analyses. Some idioms may only appear in the downstream tasks, but not in our continued pre-training data. Therefore, we explore our model's performance on idioms seen during pretraining and those unseen for different downstream tasks. As shown in

---

Published as a conference paper at COLM 2024
Table 5, across different downstream tasks, our model was able to process unseen idioms, though not nearly as well as those seen.
Beyond processing idioms, we find that our model is able to positively impact the processing of other types of figurative expressions, including metaphor detection (+0.8% on accuracy) and euphemism detection (+1.03% on accuracy). Details are provided in appendix A.4.
Method
IMPLI
12L
L21
FNB
NE
ANT
Acc
Num
R
Num
R
Num
CC
ICC
Acc
Num
76.57
90.67
86.90
93.30
89.35
551
83.33
84.33
84.67
320
Unseen
89.66
80.00
85.00
85.94
18
80.69
75.20
618
77.01
73.09
618
85.45
70,91
78.18
72
All
84.19
88.77
87.34
86.11
81.32
1169
80.79
1169
83.10
83.66
392
Table 5: Performance on seen idioms and unseen idioms. Num refers to the number of
idioms in the test set.
9 Conclusion and Future Works
In this work, propose continued pre-training method based on novel multi-view data augmentation approach, a parameter-efficien training method, Idapter and a meta- pretraining mechanism. Unlike previous works IEs that only focus on task, we empirically show that our method injects generic idiomatic knowledge that is beneficial for various tasks involving IEs, such as IE-related natural language inference, idiomatic expression paraphrasing, literal expression stylizing and figurative narrative reasoning. Besides, we also show that even with only 20% of the training data for downstream IE- related tasks, the model equipped with generalized idiomatic knowledge yields comparable performance with the model trained on the whole data on IE-related natural language inference. Future work should explore how this method can be useful in low-resource scenarios, such as the medical and legal domains.
Limitations
The main limitation of our method is that it still needs external resources to obtain the definitions of IEs, which restricts the learning of some IEs whose definitions cannot be easily accessed.
Also, the scope of this research focuses on equipping large PTLMs with idiomatic knowledge efficiently. However, due to the limitation of our computation resources, we cannot perform our experiments on larger PTLMs such as TO or GPT-4. However, we believe that while optimizing hyperparameters should yield higher performance, the trends between different pipelines should remain the same (i.e., knowledge injection should still be beneficial).
Additionally, although the scope of this paper is to understand the effect of idiomatic knowledge injection on IE comprehension, it would be more beneficial for the community to generalize our method to more domains that require domain knowledge, e.g,, medical domain, legal domain, and science domain. However, our experiments are limited by focusing only the idiom domain.
Ethics Statement
The proposed system intended to use as method to enhance pre-trained language models' ability to interpret and comprehend the figurative semantics of idiomatic expressions and thus improve the performances on various natural language understanding tasks, especially those that need to pay close attention to the presence of idiomatic expressions in the text. In case of failure, the system will misinterpret the meaning and uses of idiomatic expressions, thus negatively impacting the performances of any related downstream task. Given that we have not thoroughly studied the causes of failures or the performances of idioms not covered by our idiomatic knowledge injection phase, we do not recommend the uses in
10

---

Published as a conference paper at COLM 2024
critical domains. During the training and testing of our methods and their downstream applications, we only use publicly available datasets without collecting or generating additional data that could breach privacy rights. Hence, when used properly, our work presents no calls for ethical concern.
Acknowledgements
This research was supported by the National Science Foundation under Grant No. IIS 2230817 and in part by the U.S. National Science Foundation and Institute of Education Sciences under Grant No. 2229612.
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7319- 7328, 2021.
Miriam Amin, Peter Fankhauser, Mare Kupietz, and Roman Schneider. Data-driven identifi- cation of idioms in song lyrics. MWE 2021, pp. 13, 2021.
Isabelle Augenstein, Sebastian Ruder, and Anders Søgaard. Multi-task learning of pair- wise sequence classification tasks over disparate label spaces. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1896-1906, 2018.
Timothy Baldwin and Su Nam Kim. Multiword expressions. In Nitin Indurkhya and Fred J. Damerau (eds.), Handbook of Natural Language Processing, Second Edition, pp- 267-292. Chapman and Hall/CRC, 2010a. URL http::/wwwwwwcceeeebeebe.e com//oi/abs// 7201/9781422088593888
Timothy Baldwin and Su Nam Kim. Multiword expressions. Handbook of natural language processing, 2:267-292, 2010b.
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLI-IJCNLP), pp. 3615-3620, 2019.
Prajjwal Bhargava and Vincent Ng. Commonsense knowledge reasoning and generation with pre-trained language models: A survey. CoRR, abs /bbbb.2220.1233 2022. URL ht //arxiv.org/abs/2220 12438.
Cristina Cacciari and Maria Chiara Levorato. How children understand idioms in discourse. Journal of child language, 16(2):387-405, 1989.
Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. It's not rocket science: Interpreting
figurative language narratives. arXiv preprint arXiv:2109.0008) 2021a.
Tuhin Chakrabarty, Debanjan Ghosh, Adam Poliak, and Smaranda Muresan. Figura- tive language in recognizing textual entailment. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3364-3361, Online, August 2021b. Association for Computational LingiutttionnllIinuuiississ. doi: d htt./2222..nndins-ac..oo URL https://aalanthhooogg..ogz wtt.://i.....iinnnnnin findings-acie 297.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp.
11

---

Published as a conference paper at COLM 2024
4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
doi: 10.18653/v 11003//111191423 URL https: /wwww aclweb. org/ wwtpaaaooooonn1---- 1423.
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases.
In Third International Workshop on Paraphrasing (IWP2005). Asia Federation of Natural Language Processing, January 2005. URL.tttss:/www.iiccccc
http:///wwww....ooooooooooooo/// reseerchh/
Zi-Yi Dou, Keyi Yu, and Antonios Anastasopoulos. Investigating meta-learning algo- rithms for low-resource natural language understanding tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP), PP. 1192-1197,
2019. 2019.
Steven Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. A survey of data augmentation approaches for nlp. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 968- 988, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126- 1135. PMLR, 2017.
Michael Flor and Beata Beigman Klebanov. Catching idiomatic expressions in efl essays. In Proceedings of the Workshop on Figurative Language Processing, pp. 34-44, 2018.
Martha Gavidia, Patrick Lee, Anna Feldman, and Jing Peng. Cats are fuzzy pets: A cor- pus and analysis of potentially euphemistic terms. In 13th International Conference Language Resources and Evaluation Conference, LREC 2022, pp. 2658-22582 1671.. European Language Resources Association (ELRA), 2022.
Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4884-4896, 2021.
Hessel Haagsma, Johan Bos, and Malvina Nissim. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of The 12th Language Resources and Evaluation Conference, pp. 279-287, 2020.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Yukun Huang, Kun Qian, and Zhou Yu. Learning a better initialization for soft prompts via meta-learning. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 67-75, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arX1v:2310.06825, 2023.
Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, and Graham Neubig. Multi-lingual and multi-cultural figurative language understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 8269-8284, 2023.
Minki Kang, Moonsu Han, and Sung Ju Hwang. Neural mask generator: Learning to generate adaptive word maskings for language model adaptation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6102-61211 2020.
12

---

Published as a conference paper at COLM 2024
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022-1035, 2021.
Zhen Ke, Liang Shi, Songtao Sun, Erli Meng, Bin Wang, and Xipeng Qiu. Pre-training with meta learning for chinese word segmentation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human
Language Technologies, pp. 5514-5523, 2021.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-144 12400, 2020.
Chee Wee Leong, Beata Beigman Klebanov, and Ekaterina Shutova. report on the 2018 vua metaphor detection shared task. In Proceedings of the workshop on figurative language processing, pp. 56-66, 2018.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselín Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to- sequence pre-training for natural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1. 20202020000 a1l20acl-maai.700 703.
URL http:::s//alaathhllgg/o/o org/2020. acl-main.
Changsheng Liu and Rebecca Hwa. Representations of context in recognizing the figura- tive and literal usages of idioms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Changsheng Liu and Rebecca Hwa. Heuristically informed unsupervised idiom usage recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1723-1731, 2018
Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. Are multilingual Ilms culturally-diverss reasoners? an investigation into multicultural proverbs and sayings. arXiv preprint (xXii:2309908591 2023.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-effice fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
Pengfei Liu, Xipeng Qiu, and Xuan-Jing Huang. Adversarial multi-task learning for text classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1-10, 2017.
Filip Miletić and Sabine Schulte im Walde. Semantics of multiword expressions in transformer-bssed models: A survey. arXiv preprint arXiv:2401.15393, 2024.
Takeru Miyato, Andrew M Dai, and lan Goodfellow. Adversarial training methods for semi- supervised text classification. In International Conference on Learning Representations.
Rosamund Moon et al. Fixed Expressions and Idioms in English: Corpus-Ban -ased
Approach. Oxford University Press, 1998.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint XXXx:1803.00299 2018
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 46-54, Online, October 2020a. Asso- ciation for Computational Linguistics. doi: 10.18653/ 102/220.emmlp-demmm..... URL https://aclanthhology org/2020. emnlp-demos.7.
13

---

Published as a conference paper at COLM 2024
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations, pp. 46-54, Online, 2020b. Association for Computational Linguistics. URL. https: http:::/wwwwwwwwww.. org org/anthology/ 2020.
7.
Kun Qian, Wei Wei, and Zhou Yu. A student-teacher architecture for dialog domain adaptation under the meta-leamiing setting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 13692-13700, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-tett transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 5533-5542. IEEE Computer Soci- ety, 2017.
Ivan A Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. Multiword expressions: A pain in the neck for NLP. International conference on intelligent text processing and computational linguistics, pp. 1-15. Springer, 2002.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic com compositionality over sentiment treebank. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: /ttp:::///wwwwwww// acwweb. org / D13-1170.
Kevin Stowe, Prasetya Utama, and Iryna Gurevych. IMPLI: Investigating NLI models' performance on figurative language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 5375-5388, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/vl acl-long.
2022.acl-long.363 URL https: httpss:/aalaanthologg org.oogg///// 2022. a0t.22................... 369.
Harish Tayyar Madabushi, Edward Gow-Smith, Carolina Scarton, and Aline Villavicencio. Dataset and methods for the exploration of idiomaticity in pre-
AStitchinLanguageMenn
trained language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4477, Punta Cana, Dominican Republic, November 2021. Associ- ation for Computational Linguistics. doi: 10.18653 //////11122////// 2221..indiiggs seatings-emnl.22255 URL https://aclanthology www.hhhooooyoo.g/22111111 http:///......nnnnneee.eem. findings-emllp.
294.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint Sacikv::307.092885 2023.
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-JJCNIPP) pp. 6382-632222 6388, 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman.
rrovr-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122. Association for Compu- LirionaLLinguiste Linguistics, 2018. URL
tational
org/ wtt.://r.r..annooog///----------
Hongqiu Wu, Ruixue Ding, Hai Zhao, Boli Pengjun Xie, Fei Huang, and Min Zhang. Forging multiple training objectives for pre-trained language models via meta-leamiig, In Findings the Association for Computational Linguistics: EMNLP 2022, pp. 6454-6466, 2022.
14

---

Published as a conference paper at COLM 2024
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data aug- mentation for consistency training. Advances in neural information processing systems, 33:6256-6268, 2020.
Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. Jaket: Joint pre-training of knowledge graph and language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11630-11638, 2022.
Ziheng Zeng and Suma Bhat. Idiomatic expression identification using semantic com- patibility. Transactions of the Association for Computational Linguistics, 9:1546-1562, 2021.
Ziheng Zeng and Suma Bhat. Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressions. Transactions the Association for Computational Linguistics, 10:1120-1137, 10 2022. ISSN 2307-387X. doi: 10.1162/tacLa.000 URL
https://doi.org 100 (622/tacl...00 110/111111///ata...0000
Ziheng Zeng and Suma Bhat. Unified representation for non-compositional and composi- tional expressions. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Ziheng Zeng, Kellen Cheng, Srihari Nanniyur, Jianing Zhou, and Suma Bhat. lekg: A commonsense knowledge graph for idiomatic expressions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14243-14264, 2023.
Taolin Zhang, Chengyu Wang, Nan Hu, Minghui Qiu, Chengguang Tang, Xiaofeng He, and Jun Huang. Dkplm: Decomposable knowledge-enhancee pre-trained language model for natural language understanding. In Proceedings of the AAAI Conference on Artificial Langua Intelligence, volume 36, pp. 11703-11711, 2022.
Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1298-1308, Minneapolis, Minnesota, June 2019a, Association 000865331//// ///////1N1111111111 URL https: /atp:///alaatooooyyy
for Computational Linguistics. doi: 10. 10.18653./..1111 org/N19-1131.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1441-1451, 2019b.
Jianing Zhou, Hongyu Gong, and Suma Bhat. PIE: A parallel idiomatic expression corpus for idiomatic sentence generation and paraphrasing. In Proceedings of the 17th Workshop Multiword Expressions (MWE 2021), pp. 33 4844, Online, August 2021a. Association for URL
Computational Linguistics. doi: (0.18653/v1 org/2021. meme1...w111111 1.5.
Jianing Zhou, Hongyu Gong, and Suma Bhat. From solving problem boldly to cutting the gordian knot: Idiomatic text generation. arXiv preprint arXiv:2104.06541 2021b.
Jianing Zhou, Ziheng Zeng, Hongyu Gong, and Suma Bhat. Idiomatic expression para- phrasing without strong supervision, 2021c
Tadej Škvorc, Polona Gantar, and Marko Robnik babnik-šikonja. Mice: Mining idioms with con- textual embeddings. Knowledge-Bassed Systems, 235:107606,, 2022. ISSN 0950-7051. doi:
107666. URL https: /www./wwww///.c//////// wet.seiieieeeeet.ccccmmmmm com/
15

---

Published as a conference paper at COLM 2024
A Appendix
A.1
Experimental Settings
Our experiments are performed as follows: we use the 15-large model Raffel et al. (2020) as the backbone PTLM. We add Idapter to the backbone model, which is then continued pretrained to perform the idiomatic knowledge injection using our multi-view data augmen- tation method. For this continued pre-training stage, we train the Idapter for 5 meta-epochs within each of which the model is trained for 50 epochs with a batch size of 32 using the Adam optimizer with learning rate of 2e-5 and linear decay scheduler with 60 warm- up steps. For outer learning rate of meta-learning is 1e-5. After injection, we deploy the model on different downstream tasks, where we use the parameters initialized from the trained Idapter. We train for 20 epochs with batch size of 64 and the same optimizer and scheduler setup as before. We apply prompt templates shown in Table to all the tasks during training and inference to convert each example into instructive text-to-text format. Importantly, we apply this recipe to every downstream dataset in exactly the same way without modifications.
A.2 Downstream Tasks
IE-related Natural Language Inference. This semantic understanding task involves de- termining whether given hypothesis can be inferred from a given premise (entailment) or not (non-entailment). The overall challenge is compounded with the insertion of an IE in the Idiomatic and Metaphoric Paired Language Inference (IMPLI) dataset (Stowe et al., 2022). For this dataset, E represents entailment by paraphrasing idioms in the premise, NE represents non-entailment created from adversarial idiom definitions, and ANT represents non-entailment created by swapping keywords in idiom definitions to their antonyms.
Idiomatic Expression Paraphrasing and Literal Expression Stylizing. Proposed by (Zhou et al., 2021c), the tasks involve paraphrasing IEs into literal phrases and stylistically trans- forming the literal phrases in given sentences by replacing them with their IE counterparts, respectively. We use the dataset created by (Zhou et al., 2021c) for this task (denoted as I2L and L21 henceforth).
Figurative Narrative Benchmark. Proposed by (Chakrabarty et al., 2021a), the task is to continue a given narrative by choosing an acceptable continuation from two candidates. The dataset has narratives that use idioms and similes. Following the settings in (Zeng et al., 2023) and to account for our setting of working with idioms, we only utilize the narratives with idioms. Moreover, we transform each instance into two continuation acceptability examples: one that pairs the narrative with the correct continuation, and another that pairs the same narrative with the incorrect continuation (denoted FNB henceforth).
A.3 Performance on Compositional Tasks
Here we show that injecting idiomatic ability does not take away our model's prior ability with compositional expressions. We perform experiments on sentiment classification and paraphrase identification following the settings in Zeng & Bhat (2023). For sentiment classification, we use the SST2 (Socher et al., 2013) dataset and its default train and test splits (two classes). For paraphrase identification, we combine the MRPC (Dolan & Brockett, 2005) and PAWS (Zhang et al., 2019a) datasets and their default train /est splits with total of 53,069 train and 9,725 test instances. The experimental results are presented in Table 8. As shown in Table 8, compared with the original T5-large, our model still achieves a comparable performance for these two tasks, which proves that injecting idiomatic ability does not take away our model's prior ability with compositional expressions.
A.4 Performance on More Types of Non-compositionall Expressions
Here we show that injecting idiomatic ability could also help with the comprehension of other types of non-compositional expressions including metaphors and euphemism. We
16

---

Published as a conference paper at COLM 2024
I2L
L2I
Method
BLEU
Rouge-2
Bertscore
BLEU
Rouge-2
Bertscore
Full
82.46
77.15
81.32
82.89
76.69
80.46
Adapter
81.21
75.36
80.14
82.72
76.66
80.03
Lora
81.12
75.28
80.20
81.85
75.74
80.09
(IA)
76.44
69.82
71.74
76.38
70.19
73.08
Llama2-10shot
61.09
56.36
78.97
65.38
60.63
80.22
Mistral-10shot
77.12
69.24
71.66
77.88
70.55
73.28
Ours
86.11
81.32
86.46
85.78
80.79
85.40
Table 6: Performance of different methods for i2l and 12i downstream tasks. Here we include the Bertscore for more thorough evaluation. When evaluating using Bertscore, we only extract out the idiomatic expressions and their corresponding literal parts. For 12L, we directly calculate the bertscore based on generated literal parts and reference literal parts. For L2I, calculate the Bertscore between the generated idioms and the reference idioms.
Dataset
Model
#Para
E
NE
ANT
Acc
MNLI
Full
770M
75.04
39.11
75.81
67.41
MT
Ours
3M
82.47
40.35
77.91
75.45
Table 7: Performance of zero setting for IMPLI task.
perform experiments metaphor detection and euphemism detection. We use the VUAall dataset (Leong et al., 2018) and Euphemism dataset (Gavidia et al., 2022) for our experiments. The experimental results are presented in Table 9. As shown in Table compared with the original T5-large, our model still achieves a better performance for these two tasks, which proves that injecting idiomatic ability could also help with the comprehension of metaphors and euphemism.
A.5 Experiments LLMs
Here we we show the experimental details of Llama2-13b and Mistral-7b, including the prompts used and different performance when different numbers of in-context examples are used.
In the following we provide the prompts we used: IMPLI:
Below is an instruction that describes task. Write a response that appropriately completes the request.
### Instruction:
If the following premise and hypothsis are in entailment or contradiction:
Premise: [sentence]
Hypothesis: [sentence]
Just output the entailment or contradiction in the Response
### Response:
|entailment/ c
I2L:
Below is an instruction that describes task. Write response that appropriately completes the request.
#*# Instruction:
17

---

Published as a conference paper at COLM 2024
Sentiment Classification
Paraphrase Identification
Method
Positive
Negative
Acc
Positive
Negative
Acc
Full T5-large Ours
96.17 96.85
96.03
96.10
94.11
92.30
93.17
95.56
96.22
93.98
92.32
93.12
Table 8: Performance of different methods for sentiment classification and paraphrase identification.
Metaphor Detection
Euphemism Detection
Method
Literal
Figurative
Acc
Literal
Figurative
Acc
Full T5-large 95.29
77.22
92.73
83.82
89.06
87 87.24
Ours
96.47
75.80
93.53
86.76
89.06
88.27
Table 9: Performance of different methods for metaphor detection and euphemism detection.
Paraphrase the following sentence via only paraphrasing the idiom into its literal counter- parts: [sentence]
### Response: [sentence]
L2I:
Below is an instruction that describes task. Write a response that appropriately completes the request.
### Instruction:
Paraphrase the following sentence into its counterpart with an idiom: [sentence]
### Response:
[sentence]
FNB:
Below is an instruction that describes task. Write response that appropriately completes the request.
### Instruction:
Given the following sentences: [sentence]
If the following continuation correct not: [sentence]
Just output correct or wrong in the Response.
### Response: [correct/ /wron]]
18

---

Published as a conference paper at COLM 2024
IMPLI
FNB
Method
E
NE
ANT
Acc
R
B
R
CC
ICC
Acc
Llama2-2shot
53.32
56.17
73.91
68.88
54.96
49.53
62.04
57.80
71.27
66.76
69.01
Llama2-6shot
50.80
86.75
91.25
71.79
60.91
56.31
65.27
60.52
66.48
78.87
72.68
Ilama2-10shot
53.32
84.19
89.84
71.92
61.09
56.36
65.38
60.63
62.82
81.41
72.11
Mistral-2shot
40.61
73.91
75.13
59.10
75.75
73.16
72.44
70.40
71.55
35.77
53.66
Mistral-6shot
30.74
90.51
91.44
63.52
76.73
74.06
73.61
70.36
76.34
56.90
66.62
Mistral-10shot
36.05
84.58
90.11
64.21
77.12
74.27
73.88
70.55
74.08
59.72
66.90
Table 10: Performance of Llama2-13b and Mistral-7b.
19


Reviews:
[{"review_id": "IEeb0oYvKN", "reviewer": "Reviewer_9B9h", "summary": "This paper investigates use of idiom aware PEFT method to improve backbone model's performance over non-compositional tasks like idiom detection, idiom generation and idiom definition generation. Its essentially a training framework that is proposed in the paper each sample in the dataset is split into different tasks (multi-view data augmentation), PEFT is used to inject idiomatic knowledge in the model and meta-learning over multiple tasks is applied in order to generate a better initialized pre-trained model ideal for being trained on downstream tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really thank the reviewer for the valuable advice and suggestion.\n\nR1: Thanks for pointing this out. 3M additional parameters were added. Besides, the parameters of Adapter and Lora are at par with our proposed model\n\nR2: Thanks a lot for your suggestion. We could provide some examples in the appendix in the final version.\n\nR3: The T5-large model was chosen as the backbone PTLM due to its robust performance across a wide range of natural language processing tasks and its alignment with our multi-task pre-training methodology. As highlighted in our paper, T5-large serves as a representative model for evaluating the effectiveness of our proposed Idapter mechanism and multi-view data augmentation strategy. We acknowledge the importance of evaluating the applicability of Idapter across different language models (LMs). While our current work focuses on T5-large, our methodology is designed to be model-agnostic. This is evident from the architecture of Idapter, which leverages parameter-efficient fine-tuning (PEFT) techniques that are compatible with a variety of PTLMs. Specifically, the design of Idapter incorporates non-compositionality scaling and task-specific scaling layers that can be adapted to different LMs without significant modifications\n\nR4: While Table 2 indicates that data augmentation significantly improves performance, it is important to understand that this enhancement is maximized in conjunction with the Idapter mechanism. The multi-view data augmentation method effectively generates diverse idiomatic data, but its full potential is harnessed when combined with Idapter's ability to process and learn from this augmented data in a parameter-efficient manner. This synergy is evident from the performance gains across various tasks when both strategies are used together, as shown in our ablation studies.\n\nQ1: Although Idapter like adapter can be applied to other tasks as well, our original aim is to study idiomatic expression processing. Idiomatic expressions are inherently non-compositional, meaning their figurative meanings cannot be directly inferred from their individual components. This non-compositionality, combined with their contextual ambiguity and sparse representation in text corpora, makes idioms a particularly challenging and compelling domain for testing our method.\n\nQ2&3: Thanks for your suggestion. We would make the corresponding modification in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Very clear writing, well motivated paper and easy to follow for the readers.\n* Experiments are quite comprehensive - results are compared against multiple baselines. Ablation study (Table 2) is quite thorough!\n* Results (table 2) show substantial improvements over strong baselines like LoRA, Mistral-10 shot, Llama2-10 shot etc.", "reasons_to_reject": "* Its unclear how many additional parameters were added because of non-compositionality scoring (W_n) and task specific vectors (W_h). Are the parameters for multiple baselines (shown in Table 2) at par with your model?\n* It would be better to see some examples where the proposed approach worked and failed to identify idiom/generation idiom. \n* Nitpicking here, but results are only computed with T5-large as a baseline which begs a question - what happens when Idapter is applied to other LMs?\n* From the results in Table 2, it looks like data augmentation was the most effective, so why was the paper pitched for Idapter?", "questions_to_authors": "* Conceptually, Idapter like adapter can be applied to other tasks as well (not just idiom related tasks) like style adaptation (formal/informal/semi-formal), one just needs a good dataset. Why was this work limited to idiomatic expressions? \n* Typically with idioms, its easy to memorize them during training so generalization (over non-literal data) is quite difficult. Appendix provided necessary details like performance on seen/un-seen idioms - consider adding this table in main paper. Can you provide examples of failure and success cases in Appendix for readers?\n* Do-no-harm experiments (no regression on standard tasks) are important for productionizing any work. Table 8 is quite important in that aspect, maybe find a place to reference section A.4?", "extra_scores": {}}, {"review_id": "aVrBEOQOJW", "reviewer": "Reviewer_i4hT", "summary": "This paper introduces a data augmentation method, a model adapter, and a training scheme to first enhance a pre-trained language model (PTLM) with idiomatic knowledge, and then fine-tune it for idiomatic-aware tasks. \n\nBelow, I describe the three components of the proposed framework:\n\n1) **Data Augmentation**: Their approach for data augmentation consists of taking an existing dataset of idiomatic expressions and create 1-1 examples for three different tasks: 1) generating dictionary definitions of idiomatic expressions, 2) identifying expressions in a sentence that are used in their idiomatic sense, while ignoring literal senses and 3) a cloze task that masks phrases and recovers appropriate idiomatic expressions from their context. This multi-task views allows to learn different aspects for the same input (and idiom).\n\n2) **Network Adapter**: Their adapter includes a layer that aims to capture non-compositional information, and a derived scaling factor that allows to balance the original representation and the induced \"non-compositional information\" during training and inference. They also learn a scalar vector for each tasks to adapt to them. \n\n3) **Meta-pretraining**: This component was quite confusing to me. Authors claim it is supposed to consider aspects of the downstream tasks (but it is not standard fine-tuning?) and I don't understand why. In general, it wasn't clear to me what they do here. What are the meta-tasks? Are they different from the pre-training tasks AND the downstream tasks? How do they alleviate the disconnect between pre-training and downstream tasks?\n\nFinally, the authors perform an experimental evaluation on a diverse set of downstream tasks (tasks that require deeper understanding of idiomatic expressions) and show that their model outperforms full fine-tuning, standard parameter efficient fine-tuning and in-context learning for most tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really thank the reviewer for the valuable advice and suggestion.\n\nR1: Thanks a lot for your suggestion. \n1. The core idea behind our meta-pretraining approach is to bridge the gap between pre-training and downstream tasks more effectively than standard fine-tuning. While standard fine-tuning adjusts a pre-trained model's parameters to fit a specific downstream task, it often does not account for the diverse nature and requirements of multiple downstream tasks, especially when dealing with idiomatic expressions. Meta-pretraining, in our context, is designed to prepare the model for a variety of downstream tasks by optimizing the initial parameter settings. \n2. Meta-tasks are indeed distinct from both pre-training tasks and downstream tasks. In our framework, meta-tasks are specifically designed to encapsulate various idiomatic properties, such as idiom identification, definition generation, and idiom usage generation. These tasks are formulated to introduce idiomatic reasoning capabilities during the meta-pretraining phase. By learning through these diverse meta-tasks, the model develops a generalized idiomatic knowledge base, which helps alleviate the disconnect typically observed between pre-training and downstream fine-tuning. \n3. Our approach alleviates the disconnect by integrating a multi-view data augmentation strategy combined with meta-learning principles. This strategy ensures that the model is pre-trained on various idiomatic scenarios, preparing it for a wide range of idiomatic contexts it might encounter in downstream tasks. During meta-pretraining, the model undergoes multiple iterations of learning and adaptation through these carefully crafted meta-tasks, thus reducing the gap between the pre-trained model and the requirements of downstream idiomatic tasks. \nWe would make these clearer in the final version.\n\nR2: Thanks a lot for pointing this out. The naming of parameters, particularly the \"non-compositional layer,\" was chosen based on the specific functionality and the theoretical underpinnings of the layer. This layer is designed to specifically address the non-compositional nature of idiomatic expressions, which is a well-known challenge in natural language processing (NLP) due to the idiom's meaning not being directly inferred from its individual components. \n\nQ1: Thanks a lot for your suggestion. We would make the corresponding modification in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The idea is simple, interesting and effective. The chosen downstream tasks seem to greatly benefit from the injected idiomatic knowledge.\n2) Solid experimental settings: comparison to alternative paradigms, ablation study to show the impact of all components and pre-training tasks, data efficiency study, out of distribution study. The results are convincing.", "reasons_to_reject": "1) My only real issue with this paper is that the learning scheme is a bit unclear. I think the paper could benefit from a re-write of the meta-pretraining section (see comments and questions in summary above)\n\n2) Maybe it is me nitpicking, but I am not sure some of the names chosen for the parameters are completely justified. How can we know for sure that the added layer is capturing non-compositional information? All we can see is that it contributes to the improved performance (as seen in the ablation study) but this is not really enough evidence of this parameters capturing this information. Calling it non-compositional layer reads like a soft claim that isn't actually validated.", "questions_to_authors": "1) I suggest that you bold the highest numbers in Tab 3 as you do in the other tables, it is a bit confusing to just see the last line bolded even in cases where the performance decreases (E for IMPLI)", "extra_scores": {}}, {"review_id": "EP5RpNk7hg", "reviewer": "Reviewer_v6Fk", "summary": "The paper presents a new approach to improving the handling of idiomatic expressions by pre-trained language models. It highlights the difficulty that existing large language models (LLMs) face with figurative language, which often deviates from standard compositional rules, making idiomatic expressions particularly challenging to interpret. The authors introduce a method that integrates a multi-view data augmentation strategy to generate diverse training scenarios from a single dataset, focusing on different aspects of idiomatic usage. This is combined with a parameter-efficient tuning method and meta-pretraining techniques that tailor the model's learning process to enhance its idiomatic reasoning capabilities. Empirical tests demonstrate that this approach not only outperforms several baselines in tasks involving idioms but also enables the model to adapt more efficiently to various downstream tasks requiring the understanding of idiomatic language.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really thank the reviewer for the valuable advice and suggestion.\n\nR1: Our empirical results on diverse benchmarks, such as IMPLI, I2L, L2I, and FNB, demonstrate the efficacy of our approach. The model not only shows improved performance on tasks involving idioms seen during training but also on those involving unseen idioms. This is a strong indicator of the model’s generalizability and robustness in handling idiomatic expressions beyond the confines of the MAGPIE dataset. Additionally, our model surpasses various parameter-efficient fine-tuning baselines and outperforms larger models such as LlaMa2 and Mistral, which highlights the effectiveness of our data augmentation and training strategies​\n\nR2: Thanks for pointing this out. 3M additional parameters were added. Besides, the parameters of Adapter and Lora are at par with our proposed model. We will provide the test dataset sizes in the final version.\n\nR3: While our current submission primarily presents results on idiom-related tasks, we have conducted preliminary experiments on additional NLP tasks such as Idiomatic Expression Paraphrasing, Literal Expression Stylizing (L2I), sentiment analysis and paraphrase generation, which are indicative of text generation capabilities. These initial experiments have shown promising results, suggesting that our method retains its efficacy across different types of tasks. Due to space constraints, these results were not included in the main manuscript but can be provided as supplementary material upon request.\n\nR4: Our choice to use the T5-large model was influenced by the constraints of computational resources. The T5-large model is already a well-established, high-capacity model that provides a robust baseline for evaluating the effectiveness of our proposed methodologies. While we acknowledge that evaluating on larger models like Llama and Mistral would be ideal, the computational resources required to perform such experiments were beyond our current capacity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a new integration of multi-view data augmentation and meta-pretraining techniques, specifically tailored to improve idiomatic reasoning in pre-trained language models. This method addresses the inherent non-compositionality of idiomatic expressions by enhancing the model's ability to understand various contextual uses and interpretations.\n2. Empirical results demonstrate that the approach outperforms several baselines, achieving better performance on multiple benchmarks focused on idiomatic comprehension and reasoning.\n3. The proposed PEFT and meta-pretraining method seems can also be applied to other tasks, which may inpire their utilization in other NLP tasks.", "reasons_to_reject": "1. The research relies predominantly on the MAGPIE dataset, which might not cover the full spectrum of idiomatic expressions encountered in natural language, potentially limiting the model’s effectiveness with unfamiliar idioms. \n2. The experimental setups are not entirely clear. For example, what is the rank of LoRA? I didn't see the trainable parameter size of each compared model in Table 2. In this context, it is not clear whether the improvements are comparable within similar trainable parameter size. The test data size (#samples) of the downstream tasks is also not provided (nor in the appendix). Also, it would be better to also analyze the training efficiency of other PEFT methods.\n3. The proposed method is not tested under broader tasks, such as text generation, etc. So it is doubtful if the proposed method can be applied to some real application scenario of LLMs (not just doing some classifications or simple text extractions). As the proposed PEFT method and meta-pretraining mechanism seem to be applicable to other general NLP tasks, adding more types of tasks would have strengthened the generalizability of the proposed method.\n4. The proposed method is not performed on bigger LLMs, such as Llama, and Mistral, where the authors only perform on T5-large. It is questionable whether the proposed method can still be effective on a stronger baseline.", "questions_to_authors": "Please address the weaknesses illustrated in the previous section.\n\n\n\nOther comments:\n\n1. PTLM seems not to be a common abbreviation for pre-trained language model. => PLM\n2. Section 1: LlaMa2 -> Llama-2 or Llama 2\n3. Section 2: missing year info for (Hu et al.)\n4. Section 6 baselines: Lora -> LoRA", "extra_scores": {}}, {"review_id": "xX8jjd27I8", "reviewer": "Reviewer_Wnju", "summary": "This is a good paper introducing a new fine-tuning method to inject idiomatic knowledge into pre-trained language models. The author(s) propose(s) a multi-view data augmentation strategy that automatically converts a single-task dataset into labeled data for three tasks, and variations of parameter efficient and meta-learning to improve the fine-tuning process. It is worth to mention that after reading the paper several times, I could not figure out in which language the experiments were done, or whether the author(s) believe that the approach is language-independent, in the sense that with labeled data (like MAGPIE) it would work in a large number of languages.\n\nThe paper is well motivated, has a clear structure and it is easy to read. The related work is not bad, and the experiments seem solid. They include competitive baselines, an ablation study, and additional experiments shown in the appendix. I lack information about the availability of the code: without code and data, the experiments are not replicable, and without replicability science does not progress.\n\nI miss some general evaluations of the proposed model vs. the original T5 on non-idiomatic tasks. This would inform the readers about the performance of the model in both scenarios (compositional and non-compositional). In this regard, I assume that the objective is to have a good model that adequately represents \"multiword expressions\", both compositional or not, and not focus only on idioms.\n\nThe analysis in Table 7 (performance in seen vs. unseen idioms) is interesting. It would be nice to include in this table the results of the \"Full\" model, in order to compare the effect of the proposed strategy for seen/unseen cases.\n\nAs an aside, I wonder to what extent we can rely on a dataset like MAGPIE (which is a great resource!) for data augmentation. For instance, in the \"Idiom identification\" example of Table 1, \"catch a cold\" is the only expression in this sentence labeled as an idiom. It can certainly be considered an idiom, but I would say that most scholars would classify it more as a \"collocation/light verb construction\", and only partially idiomatic (\"cold\" conveys its standard meaning in \"catch a cold\", right?). Moreover, an expression like \"looks like\" could also be considered idiomatic, and \"temperature [...] drop\" metaphorical. With the data augmentation technique we lose this information, and implicitly tell the model that these are compositional (non-idiomatic, non-metaphorical) expressions.\n\nFinally, I would remove the last sentence of the conclusion (\"our method can be applied to [...]\"): if you have already applied your method to other domains, please say so and comment on some preliminary results. Otherwise, just saying that it could be applied to other domains is not informative, unless you have some hypotheses about the adaptation process or about the expected performance.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really thank the reviewer for the valuable advice and suggestion.\n\nHere we provide explanations to your questions. \n1. which language the experiments were done: we perform our experiments on English.\n2. the availability of the code: we would release our code upon acceptance.\n3. general evaluations of the proposed model vs. the original T5 on non-idiomatic tasks: we provide the corresponding evaluations in the Section A.4 in the appendix. We show that injecting idiomatic ability does not take away our model’s prior ability with compositional expressions. We perform experiments on sentiment classification and paraphrase identification.\n4. to what extent we can rely on a dataset like MAGPIE: While it is true that \"catch a cold\" involves a degree of compositionality since \"cold\" retains its standard meaning, it is widely regarded in idiomatic studies as a fixed expression that conveys a meaning not entirely predictable from its parts. This nuanced interpretation aligns with idiomaticity in the broader sense, as it represents a conventionalized phrase with figurative aspects, even if not fully idiomatic in the strictest sense. Regarding the example \"looks like\" and \"temperature [...] drop,\" it is important to clarify our data augmentation strategy. Our multi-view data augmentation is designed to transform the same set of idiomatic expressions into multiple tasks, each emphasizing different aspects of idiomaticity, such as contextual ambiguity, non-compositionality, and usage patterns. This method allows the model to learn the nuances and contextual dependencies of idiomatic expressions effectively. While \"looks like\" might be considered idiomatic in some contexts and \"temperature [...] drop\" metaphorical, our focus is on systematically labeled idiomatic instances to avoid overgeneralization and ensure task-specific learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well motivated, has a clear structure and it is easy to read. The related work is not bad, and the experiments seem solid. They include competitive baselines, an ablation study, and additional experiments shown in the appendix.", "reasons_to_reject": "I do not see solid reasons to reject the job. Some minor issues and doubts that can be seen in my review, but I think the paper should be accepted.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The work introduces a pretraining methodology aimed at enhancing language models with better idiomatic reasoning capabilities. The method involves training an adaptor using a MAML-style algorithm, specifically REPTILE, a more efficient first-order method. The effectiveness of this method is then assessed across downstream tasks that require idiomatic processing.  All the reviewers agree on the merits of the work.


However, there is a significant issue that has only partially raised during the author-reviewer discussion stage, but has been discussed between reviewers and AC in the follow-up period. We strongly encourage you to address it. The paper fails to discuss the fact that the pretraining method and at least three of the four evaluation tasks (IMPLI, L2I, and I2L) partially originate from the same source, MAGPIE. This suggests the possibility of an unrealistic overlap between the idiomatic expressions seen in pretraining and those seen in testing. Although the appendix (Table 7) presents results separately for unseen and seen PIEs, it only evaluates the proposed system in isolation. I believe that evaluating the unseen subsets against alternatives in Table 2 is crucial and needs to be included in the paper.

Additionally, it is important to openly discuss and clarify that these datasets are created from a common resource. The proportion of unseen and seen examples in the test set also needs to be mentioned. If there is no substantial improvement on unseen constructions, the claims should ideally be corrected accordingly. Currently, the approach is positioned as improving the acquisition of generic IE knowledge, which would be debatable if the improvements are observed only on 'seen' PIEs.

[comments from PCs] Please make sure to follow the AC recommendations, especially with regard to data and evaluation.
Decision: Accept