Unified Representation for Non-compositional and Compositional Expressions
Authors: Ziheng Zeng
Suma Bhat
Venue: EMNLP 2023 Findings (2023)
Keywords: Potentially Idiomatic Expression; Non-compositionality; Phrase Embedding; Idiomatic Expression Processing
TL;DR: We propose a new method to produce semantically meaningful and contextually appropriate embeddings for potentially idiomatic expressions that benefits the performances of the downstream idiomatic processing tasks.

Abstract: Accurate processing of non-compositional language relies on generating good representations for such expressions. In this work, we study the representation of language non-compositionality by proposing a language model, PIER+, that builds on BART and can create semantically meaningful and contextually appropriate representations for English potentially idiomatic expressions (PIEs). PIEs are characterized by their non-compositionality and contextual ambiguity in their literal and idiomatic interpretations. Via intrinsic evaluation on embedding quality and extrinsic evaluation on PIE processing and NLU tasks, we show that representations generated by PIER+ result in  33\% higher homogeneity score for embedding clustering than BART, whereas 3.12\% and 3.29\% gains in accuracy and sequence accuracy for PIE sense classification and span detection compared to the state-of-the-art IE representation model, GIEA. These gains are achieved without sacrificing PIER+'s performance on NLU tasks (+/- 1\%  accuracy) compared to BART.

Unified Representation for Non-compositional and Compositional Expressions
Ziheng Zeng and Suma Bhat
Department of Electrical and Computer Engineering University of Illinois Urbana-Champaign, Urbana, Illinois, USA (zzeng13, spbhat2)eillinois..
Abstract
primarily driven by the notion of compositional- ity, which at the core of several system com- ponents, including tokenization (Sennrich et al., 2016; Wu et al., 2016) and the self-attention mech- anism (Vaswani et al., 2017). More fundamentally, recent studies (Zeng and Bhat, 2022) reveal that the pre-trained language models (PTLMs), such GPT-3 (Brown al., 2020) and BART (Lewis al., 2020), are ill-equipped represent (and compre- hend) idiomatic expressions' demonstrated by the lack of correspondence be- tween the IE meanings and their embeddings; IEs with similar meanings are not close in the embed- ding space. Conversely, IEs close in the embedding space have significant token or syntactic overlap. From representation standpoint, this highlights the need for language models (LMs) to handle compositionality through valid representations.
Accurate processing of non-compositional lan- guage relies generating good representa- tions for such expressions. we study the representation of language non- compositionality by proposing language model, PIER, that builds on BART and can create semantically meaningful and contextu- ally appropriate representations for English po- tentially idiomatic expressions (PIEs). PIEs are characterized by their non-compositionallty and contextual ambiguity their literal and idiomatic interpretations. Via intrinsic evalua- tion embedding quality and extrinsic eval- uation on PIE processing and NLU tasks, we show that representations generated by PIER result in 33% higher homogeneity score for embedding clustering than BART, whereas 3.12% and 3.29% gains in accuracy and se- quence accuracy for PIE sense classification and span detection compared the state-of- the-art IE representation model, GIEA. These gains are achieved without sacrificing PIER's performance on NLU tasks (+/- 1% accuracy) compared to BART.
this work,
(IE) meanings. This
Efforts to generate semantically congruent rep- resentations for IEs are now coming to the fore. For instance, GIEA (Zeng and Bhat, 2022) uses frozen pre-trained BART that is injected with trainable adapter layers (Houlsby et al., 2019; Pfeiffer et al., 2020a) to generate IE embeddings for non-compositional expressions. With better meaning-represennttionn correspondence, the non- compositional expert GIEA performs better than BART in downstream IE processing tasks. Yet, this advance is limited by the assumption that all IEs occur in their idiomatic sense and ignores their contextual ambiguity that makes them potentially idiomatic expressions (PIEs)- their meanings can be understood either literally or idiomatically in a context-dependen manner (Haagsma et al., 2 For example, the PIE "behind closed doors" should be interpreted literally in Always lock valuables behind closed doors and idiomatically in They avoided any publicity and made all deals behind
1 Introduction
Non-compositionality is characteristic of natural language. where the meaning of the expressions cannot be deduced from its components (Baldwin and Kim, 2010). These non-compositional expres- sions, often referred as being idiomatic, assume figurative meanings and are collectively common occurrence appearing in nearly three out of ten sen- tences in English (Moon et al., 1998) across various genres (Haagsma et al., 2020) The challenges they pose to NLP systems have been acknowledged as the classical 'pain in the neck' (Sag et al., 2002) and recently found to impact various NLP tasks negatively, such as sentiment analysis (Liu al., 2017: Biddle et al., 2020), dialog models (Jham- tani et al., 2021), and paraphrase generation (Zhou et al., 2021). Modern NLP systems, however, are
For simplicity, we hereafter refer to a sentence with an idiomatic//iteral PIR as an idiomatic/lieral sentence and their PIE embedding as idiomatic/literal embedding.

---

closed doors. Ideally, their representations ought to be distinct these two contexts. However, examin- ing the representation of 235 PIEs that are largely unrelated in their literal and idiomatic context (their literal PIE embeddings and idiomatic definitions have mean cosine similarity of 0,0047), we notice that their representations generated by the state-of- the-art (Zeng and Bhat, 2022) exhibit a high cosine similarity between their idiomatic and literal PIE embeddings (mean cosine similarity of 0.82).
ing IE embeddings' semantic quality by clustering them into meaning groups; the idiomatic embed- dings of PIER are superior compared to those of pre-trained BART in terms of homogeneity score (+0.1 15);; additionally, we evaluate the distinctive- ness between the literal and idiomatic embeddings and found that PIER can better differentiate PIE usage and has, average, +0.49 larger cosine distance for idiomatic and literal PIEs in the em- bedding space than GIEA.
(4) Extrinsic evaluations validate PIER'S utility; PIER outperforms both BART and GIEA on two PIE processing tasks- sense classification (ac- curacy +3.12% over GIEA and +2.67% over BART) and PIE span detection (sequence accuracy +3.29% over GIEA and +28.54% over BART). In two classic NLU tasks of sentiment classification and paraphrase identification, PIER compares more favorrbly with BART than GIEA, demonstrating that its NLU capabilities do not suffer at the cost of refining its PIE reprexentation".
Towards addressing this discrepancy, this study extends GIEA's ability in two concrete ways. First, through semantically meaningful representations for non-compositional expressions we enable ef- fective handling of non-composittonalliyy Second, by generating context- -aprooprate PIE represen- tations distinct for idiomatic and literal PIEs we enable effective contexiual disambiguation of PIEs. Addressing these issues involves attending to the following challenges. (1) BART and GIEA's abili- ties should be combined to generate good embed- dings for PIE in a context-dependent manner. (2) With the supervied reconstruction task as the sole objective, PTLM parameters are already op- timized for token reconstruction from their token embeddings. To represent PIEs, BART and GIEA's learning objectives should be revamped. To address these challenges, we propose Potentially Idiomatic Expression Representation generator (PIER). In- spired by AdapterFusion (Pfeiffer et al., 2021), which has been used to combine task-specific adapters, PTER generates embeddings by combin- ing the output from each GIEA adapter layer and pre-trained BART transformer layer with an atten- tion fusion layer serving routing mechanism that passes compositional non-compositional embeddings based on the context.. It is trained un- der the supervision of external knowledge, e.g., IE dictionary definition and PIE senses, that helps the model to disambiguate and comprehend PIEs' lit- eral and idiomatic meanings via a cosine-similarity based learning objective and a set of mask-infilling tasks with prompts.
2 Related Work
Non-compositional Phrase Embedding. Tradi- tional methods for non-compositional phrase em- bedding include learning adaptive weights com- bine the compositional (averaging word embed- dings) and non-compositional representation of the phrase (representing phrases with single to- kens) (Hashimoto and Tsuruoka, 2016; Li et al., 201 8a,b). These methods cannot be adopted for contextualized embeddings. PTLMs, though pro- ducing contextualized representations, are known for their inability to handle non-compositional phrases (Zeng and Bhat, 2022: Liu and Neu- big, 2022). GIEA (Zeng and Bhat, 2022), the first contextualized representation model for non- compositional phrases, efficiently adapts BART using adapter modules (Pfeiffer et al., 2020a) con- sisting of simple, parameter-efficient projection layers added between the trained transformer lay- ers, produce semantically meaningful IE embed- dings in a data-efficient manner compared to LM pre-training (~60MB vs. ~ 160GBB. Despite out- performing a fine-tuned full BART model, GIEA remains challenged by PIES" semantic ambiguity. PIER addresses this limitation through architectural modifications and additional prompt-based learn- ing objectives as detailed in Section 3.
Our main contributions are as follows.
(1) We propose PIER, a unified language model that combines pre-trained BART's compositional and GIEA's non-compositional representation abil- ities to generate semantically meaningful repre- sentations for both literal and idiomatic PIES in a context-dependent manner.
"The code for PIER can be found at https: //gi thub. com/zzeng13/Pe 1//PIER.
(2) We perform an intrinsic evaluation of the result-

---

phrase doors" neans crasiio.
GIEA Adapter
Attention Fusion of BART and GIEA
Query
Key
nade behind closed
The "bchind closed doors"
Senteen
Mean Pooling
PIE Definitin
Safema
Attention
BART
MPHet
Fusion
Fribedding
Defn Embedding
Cosine Similarity Loss
Cross-Entropy Loss
Final Losa Loss
Figure Overview of the PIER training framework.
Architectures for Information Fusion. Prior stud- ies have explored different architectures to fuse information in neural networks. For instance, an attention flow module (Sco et al., 2017) is pro- posed combine and fuse information from two vectors (query and context) for reading comprehen- sion. Yuan and Liu (2022) infuses external graph knowledge into pre-trained BART by adding a cross-attention module inside each BART decoder layer to infuse graph entity representation. In this work, we follow GIEA and use adapters to combine and route GIEA and BART embeddings. Adapters have also shown effectiveness in multi-task and multi-lingual transfer (Pfeiffer al., 2020b; Ansell al., 2021). Specifically, an AdapterFusion mod- ule (Pfeiffer et al., 2021) combines multiple trained task-specific adapters with a single attention layer to automatically select appropriate adapters for a given task. In PIER, we utilize an attention fusion layer, simplified version of an AdapterFusion module, to allow the LM to (a) combine BART and GIEA as the compositional and non-compositional language experts and (b) contextually select proper PIE PIE representation depending on whether the PIE is used idiomatically or literally. The attention fusion layer is explained Section 3 (See Figure I).
Zeng and Bhat, 2022). Prior work also suggests that semantically meaningful latent embeddings can be learned by optimizing the cosine similarity between source and target embeddings (Radford et al., 2021). Similarly, PIER utilizes dictionary definition for IEs compensate for the rarity of IEs and the relatively small IE-type training in- stances. We also guide the PIE representation learn- ing by optimizing the cosine similarity between the PIE embeddings and their corresponding defini- tion/PTLM embeddings.
3 Unified PIE Representation Generator
To create single language model that produces contextually appropriate embeddings for PIEs, in PIER, we combine BART's ability to generate em- beddings appropriate for compositional meanings and GIEA's non-compositional meanings such that PIER should output GIEA-style embeddings when the PIE is used idiomatically and BART's embedding otherwise.
We implement an attention layer that acts on the outputs from each frozen GIEA's adapter layer and frozen BART's transformer layer and serves as a "routing" mechanism for the compositional and non-compositional type embeddings. To train the attention layer, the overall loss is the sum of (1) cosine similarity-based part for the embedding to encode meaning via external dictionary defini- tions, and (2) reconstruction cross-entropy part that teaches the embedding of the association be- tween the PIE senses and sentence contexts. Opti- mizing these two losses jointly allows the model to link PIEs' meanings to their contextual uses. The
Auxiliary Guided Representation Learning. Auxiliary information to aid learning of language representations has been explored by using phrase knowledge mask and reconstruct token spans, e.g., noun phrases named entities, during train- ing to learn phrase representation (Joshi et al., 2020), and by including dictionary definitions to learn representations of rare words (Yu et al., 2022;

---

overview of PIER framework is shown in Figure 1.
have developed and incorporated the similarity learning objective and prompt infilling objectives, which we will describe in the subsequent sections.
3.1 Attention Fusion Layer
We implement attention fusion layers to route and propagate BART or GIEA's outputs layer by layer. insert attention fusion
3.2 Similarity Learning Objective
As shown in Figure layer after each GIEA's adapter layer and BART's transformer layer to combine them with attention weights into a single embedding vector that is sent to the next BART's transformer layer; the last at- tention layer outputs the final embedding vector. Specifically, each attention layer has three train- able weight matrices, namely, Key (K1), Value (Vi) and Query (Q:). The attention layer I takes two inputs, namely, GIEA's (-th adapter layer output at each token position i, gi,i and BART's I-th trans- former layer output, bi, then, it computes the contextually attention weighted representation as
From prior work (Zeng and Bhat, 2022), we in- fer that the quality and quantity of sentences with PIE are insufficient for unsupervised representation learning. This prompts us to use dictionary deñin- tions for idiomatic and original BART's embedding for literal PIES to create contextual awareness.
Specifically, given sentence with a PIE train- ing time, we first generate the PIE embedding by mean pooling PIER final output embeddings of the PIE tokens. Then, we generate two embeddings that aid the refining of the PIE embedding: (1) we generate an idiomatic embedding that encodes the non-compositional meaning of the PIE by using MPNet (Song al., 2020) to produce sentence embedding the PIE's idiomatic dictionary defini- tion. We use MPNet because prior work (Zeng and Bhat, 2022) found the resulting definition embed- dings help representation learning more than other models such as BART; and (2) we generate literal embedding that encodes the compositional mean- ing of the PIE by mean pooling a regular PTLM's (here, BART) final PIE token embeddings. Note that since the "literal" embeddings contextual- ized, they may already encode idiomatic meanings for frequently used idioms, including idioms ex- clusively used figuratively. Our model should still provide accurate semantics for these idioms since the attention fusion layer could pass the idiomatic semantics through. However, for the vast majority of idioms that are rare in text, BART's embeddings considered "compositional, not capturing their figurative meanings, hence we refer them lit- eral embeddings.
hr. [br, Bt,1
a = softmax (b, -,,
hy, b,,,V,
O1i ar. bi,
Note that our attention fusion layer a special, simplified case of AdapterFusion module; instead of fusing outputs from multiple adapters, our mod- ule fuses the embeddings from before (BART's transformer layer output) and after GIEA adapter layer. Intuitively, the attention layer at each layer uses linearly transformed BART's transformer layer output as query to the GIEA and BART representation to determine how much of each to- ken's BART's compositional representation needs to be substituted with GIEA's non-compositional representation based on its context. The atten- tion weight oi acts similarly to the PIE-specific weight that combines the compositional and non- compositional representations for computing PIE embeddings adjust the balance and mixture be- tween the compositional and non-compositional meaning prior works (Hashimoto and Tsuruoka, 2016; Li et al., 2018a.] b0. But, our layer-wise atten- tion weight is more contextualized and flexible.
Finally, we introduce learning objective for sentences with a literal PIE that maximizes the co- sine similarity between PIE and literal embeddings while minimizing the cosine similarity between the PIE and the idiomatic embeddings. For sentences with idiomatic PIE, we do the opposite: en- courage higher cosine similarity between PIE and idiomatic embeddings and lower cosine similarity between PIE and literal embeddings.
With the attention fusion layer, we can train the model using the copy objective, where the in- put and output sequences are identical, just as the GIEA model does. However, our experiments later demonstrate that simply adding the attention fu- sion layer with the copy objective is not enough to effectively learn PIE representations. Therefore,
3.3 Prompt Infilling
To directly provide the PIE sense information to the model and help it relate PIE senses with sentence contexts, we design two types of prompt-based

---

mask infilling tasks: (1) type classification prompts and (2) definition generation prompts.
(2022) to perform an intrinsic evaluation of the embeddings; 129 IEs form 20 groups with distinct meanings such that any two IEs from two different groups have different meanings while two IEs from the same group have similar meanings.
For the type classification prompts, we append the original sentence with another sentence that has a mask token, e.g., the phrase "see red" is used in its [MASK] sensee. and ask the model to infill the correct PIE sense, i..., "idiomatic" "literal", ac- cording to the context of the original sentence. As such, we directly inform the model of the existence and the distinction of the two PIE senses.
4.2 Models
We compare the performances of BART, GIEA. PIER and its variants to demonstrate the usefulness of the components in PIER.
For the definition generation prompts, we ap- pend a masked sentence, c.g., the phrase "see red" is used to mean [MASK]., and we ask the model to generate the definition of the idiomatic meaning in the place of the mask token if the PIE is idiomatic in the context; otherwise, the model should fill the mask with the PIE itself since the meaning is com- positional. Through these prompts, we allow the model to learn the two PIE senses' meanings and relate them with their contexts.
BART the pre-trained BART-base language model with six encoder and decoder layers.
GIEA is the non-compositional embedding gener- ator trained with a BART-base model and adapters using the MAGPIE train set.
BA BART-FT is BART-base model fine-tuned with the copy, similarity learning, and prompt infilling objectives.
FusionAttn is the model that combines BART and GIEA with the attention fusion layer and is trained with the copy objective with the cross-entropy loss. FusionSim combines BART and GIEA with the attention fusion layer and is trained with the copy and similarity learning objective.
We pre-defined five prompt templates for each prompt type (see Appendix A) based on our empir- ical observation that the variety of the prompt tem- plates positively influences the evaluation perfor- mances (see Section 4). We append these prompts to the end of the original idiomatic or literal sen- Sence.During training, we compute the mean cross- entropy loss for all tokens from the mask-in-filled output sentence, which we then add to the cosine similarity losses introduced in the last section to serve as the final loss. Note that unlike prompt- prompts
F FusionPrompt combines BART and GIEA with the attention fusion layer and is trained with the prompt infilling objective. The above four models are used to show the usefulness of the different components of our model.
PIER and PIER+ PIER combines BART and GIEA with the attention fusion layer and is trained with the type classification and definition gener- ation prompts with the reconstruction, copy, and similarity learning objectives. Only a single prompt template is provided for each prompt type. PIER+ is similar to PIER, but for each prompt type, we provide five templates. This model shows the bene- fit of using multiple prompts for each prompt type and is considered our final model.
based learning (Liu et al., 2022), teach LMs informative representations.
4 Experiments
4.1 PIE Datasets
Similar to Zeng and Bhat (2022), we use MAGPIE (Haagsma al., 2020), the largest-to-date dataset for English PIES with sentences sampled from the BNC (BNC Consortium, 2007). We selected all sentences with PIÉS that were unanimously labeled as idiomatic or literal by the MAGPIE annotators and have a single idiomatic definition according to Google dictionary and Wiktionary. In all, we had 32,693 sentences (77.4% idiomatic) with 1,480 PIES in the train set and 4,102 (77.57% idiomatic) sentences with 1,001 idioms in the test set. We use MAGPIE's official random split to divide the data into train and test sets where all the PIEs in the test data appear in the train data. We also use idiom meaning groups proposed by Zeng and Bhat
4.3 Evaluation Tasks
We conduct intrinsic and extrinsic evaluation tasks.
4.3.1
Intrinsic Evaluation
An intrinsic evaluation indicates if PIE embeddings are semantically meaningful and distinctive in the respective literal and idiomatic contexts. Embedding Generation. We evaluate the embed- ding quality by the competing models. We use a candidate model for each sentence to compute
and mean pool the PIE token embeddings to get a single embedding vector. Then, we compute and

---

mean pool across all idiomatic sentences get the idiomatic embedding for the PIE. Similarly, we get the literal embeddings for the PIEs. With the embeddings, we perform two intrinsic evaluations. Embedding Clustering, The procedure is in line with Zeng and Bhat (2022). Specifically, given model, we compute the idiomatic PIE embeddings for 129 idioms and cluster them into 20 distinct meaning groups using agglomerative clustering with complete linkage and pairwise embedding co- sine similarity as the distance metric. We measure clustering quality using a homogeneity score and the mean inter-group cosine distance between the embeddings for IEs from different groups. Because the ground truth meaning groups are distinct, the larger the homogencity scores (the score is 1.0 if all clusters contain only IEs from the same meaning group) and the mean inter-group cosine distances, the better the clustering quality.
tion. The classifier's linear layer is trained with the MAGPIE train set and is evaluated with Fl score and accuracy on the MAGPIE test set.
PIE Span Detection (SpanDET) is a more recent PIE processing task, a.k.a. IE identification (Zeng and Bhat, 2021; Škvorc et al., 2022), which is a spe- cial case of MWE identification Baldwin and Kim (2010) focusing on PIEs. Given sentence with PIE, token-level classifier is asked to classify every token as either idiomatic (positive) or literal; when a PIE is used literally, all tokens are classified as literal; otherwise, the tokens from the PIE are la- beled as idiomatic. To succeed, the classifier must correctly classify every token in the input sentence, effectively identifying the presence of an idiomatic PIE and precisely detecting its boundary simulta- neously. Since each MAGPIE sentence annotates single PIE, our models identify one idiomatic PIE per sentence. For the classifier, input each token embedding generated by the tested LM to two-layer MLP using ReLU activation, whose input dimension is the embedding dimension, and the hidden dimensions are halved after each layer. The classifier is trained with the MAGPIE train set, and only the MLP weights are trainable while the associated language model's weights are frozen. The performance is evaluated by sequence accu- racy and token recall. In sequence accuracy, an instance is considered correct if and only if all the tokens in the sequence are classified correctly. To consider model's ability to classify the sequence partially correct, we consider the token recall score by computing it for each test sequence and then averaging it across all test sequences.
Embedding Differentiation. The clustering evalu- ation examines only the model's ability to produce high-quality idiomatic embeddings. As discussed in Section 1, it is important for the language model to become innately aware of the difference between the idiomatic and literal meanings of the same PIE based on their context, Hence, given a model, we generate idiomatic and literal PIE embeddings for PIES with both idiomatic and literal sentences from the MAGPIE test set. We compute mean inter-type cosine similarity between pair of idiomatic and literal PIE embeddings across all PIEs with both literal and idiomatic sentences from the MAGPIE test set (there are 235 such PIEs). Assuming weak correlation between the literal and idiomatic meanings for PTEs, the smaller the mean inter-type cosine similarity, the better the differentiation.
To show that PIER does not sacrifice perfor- mance on NLU tasks, we consider two NLU tasks.
4.3.2 Extrinsic Evaluation
Sentiment Classification (SentCLF) classifies a given sentence into positive or negative sentiment. We use the SST2 (Socher et al., 2013) dataset and its default train and test splits (two classes) with 67,349 and 1,821 instances.
We include two classic PIE processing tasks and two NLU tasks for the extrinsic evaluation. PIE Sense Classification (SenseCLF) is a classic PIE processing task (Fazly et al., 2009; Feldman and Peng, 2013; Rajani et al., 2014; Peng and Feld- man, 2016; Salton et al., 2; Liu and Hwa, 2017; Taslimipoor et al., 2018; Peng et al., 2014; Liu and Hwa, 2019), a.k.a. idiom type classification. Each sentence with PIE is classified into two classes, idiomatic (positive) and literal, based on the PIE uses. Given a sentence with PIE and its location, we first use the model to generate its PIE embed-
Paraphrase Identification (ParalD) classifies a pair of given sentences into paraphrase non- paraphrase classes. We combine the MRPC (Dolan and Brockett, 2005) and PAWS (Zhang et al., 2019) datasets and their default train/test splits with a total of 53,069 train and 9,725 test instances.
For SentCLF and ParalD, we train a new adapter with the default Pfeiffer configuration (Pfeiffer et al., 2020a) stacked atop the testing models, mak- ing only the paraphrase classifier adapter trainable
ding: then, the PIE embedding is passed to a lincar and softmax layer to perform the binary classifica-

---

during training. Performances are evaluated using the Fl score and accuracy.
BART by 2.66% and GIEA by 3.12%. Note that BART's type classification accuracy is already high at 93.71% %, and GIEA's performance only com- parable with that of BART (not better). This is because BART and GIEA are only compositional and non-compositional expression experts (treating all PIEs as either literal or idiomatic), respectively. As shown by the intrinsic evaluation, none of their embeddings are distinctive enough for idiomatic and literal PIE embeddings. Because PIER+ pro- duces different PIE embeddings based on context, improves over the already high type classification accuracy from BART and GIEA.
Note that we freeze the testing language model and deliberately constrain the complexity of the classifiers to a linear layer, MLPs, or adapter layer to ensure that the performance primarily reflects the quality of the PIE embedding. See Appendix B for more details on the general setup.
5 Results and Analyses
Intrinsic Evaluation. As shown in Table 1, BART has the lowest homogeneity score (0.45) and inter- group cosine distance (0.037), indicating that the groups using the idiomatic embeddings do not cor- respond to those grouped by meaning. PTER+ has an absolute 0.15 0.15-point gain in both the homogene- ity score and the inter-group cosine distance. While GIEA has a larger homogeneity score and inter- group cosine distance than PIER+, its inter-type cosine similarity is very high (0.82). This confirms that GIEA cannot generate contextually appropri- ate embeddings for idiomatic and literal PIEs and instead treats them idiomatic in all contexts, thus ignoring their contextual ambiguity. In compari- son, PIER+ achieves an inter-type cosine similar- ity of 0.49 lower, generating more contextually distinctive PIE embeddings. We hypothesize that PIER+ achieves a lower homogeneity score and inter-group cosine distances than GIEA because it contextually fuses BART and GIEA embeddings, thus making its idiomatic embeddings less distinc- tive in terms of idiomatic meanings. However, as we will show in Section 5, PIER+ embeddings en- code information that helps it to achieve even better performance PIE processing tasks. Additionally, we emphasize that PIER+ is not a mere interpola- tion between BART and GIEA, as it goes beyond simple combination techniques. It disambiguates idioms' literal and figurative senses based on the sentence context and generates appropriate embed- dings with accurate semantics. A naïve interpola- tion approach, such as concatenation or averaging BART and GIEA's embeddings, would indeed re- sult in high H-scores (0.6214 4 and 0.6154) and Cos- Dist (0.1 1503 and 0. 031555), but also high DiffSim scores (0.7934 and 0.7954), which is undesirable. Performance on PIE Processing Tasks. Un- surprisingly, PIER+ outperforms both BART and GTEA in the classic PIE processing tasks- SenseCLF and SpanDET-in all metrics as shown in Table I. For the SenseCLF, PIER+ outperforms
Similarly, for SpanDET, a much more demand- ing task requiring detection and tagging simulta- neously, PIER+ has sequence accuracy that is 28. 28.54% higher than BART and 3.29% higher than GIEA. We point out that 22.43% of sentences in the test set have literal PIEs, over which GIEA's sequence accuracy is only 71.84% while PIER+ + has a sequence accuracy of 85.76%, gaining 13.91% 1 absolute points in accuracy. Observing the token- level recalls, PIER+'S +'s performance is only slightly better than that of GIEA, yet leads to 3%++ gain in sequence accuracy. Plausibly, this is because of FusionMultiPrompt's better ability to recognize literal PIEs (as shown by the ~ 14% sequence ac- curacy gain on literal sentences). So, with its abil- ity to produce meaningful idiomatic embeddings, GIEA achieves high sequence accuracy in Span- DET, whereas PIER+'S 's ability to generate appropri- ate literal embeddings allows it to improve further.
Performance on NLU tasks. PIER+ performs competitively with BART and GIEA (F1 and ac- curacy differing by around +/- 1%). Given that the main purpose of PIER+ is for producing high- quality PIE embeddings and enhancing IE process- ing ability, the results on the NLU tasks lead us to conclude that (1) PIER+ (and GIEA to lesser tent) adequately processes sentences with with- out PIEs and thus performs comparably with BART on classic NLU tasks, i.e., PIER+ does not break- down on sentences without PIEs; and (2) given that PIER+ produces PIE embeddings with supe- rior semantic propertics and performs very well on PIE processing tasks, we believe PIER+ is overall better LM than BART for PIE processing.
Effect of Individual Components. As shown by FusionAttn's 's performances in Table and 2, naïvely adding an attention fusion layer to com- bine GIEA and BART with a copy objective does

---

Model
H-Score (?)
CosDist
DifiSim (L)
BART
0.4546
0.0379
0.7495
GIEA
0.6450
0.2284
0.8224
BART-FT
0,4510
0.0331
0.8198
AusionAtto
0.4306
0.0357
0.8495
FusionSim
0.5015
0.0924
0.6428
PusionPrompt
0.4160
0.0495
0.7843
P-CIs
0.5756
0.1527
0.3468
P-Defn
0.3751
0.1546
0.3547
PIEK
0.5844
0.1782
0.3272
PIER+
0.6095
0.1838
0.3230
Table I: Intrinsic evaluations measured by clustering homogeneity score (H-Score 1), mean inter-group cosine distance (CosDist 1), and mean inter-type cosine similarity (DiffSim L). Best performances are boldfaced.
SLn
SpanDET
SentCLF
Parall
Model
FT
Åcc
SA
TR
F1
Ace
FT
Acc
BART
0.9589
0.9371
0.5076
0.7545
0.9246
0.9232
0.9165
0.9225
GIEA
0,9573
0,9325
0.7601
0.9075
0.9145
0,9117
0.9046
0,9103
BART-FT
0.9614
0,9408
0.4720
0.7907
0.9364
0.9346
0.9152
0.9207
FusionAtln
0.9605
0,9386
0.5651
0.6343
0.9158
0,9140
0.9031
0.9098
FusionSim
0.9642
0.9447
0.6131
0.6415
0.9243
0,9232
0.9068
0.9121
FusionPrompt
0.9632
0.9430
0.5405
0.8149
0.9025
0.9084
0.9270
0.9243
P-CIs
0.9712
0.9558
0.7370
0.8848
0.9208
0.9197
0,9069
0.9128
P-Defn
0.9720
0.9571
0.7190
0.8810
0.9315
0.9300
0.9060
0.9118
PIER
0.9749
0.9612
0.7864
0.9029
0.9181
0.9163
0.9027
0.9096
PIER+
0.9765
0.9637
0.7930
0.9101
0.9290
0.9278
0.9068
0,9122
Table 2: Performances extrinsic evaluation tasks; binary classification tasks, namely, PIE classification (SenseCLF), sentiment classification (SentCLF), and paraphrase identification (ParalD), are measured by F1 score (FI) and accuracy (Acc), and the PIE span detection (SpanDET) measured by sequence accuracy (SA) and token-level recall (TR). Best performances are boldfaced.
not work; the homogeneity score is lower even than BART while the inter-type cosine similarity is higher than GIEA; also, FusionAttn's sequence accuracy for SpanDet is only 5.75% higher than BART yet 19.5% lower than GIEA. FusionAttn's poor performance highlights the fact that the re- construction task with the copy objective alone is insufficient for the model learn the intended em- beddings as discussed in Section 1. Although after adding the similarity learning objective, FusionSim shows the effectiveness of similarity learning ob- jective compared to FusionAttn ((c.g., +4.7% in SpanDET sequence accuracy), it underperforms GIEA. Similarly, FusionPrompt achieves marginal gains over BART (e.g., +3.39% in SpanDET se- quence accuracy) by combining attention fusion and prompt infilling objectives, yet it severely un- derperforms GIEA. Moreover, having both the sim- ilarity learning and the prompt infilling objectives, BART-FT model exhibits an intrinsic quality that is similar to GIEA or BART, yet, without the at- tention fusion layer, it underperforms PIER+ in all intrinsic evaluation tasks and PIE processing tasks (i.e., SenseCLF and SpanDET). These results
indicate that neither cosine similarity forcing nor prompt infilling objective alone sufficient, and that it would require the utilization of the attention fusion layer to route GIEA and BART to produce appropriate PIE embeddings.
Finally, we tested the usefulness of the classifica- tion and the generation prompts through an ablation study, where we compare PIER with only the type classification prompt (P-CIs) and PIER with only the definition generation prompt (P-Defn). Each prompt type utilizes the same five prompt templates as the PIER+ model. As shown in Tables and 2, even with a single prompt type, our models achieve significant improvements in both intrinsic evalua- tion and PIE processing tasks while maintaining a competitive performance in NLU tasks. However, when combining both prompt types, PIER+ out- performs P-CIs and P-Defn, especially in the most difficult SpanDET task, with gains of 5.6% and 7.4% sequence accuracy, respectively. These re- sults lead us to infer that combining the two types of prompts is beneficial and leads to further perfor- mance gains.
Effect of Combined Components. The salient

---

effect of combining all components is shown by comparing FusionSim and PIER. PIER gains 16.5% in homogeneity score and 0.32 inter-type cosine similarity while achieving 1.65% higher accuracy in SenseCLF and 17.3% higher sequence accuracy in SpanDET. Moreover, comparing PIER+ to PIER, we observe further meaningful gain in all metrics across all tasks, indicating the benefit of including multiple prompt templates.
'he remaining 16 PILs exhibit no discernible prop- erties, and we leave deeper dive into these hard- to-learn PIEs for a future study.
6 Conclusion and Future Work
We propose PIER, a language model that uses at- tention fusion to combine BART and previously proposed adapter to produce semantically mean- ingful and contextually appropriate representations for PIEs. Training using a prompt-infiling objec- tive for contextual PlE-type awareness and co- sine similarity objective guide the generated PIE embeddings toward their idiomatic literal mean- ings resulted in PIER generating PIE embeddings with superior semantic properties. IE-aware PTeR outperforms both BART and GIEA on IE process- ing tasks with BART-level performance on classic NLU tasks. These results demonstrate PIER'S use- fulness idiom-aware LM.
5.1 Performance and Error Analyses
Effect of PIE Properties. Psycholinguistic find- ings shed light on how idioms' frequency, semantic and syntactic properties affect human IE compre- hension Slaban-Bezale and Mashal, 2019; Lada et al., 2023). Informed by these results, we an- alyze the effeet of PIE training data size and IE semantic/syntactic properties on PIER'S IE process- ing competence through correlational analyses. We found that PTE frequency the train set does affect the PIER+'S intrinsic embedding quality but not the downstream PIE processing task performances. Additionally, we examined the correlation between PIER+'s performances from all evaluation tasks to three IE properties decomposability. i.e., the degree to which an IE's constituent words contribute to its figurative meaning, literalness, i.c., the extent an IE can be used in a literal sense, and flexibility, i.e., how flexible are IEs to morphosyntactic or internal modifications. We found little to no correlation berween model performance and these properties. See Appendix C for more details.
Future directions could explore methods fur- ther enhance the IE awareness of broader types of non-compositional constructions beyond idioms (c.g., metaphors and similes).
Limitations
PIER has two main limitations. First, PIER not expected to produce embeddings for PIEs unseen during training when used in their figurative sense. This is because each PIE has conventionalized figurative interpretation stemming from its unique origins and metaphorical linking, which requires external and PIE-specific knowledge (e-g.. PIE def- initions) for PIER'S 's learning of high-quality PIE embeddings. It is likely that PIER can 'guess' the figurative meanings for certain PIES with high de- composability or when BART already encodes their semantics during its pre-training. As such, this generalizability to unseen PIEs is not guaranteed with PIER, and we do not currently see a practical way to enable it. Second, PIER requires supervi- sion the form of sentences with the PIEs clas- sified as literal or figurative. Given that BART's self-supervised learning objective during its pre- training fails to take this into account, we argue that using supervised learning with adapters is a practical alternative to capture IE semantics while reducing the number of trainable parameters and associated training data. With the availability of multi-lingual resources (Tedeschi al., 2022) for PIEs now becoming available, and automatic PIE sense classification methods that generalize to un-
PIE Embedding Error Analysis. Given that lin- guistic properties PIEs are not the main con- tributing factors to PIER+'S embedding quality, we conduct further analyses on PIE embedding errors. Specifically, we analyze the PIÉs that are poorly differentiated by PIER+. Selecting from the 235 PIES used in the embedding differentiation test, we pick PIEs whose PIER+ embeddings have inter- type cosine similarity larger than 0.7 (highly similar idiomatic and literal embeddings). Observing the resulting 60 (25.5%) PIEs, we find that 44 (73.3%) have a very skewed idiomatic/literal sentence dis- tribution in the training data. Notably, the number of idiomatic sentences divided by the number of training sentences for that PIE is either over 0.85 lower than 0.15, suggesting that these are either only used idiomatically (idioms) or only as literal expressions in the training data, resulting in a low embedding differentiation among the PIE senses.

---

seen PIES (Zeng and Bhat, 2021), we believe this requirement of training corpus may be less of a bottleneck. More broadly, although we show that PIER does not lose its NLU ability on avail- able tasks, we leave the application of PIER to IE-centered NLU tasks (e.g., NLI with figurative language) to future studies.
Rhys Biddle, Aditya Joshi, Shaowu Liu, Cecile Paris, and Guandong Xu. 2020. Leveraging sentiment dis- tributions to distinguish figurative from literal health reports on Twitter. In Proceedings of The Web Con- ference 2020, pages 1217-1227
BNC Consortium, 2007. British national corpus, XML edition. Oxford Text Archive.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford. 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Ethics Statement
The intended use of our system is to serve as a pretrained language model capable of adequately handling idiomatic language. As such, the intended users are those interested in fine-tuning PIER or us- ing PIER's embeddings rich in idiomatic semantics for downstream NLP applications, such detect- ing idiomatic expressions in text, sentiment anal- ysis of text with idioms. In case of model failure, PIER may produce embeddings that do not accu- rately reflect the true figurative meanings of the IEs and thus negatively impact the downstream perfor- mance. Therefore, we advise against using PIER part of decision models in critical situations, such medical or financial scenarios. To ensure PIER performance as expected, idioms covered by PIER's training should be used. Beyond this, the best of our knowledge, PIER does not introduce or contain any additional bias. PIER is trained using datasets that are publicly available and reputable. We do not collect or use any data that can violate privacy rights.
Nyssa Z. Bulkes and Darren Tanner. 2017. "going to town": Large-scale norming and statistical analysis of 870 american english idioms. Behavior Research Methods, 49(2):772-783.
Mathieu Constant, Gülşen Eryigit, Johanna Monti, Lon- neke Van Der Plas, Carlos Ramisch, Michael Rosner. and Amalia Todirascu. 2017. Multiword expression processing: A survey. Computational Linguistics, 43(4):837-82..
Bill Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Third International Workshop Paraphrasing (IWP2005). Asia Federation of Natural Language Processing.
Acknowledgements
Afsanch Fazly, Paul Cook, and Suzanne Stevenson. 2009. Unsupervised type and token identification of idiomatic expressions. Computational Linguistics, 35(1):61-103.
This research was supported in part by the National Science Foundation under Grant No. IIS 2230817 17 and by U.S. National Science Foundation and Institute of Education Seiences grant (2229612).
Anna Feldman and Jing Peng. 2013. Automatic detec- tion of idiomatic clauses. In International Confer- ence on Intelligent Text Processing and Computa- tional Linguistics, pages 435-446. Springer.
References
Berit Gehrke and Louise McNally. 2019. Idioms and the syntax/semantics interface of descriptive content reference. Linguistics, 57(4)769-8814
Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Se- bastian Ruder, Goran Glavaš, Ivan Vulié, and Anna Korhonen. 2021. MAD-G: Multilingual adapter gen- eration for efficient cross-lingual transfer. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 4762-4781, I, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 279- 287.
Timothy Baldwin and Su Nam Kim. 2010. Multiword expressions. In Nitin Indurkhya and Fred J. Dam- erau, editors, Handbook of Natural Language Pro- cessing, Second Edition, pages 267-292. Chapman and Hall/CRC.
Kazuma Hashimoto and Yoshimasa Tsuruoka. 2016. Adaptive joint learning of compositional and non- compositional phrase embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers),

---

pages 205-215, Berlin, Germany, Association for Computational Linguistics.
Emmy Liu and Graham Neubig. 2022. Are representa- tions built from the ground up? an empirical exami- nation of local composition in language models. In Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), Abu Dhabi, UAE
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for Nİ.P In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Pengfei Liu, Kaiyu Qian, Xipeng Qiu, and Xuan-Jing Huang. 2017. Idiom-aware compositional distributed semantics. In Proceedings of the 2017 conference on empirical methods in natural language processing. pages 1204-1213
Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Tay- lor Berg-Kirkpatrickk 2021. Investigating robustness of dialog models to popular figurative language con- structs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1476-7485. Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Pengfei Liu. Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. 2022. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv. Just Accepted.
Rosamund Moon et al. 1998. Fixed Expressions and Id- ioms in English: A Corpus-Based Approach. Oxford University Press.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zeulemoyer, and Omer Levy. 2020. Span- BERT: Improving pre- pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Jing Peng and Anna Feldman. 2016. Automatic id- iom recognition with word embeddings. In Infor- mation Management and Big Data Second Annual International Symposium, SIMBig 2015, Cusco, Peru, September 2-4, 2015, and Third Annual International Symposium, SIMBig 2016, Cusco, Peru, September 1-3, 2016, Revised Selected Papers, volume 656 of Communications in Computer and Information Sci- ence, pages 17-29. Springer.
Anastasia Lada, Philippe Paquier, Christina Manouili- dou, and Stefanie Keulen. 2023. A systematic re- view: Idiom comprehension in aphasia: The effects of stimuli and task type. Journal of Neurolinguistics 65:101115.
Jing Peng, Anna Feldman, and Ekaterina Vylomova. 2014. Classifying idiomatic and literal expressions using topic models and intensity of emotions. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 2019-2027. Association for Computational Linguistics.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyr.. 2020. pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
BART: Denoising
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho. and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volune, pages 487-503, Online. Association for Computational Lin- guistics.
Bing Li, Xiaochun Yang, Bin Wang. Wei Wang, Wei Cui, and Xianchao Zhang. 2018a. An adaptive hier- archical compositional model for phrase embedding. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, 1JCAI-18, pages 4144-4151, International Joint Conferences on Artificial Intelligence Organization.
Jonas Pfeiffer, Andreas Rücklé Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations. pages 46-54, Online. Association for Computational Linguistics.
Minglei Li, Qin Lu, Dan Xiong, and Yunfei Long. 2018b. Phrase embedding learning based on external and internal context with compositionality constraint. Knowledge-Baan Based Systems, 152: 106-116.
Changsheng Liu and Rebecca Hwa. 2017. Represen- tations of context in recognizing the figurative and literal usages of idioms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Se- bastian Ruder. 2020b. MAD-X: An Adapter- Adapter-Based Framework for Multi-Task Cross- Pross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages
Changsheng Liu and Rebecca Hwa. 2019. A general- ized idiom usage recognition model based on seman- tic compatibility. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pages 6738-6745.
Online, Association for Computa- tional Linguistics.

---

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In ICML.
Shiva Taslimipoor, Omid Rohanian, Ruslan Mitkov, and Afsaneh Fazly. 2018. Identification of multiword expressions: A fresh look at modelling and evalua- tion. In Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop, vol- ume 2, page 299. Language Science Press.
Nazneen Fatema Rajani, Edaena Salinas, and Raymond Mooney. 2014, Using abstract context to detect figu- rative language.
Simone Tedeschi, Federico Martelli, and Roberto Nav- igli. 2022. ID10M: Idiom identification 10 lan- guages. In Findings of the Association for Computa- tional Linguisties: NAACL 2022, pages 2715-2726, Seattle, United States. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguisties.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Ronit MaraallBezzaellaae and Nira Mashal. 2019. Different factors predict idiom comprehension in children and adolescents with ASD and typical development. Autism Dev Disord, 49(12):4700477) 4750..
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-ar natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguistics.
Ivan A Sag. Timothy Baldwin. Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions: A pain in the neck for NLP. In Interna- fional conference on intelligent text processing and computational linguistics, pages 1-15. Springer
Giancarlo Salton, Robert Ross, and John Kelleher. 2016. Idiom token classification using sentential distributed semantics. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 194-204.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1715-1725, 1725, Berlin, Germany. Association for Computational Lin- guistics.
Yonghui Wa, Mike Schuster, Zhifeng Chen, Quoc V. Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah. Melvin Johnson. Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. CoRR, 0///1669984444
Minjoon Sco, Aniruddha Kembhavi, Ali Farhadi, and Hannanch Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In 5th Inter- national Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Con- ference Track P Proceedings.
Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang. Yichong Xu, Michael Zeng, and Meng Jiang. 2022. Dict-bert: Enhanccing language model pre- are-training with dictionary. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022. pages 1907- 1918, Association for Computational Linguistics.
Richard Socher, Alex Perelygin. Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Weizhe Yuan and Pengfei Liu. 2022. Kid-review: Knowledge-guided scientific review generation with oracle pre- pre-training. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 36(10):11639-110 11647.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. In Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020. viriual.
Ziheng Zeng and Suma Bhat. 2021. Idiomatic ex- pression identification using semantic compatibility. Transactions of the Association for Computational Linguistics, 9:1546-1562.

---

Ziheng Zeng and Suma Bhat. 2022. Getting BART to Ride the Idiomatic Train: Learning to Represent Id- iomatic Expressions. Transactions of the Association for Computational Linguistics, 10:1 120-1137.
A Prompt Templates
Table 3 shows the prompt templates we used for the type classification and definition generation prompt infilling objective. During training, we ram- domly sample third of the sentences to append the type classification prompts, third to append the definition generation prompts, and the rest of the sentences are retained without appending prompts. We uniformly sample one of the five prompt tem- plates for each chosen prompt type. We reserve sentences without masked prompts appended to sure the model can properly generate embeddings for sentences without prompts after training. Cross- entropy and cosine similarity losses were also ap- plied to these unaltered sentences for generating appropriate embeddings during training.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1988-1388, Minncapolis, Minnesota. Association for Computa- tional Linguistics.
Jianing Zhou, Ziheng Zeng, Hongyu Gong, and Suma Bhat. 2021. Idiomatic expression paraphrasing with- out strong supervision.
Tadej Škvore, Polona Gantar, and Marko Robnik- Sikonja. 2022. Mice: Mining idioms with con- textual embeddings. Knowledge-Bass Systems, 235:107606.
B Experiment Setup
We use the following implementation and check- points: BART-base and MPNet as maintained by Huggingface (Wolf et al., 2020) and Sentence- Transformers (Reimers and Gurevych, 2019); the adapters based on the code released by AdapterHub (Pfeiffer et al., 2020a); and GIEA as released by its authors. With the exception of BART, which requires no fine-tuning, we train all language mod- els for 35 epochs with batch size of 8. For PIE sense classification, all classifiers are trained for 55 epochs with a batch size of 32. For PIE span detection, all classifiers are trained for 100 epochs with batch size of 16. For the NLU tasks, we train all classifiers for 30 epochs with a batch size of 16. We use the Adam optimizer across all ex- periments and set other hyperparameters their default values.
C Effect of IE properties on Performances
Effeet of Training Data Size. To understand how the availability of training data impacts PIER'S per- formance, we compute the correlation between the PIER's (FusionMuultiPrompp)) per-PIE performance (averaging performances over each PIE type) to the number of training sentences for each PIE (as proxy for their frequency in the wild). For the intrinsic evaluation, we focus correlating each PIE's mean inter-type cosine similarity, which is in- dicative of the contextual appropriateness of PieR'S embeddings, to its number of training instances; we find that the embedding differentiation for a

---

Type Classification Prompts
Definition Generation Prompts
The phrase "[PIE]" quite [MASK].
The meaning of the phrase "[PIE] IS [MASK].
The phrase "[PIE]" used its [MASK] sense.
The definition of the phrase "[PIE]" is [MASK].
The phrase '[PIL]" is used as the [MASK] expression.
The phrase "[PIE]" means [MASK].
The phrase '[PIE]" is the [MASK] way of saying it.
The phrase "[PIE]" is defined as [MASK].
The phrase "[PIF]" takes on its [MASK] meaning.
The phrase "[PIE]" is used to express [MASK].
Table 3: Prompt templates for type classification and definition generation prompts. Each prompt template has a placeholder [PIE], which will be substituted into the actual PIE when appending to training sentences.
given PIE has low correlation with the data quan- tity for that PIE (Pearson correlation coefficient less than 0.15). However, for the PIE processing tasks, PIER'S 's accuracy for both type classification and span detection positively correlates with the number of training sentences; the Pearson corre- lation coefficients and p-values are (0.73, 6c-21) and (0.66, 2e-69), respectively. So, despite the low correlation of training data size with the intrin- sic embedding quality, it has a positive impact on downstream processing performance.
for PIE span detection, the per-PIE sequence accu- racy is weak but positively correlated with decom- posability (correlation = 0.2537, p-value 6c-4). Hence, PIE decomposability only weakly impacts the PIE span detection performance while having significant impact on the intrinsic quality of PIE embeddings or PIE sense classification.
Effect of PIE Literalness. Here, we study the impact of PIEs contextual ambiguity (represented as their literalness scores from Bulkes and Tanner (2017)) on PIER'S performance. Literalness cap- tures the extent to which a phrase can be used in its literal sense on a scale of to 5, with a 5 indi- cating IE has a clear, well-formed, and plausible literal interpretation. We hypothesize that literal- ness would not affect the embedding quality and downstream performances since PIER is designed to address PIEs' contextual ambiguity. We use the same PIE types and per-PIE performance metrics from the decomposability study above. Computing the Pearson correlation between the literalness and PIER'S 's performance for the intrinsic and PIE span detection tasks, we again found both correlation coefficients to be negligible with magnitudes less than 0.1. For PIE sense classification, the corre- lation coefficient between per-PIE accuracy and literalness is -0.1213 with a p-value of 0.1068 (not statistically significant at a 0.05). This sug- gests that PIE's literalness has no overall impact on PIER'S ' PIE embedding and IE processing abilities. Effect of PIE Flexibility. We examine the im- pact of PIE's syntactic flexibility (or frozenness) (Constant et al., 2017: Gehrke and McNally, 2019) on PTER 's performance. We analyze the PIE span detection performance with respect to the idiom fixedness levels. According to the definitions given by Sag et al. (2002), PIEs (and lexicalized phrases in general) can be categorized into three levels: (1) fixed (e.g.. ahead of the game)-PlB: with no morphosyntactic or internal modification, (2) semi- fixed, (e.g., go down the rabbit hole) re- stricted lexical variations (went down the rabbit hole) such as inflection and determiner selection,
Effect of TE Decomposability. Decomposability is a semantic property of IEs capturing the degree to which an IE's words contribute to its figurative semantics. For example, quick as a flash is highly decomposable because its constituent words quick and flash both contribute to its overall figurative meaning, i.e., very quickly. On the other hand, get your feer wet, which is used mean begin to par- -on-ppcomp...be.. To
ticipate in an activity, understand the impact of IE semantic properties, we study the association (in terms of Pearson's 's cor- relation coefficient) between PIER's 's per-PIE perfor- mance to IE decomposabilityy For the quantitative decomposability scores, we use the largest-to-date linguistic resource available (Bulkes and Tanner, 2017), which contains descriptive norms for 870 American English idioms gathered from 2,100 hu- man participants (several idioms have 100+ anno- tations). Of these, 39 overlap with the PIE types from the intrinsic evaluation set and 178 with the PIE types from the PIE processing tasks. The de- composability scores are binary, with being de- composable and O being non-decomposable and were averaged over more than 100 human-assigned scores per PIE.
For the intrinsic evaluation, we find that the mean inter-type cosine similarity has low asso- ciation with PIE's decomposability (correlation of 0.2012 with a p-value of 0.2194). Similarly, for PIE sense classification, the per-PTE accuracy also not correlated with decomposability (the magnitude of the correlation is less than 0. 1).. On the other hand,

---

while maintaining strict word ordering and com- position, and (3) syntactically-llxibbee (c.g., stab someone the maita-mi only the basic word order while allowing a multitude of syntactic variations on the internal words. We use the flex- ibility measurements provided by Zeng and Bhat (2021), which contains 139 PIE types. Intersect- ing with PIE types in our experiments, we find 60 PIE types for the intrinsic evaluation and 129 PIE types for PIE processing tasks. After computing the Pearson correlation for PIER'S performance from each evaluation task, we found all correlation coef- ficients have magnitudes of less than 0.1, indicating that the PIE flexibility does not affect the quality of PTER's ' PIE embedding, nor does it impact the performance of the downstream processing task.


Reviews:
[{"review_id": "WH0xXoXklh", "reviewer": "Reviewer_mdhy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Question**: PIER works as an interpolation of BART and GIEA; its performance is somehow expected on intrinsic evaluation.*\n\nThank you for the review. We would like to clarify that our PIER model is not a mere interpolation between BART and GIEA, as it goes beyond simple combination techniques by disambiguating idioms' literal and figurative senses based on the sentence context and generating appropriate embeddings with accurate semantics. A naïve interpolation approach, such as concatenation or averaging BART and GIEA's embeddings, would indeed result in high H-scores and CosDist in the intrinsic evaluation due to inheriting GIEA's embedding. However, this would also lead to an undesirable, high DiffSim score, which represents the difference between the embeddings for the literal and figurative meanings of the same PIE that are not clearly distinguished.\n\nAs demonstrated in the table below (results were not part of the paper), averaging or concatenating BART and GIEA's embeddings results in a DiffSim score that is approximately 0.47 higher than our proposed PIER model. This indicates that our PIER model surpasses simple interpolation methods by effectively discerning contextual ambiguity, leading to superior embedding quality.\n\n| Model         | H-Score (↑) | CosDist (↑) | DiffSim (↓) |\n| ------------- | ----------- | ----------- | ----------- |\n| BART          | 0.4546      | 0.0379      | 0.7495      |\n| GIEA          | 0.6450      | 0.2284      | 0.8224      |\n| Concatenation | 0.6214      | 0.1503      | 0.7934      |\n| Averaging     | 0.6154      | 0.1355      | 0.7954      |\n| PIER+         | 0.6095      | 0.1838      | 0.3230      |\n\nTo provide better clarity on this point, we will include the results from the interpolation methods in our revision and explain their significance.\n\n-----\n\n***Question**: PIER still faces the same challenge as GIEA, that the method is applicable only if a PIE is seen during the training stage. Although the authors pointed out this in Limitations, it's better to explore to what extent that idiomaticity can be sensed by models simultaneously.*\n\nThank you for your comment. We acknowledge that like GIEA, our proposed PIER model, is limited to working with PIEs that are encountered during training. However, we contend that this constraint is necessary to ensure high-quality representations for PIEs when using smaller-scale pretrained language models that have not adequately learned the figurative meanings of PIEs during their pre-training phase.\n\nThe main reason for this limitation lies in the nature of PIEs being collectively frequent yet individually rare. Due to the individual scarcity of a large number of PIEs in natural language, pretrained language models struggle to capture their figurative meanings and to associate them with the appropriate sentence contexts. Therefore, instead of increasing the amount of training data during pre-training to include a sufficient number of idiomatic sentences, which would be costly and impractical in most working scenarios , our method supplies explicit knowledge of the PIE's figurative meaning (i.e., their dictionary definitions) to facilitate the model's learning of PIE meanings and their association with corresponding contexts.\n\nWhile it would be interesting to explore the extent to which our model can generalize to unseen idioms during training, we do not anticipate strong generalizability due to the design of our method (highlighted above). Any generalization observed would likely be more indicative of what is already encoded in the pretrained language models rather than the merit of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a unified view on learning PIE representations, it integrates several past works like BART, GIEA and MPNet. To make comparisons with these methods, the paper designs different baselines addressing possible directions of fusion. \n\nFrom evaluation perspective:\n[1] PIER shows consistent improvement over GIEA on generating contextually appropriate embeddings for idiomatic and literal PIEs.\n[2] PIER is useful on downstream tasks without suffering its NLU capabilities.\n", "reasons_to_reject": "[1] PIER works as an interpolation of BART and GIEA, its performance is somehow expected on intrinsic evaluation. \n[2] PIER still face the same challenge as GIEA that the method is applicable only if a PIE is seen during training stage. Although the authors pointed out this in Limitations, it's better to explore to what extend that idiomaticity can be `sensed` by models simutaneously.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper studies noncompositionality of phrases and proposes a model PIER to generate contextualized representations for them. PIER is constructed over BART and GIEA (BART with Adapter) to address the discrepency that GIEA's representations for the idiomatical and literal PIE are highly correlated (has a high similarity). Experiments for intrinsic evaluation show that the new method is effective in separate literal and idiomatic embeddings. Extrinsic evaluations indicate that PIER can achieve significant improvement on PIE-realted tasks. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "[1] Line 116: duplicate full stops.\n[2] Line 141: Contribution (3) is missing."}, "extra_text": {"Missing_References": "At BERT age, \n[1] there have been similar methods motivated by using both literal and idiomatic representations for downstream tasks, i.e. {tan-jiang-2020-bert,tan-jiang-2021-learning}.\n[2] there have been datasets that also discussed methods on compositional representations {tayyar-madabushi-etal-2021-astitchinlanguagemodels-dataset} and other methods from SemEval 2022 Task 2.\n[3] As the model does not lose NLU abilities but achieve better embeddings, I am wondering how the model performs on an idiom paraphrase identification task, which is proposed by {pershina-etal-2015-idiom} and {tan-jiang-2021-bert}."}}, {"review_id": "EwZtr6IMO3", "reviewer": "Reviewer_uedZ", "summary": "", "questions": "- line 303: when you generate a literal embedding, do you use token embeddings (layer 0) or contextualised embeddings (final layer)? I was thrown off by the usage of the word “final” in line 306, hence the question. When using the contextualised embeddings, this may, after all, not be a compositional embedding, considering that BART would have some accurate idiomatic representations, particularly for the more frequent idioms (whose idiomatic meaning is highly conventionalised).\n- line 367 “...where all the PIEs in the test data appear in the train data”: have you run some experiments where that was not the case? I’m curious to see whether there can be non-iid improvements as well. I consider it somewhat unlikely, since without having seen training data, the meaning of an idiom can be somewhat unpredictable, but still.\n- For the training techniques that are not specific to fusing the two models (copy objective, infilling, cosine similarity objective) have you applied those to BART or GIEA without including the fusion attention layer? That could help to disentangle your various contributions, because running full PIER might be impractical for future work, whereas adding one of those components might be quite a bit easier. (e.g. I can imagine that the copy objective in itself could help BART improve)\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Questions**: The model is presented as one that preserves strong compositional representations while also improving on capabilities to disambiguate between idiomatic usage of PIEs and literal usage of PIEs. However, to achieve that, all of the different components of PIER (multiple losses, merging representations via attention fusion, and multiple prompting techniques combined during training) must be combined. This limits its potential to be applied outside of figurative language processing tasks: it would require quite a lot of engineering to maintain the performance on idioms while also improving on other downstream tasks.*\n\nWe appreciate your observation that obtaining high-quality PIE representations requires the combination of all three components introduced in this paper: attention fusion, similarity learning, and multiple prompt infilling techniques, as supported by our experimental results and ablation study. We would like to emphasize that the primary objective of this paper is to enable pretrained language models to generate high-quality PIE representations and enhance their PIE processing capabilities while preserving pretrained models’ original proficiency in non-idiom specific NLU tasks, such as sentiment classification and paraphrase identification.\n\nAchieving this goal, which entails developing a BART-based model capable of producing superior PIE embeddings, does not preclude further fine-tuning for downstream applications and, therefore, does not limit the model's potential applications to other tasks.\n\nThe question of whether our model can maintain its PIE processing ability after fine-tuning is an interesting yet unexplored empirical question, as it is expected that a language model's embeddings may shift to achieve high performance for a specific downstream task after fine-tuning. Addressing this question is somewhat beyond the scope of this paper, as our primary aim is to improve the representation of idiomatic expressions and demonstrate their impact on PIE processing tasks. We have also shown that, in doing so, we did not compromise the language model's NLU capabilities. Nevertheless, investigating the changes in representation after fine-tuning in general could be a valuable direction for future research.\n\n-------\n\n***Question**: line 303: when you generate a literal embedding, do you use token embeddings (layer 0) or contextualised embeddings (final layer)? I was thrown off by the usage of the word “final” in line 306, hence the question. When using the contextualised embeddings, this may, after all, not be a compositional embedding, considering that BART would have some accurate idiomatic representations, particularly for the more frequent idioms (whose idiomatic meaning is highly conventionalised).*\n\nThank you for raising this point. To clarify, we utilized the output from the final layer when generating the literal embeddings, so they are indeed contextualized. It is possible that for the most frequent idioms in the pretraining corpus, the embeddings already encode their idiomatic meanings. This could also be true for any idioms that frequently appear in text and are used exclusively in their figurative sense (i.e., are highly conventionalized).\n\nHowever, we argue that for these idioms, our method would not negatively impact their semantics, as both GIEA and BART already contain their figurative meanings, and they appear in sentences solely in their figurative sense. Consequently, the output of our proposed PIER model should remain semantically accurate for these idioms. Moreover, for most idioms that are individually rare in text, the BART embeddings encode only their literal meanings. This is supported by the intrinsic evaluation in our paper and was also demonstrated in the original GIEA paper, where the author presented a case study showing that in BART's embedding space, idiomatic expressions are grouped by their token or syntactic structure overlaps, rather than their semantic affinities. Therefore, we generally consider BART's embeddings to be \"compositional,\" as they do not capture the non-compositional, figurative meanings of idioms.\n\nNonetheless, to avoid potential confusion and enhance clarity, we will provide additional clarification regarding the use of BART's embeddings and the possible cases where BART already encodes the figurative meanings for frequent idioms in our revision. \n\n-----\n\n***Question**: [4] line 367 “...where all the PIEs in the test data appear in the train data”: have you run some experiments where that was not the case? I’m curious to see whether there can be non-iid improvements as well. I consider it somewhat unlikely, since without having seen training data, the meaning of an idiom can be somewhat unpredictable, but still.*\n\nThank you for your comment. We acknowledge that our proposed PIER model, like GIEA, is limited to working with PIEs that are encountered during training. However, we contend that this constraint is necessary and the most feasible approach to ensure high-quality representations for PIEs when using smaller-scale pretrained language models that have not adequately learned the figurative meanings of PIEs during their pre-training phase.\n\nThe main reason for this limitation lies in the nature of PIEs being collectively frequent yet individually rare. Due to the individual scarcity of each PIE in natural language, pretrained language models struggle to capture their figurative meanings and associate them with the appropriate sentence contexts. Therefore, instead of increasing the amount of training data during pre-training to include a sufficient number of idiomatic sentences, which would be costly and impractical in most working situations, our method supplies explicit knowledge of the PIE's figurative meaning (i.e., their dictionary definitions) to facilitate the model's learning of PIE meanings and their association with corresponding contexts.\n\nWhile it would be interesting to explore the extent to which our model can generalize to unseen idioms during training, we do not anticipate strong generalizability due to the design of our method. Any generalization observed would likely be more indicative of what is already encoded in the pretrained language models, rather than the merit of our approach.\n\n-----\n\n***Question**: For the training techniques that are not specific to fusing the two models (copy objective, infilling, cosine similarity objective) have you applied those to BART or GIEA without including the fusion attention layer? That could help to disentangle your various contributions, because running full PIER might be impractical for future work, whereas adding one of those components might be quite a bit easier. (e.g. I can imagine that the copy objective in itself could help BART improve).*\n\nThank you for highlighting this point. Indeed, applying only similarity learning and prompt infilling objectives to fine-tune a BART model without the attention fusion layer is an important and informative baseline. Consequently, we have incorporated this as a new baseline model, named \"BART-FT,\" in our experimental results, which we will include in the paper.\n\nAs demonstrated in the augmented tables below, the BART-FT model exhibits similar intrinsic quality to GIEA or BART and underperforms PIER+ in all intrinsic evaluation tasks and PIE processing tasks (i.e., SenseCLF and SpanDET). However, it performs competitively with all other models in the NLU tasks (i.e., SentCLF and ParaID). This outcome indicates that our core architectural design, which involves applying attention fusion to contextually combine BART and GIEA's embeddings, is essential. Merely fine-tuning BART with auxiliary learning objectives cannot produce PIE embeddings with accurate semantic meanings and robust PIE processing capabilities.\n\n**Intrinsic Evaluation**\n\n| Model   | H-Score (↑) | CosDist (↑) | DiffSim (↓) |\n| ------- | ----------- | ----------- | ----------- |\n| BART    | 0.4546      | 0.0379      | 0.7495      |\n| GIEA    | 0.6450      | 0.2284      | 0.8224      |\n| BART-FT | 0.4510      | 0.0331      | 0.8198      |\n| PIER+   | 0.6095      | 0.1838      | 0.3230      |\n\n**Extrinsic Evaluation**\n\n| Model   | SenseCLF |        | SpanDET |        | SentCLF |        | ParaID |        |\n|---------|----------|--------|---------|--------|---------|--------|--------|--------|\n|         | F1       | Acc    | SA      | TR     | F1      | Acc    | F1     | Acc    |\n| BART    | 0.9589   | 0.9371 | 0.5076  | 0.7545 | 0.9246  | 0.9232 | 0.9165 | 0.9225 |\n| GIEA    | 0.9573   | 0.9325 | 0.7601  | 0.9075 | 0.9145  | 0.9117 | 0.9046 | 0.9103 |\n| BART-FT | 0.9614   | 0.9408 | 0.472   | 0.7907 | 0.9364  | 0.9346 | 0.9152 | 0.9207 |\n| PIER+   | 0.9765   | 0.9637 | 0.793   | 0.9101 | 0.929   | 0.9278 | 0.9068 | 0.9122 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- PIER is a carefully crafted model that, across the board, outperforms its main competitor, GIEA. This model will be of great interest to the subcommunity of *ACL conferences that are interested in figurative and idiomatic language modelling or modelling multi-word expressions, more generally.\n- Apart from full PIER, the paper presents a range of techniques that themselves can be more widely applied, even outside of the context of PIER, specifically, such as the definition generation prompting setup, or the cosine similarity losses introduced.\n- The paper presents a very thorough analysis via the various intrinsic and extrinsic evaluations, highlighting both where PIER does and does not outperform previous models.", "reasons_to_reject": "- The model is presented as one that both preserves strong compositional representations while also improving on capabilities to disambiguate between idiomatic usage of PIEs and literal usage of PIEs. However, in order to achieve that, all of the different components of PIER (multiple losses, merging representations via the attention fusion, multiple prompting techniques combined during training) need to be combined. This limits its potential to be applied outside of figurative language processing tasks: it would require quite a lot of engineering to maintain the performance on idioms while also improving on other downstream tasks.\n- It is unclear how the individual components contribute to the overall performance, since all variants use the attention fusion. Potentially, a subset of the techniques but without the attention fusion could already improve GIEA and/or BART.", "questions_to_authors": "- line 303: when you generate a literal embedding, do you use token embeddings (layer 0) or contextualised embeddings (final layer)? I was thrown off by the usage of the word “final” in line 306, hence the question. When using the contextualised embeddings, this may, after all, not be a compositional embedding, considering that BART would have some accurate idiomatic representations, particularly for the more frequent idioms (whose idiomatic meaning is highly conventionalised).\n- line 367 “...where all the PIEs in the test data appear in the train data”: have you run some experiments where that was not the case? I’m curious to see whether there can be non-iid improvements as well. I consider it somewhat unlikely, since without having seen training data, the meaning of an idiom can be somewhat unpredictable, but still.\n- For the training techniques that are not specific to fusing the two models (copy objective, infilling, cosine similarity objective) have you applied those to BART or GIEA without including the fusion attention layer? That could help to disentangle your various contributions, because running full PIER might be impractical for future work, whereas adding one of those components might be quite a bit easier. (e.g. I can imagine that the copy objective in itself could help BART improve)\n", "paper_topic_and_main_contributions": "This paper proposes PIER, a model that fuses representations from BART and GIEA (a model specialised in creating idiomatic representations and makes use of adapters in order to do so), to maintain performance on compositional data (both literal PIEs and compositional data in NLU tasks), while improving representations for non-compositional phrases (specifically, idioms included by Haagsgma in the MAGPIE dataset). Fusing representations is achieved using an Attention Fusion layer that, for each layer individually, mixes the two representations using layer-specific attention weights. The model is trained using a loss consisting of\n1. cosine distances that encourage similarity between idiomatic PIEs and the idiom’s dictionary definition, encourage similarity between literal PIEs and compositional representations, discourage similarity between literal PIEs and the idiom’s dictionary definition, and discourage similarity between idiomatic PIEs and compositional representations.\n2. a reconstruction loss that is meant to strengthen the association between the context and the PIE\n\nThe data is presented via a prompt infilling task (where the model indicates whether a phrase is used idiomatically or literally) and a definition generation task, e.g. “the phrase `see red’ is used to mean …”, where the model should output the literal meaning or the idiomatic interpretation. The mean cross-entropy loss computed over these “definitions” serves as the reconstruction loss mentioned above.\n\nThe models trained are BART, GIEA, fusing combined with the reconstruction loss, fusion combined with the reconstruction and similarity losses, fusion combined with the prompt infilling task, full PIER, and PIER+ (that uses multiple prompts per input). The models are trained using MAGPIE data.\n\nThe evaluation consists of intrinsic and extrinsic evaluation:\n1. The intrinsic evaluation measures the quality of clusters of idioms with related meanings, and measures the correlation between literal and idiomatic occurrences of a PIE.\n2. Extrinsic evaluation using PIE processing tasks, specifically PIE sense classification and span detection.\n3. Extrinsic evaluation using sentiment classification and paraphrase identification.\n\nPIER outperforms GIEA on most of the extrinsic evaluation tasks, while GIEA is better from an intrinsic point of view. Note that the versions of PIER that don’t include all components (apart from the 5 prompts) typically underperform, full PIER is needed to obtain the improvements.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- In the equations in section 3.1 it looks like the value vectors are only bart-based (h_tilde) whereas in the figure it looks like the value vectors are both bart- and giea-based (h). Could you double-check that and revise if needed? (Unless I misunderstood something)\n- line 141: Wait... where is contribution number 3?\n- line 241/242: rephrase, “teaches” is very vague\n- line 87 - 89: the exact cosine similarity here is a bit meaningless without a baseline... all sentence / phrase embeddings tend to have high similarity anyways, and it’s hard to tell whether 0.82 is relatively good/bad due to an issue known as the high anisotropy problem or the representation degeneration problem: LMs contextualised embeddings all end up in a small portion of the hidden space and are, therefore, more similar than one would expect. (hence the need for a baseline)\n"}}, {"review_id": "cAIeUJUJ0e", "reviewer": "Reviewer_ZgHk", "summary": "", "questions": "- Please see above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Question**: The current baselines cannot show the effectiveness of attention fusion layers. All FusionAttn, FusionSim, FusionPrompt, PIER, and PIER+ have the attention fusion layers. My hypothesis is that fine-tuning BART with prompt infilling + similarity learning objective is enough. This can be also observed from Table 2, which FusionAttn does not give us any improvement. The authors should design more baselines to figure out the necessary of attention fusion layers.*\n\nThank you for pointing this out. We agree that applying only similarity learning and prompt infilling objectives to fine-tune a BART model without the attention fusion layer is an important and informative baseline. Consequently, we have added this as a new baseline model, named \"BART-FT,\" to our experimental results, which we will include in the revised version of our paper.\n\nAs shown in the augmented tables at the end, the BART-FT model exhibits similar intrinsic quality to GIEA or BART and underperforms PIER+ in all intrinsic evaluation tasks and PIE processing tasks (i.e., SenseCLF and SpanDET). However, it performs competitively with all other models in the NLU tasks (i.e., SentCLF and ParaID). This result indicates that our core architectural design, which involves applying attention fusion to contextually combine BART and GIEA's embeddings, is essential. Simply fine-tuning BART with auxiliary learning objectives cannot produce PIE embeddings with accurate semantic meanings and strong PIE processing ability.\n\nIn other words, although FusionAttn alone is not sufficient to provide improvements, the combination of attention fusion and our proposed learning objectives leads to significant enhancements.\n\n-----\n\n***Question**: The results of SpanDET is a bit weird. Using only FusionSim or FusionPrompt is worse than GIEA. However, Using PEIR (which is kind of FusionSim+Prompt) can perform better than GIEA. Can authors provide some explanations?*\n\nThrough our experiments, we have discovered that all three components introduced in our paper – the attention fusion layer, similarity learning objective, and prompt infilling objective – are essential for generating semantically accurate representations for PIEs. Therefore, relying solely on prompt infilling or the similarity learning objective is insufficient for learning embeddings with high intrinsic quality, which in turn would be beneficial for downstream PIE processing ability.\n\nIn addition to span detection, we observe a similar trend in other intrinsic and PIE processing applications, where the model achieves significant performance improvement only when all three components are combined. Consequently, the results from the span detection are consistent with the findings from our other experiments.\n\n-----\n\n***Question**: It's worth having some ablation studies for type classification prompts and definition generation prompts.*\n\nThank you for the suggestion. In addition to the ablation study on the number of prompts, we have included an ablation study comparing PIER with only the type classification prompt (P-Cls) and PIER with only the definition generation prompt (P-Defn). Each prompt type utilizes the same five prompt templates as the PIER+ model.\n\nAs demonstrated in the table below, our model achieves significant improvements in both intrinsic evaluation and PIE processing tasks when using either type of prompts, while maintaining competitive performance in NLU tasks. However, when combining both prompt types, PIER+ outperforms P-Cls and P-Defn in intrinsic evaluation metrics, particularly in the SpanDET task, with gains of 5.6% and 7.4% in sequence accuracy, respectively. These results support the conclusion that combining the two types of prompts is beneficial and leads to further performance improvements.\n\nFurthermore, when comparing P-Cls and P-Defn, their performances are mostly competitive, with no clear winner. Thus, we conclude that both prompt types are approximately equally beneficial, and their advantages are amplified when used together.\n\n\n**Intrinsic Evaluation**\n\n| **Model** | **H-Score (↑)** | **CosDist (↑)** | **DiffSim (↓)** |\n|-----------|-----------------|-----------------|-----------------|\n| BART      | 0.4546          | 0.0379          | 0.7495          |\n| GIEA      | 0.645           | 0.2284          | 0.8224          |\n| BART-FT   | 0.451           | 0.0331          | 0.8198          |\n| P-Cls     | 0.5756          | 0.1527          | 0.3468          |\n| P-Defn    | 0.5751          | 0.1546          | 0.3547          |\n| PIER+     | 0.6095          | 0.1838          | 0.323           |\n\n**Extrinsic Evaluation**\n\n| **Model**   | **SenseCLF** |        | **SpanDET** |        | **SentCLF** |        | **ParaID** |        |\n|---------|----------|--------|---------|--------|---------|--------|--------|--------|\n|         | F1       | Acc    | SA      | TR     | F1      | Acc    | F1     | Acc    |\n| BART    | 0.9589   | 0.9371 | 0.5076  | 0.7545 | 0.9246  | 0.9232 | 0.9165 | 0.9225 |\n| GIEA    | 0.9573   | 0.9325 | 0.7601  | 0.9075 | 0.9145  | 0.9117 | 0.9046 | 0.9103 |\n| BART-FT | 0.9614   | 0.9408 | 0.472   | 0.7907 | 0.9364  | 0.9346 | 0.9152 | 0.9207 |\n| P-Cls   | 0.9712   | 0.9558 | 0.737   | 0.8848 | 0.9208  | 0.9197 | 0.9069 | 0.9128 |\n| P-Defn  | 0.972    | 0.9571 | 0.719   | 0.881  | 0.9315  | 0.93   | 0.906  | 0.9118 |\n| PIER+   | 0.9765   | 0.9637 | 0.793   | 0.9101 | 0.929   | 0.9278 | 0.9068 | 0.9122 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting research direction.", "reasons_to_reject": "- The current baselines cannot show the effectiveness of attention fusion layers. All FusionAttn, FusionSim, FusionPrompt, PIER, and PIER+ have the attention fusion layers. My hypothesis is that fine-tuning BART with prompt infilling + similarity learning objective is enough. This can be also observed from Table 2, which FusionAttn does not give us any improvement. The authors should design more baselines to figure out the necessary of  attention fusion layers.\n- The results of SpanDET is a bit weird. Using only FusionSim or FusionPrompt is worse than GIEA. However, Using PEIR (which is kind of FusionSim+Prompt) can perform better than GIEA. Can authors provide some explanations?\n- It's worth to have some ablation studies for type classification prompts and definition generation prompts.", "questions_to_authors": "- Please see above.", "paper_topic_and_main_contributions": "This paper proposes PIER and PIER+, which aim to improve the representations for potentially idiomatic expressions (PIEs). They provide two main components: attention fusion layers, similarity learning objective, and prompt infilling. They also report the experimental results of intrinsic and extrinsic evaluation, which show the potential of PIER and PIER+ to represent PIEs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "None."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model based on BART to generate contextualized representations of non-compositional expressions (specifically, English potentially idiomatic expressions; PIEs), showing good results on PIE processing tasks. Reviewers mostly agreed that the results were sound, though the paper could benefit from more ablations/analysis demonstrating the need for certain modeling decisions. This paper represents a focused work providing a new state-of-the-art for PIE identification, which will be of interest to those in the *CL community focused on modeling idiomatic/MWE expressions, but may be less exciting to the community more broadly.
Decision: Accept-Findings