Metaphor Understanding Challenge Dataset for LLMs (2024)

Abstract: Metaphors in natural language are a reflection of fundamental cognitive processes such as analogical reasoning and categorisation, and are deeply rooted in everyday communication. Metaphor understanding is therefore an essential task for large language models (LLMs). We release the Metaphor Understanding Challenge Dataset (MUNCH), designed to evaluate the metaphor understanding capabilities of LLMs. The dataset provides over 10k paraphrases for sentences containing metaphor use, as well as 1.5k instances containing inapt paraphrases. The inapt paraphrases were carefully selected to serve as control to determine whether the model indeed performs full metaphor interpretation or rather resorts to lexical similarity. All apt and inapt paraphrases were manually annotated. The metaphorical sentences cover natural metaphor uses across 4 genres (academic, news, fiction, and conversation), and they exhibit different levels of novelty. Experiments with LLaMA and GPT-3.5 demonstrate that MUNCH presents a challenging task for LLMs. The dataset is freely accessible at https://github.com/xiaoyuisrain/ metaphor-understanding-challenge.

heading: 1 Introduction
text: Large language models (LLMs), such as BERT (Devlin et al., 2019), GPT-3 (Brown et al., 2020) and LLaMA (Touvron et al., 2023), have become a common paradigm in natural language processing (NLP). Several benchmarks have been proposed to investigate the capabilities of LLMs (Srivastava, 2022; Liang et al., 2022; Hendrycks et al., 2021); and comprehensive analyses have been conducted, evaluating their performance on a range of NLU tasks (Zhong et al., 2023; Qin et al., 2023; Kocoń et al., 2023; Ye et al., 2023; Bang et al., 2023). The community has extensively examined LLM performance on question answering, summarisation, sentiment analysis, natural language inference; a
metaphor sample has a ∗highlighted∗ word that is metaphorically used, and is accompanied by up to 5 crowdsourced paraphrases : Substituting the highlighted word with one of the provided words should result in an apt paraphrase. For a selection of metaphor samples, we also provide expert annotation : a pair of correct and incorrect substitution words.
few studies have also shed light on LLMs’ analogical reasoning capabilities (Czinczoll et al., 2022; Webb et al., 2023). However, the ability of LLMs to comprehend metaphor—a fundamental linguistic and cognitive tool—is still poorly understood.
Metaphors are linguistic expressions based on conceptual mappings between a target and a source domain (Lakoff and Johnson, 1980). The verb phrase to stir excitement, for example, is based on the conceptual metaphor FEELING IS LIQUID, with FEELING (excitement) being the target domain and LIQUID (something that can be stirred) the source domain. The metaphor compares FEELING with LIQUID, introducing vividness into the description of an otherwise intangible emotional impact. Such cross-domain mappings are sets of systematic ontological correspondences, mapping concepts and their relational structure across distinct domains. Performing this mapping is an essential part of reasoning involved in the interpretation of metaphorical language (Lakoff, 2014; Grady et al., 1999; Gentner and Markman, 1997).
Humans use metaphors so naturally and frequently that they largely fly under our radar. In
3517
one of the largest metaphor corpora annotated by linguists, the VU Amsterdam Metaphor Corpus (VUA; Steen et al., 2010b), every 8th word is metaphorical, as averaged over four different genres, including academic and conversation. LLMs, therefore, require the ability to comprehend metaphor in order to have a full command of language. As such, metaphor understanding is an essential task for evaluating the capabilities of LLMs.
Several corpora have been created that contain metaphor annotations at either word or sentence level. These include the VUA corpus (Steen et al., 2010b), the LCC metaphor datasets (Mohler et al., 2016) and the metaphor-emotion dataset of Mohammad et al. (2016), among others. These datasets have been widely used to develop and evaluate automated metaphor identification systems (see Tong et al. (2021) for a survey), but they do not contain information of how the annotated metaphors are interpreted. On the other hand, several works developed datasets with a focus on interpretation, typically casting the problem as a paraphrasing task (Shutova, 2010; Bizzoni and Lappin, 2018; Joseph et al., 2023). Yet, those datasets were often small in scale (containing 200–1000 instances) and were not designed to test the reasoning process by which a metaphor is interpreted, which remains an open question.
This paper presents a novel Metaphor Understanding Challenge Dataset for LLMs (MUNCH). It provides over 10k paraphrases for metaphorical sentences and 1.5k triples of a metaphorical sentence and two candidate paraphrases, which could be apt or inapt (for dataset examples see Figure 1; for statistics see Appendix A). The metaphorical sentences were extracted from VUA texts, spanning four genres (academic, news, fiction, and conver-
sation) and featuring metaphors at different levels of novelty. Each metaphorical sentence contains a content word that is marked as metaphorically used. A candidate paraphrase replaces the metaphorical word with another word, so that the resulting sentence is the same as the reference sentence except for that one word, therefore representing a lexical substitution task. An apt paraphrase shows correct contextual interpretation of the metaphor while an inapt paraphrase uses a word that is related to a literal, source domain sense of the metaphorical word (see the examples of correct and incorrect substitution words in Figure 1). Such a setup of the task is inspired by the conceptual metaphor theory (Lakoff and Johnson, 1980) and allows us to investigate whether the model performs full metaphor interpretation by cross-domain mapping or rather resorts to more shallow lexical similarity. In order to investigate this in a more controlled fashion, we opted for a lexical substitution task. Specifically, we test whether the model consistently chooses the correct target domain paraphrase (therefore, fully interpreting the metaphor) or rather bases its decisions on lexical similarity and chooses the inapt paraphrase that is similar in meaning to the literal use of the metaphorical word.
We set up a fill-in-the-blank task to crowdsource apt paraphrases, and manually selected the best paraphrases using expert knowledge. We also manually created inapt paraphrases from WordNet synsets, so that the apt and inapt paraphrases reflect the target and source domains of the metaphors respectively. Specifically, the inapt paraphrases are synonyms or hypernyms associated with the word’s literal use (the source domain).
Using this dataset, we tested the metaphor understanding capabilities of LLaMA-13B, 30B, and GPT-3.5 zero-shot in two tasks: paraphrase judgement and paraphrase generation, as illustrated in Figure 2. Our results show that both tasks are challenging for the models. In particular, the models are prone to confuse the target and source domains of the metaphors, as they often fail to distinguish the inapt paraphrases from the apt paraphrases or reference sentences. Our experiments also reveal that LLMs’ metaphor understanding capabilities are associated with genre, metaphor novelty, and POS of the metaphorical word. The MUNCH tasks thus allow us to gain insight into how LLMs process metaphors as well as how this remarkable ability can be improved in the future.

heading: 2 Related work
text: Steen et al. (2010b) created VUA, which marks out metaphor-related words (MRWs) in a 4-millionword subset of the British National Corpus. MRWs are lexical units implicative of underlying crossdomain mappings; they can be directly or indirectly used, depending on their contextual meaning. Consider the sentence “A small five-year-old perched like a mosquito on the beginners’ pony”. The noun mosquito is a direct metaphor, as it literally means mosquito (the source domain) in this context. The verb perched is an indirect metaphor, because it has a more basic usage, as in “A pair of glasses were perched on the bridge of his nose”.
VUA has been widely used in studies on automated metaphor detection (Leong et al., 2018, 2020; Choi et al., 2021; Zhang and Liu, 2022; Li et al., 2023). However, the corpus does not specify the conceptual metaphors indicated by the MRWs or provide annotation for interpreting the metaphors. The corpus is not directly applicable to automated metaphor interpretation.
Shutova (2010) defined automated metaphor interpretation as a paraphrasing task: Given a metaphorical expression where a word is marked as metaphorically used, the model should replace this word with another word to render a literal paraphrase of the expression. For example, the verb phrase stir excitement, where stir is used metaphorically, should be paraphrased as provoke excitement.
Bizzoni and Lappin (2018) created a Metaphor Paraphrase Evaluation Corpus (MPEC), which provides correct and incorrect paraphrases for ∼200 short sentences containing metaphor use; paraphrases could greatly differ from the reference sentences. Joseph et al. (2023) created the NewsMet dataset, which consists of 1k verbal metaphors in news headlines as well as their literal equivalents; incorrect paraphrases are not provided.
Several recent studies approached metaphor understanding as an inference task. The IMPLI dataset (Stowe et al., 2022) includes entailed and non-entailed sentences for ∼900 metaphorical sentences. The FLUTE dataset (Chakrabarty et al., 2022b) provides entailment and contradiction pairs for 1500 metaphorical sentences (including 750 similes). Fine-tuned transformer-based models reached > 0.8 accuracies in these 2 studies in predicting the class of a given sentence pair.
Recent studies also employed multiple-choice and generative tasks to assess LLMs’ ability to rea-
son with metaphorical language. The MiQA benchmark (Coms, a et al., 2022) uses such tasks to test whether models can distinguish metaphorical and literal uses of the same words; 150 conventional metaphors are involved. The Fig-QA task (Liu et al., 2022) includes 10k similes (a type of direct metaphor) and requires models to distinguish a pair of metaphors of opposite meanings. Chakrabarty et al. (2022a) examined LLMs’ figurative language understanding by asking them to generate text after encountering an idiom or simile.
The MUNCH dataset provides 3k samples of indirect metaphors, 10k correct paraphrases, and 1.5k incorrect paraphrases. It is therefore one of the largest datasets for paraphrasing of indirect metaphors. The candidate paraphrases are also systematically different from the ones in previous datasets, as we tailored the dataset for testing LLMs’ understanding of metaphors as crossdomain mappings and correctly capturing the underlying relational structures. We summarise differences between MUNCH and previous datasets and provide more details for the latter in Appendix A.

heading: 3 Data collection: metaphor samples
text: The metaphor samples in our dataset were selected from the publicly available metaphor corpus VUA. Each metaphor sample is a sentence containing a highlighted MRW, the metaphorical word to be interpreted; a paraphrase uses a single word to substitute the metaphorical word. We use two criteria for selecting metaphorical sentences: novelty of the metaphor and possibility of single-word substitution. We explicate our selection process below.
The novelty criterion. We employed novelty scores from Do Dinh et al. (2018) to increase the proportion of novel metaphors in our dataset. Scores range from -1 to 1. VUA contains a large proportion of conventional metaphors: The metaphorical use of the word can be found in a dictionary of contemporary language use (Steen et al., 2010a). As LLMs might have encountered enough data for such conventional metaphor uses during pre-training, the understanding of such metaphors should be relatively easy. To render a more challenging dataset, we excluded MRWs with novelty scores below -0.3. Metaphors with a novelty score higher than -0.3 could still be conventional: The crowd workers who provided the novelty annotations in Do Dinh et al. (2018) relied on their intuition instead of a dictionary like Steen et al. (2010a).
And metaphorical uses included in dictionaries may still be considered novel by lay people. We chose -0.3 as the threshold in order to collect a large and diverse dataset as a starting point.
The single-word criterion. To ensure that the metaphorical sentences can be paraphrased via single-word substitution, we excluded MRWs that are marked as direct metaphors, as well as a portion of indirect metaphors. Directly used MRWs usually occur in a sequence, such as “I knew the pathway like the back of my hand”. They are thus not suitable for single-word substitution. Also, the direct metaphor back of my hand refers literally to the back of the speaker’s hand—its contextual meaning is directly associated with the source domain. This is contrary to our task setup, where apt paraphrases (contextual meaning) should be associated with target domains. We therefore opted to focus on indirect metaphors in this study.
Within the category of indirect metaphors, we filtered out new-formations, consecutive MRWs, and proper names. New-formations are words that do not have an entry in dictionary, so VUA annotated the parts that do have corresponding entries. For example, in the phrase a rose-tinted vision of the world, the word rose-tinted was a new-formation; so rose and tinted are marked as separate MRWs in VUA and received their separate novelty scores. We filtered these out because a single metaphorical word should have a single novelty score (rosetinted has two), yet it is hard to paraphrase rose or tinted instead of rose-tinted altogether.
Likewise, we excluded cases where multiple content words marked as indirect metaphors occur consecutively, such as take place, long road home, great leap forward. These often involve fixed expressions or phrases that either should be replaced as a whole or should not be marked as consecutive indirect metaphors. We also excluded metaphorical words that are part of a proper name, which, like fixed expressions, need to be treated as a whole. For example, the proper name Nord Stream would lose its meaning if one changed the metaphorical word stream into another word.

heading: 4 Annotation of apt paraphrases
text: Crowdsourcing task. We constructed a fill-inthe-blank task to crowdsource (apt) paraphrases for the metaphorical sentences. Each task included 30 sentences to be paraphrased, so that the task can be finished within 30 minutes. Under each sentence,
the workers were presented with a copy of the sentence where the metaphorical word is replaced with a blank; they were asked to fill the blank with a single word so that the new sentence is a semantically and grammatically apt paraphrase of the reference sentence. If they were not able to paraphrase the sentence, they were asked to explain why it was difficult. Examples of good and bad answers were provided in the instructions (see Appendix B).
The workers were recruited via Prolific1. We set prescreening criteria to only include adult (age > 18) native English speakers who were living in an English-speaking country and did not have any language-related disorders. The workers were asked to confirm within the task that they met these criteria. After giving consent to participate and reading the instructions, they were also required to correctly paraphrase a trial sentence in order to access the task. More details (worker’s consent, the trial sentence) are given in Appendix B.
We released 99 tasks in total and collected 5 data points for each of the 2970 reference sentences. We received single-word substitutions for 2953 sentences (the other 17 are presented and explained in Appendix B), and 61% of them got repeated answers—multiple workers submit the same paraphrase despite the question being open-ended. This confirms the reliability of our task.
Expert validation. For a selection of the reference sentences for which we later annotated inapt paraphrases (Section 5), we further validated the crowdsourced paraphrases to determine the best paraphrase for creating the triples (one metaphor sample, two candidate paraphrases).
We used both majority vote and expert knowledge to find one best paraphrase for each reference sentence. For each sentence, we first sorted the collected single-word substitutions from the most to the least popular (in terms of how many participants proposed that substitution). The apt paraphrase that was proposed by the highest number of participants was verified by the authors and selected as the best paraphrase for that reference sentence.
When multiple apt paraphrases have the same number of votes, we chose the one that is clearly within the target domain—that is, the paraphrase clearly shows that the metaphorical word is interpreted in its contextual sense. For instance, we received 5 different single-word substitutions for the metaphorical word attack in the sentence “. . . he
1https://www.prolific.co/
has become involved in a row over his attack on the “Pharisees” of British society”. These are remarks, views, offense, incursion, and disagreement, each proposed by a single participant. All of them can be considered apt paraphrases. We chose remarks because it clearly shows the metaphorical word attack is interpreted in the ARGUMENT domain. The meaning of offense and disagreement are more abstract and could involve other conceptual domains; the paraphrases that replace attack with views and incursion respectively are still metaphorical, as view can be associated with VISION and incursion, like the metaphorical word itself, is still in the domain of BATTLE. These four are thus less preferable with regard to the purpose of our dataset.
While we managed to find one best paraphrase for most reference sentences, there are 45 for which we selected two paraphrases as the best, as the two received the same votes and are equally apt. There are also 11 sentences for which no paraphrase was selected. These are cases where the given context is insufficient for determining the contextual meaning of the metaphorical word.

heading: 5 Annotation of inapt paraphrases
text: Tong (2021) shows that incorrect paraphrases based on the basic sense of the metaphorical word are the least distinguishable from correct ones (i.e., paraphrases based on the contextual sense) with respect to aptness. To render truly challenging inapt paraphrases for our task, we therefore created inapt paraphrases exclusively from basic senses.
We employed WordNet for identifying basic senses and obtaining sense-specific synonyms, following the annotation guidelines presented in Appendix C. For each metaphorical word, we first locate the WordNet synsets that correspond to its more basic meaning (relative to its contextual meaning in the reference sentence). Then we go through the synonyms (or hypernyms when no synonyms are provided) under the basic-sense synsets and select those that are clearly associated with the metaphor’s source domain and would render a grammatical (but inapt) paraphrase.
We went through all 2970 sentences released for the crowdsourcing task and found inapt paraphrases for 991 of them. After removing items lacking apt paraphrases (either because no single-word substitutions were crowdsourced or because none of the collected ones are of sufficient quality), we created 1492 triples for 728 metaphorical sentences, includ-
ing 1072 triples with an apt and an inapt paraphrase, 375 triples with two inapt paraphrases, and 45 with two apt paraphrases.
Inter-annotator agreement. From the 991 sentences for which inapt paraphrases were identified, we randomly selected 200 to be annotated by a second annotator. The annotator was a PhD candidate in linguistics specialising in metaphor research. We explained the annotation process to the second annotator through a meeting and the guidelines in Appendix C. The Gwet’s gamma coefficient for the agreement between the two expert annotators is 0.84.

heading: 6 Data analysis
text: The dataset contains approximately the same number of samples from academic and news genres, and fewer samples from fiction and conversation, as shown in Table 1. These metaphor samples cover metaphorical use of content words in all four parts of speech. Noun and verb MRWs are of a higher proportion compared to adjectives and adverbs. In news and fiction, these two categories have similar percentages. The academic genre contains more noun MRWs than verbs whereas in conversation the situation is reversed: Half of the metaphorical words are verbs, while the percentage of nouns is similar to that of adjectives and adverbs.
As we excluded MRWs of novelty scores lower than -0.3, the metaphor samples exhibit a wider range of novelty scores above 0 than below 0 (see Appendix D). Meanwhile, a large proportion of the metaphor samples could be considered only slightly novel or conventional (novelty scores between -0.3 and 0.3). Metaphor samples of the highest novelty scores can be from any of the four genres. Despite their different proportion in the entire dataset (Table 1), all four genres include metaphor samples across all levels of perceived novelty.
We also examined the cosine similarity between the metaphorical words and apt and inapt substitu-
tions. Since the inapt paraphrases were based on the more basic meaning of the MRWs (section 5), we expected inapt substitution words to be more similar to the metaphorical words than apt substitution words. We computed the cosine similarity scores using glove-wiki-gigaword-300 embeddings (Pennington et al., 2014), accessed through the gensim Python library. Figure 3 shows the distribution of cosine similarity scores for 1006 triples, excluding the ones containing out-of-vocabulary words. Surprisingly, the target-apt pairs tend to have higher cosine similarity scores than the targetinapt pairs. The plot suggests that the 3 pairs are distinguishable in terms of cosine similarity scores, with target-apt pairs being the most similar, and apt-inapt the least similar. This might be associated with the fact that our metaphorical sentences were sampled from VUA, which, being representative of metaphor use in natural discourse, includes a large percentage of conventional metaphors. Nonetheless, the majority of the cosine similarity scores are above 0, and the 3 pairs still share a wide range of similarity scores. The distribution plot is therefore also suggestive of the reliability of our dataset, as well as its potential challenge for LLMs.

heading: 7 Model evaluation
text: We evaluated LLaMA-13B, LLaMA-30B, and GPT-3.5 (text-davinci-003) on two tasks: (Task 1) paraphrase judgement, which requires a model to select correct paraphrases for a given reference sentence from given candidates; and (Task 2) paraphrase generation, which asks a model to generate correct paraphrases for a given reference sentence. The paraphrase judgement task used the 1492 triples that include inapt paraphrases; the generation task used all 2953 metaphorical sen-
tences. Details regarding computational budget is given in Appendix E.

heading: 7.1 Paraphrase judgement
text: We evaluate the LLMs in a prompting setup. We test the models’ ability to interpret metaphor under different conditions. In the first scenario, we prompt the model by providing the reference sentence with the metaphorical word highlighted and two candidate replacement words for it (Wordjudgement). In the second scenario, each of the candidate replacement words is embedded in the sentence (Sentence-judgement). In both cases the model needs to solve a multiple choice task. Besides providing the apt and inapt paraphrases (Options A and B) as answer options, we also complement them with Option C, that both candidates are correct, or Option D, that neither are correct. See Figure 2 for an example. We expect Wordjudgement to be more challenging, as the model would need an additional inference step compared to sentence judgement, to replace the metaphorical word with the two given options and (implicitly) form the intended paraphrases.
For both Word-judgement and Sentencejudgement setups, we further investigate whether it makes a difference if the model is explicitly “told” that the task is to paraphrase a metaphor or not. This results in three further conditions: Implicit (not mentioning metaphor in the prompt), Metaphor-Sent (revealing that the reference sentence contains a metaphor), and Metaphor-Word (revealing that the specific highlighted word in the sentence is metaphorically used). The Implicit condition corresponds best to the real-life application of LLMs, where the model needs to be able to interpret metaphors without being instructed that metaphors are there.
We tested LLaMA-13B and 30B, and GPT-3.5 in each of the 6 conditions, using 3 prompts for each condition (the prompts are listed in Appendix E).
Table 2 shows the mean accuracy and standard deviation for each model in each condition. The random baseline achieves an accuracy of 0.25, as there is always only one correct option out of the given four. The performance of all three models was below the random baseline in most cases, except for LLaMA-13B in the Word-judgement tasks and LLaMA-30B in the Metaphor-Word condition of the Sentence-judgement task. Meanwhile, the accuracy of LLaMA-13B varied a great deal across different prompts in the Word-judgement tasks.
The Sentence-judgement task seems to be more challenging than Word-judgement for the models. For LLaMA-30B and GPT-3.5, the task was particularly difficult when they were not instructed to focus on the metaphorical word, and were not informed that the word is metaphorically used (the Metaphor-Word condition). For LLaMA-13B, all 3 Sentence-judgement conditions are similarly difficult. However, its higher accuracies in the Wordjudgement tasks also indicate the benefit of instructing the model to focus on the metaphorical word.

heading: 7.2 Paraphrase generation
text: The purpose of this task is to compare model and human performance in paraphrasing metaphorically used words. The prompts were thus designed to be semantically close to the instructions in our crowdsourcing task (Section 4). The model answers were generated in two steps (see Figure 5). We first let the models generate a single token— this allowed us to access the models’ ranking of all tokens in their vocabulary. Of these, we selected the ones that match human annotations and let the models complete them into words. The completions were then compared with human annotations to determine the rank of each expected answer.
We tested the models on 3 prompts (see Appendix E) and their mean performance in terms of mean reciprocal rank (MRR), recall at top 5 paraphrases and recall at top 10 paraphrases is shown in Table 3. GPT-3.5 performed best and LLaMA-30B came second. The models’ performance was also more stable across different prompts as compared to the paraphrase judgement task. Nonetheless, all three models clearly preferred different answers as compared to human annotators.

heading: 8 Discussion
text: Paraphrase judgement. We looked into the type of errors the models made in paraphrase judgement. The number of each combination of expected and predicted answers for each model is in Appendix F. We found that LLaMA-30B and GPT-3.5 could ignore the semantic differences between a given sentence and an inapt paraphrase, as they tend to predict both candidates as correct when presented with one or more inapt paraphrases. LLaMA-13B, on the other hand, tends to assume that the two given candidates always contain one apt and one inapt paraphrase. Nonetheless, it did not seem capable of distinguishing the two, as it made a similar number of Option A and Option B predictions.
Paraphrase generation. We examined the topranked answers of the models and found 4 cate-
gories that the ‘incorrect’ or unexpected answers could fall into. 1) Nonsensical: For the sentence “...for this number line I would say...”, GPT-3.5 gives thus as the best substitution word, ignoring the meaning of line, whereas LLaMA-13B repeats number. 2) Lack of contextual understanding: In “...he touched both sides of the coin...”, GPT-3.5 replaces the word sides with facets, suggesting that it neglects details of the meaning of the sentence (that a coin only has 2 sides). 3) Ungrammatical: On the other hand, the model may have understood the metaphor, but fails to convert its understanding into a suitable substitution word. In “...they all shared the emphasis on ‘her’...”, LLaMA-30B suggests concurred as the best answer, implying that the meaning of shared is understood, but that grammatical agreement has been sacrificed. 4) Preference: Finally, the disagreement between the models and human annotators may simply be a matter of preference. For “For a man whom Rebecca West ... called ‘repulsive’ and ‘treacherous’...”, crowd workers provide 4 possible answers: revolting, disgusting, awful, and grotesque. Both LLaMA-30B and GPT-3.5 give odious as the best answer. Here, both the human annotators and the models understand and can paraphrase the sentence, and it is hard to say whose answer is best.
Factors associated with model performance. We also examined the association between model performance and 3 factors: genre, metaphor novelty, and the POS of the metaphorical word. The details are available in Appendix F. We found metaphors of higher novelty scores to be more difficult for LLaMA-30B in paraphrase judgement, and for GPT-3.5 in paraphrase generation. The association between genre or POS and model performance tends to differ per model and task. The fiction genre, for example, is the easiest for the LLaMA models in paraphrase generation; yet it is the most difficult for GPT-3.5 in the generation task and for LLaMA-30B in the judgement task. Similarly, noun metaphors are the easiest for LLaMA30B in the generation task and for GPT-3.5 in the judgement task. Meanwhile for GPT-3.5 in the generation task, adverb metaphors become the easiest.
To sum up, the results of the two paraphrase tasks indicate that the LLMs are unable to (fully) understand some of the metaphors in our dataset. The paraphrase judgement task further reveals that the models have difficulty distinguishing the metaphors’ source domains (implied by the inapt
paraphrases) and target domains (implied by the reference sentences and apt paraphrases). This further suggests that the models are unlikely to perform reasoning across semantic domains; when they succeed in understanding the metaphor, they may still reason in ways that are different from humans. This means that for downstream NLP tasks such as opinion mining, bias detection, humour detection, and intent recognition, the LLMs could overlook the entailment of a metaphor. In machine translation as well as summarisation of highly figurative or poetic texts, the problems may manifest as incorrect or peculiar explanation of metaphors.
A direction for improvement is to mark out metaphor uses in texts and direct the model’s attention to them: In the paraphrase judgement task, the models reach higher accuracies when the metaphorical word is marked out (in the Word-judgement task or in a Metaphor-Word condition). However, since the models generally performed poorly in the experiments, the LLMs may need to be fine-tuned in order to better understand metaphors. When finetuning, one can consider increasing the proportion of certain metaphor types in training data, as genre, metaphor novelty, and POS of the metaphorical word are all associated with model performance. Future studies could first employ MUNCH to detect the weak points of an LLM and then curate training data accordingly.

heading: 9 Conclusion
text: We release a dataset of manually created apt and inapt paraphrases for metaphorical sentences and present two metaphor understanding tasks, which we demonstrate to be challenging for current LLMs. The errors the models make in the paraphrase generation task indicate various levels of misunderstanding of the metaphors. In the paraphrase judgement task, the models’ accuracy was lower than the random baseline in the majority of the cases; a closer look at their errors reveals that the models had difficulty in detecting the inaptness of the inapt paraphrases. The experiments also show that the models performed better when being instructed to focus on the metaphorical word, and that genre and the POS and novelty of the metaphorical word are all potential factors that affect model performance.

heading: 10 Limitations
text: We designed the metaphor understanding tasks to be representative of a lexical substitution task: The
metaphorical word is the only difference between a reference sentence and a candidate paraphrase. This setup makes it possible to examine whether LLMs indeed perform metaphor interpretation or resort to lexical similarity when they encounter metaphorically used words. At the same time, however, it also means that the models’ understanding of multi-word metaphors and direct metaphors (e.g., similes) could not be tested using our dataset.
We tested the lastest and state-of-the-art LLMs available at the time that our study was ongoing, but newer LLMs such as GPT-4 and Llama 2 emerged shortly after the completion of our study. We suggest running a data contamination test before evaluating newer LLMs using our dataset.
This study reveals that LLMs have difficulty distinguishing the target and source domains of linguistic metaphors. A more extensive analysis is desirable to uncover more differences between LLMs and humans in terms of metaphor interpretation.

heading: 11 Ethics statement
text: We abide by the ACL Code of Ethics. The metaphor resources used in this study are publicly available. The crowdsourcing task was approved by an ethics committee. The crowd workers received fair payment (9 GBP per hour), and no personal information was collected or stored in our database.
The metaphor samples in our dataset come from excerpts of natural discourse. They may therefore involve bias, taboo, violence, or other aspects of everyday language use that could be considered negative (we also pointed this out to the crowd workers before they gave their consent to participate, as presented in Appendix B). Nonetheless, these are integral parts of language use, and should be properly understood by NLP systems, which is precisely what this paper aims at.

heading: Acknowledgements
text: The crowdsourcing task (for collecting apt paraphrases) was funded by the Faculty of Engineering at the University of Bristol, through the faculty’s Pump-Priming Fund. We thank Dr. Jie Fu for coannotating the inapt paraphrases. We also thank all anonymous reviewers who provided constructive feedback for previous versions of this paper.

heading: A Previous metaphor understanding datasets and tasks
text: Table 4 summarises the differences between MUNCH and previous datasets.
Example (1) is extracted from MPEC. The correct paraphrase, sentence (1-a), is almost completely different from the original sentence. The two distractor sentences that follow indicate different types of misinterpretation: Sentence (1-b) wrongly interprets the meaning of the original sentence, while the last sentence is based on a literal use of the word wheels.
(1) the wheels of justice turn slowly
a. it might take time but eventually justice prevails b. ¿ justice prevails in very little time c. ¿ the wheels of a car turn slowly
The MPEC corpus is employed by two metaphor understanding tasks in BIG-Bench (Srivastava et al., 2022). The metaphor-boolean task uses a binary classification setup: Given a pair of sentences, is the second sentence a paraphrase of the first? GPT-2 only reached 0.41 accuracy on this task in a zero-shot scenario. The metaphor-understanding task consists of two subtasks: metaphor to paraphrase, which asks the model to select the correct paraphrase from 4 candidates; and paraphrase to metaphor, which requires the model to distinguish the metaphorical sentence corresponding to a given paraphrase from 3 other metaphors. GPT-2 large performed poorly on both subtasks: In a zeroshot scenario, the model gave 0.27 accuracy on the metaphor-to-paraphrase task, and 0.67 accuracy on the paraphrase-to-metaphor task.
The metaphor-literal pairs in the NewsMet dataset was created with the help of LLMs. Each news headline has a verb considered as the focus word. They first passed the headlines with the focus words masked to ALBERT (Lan et al., 2020) to obtain the first 200 words that can replace the focus word. These 200 words were then passed to a metaphor detector to obtain the top-6 metaphorical and top-6 literal candidates. Human annotators then identified the best literal counterparts for
metaphorical focus words and the best metaphorical counterpart for literal focus words.
In the IMPLI example (2), the correct paraphrase (2-a) uses a phrase, paid for, to explain the metaphorically used word absorbed in the original sentence. The distractor, on the other hand, is based on the literal meaning of absorbed. Fine-tuned RoBERTa base and RoBERTa large achieved high accuracies (> 0.8) on labelling these metaphorparaphrase and metaphor-distractor sentence pairs.
(2) he absorbed the costs for the accident a. he paid for the costs for the accident b. ¿ he absorbed the sunlight after the
accident
Example (3) is extracted from the FLUTE dataset; included in the parentheses are explanations for the paraphrase and the contradict respectively. Contrary to the MUNCH dataset, the authors aimed at paraphrases that use more than one word to replace a metaphorically used word. Note that sentence (3-b) is more of a direct contradiction of the reference metaphor than the paraphrase, as it preserves the metaphorically used word louder. The difference between the contradict and the reference metaphor may thus be easier to detect as compared to a contradict that is more similar to the paraphrase (e.g., Actions are not more important than words).
(3) Actions speak louder than words.
a. Actions are more important than words. (This phrase is used to say that what someone does is more important than what they say.) b. ¿ Actions are not louder than words. (The metaphor suggests that deeds or actions are more important than words, while the contradiction suggests that words are more important than deeds or actions.)
As example (4) shows, each metaphorical premise in the MiQA dataset is paired with a literal premise exemplifying literal use of the metaphorical word; the dataset also includes implications (the text in parenthesis) of the metaphorical and literal premises:
(4) a. I see what you mean (I understand you)
b. I see what you are pointing at (My eyes are working well)
Coms, a et al. (2022) set up 2 binary-choice tasks using the MiQA dataset: (1) Given a metaphorical premise, select the correct implication; (2) given an implication, select the corresponding premise. They also set up a generative task: Given a metaphorical premise, answer whether it implies the literal conclusion. LLMs performed well on these tasks.
The Fig-QA dataset provides similes of opposite meanings as well as their implications (given in parentheses):
(5) a. The meteor was as bright as New York City (The meteor was very bright)
b. The meteor was as bright as coal (The meteor was not bright at all)
The binary-choice task is similar to MiQA: Given a metaphorical premise, select the correct implication. They also develop a generative task which prompt models to generate implications freely. Liu et al. (2022) found these tasks challenging for LLMs in zero-shot settings.

heading: B Crowdsourcing task
text: The participant information sheet, which was presented to the crowd workers prior to the consent form, has a section dedicated to potential disadvantages and risks involved in participating in the study—
The sentences you will paraphrase were from a wide range of sources, including newspapers, fiction, and dialogues. You may occasionally encounter violence or taboo topics (e.g., war, crime, sex), as well as potentially disturbing opinions.
If you are concerned, you do not have to give consent; you can also withdraw anytime during the experiment.
The information sheet also explains how data collected from the study will be used. The workers were informed that their participation would remain confidential, that their response would be anonymised, and that the data would be made open access at the end of the study.
The annotation guidelines are shown in Figure 6. The trial sentence is provided in example (6), where introduce is the metaphorical word to be interpreted. Our final list of acceptable answers includes: address, advance, clarify, convey, cover, define, describe, discuss, elucidate, establish, explain, mention, present, propose, reveal, share, show, state, submit, suggest, teach, unveil.
(6) I shall now introduce the concept of an elementary charge, 1.6 × 10 -19 C, carried by an elementary particle called the electron.
Table 5 presents the 17 sentences for which none of the crowd workers were able to provide single-word substitutions for the metaphorical words. These are mainly highly conventionalised metaphors, for which it is usually difficult to find an alternative expression. There are also cases where the target word is part of a multi-part word (e.g., carry out, point of view) or a phrase (e.g., put in an appearance, get rid of ). These stem from annotation mistakes in VUA: According to the MIPVU procedure, VUA should have marked the entire word or phrase as a single annotation unit. We still collected paraphrases for these cases as there were no suitable way to filter them out automatically.
C Inapt paraphrase annotation
The guidelines for inapt paraphrase annotation are presented in Figure 7.

heading: D Novelty distribution of MUNCH metaphor samples
text: Figure 8 presents the novelty distribution of the metaphor samples in MUNCH.

heading: E Model evaluation details
text: We accessed the LLaMA models through Hugging Face; the queries used ∼880 GPU hours. Our GPT-3532
3.5 queries through the OpenAI API cost ∼255 USD.
We provide all the prompts used in this study: three prompts for each condition of the paraphrase judgement tasks, including word judgement (Table 6) and sentence judgement (Table 7); and three prompts for the paraphrase generation task (Table 8).

heading: F Error analysis details
text: F.1 Paraphrase judgement
Table 9 shows the number of each expectationvs-prediction combination for each model when it achieved the highest accuracy score across all conditions and prompts: The Metaphor-Word condition of Word-judgement for LLaMA-13B, using the third prompt (see Table 6); the Metaphot-Word condition of Sentence-judgement for LLaMA-30B, using the second prompt (see Table 7); the Implicit condition of Word-judgement for GPT-3.5, using the third prompt.
F.2 Factors associated with model performance
Table 10 summarises the novelty scores of the metaphor samples that receive correct versus incorrect answers from the models in the two paraphrase tasks. Table 11 and 12 show model accuracies in different genres and for different POS of the metaphorical word respectively. Like in F.1, the statistics are based on the best performance of each model. In paraphrase generation, the LLaMA models achieve their respective best performance when given the first prompt (see Table 8); for GPT-3.5, it is the second prompt.

heading: Future Work
text: Future studies could first employ MUNCH to detect the weak points of an LLM and then curate training data accordingly