Can LLMs Understand Sense Ambiguity in Figurative Language? Evaluating LLMs' Figurative Word Sense Knowledge
Authors: Seokwon Song, Taehyun Lee, Jaewoo Ahn, Jae Hyuk Sung, Gunhee Kim


Reviewer paper summaries: The paper presents a dataset for the recognition of figurative language. 

To create the dataset, the authors first extracted the vehicles from 700 sentences; then they checked if the main vehicle conveyed the figurative meaning. Finally, to verify this condition, they hired 5 annotators via Amazon Mechanical Turk. They also tested 5 state-of-the-art (SotA) Large Language Models on the dataset (LLama3 8B, Mistral 7B, Claude 3 Opus, and ChatGPT 3.5 and 4o).

The paper faces a relevant challenge and it carries some positive contributions. However, it can be improved in several ways (see comments)
Summary of strengths: - the paper focuses on an important and relevant problem
- the experimental section is good (however there is some lack in comparing what proposed with sota datasets, systems and models)
Summary of weaknesses: - the paper introduces some ambiguity since the beginning, creating confusion by using the term vehicle as a word, and as a concept in the definition of the task
- In the related work section, the authors describe that this is not the first work on figurative meaning. How do existing datasets differ from this one? What is the novelty proposed in this dataset? This should be better stated
- Since it is not the first work on figurative meaning, how do other SOTA models perform on the presented dataset?
- In the LLM comparison, the authors tested Mistral-7B and LLama3-8B. Why did they not use bigger models such as Mistral 22B and LLama3-70B?
Comments, suggestions and typos: It could also be interesting to explore how the same model, with a different number of parameters (e.g., LLama 8B, 13B, 70B), performs on the dataset.
Overall assessment scores: 2.5