IRFL: Image Recognition of Figurative Language
Authors: Ron Yosef
Yonatan Bitton
Dafna Shahaf
Venue: EMNLP 2023 Findings (2023)
Keywords: Figurative Language
Multimodal Figurative Language
Resources
TL;DR: A dataset of multimodal idioms, metaphors, and similes and a benchmark to evaluate vision and language models' understanding of figurative language.

Abstract: Figures of speech such as metaphors, similes, and idioms are integral parts of human communication. They are ubiquitous in many forms of discourse, allowing people to convey complex, abstract ideas and evoke emotion. As figurative forms are often conveyed through multiple modalities (e.g., both text and images), understanding multimodal figurative language is an important AI challenge, weaving together profound vision, language, commonsense and cultural knowledge. 

In this work, we develop the Image Recognition of Figurative Language (IRFL) dataset. We leverage human annotation and an automatic pipeline we created to generate a multimodal dataset, and introduce two novel tasks as a benchmark for multimodal figurative language understanding. We experimented with state-of-the-art vision and language models and found that the best (22%) performed substantially worse than humans (97%). We release our dataset, benchmark, and code in hopes of driving the development of models that can better understand figurative language.

IRFL: Image Recognition of Figurative Language
Ron Yosef, Yonatan Bitton, Dafna Shahaf The Hebrew University of Jerusalem yosef, yonatan. bitton, (esha huji.ac. il
Abstract
Jumped off the sinking ship just in time
A performing clown
Figures speech such as metaphors, similes, and idioms are integral parts of human commu- nication. They are ubiquitous in many forms of discourse, allowing people to convey com- plex, abstract ideas and evoke emotion. As figurative forms are often conveyed through multiple modalities (e.g.. both text and images), understanding multimodal figurative language is an important challenge, weaving together profound vision, language, commonsense and cultural knowledge.
Two social media posts that require multi- modal figurative understanding to comprehend. The left photo depicts a soccer player who left his struggling club. The right photo shows amateur boxer who lost a boxing match to a professional boxer.
Figure 1:
In this work, we develop the Image Recogni- tion of Figurative Language (IRFL) dataset. We leverage human annotation and matic pipeline we created to generate multi- modal dataset, and introduce two novel tasks benchmark for multimodal figurative lan- guage understanding. We experimented with state-of-the-an vision and language models and found that the best (22%) performed substan- tially worse than humans (97%). We release our dataset, benchmark, and code'. in hopes of driving the development of models that can better understand figurative language.
left image, the caption reads "Jumped off the sink- ing ship just in time", and the image shows a soc- cer player who has just left his struggling club. The right image, captioned "A performing clown", shows famous YouTuber losing a boxing match to professional boxer.
Due to its integral part in human communication, detecting and understanding multimodal figurative language could play important role in various multimodal challenges, such hate-speech detec- tion in memes (Das al., 2020), fact checking (Yao et al., 2022), sentiment analysis (Soleymani et al., 2017), humor recognition (Reyes et al., 2012; Shahaf et al., 2015; Schifanella et al., 2016), and tasks focusing on the mental state of social media users (Yadav et al., 2020; Cheng and Chen, 2022).
1 Introduction
Figures of speech such as metaphors, similes, and idioms are integral parts of human communication. They are ubiquitous in many forms of discourse, allowing people to convey complex, abstract ideas, compare situations, provke thought and evoke emo- tion (Lakoff and Johnson, 1980; Hoffman and Kem- per, 1987; Roberts and Kreuz, 1994; Fussell and Moss, 1998). Figurative language research often focuses on text alone; however, figurative language is often conveyed through multiple modalities (usu- ally text and images) for example, in areas such social media, advertising, and news.
Vision and Language Pre- Pre-Trained Models' (VL- PTMs) understanding of figurative language com- bined with images has not been thoroughly ex- plored, partly due to the lack of large-scale datasets. In this work. introduce the IRFL dataset (Im- age Recognition of Figurative Language) of idioms. metaphors, and similes with matching images both figurative and literal. We developed a pipeline to collect candidate figurative and literal images and annotated them via crowdsourcing.
Figure 1 shows two social media posts that re- quire multimodal figurative understanding. In the
thaps://irff-datsst.gihhh
Next, we used the dataset to design two novel

---

A simile also compares two things, often intro- duced by "like" "as". simile open when the shared properties are not explicitly revealed ("Her tasks, multimodal figurative language detection and multimodal figurative language retrieval, to assess the ggurative-languaae capabilities of VL- PTMs. The detection task is to choose the image that best visualizes the figurative phrase. See Fig- ure 2 for an example for an idiom, metaphor, and a simile, with the conrect answers highlighted.
a
Defimiince
diadrastagsi difiiid
Blanket snow
As we noticed that VL-PTMs tend to select im- ages containing objects that appear in the figurative phrase, we designed a second task targeting this behavior. In the multimodal figurative language retrieval task, the goal is to rank figurative images higher than images with objects from the phrase.
The is f'ast cheetah
We experiment with several VL-PTMs and find that the best model (22% accuracy) fails to reach human performance (97%). We also find that gener- ative models have diffieulties generating figurative images for idiomatic phrases.
Figure 2: Examples of the multimodal figurative lan- guage detection task for idiom, metaphor, and simile. The input is figurative phrase and four candidate im- ages (for idiom, we also show the definition). The cor- rect answer is marked with orange square.
We hope our dataset and benchmarks will drive the development of multimodal models that can bet- ter understand figurative language, closing the gap with human performance. More broadly, metaphor- ical reasoning is strongly tied to problem-solving and creativity; we believe such models, able to see analogies between situations that share very little on the surface, could greatly advance the field.
The IRFL Dataset
Our goal is to create a dataset with idioms, metaphors, and similes paired with figurative and literal images. This dataset can then serve as a benchmark to evaluate Vision and Language mod- els multimodal figurative language.
2 Background
Labels. Initially, we intended to have annota- tors label images "literal" or "figurative". However, after initial experimentation with the data gener- ated by our pipeline, we realized the necessity a more nuanced classification system. Hence, we introduced two additional categories.
We start with short introduction the main types of figurative language (Lakoff and Johnson, 1980; Paul, 1970; Philip, 2011).
A metaphor is a comparison between concepts that makes us think of the target concept in terms of the source concept. For example, in "You'r 're peach!", person is equated with a peach, suggest- ing that they are pleasing or delightful.
The first new category, Triurraive+Literall en- compasses images that express the figurative mean- ing of an expression while also maintaining some aspects of the literal interpretation. The second, "Partial Literal," includes images that visualize some (literal) elements or objects from the expres- sion.
heart is like a stone"), and closed when they are ("Her heart is hard as stone").
Table 1 illustrates our categories for the expres- sion "Touch wood". For example, image of someone literally touching wood while crossing his fingers for luck is classified as Figurative+Literal. This distinction also allows us to later perform a richer analysis of model performance.
An idiom is a group of words with a non-literal meaning that can not be understood by looking at individual words. E.g.. "We're re on the same page" means "Agreeing about something".
Understanding figurative language requires com- monsense, general knowledge, and the ability to map between domains. Understanding idioms, in particular, requires profound language and cultural knowledge (Paul, 1970; Philip, 201
3.1 Pipeline: Idioms
We collected 628 idioms from the MAGPIE. cor- pus (Haagsma et al., 2020) of idiomatic expres- sions. The MAGPIE corpus contains 56, 622

---

Idiom: Touch wood
Definitions:
Hopefully 2) Said while touching something wooden,
to avert superstitious bad luck from what has just been said
Literal
Figurative
Partial Literal
Figurative+Liteal
None
Some objects/ enti- ties of the phrase are visualized (here, wood)
Fits the "Figurative" definition also "Literal"/" Lit- eral"
The image illus- phrase
The image conveys one or more defini- liony of the idiom
The image does not fit any of the other categories
trates the literally
Table The table shows the different categories of the relation between image and phrase, along with matching images for the idiom "Touch wood". Workers were guided to choose the most suitable relation category by scheme uree that illustrates the correct thinking process (Figure 5, Appendix A.
crowdsourced potentially idiomatic expressions, covering 1, 756 unique idioms that appear in at least two of the following dictionaries: Wiktionary, Oxford Dictionary of English Idioms, and Usin- gEnglish.com. After collecting the idioms, we feed them into our pipeline.
literal images. To find figurative images, we need to understand the meaning of the idiom; however, the MAGPIE dataset does not contain idiom defi- nitions, we crawl them from online dictionaries (Wiktionary definitions tagged with 'figurative" "idiomatic" no such definitions exist, we try the Oxford Dictionary).
Our pipeline consists of four main steps, illus- trated in Figure 3. Given an idiom, we first get its definitions from online dictionaries and parse them into search queries 1,). Second, search for candidate images using the queries. Third, filter the images and select the best & literal and figura- tive candidates for annotation 1.3...3). Lastly, we annotate the different images via crowdworkers.
For example, in Figure 3, the idiom "white hat" nd is defined as "A good person; hero" (tagged with "idiomatic"), and also "a sailor" and "A well-meaning hacker" (tagged with "slang").
We split concatenated definitions (c.g.. "A good person; hero" is split into two definitions). some rare cases, a definition may be another idiom, and then we replace that idiom with its definitions.
3.1.1 Searching for Images
We then searched Google images for the idioms and their definitions, taking up 20 images per search query. Images were searched with "Safe- Scarch" flag "on", and "United States" region.
Our goal is to find literal and figurative images for each idiom from the MAGPIE dataset. Searching for idiom using image search often results in
3.1.2 Image Filtering
Idiam White hat
Dictionary (chometis) A
We noted that many of the retrieved images con- tained the search query in textual form. We used optical character recognition (OCR) tool EasyOCR to extract text from the images, and TextBlob correct spelling errors the OCR made. We then fil- tered images that contained objects or entities from the idiom or its definitions textual form (50% of the images). Such images are problematic be- cause they may cause the model to select an image solely based on its textual signal. Following this filter, 15% of the resulting images contained mostly
2. Search images
Select best
4. Human annotation White hat
White hat
A 20od person
A 200d person
A good person
WB
hero
hero
Figure 3: The flow of our idiom pipeline: getting defini- tions, looking for image candidates using the idiom and definitions, filtering an selecting candidate images. In the human annotation stage, blue represents Literal, Green - Figurative, and red None.
We also construct search queries from untagged defini- tions. Even though untagged definitions are rare (<3%), they are typically idiomatic.

---

Fig. Lit.
Part. Lit.
3.2 Pipeline: Metaphors and Similes
Fig.
Lit.
None
We collected 35 textual metaphors and 142 textual similes, compiled from online lists. Generating search queries from definitions (to find figurative images) is central part of our pipeline for idioms (Section 3.1). However, idioms are fixed expres- sions, but metaphors and similes are much more flexible, as the number of possible comparisons between two things is vast.
#
1970
751
434
487
2638
6697
3-maj
100%
100%
100%
100%
100%
94%
4-maj
75.5%
63%
68%
63%
80%
70%
5-maj
45%
33%
35%
38%
53%
43%
Mean
1.2
0.7
0.8
Mcdian
0
0
0
Table 2: IRFL statistics on 628 idioms. The majority of the images are related to the figurative phrase, most images are Figurative. (k-maj means k-majority)
For this reason, we had to adapt our pipeline. For metaphors, we asked three expert annotators to agree upon definitions. For similes, we use the simile itself and the target concept with the shared property ("fast") as search queries to find figurative images. For literal images that serve as distractors, we use the source and target without the shared property. In some cases, the target concept images inadequate literal distractors (an image of car might still be considered figurative for the simile "The car is as fast as a cheetah"). To solve this problem, we include the antonym of the property ("A slow car").
text. To tackle this problem, we used OCR (See Appendix A.2) to remove images with more than couple of words, as well as images with more than 30% of their space containing text.
For the remaining images, we calculated the matching score of each image with its phrase and search query using ViLT. Top-k images with a high "phrase-image" score (that passed threshold, see Appendix A.3) were tagged as potentially lit- eral. We chose the top k images with the highest "definition-image score as Figurative candidates.
Annotation. As the number of images was rela- tively small, we had two experts from our team manually annotate images. We obtained 1107 fig- urative and 1816 partial literal images for similes, 333 figurative and 729 partial literal for metaphors (the other categories were less relevant for the spe- cific data generated by our pipeline).
3.1.3 Human Annotation
We hired Amazon Mechanical Turk (AMT) work- ers annotate the relation between each idiom and its candidate images using the user interface seen Appendix A. (Figure 6). Five workers annotated each image in batches of five images per sample. They received payment of $0.15 per sample, which resulted in an average hourly wage of S15. We created qualification test to select quality annotators and provided them with inter- active training platform* to understand the task and the different categories better.
Experiments
4.1 Multimodal Figurative Language Detection Task
The Multimodal Figurative Language Detection Task evaluates VL.-PMMs' ability to choose the im- age that best visualizes the meaning of a figurative expression. Figure 2 shows an example of the task for an idiom, a metaphor, and a simile.
We split the annotation process into batches with an average size of 60 idioms per batch. After each batch, we provided each worker with a personal profile page (Appendix A. Figure 7) to view their statistics and some examples where their choice was different from the majority of workers.
Our goal to create difficult and diverse task representing the richness of our dataset (Section 3). We constructed 810 "mixed" task instances for idioms, metaphors, and similes. Each "mixed" instance contains four candidates: one is the correct answer, partially literal distractors, and random images.
Full annotation results and statistics are pre- sented in Table 2. Despite the subjective nature of the task, in 94% of the instances, there was a majority of 3 workers or more out of 5 compared to random chance of 29%.
Idiom instances have 1-2 partially literal distrac- tors. Simile instances contain two literal distractors, one of the target concept without the compared property or with its antonym visualized, and one of the source concept. Metaphor "mixed" instances
hhups:///rfll datasl--aaattttggte. dataaet.githuu........... ko/mur//imagglluuiiiccttoon hhups:///rtt datasct github. in

---

consist of between 1-3 partially literal distractors.
Categories
Idiom
Metaphor
Simile
Zero-Shot Baselines. We evaluate several state-of- the-art visionaand-language models. We use four versions of CLIP models (Radford et al.. 2021): RN50, ViT-B/32, ViT-L/14, and RN50x64/14 with 100M, 150M, 430M, and 620M parameters, respec- tively. We use the official implementations of ViLT (Kim al., 2021), BLIP (Li al.. 2022), CoCa ViT-L-14 (Yu et al., 2022), and BLIP2 (Li al., 2023b). We evaluate all models with their default hyper-parametent except ViLT on idioms, due to its maximum sequence length of 40 tokens.
Fig-
Fig.
CL.
Op.
Lit.
Humans
97
90
99.7
100
CLIP-VIT-L/1E
CLIP-VIT-B/22
38
CLIP-RN50
35
CIJI-RN50X64
52
41
BLIP
18
57
22
66
44
BLIPP
19
40
CoCa VIT-L-14
17
53
18
45
33
VILT
33
# Phrases
48
30
35
142
137
The models encode the figurative phrase and the image, producing matching score for each pair. We choose the image with the highest score as the one best matches the expression.
# Tasks
135
65
333
277
277
Table 3: Zero-shot models performance on the IRFL. "mixed" multimodal figurative language detection task. There are columns for idioms and similes. "Closed" and "Open" refers to the simile type. "Figurative" and "Figurative+Litera"" refer to the correct image category. Numbers are the percentage of instances annotated cor- rectly. Models fail to reach human performance across all figures of speech.
We also experimented with multimodal chat- bot models, including LLaVA (Liu al., 2023), InstructBLIP (Dai et al., 2023), OpenFlamingo (Awadalla al., 2023), and Otter (Li al., 2023a). We found that the first two do not support our set- ting, as they can not handle questions about mul- tiple images; the latter two do support the setting, but did not seem to understand the task, returning mostly nonsense answers.
the data sampled for all figures of speech ranges between 90% 100% (Table 3). Additionally, in 82% 99% of the instances, there an agree- ment between at least four annotators compared random chance of 6%. Samples from the valida- tion process are presented in Appendix A.5.
Supervised Models. We train supervised model for figurative classification of idioms. We add bi- nary classifier on top of pre-trained embeddings to classify whether given image is figurative. We use CLIP (VTT-B/32) model, concatenating the textual idiom embedding to the visual image embedding. followed by a classifier that produces a matching score. A score above 0.5 is labeled "Figurative". We use the Adam optimizer (Kingma and Ba, 2014) with learning rate of 0.001, batch size of 12, and train for epochs. We run the fine-tuned model on the multimodal figurative language detection ($4.1) task using the model's s matching score. We train the binary classifier on 4790 images, making sure the training data does not contain any of the images or idioms that appear in the task. We repeat five experiments with different random seeds for each task and take the mean score and std.
4.1.2 Results and Model Analysis
Zero-shot results the "mixed" multimodal figura- tive language detection task are presented Table 3. The best model achieved 22%, 30%, and 66% accuracy on the idioms metaphors, and similes tasks compared to a random chance of 25%. These results suggest that models do not understand the connection between figurative phrase and an image as humans do. We next conduct fine- grained analysis to examine if models failed be- cause they do not see any connection the figura- tive images or rather because they prioritize literal connections over figurative ones.
Models prefer partially literal images over fig- urative ones. We analyze the models' choices on the "mixed" multimodal figurative language detec- tion task and found that in all models, a partially literal distractor selected in 92% 100% of the instances where the models failed across all figures of speech (idioms, metaphors, and similes). This shows that models prefer partially literal images
4.1.1 Human Evaluation
We asked annotators that did not work previous IRFL tasks to solve the multimodal figurative lan- guage detection task. Each instance of the "mixed" multimodal detection task was annotated by 5 an- notators, and the correct answer was chosen by the majority. We find that human performance on
Idioms were pussed along with their definitions as input.

---

Fig.
Categories
Metaphors
Similes
Categories
Fig.
Lit.
Closed
Open
Candidates
2
4
6
4
Candidates
2
4
2
4
2
4
Random
50
25
16.6
25
CLIP-VIT-L/14
87
72
99
97
CLIP-
CLIP-VIT-B/22
86
97
64
87
46
71
33
53
76
86
VIT-L/14
CLIP-RN50
83
66
99
97
98
CLIP-
CLIP-RN50x64
88
98
96
96
61
84
38
67
30
53
65
82
VIT-B/32
BLIP
76
58
99
98
98
94
CLIP-RN50
56
75
30
60
24
46
78
86
BLIP2
72
55
99
93
95
88
CLIP-
CoCa ViT-I.-14
83
99
97
99
67
79
38
67
27
51
69
85
RN50x64
ViLT
72
53
96
91 97
89
BLIP
$7
79
30
62
19
51
72
88
BLIP2
58
75
25
58
14
40
75
82
Table 5: Zero-shot models performance on the mul- timodal figurative language detection task, metaphors and similes with random candidates. Numbers are % instances annotated correctly. Models' performance on similes is competitive with humans.
COCA
82
32
91
VIT-L-14
Table 4: Zero-shot models performance on different configurations of the multimodal figurative language de- tection task, idioms with random candidates. Numbers are % instances annotated correctly. The left column of each pair shows the score for idiom alone as input, and the right column shows the score for the idiom and definitions. Models fail to reach human performance.
that Figurative+ images have both a literal and figurative connection to the phrase.
Models understand metaphors but fail to reach human performance. Table shows the models' performance on metaphors with random candidates. The accuracy of all models on the Fig- urative category with candidates is 72% 88%, and 53% 76% with 1 candidates. We see a sig- nificant performance drop with all models when the number of candidates increases. The results suggest that models understand metaphors but fail to reach human performance.
over figurative ones. We find the case of idioms to be particularly interesting. Models receive a rela- tively long prompt (idion-definitiions), and often choose an image that is a literal interpretation of only 1-2 words from the prompt.
Models partially understand the figurative connection between idioms and images. To ex- amine whether models can comprehend figurative connection between an image and an idiom, we ex- periment with random candidates and several con- figurations of the multimodal figurative language detection task (Table 4). When provided with an idiom and its definitions as input, the accuracy on the Figurative category ranges between 75% 87% with 2 candidates and 58% 71% with candi- dates. These results are above chance level but still below human performance on the "mixed" task.
Models understand similes well. Table shows the models' performance on the similes with ran- dom candidates. The accuracy of all models the Figurative category with 2 candidates is 95% 99%, and 88% 98% with candidates. Models' performance is competitive with that of hu- mans, and the models maintain their performance when increasing the number of candidates. con- trast to the multimodal figurative language detec- tion task with random images, the "mixed" task shows a performance gap between closed and open similes due to open similes concealing the com- pared property, making it harder for the model to choose the figurative image. Analyzing the "mixed" task results on closed similes, we found that fig- urative images scored higher than source concept images in 52% - 74% of cases across all models. Additionally, source concept images scored higher than target concept distractor images in 51% 70% of cases. This pattern suggests a model prioritization for simile images: firstly, tar- get concept images with the compared property, then source concept images, and finally, target con-
When given the idiom alone as input, the accu- racy ranges between 56% - 67% with 2 candidates and 25% 46% with 4 candidates. These results suggest that models partially understand the figu- rative connection between idioms and images. We significant performance drop with all models when the number of candidates increases.
In the Ligurative+Literal category, with only the idiom input, models registered an accuracy of 65% 78% with 4 candidates. This performance significantly exceeds the accuracy recorded on the Figurative category with 2 and 4 candidates. The performance increase can be explained by the fact

---

Fig.
ruffle someone's feathers
Categories
Fig.
Lit.
Zero-Shot Idiom
5%
36%
Supervised Idiom
46.2% + 3.6
41.1% +:
Zero-Shot Idiom Def.
16%
41%
Supervised Idiom Def
58% + 4.2 49% 2.6
Table 6:
The performance of Supervised and Zero-shot models, both when provided with only idioms and when provided with idioms along with their definitions. Dur- ing training, the supervised model received the same in- put configuration shot results, the supervised results about 3.6 9x better in the figurative category, while figurative-literal results improved by 13 20%.
Figure 4: Example of multimodal figurative language retrieval task for the idiom "ruffle someone's ' feathers" (to unease, cause discomfort to someone). The task is to rank the figurative images above the partial literal ones, based on the images' matching score with the idiom.
was lested on. Compared to zemo
cept images lacking the compared property. Fine-tuning improves figurative understand- ing and reduces literal preference. The super- vised model results are presented in Table 6. Previ- ously we did not display the models' performance on the "mixed" task when taking the idiom alone as input due their poor performance (5% 7% ac- curacy). However, when training on idioms alone, the supervised model scored a mean accuracy of 16.2%, 9x the zero-shot score of 5%. This large performance increase might suggest that VL-PTMs representation of an idiom encodes its definitions. Training and testing with the idiom and its defi- nitions as input resulted in mean accuracy of 58% compared to 16% in the Zero-shot configuration. After analyzing the supervised model results, we found that its literal preference has improved sig- nificantly. In 41% % + 4.3 of the instances where the model failed, a partially literal distractor was selected compared to 96% in the zero-shot config- uration. Along with this improvement in literal preference, Figurative+Literal category accuracy raised from 41% in zero-shot 49%. These results show that models can improve their preference for partially literal images and recognize idiomatic fig- urative connections better via training. Moreover, the results suggest that the data is a useful training signal for our task.
images, the task is to rank the images using the model-matching score such that the figurative im- ages are ranked higher, and calculate the precision at k, where k is the number of figurative images in the input.
Figure shows an example of the task for the idiom "ruffle someone's feathers". We wish to have images of people causing discomfort ranked higher than pictures of birds and feathers. This task provides an opportunity for deeper probe into how the model comprehends figurative language in terms of its preferences.
In this task, we use the same basclines and train- ing methods mentioned in the previous task. We train the supervised model on 3802 images, mak- ing sure the training data does not contain any of the images or idioms that appear in the task.
4.2.1 Results and Model Analysis
Zero-shot results are presented Table 7. We eval- uate all figurative phrases that have both Figurative and Partial Literal images. Models' scores on the preference task are low (<61%). We expect models with proper figurative preference achieve better results. Models' success in the Figurative+Litraal category can be attributed to the literal connections of the Figurative+Litera images.
We have discovered that VL-PTMMs tend to prefer partially literal images. In the next section, design task tackle this issue.
The supervised model achieved a score of 68 + 3.8 in the Figurative category. almost double the zero-shot score of CIIP-ViT-B/32 (36). Addi- tionally, the score in the Figurative+Literal cate- gory was improved by 10 + 2.25 points. These results align well with the observation that the mul- timodal figurative language detection task super- vised model, which was trained using the same method on a different training set, also showed
4.2 Multimodal Figurative Language Retrieval Task
The Multimodal Figurative Retrieval Task ex- amines VL-PTMs preference for figurative im- ages. Given a set of figurative and partially literal

---

Calegories
Idioms
Metaphors
Similes
Stable
Categories
Dall-E
IRFL
Diffusion
Fig. Lit.
Fig.
Cl.
Op.
Figurative
42.5
0
11
4
46
Figurative+Literal
10
5
20
CLIP-VIT-L/14
57
37
26
44
34
Literal
31
0
17
0
35
CLIP-VIT-B/32
54
36
22
38
30
Partial Literal
48
2
42
2.5
23
1.5
CLIP-RN5O
54
37
25
38
31
None
19
44
27
85
43
CLIP-RNSOKER
61
39
29
43
32
BLIP
58
39
24
54
33
Number
48
120
59
118
69
126
BLIP2
57
39
22
42
29
CoCa VIT-L-14
56
36
20
39
24
Table 9: The table is double-columned, the first col- umn describes the percentage of images generated by idioms, and the second column describes percentage of images generated by the idioms' definitions. The results show that our pipeline extracted more Figurative, Figurative+Litera and Literal images and fewer None images than the generative models.
ViLT
25
34
28
# Phrases
35
142
137
Table Models performance on the multimodal figura- tive language retrieval task, the scoring metric is mean precision at k. where k is the number of figurative im- ages. There are two columms for idioms and similes. "Closed" and "Open" refers the simile type. "Figura- tive" and FFiguratie+Liiteen itteral refer to the correct image category. The results are low, we expect better results from models with proper figurative preferences.
The results show that our pipeline extracted more Figurative, Figurative+Litrall and Literal images and fewer None images than the generative models managed to generate.
substantially moderate literal preference. Table 8 shows the fine-tuned model results.
Related Work
Several papers have examined pre-trained LMs' ability to represent idioms. Shwartz and Dagan (2019) found that LMs' representation idiomatic expressions was of lower quality than that of literal ones. Chakrabarty et al. (2022) in- troduced a narrative understanding benchmark fo- cused on interpreting figurative language and found that pre-trained LMs struggle to perform well in zero-shot and few-shot settings. To the best of our knowledge, Vision and Language Pre-trained mod- els (VL-PTMs) understanding of idioms has not been investigated until this work.
Idioms.
Categories
Fig-
Fig. Lit.
Zero-Shot
36
54
Supervised
68 + 3.8
61 = 2.25
Table 8: Supervised models performance. The scoring metric is mean precision at k where the number of figurative images. of five experiments. Compared to zero-shot results, the supervised results increased by about 88% and 16% in the figurative and figura- tive+literal categories.
4.3 Generative Models Analysis
Metaphors and Similes. Recently there have been several works exploring the ability of VL-PTMs to understand similes and metaphors, and several datasets have been introduced (Zhang et al., 2021; hen Liu al., 2022; Chakrabarty et al., 2023; Hwang and Shwartz, 2023). These datasets often focus on different types images (memes, poli- tics, advertising), sometimes containing syntethic images (Akula et al., 2022). In contrast, we use natural images from search engine. In addition, our tasks introduce the new aspect of retrieval.
In our work so far, we focused on finding existing images matching figurative expression. We now explore the question of whether generative models can generate figurative images. We sampled 15 id- ioms from the IRFL dataset and experimented with the idioms and their definitions as input to Dall-E (Ramesh et al., 2021) and Stable Diffusion (Rom- bach et al., 2022). We annotated 315 generated images and found that generative models failed to generate figurative images for given idioms, gener- ating literal images instead. When provided with the definitions as input, the models had some more success in creating figurative images. Statistics on the generated images can be seen Table We
Commonsense. Commonsense is topic of in- creasing interest. Particularly relevant lines of work deal with abstractions. associations, and analogies (Mitchell, 2021; et al., 2022; Bitton et al., 2022). all required for understanding figurative language. For example, understanding "as stubborn as a mule"
also included the percentage of images from each category found by our pipeline.

---

requires the commonsense (false) association be- tween mules and stubbornness.
Council (ERC) under the European Union's Hori- zon 2020 research and innovation program (grant no. 852686, SIAM, Shahaf).
6 Conclusions and Future Work
In memory of the more than one thousand vic- tims of the horrific massacre carried out by Hamas terrorists on October 7th, 2023.
In this work we introduced IRFL, datasct of Fig- urative and Literal images for idioms, metaphors, and similes, We developed two novel tasks as a benchmark for multimodal figurative language un- derstanding. Our experiments demonstrate that the tasks are easy for humans and challenging for state- of-thc-art vision and language models. We publish our dataset, benchmark, and code.
References
Arjun R. Akula, Brendan Driscoll, Pradyumna Narayana, Soravit Changpinyo, Zhiwei Jia, Suyash Damle, Garima Pruthi. Sugato Basu, Leonidas Guibas, William T. Freeman, Yuanzhen Li, and Varun Jampani. 2022. Metaclue: Towards comprehensive visual metaphors research.
In the future, we hope to extend this work to other modalities and different forms of figurative speech. In addition, there are interesting cross- cultural connections between figurative expres- sions. For example, the English expression "cost an arm and a leg" (meaning expensive) has corre- sponding expression in French: "Coûter les yeux de la tête" (literally, cost the eyes of the head). Adapting our ideas languages other than English, taking advantage of such connections, is another promising direction.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hes- sel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, al. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint 2rXiv:2308.01390
Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, and Roy Schwartz. 2022. Winogavil: Gamified association benchmark to challenge lad--uugeg models. ArXiv, abs/2207. 12576
We believe that multimodal figurative language is an essential aspect of human communication that is under-explored in AI; we hope that this work will encourage the development of multimodal models that can better understand figurative language.
Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. 2022. It's not Rocket Science: Interpreting Figu- rative Language in Narratives. Transactions of the Association for Computational Linguisties, 10:589- 606.
More broadly, metaphorical reasoning is strongly tied to problem-solving and creativity; we believe that models that can see analogies between situations that share very little on the surface could find many potential applications.
Tuhin Chakrabarty, Arkadiy Saakyan, Olivia Winn, Artemis Panagopoulou, Yue Yang, Marianna Apid- ianaki, and Smaranda Muresan. 2023. I spy metaphor: Large language models and diffusion mod- els co-create visual metaphors.
Ju Chun Cheng and Arbee L. P. Chen. 2022. Multi- modal time-aware attention networks for depression detection. Journal of Intelligent Information Systems.
7 Limitations
Our dataset focuses on English idioms. As transla- tion of figurative expressions is a particularly del- icate task, it is not straightforward to expand our dataset to other languages, and further research is needed explore the effectiveness of our pipeline other languages. In addition, our method heavily relies on sources of figurative expressions, their definitions, and the image search engine.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Wcisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. In- structblip: Towards general-purpose vision-language models with instruction tuning.
Abhishek Das. Japsimar Singh Wahi, and Siyao Li. 2020. Detecting hate speech in multi-modal memes. CoRR, 2is/20122.4891..
Susan R Fussell and Mallie M Moss. 1998. Figurative language in emotional communication. Social and cognitive approaches to interpersonal communica- tion, pages 141.
Acknowledgements
We want thank Nir Sweed for his valuable feed- back. We would also like to thank Nissim Barzi- lay for his contribution to the collection of figura- tive and literal images for metaphors and similes. This work was supported by the European Research
Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages

---

279-287. Marseille, France. European Language Re- sources Association.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning, arXiv preprint arXiv:2304.08888
hen Liu, Gregor Geigle, Robin Krebs, and Iryna Gurevych. 2022. Figmemes: A dataset for figura- tive language identification in politically-opinionaeee memes. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing. page 7069-7086. Association for Computational Linguistics.
Melanie Mitchell. 2021. making artificial intelligence. CoRR.
Abstraction and analogy-
10111702.10717.
Anthony M. Paul. 1970. Figurative language. Philoso- phy and Rhetoric, 3(4):225-248
G. Philip.. 2011. Colouring Meaning: Collocation and connotation in figurative language. Studies in Cor- pus Linguistics. John Benjamins Publishing Com- pany.
Robert R. Hoffman and Susan Kemper. 1987. What could reaction-time studies be telling about metaphor comprehension? Metaphor and Symbol, 2:149-186.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence machine learning, pages 8748-8763 PMLR.
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Chris- tos Christodoulopoulls, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schotmmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2023. State-of-the-art gen- cralisation research in nip: A taxonomy and review.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot eext-to-mmae gen- eration. In International Conference on Machine Learning, pages 8821-8831. PMLR.
EunJeong Hwang and Vered Shwartz. 2023. Memecap: A dataset for captioning and interpreting memes.
Anya Ji. Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert D. Hawkins, and Yoav Artzi. 2022. Abstract visual reasoning with tangram shapes. In Proceedings of the Conference on Empirical Meth- ods in Natural Language Processing. Association for Computational Linguistics.
Antonio Reyes, Paolo Rosso, and Davide Buscaldi. 2012. From humor recognition to irony detection: The figurative language of social media. Data and Knowledge Engineering. 74:1 Applications of Natural Language to Information Systems.
Richard M. Roberts and Roger J. Kreuz. 1994, Why do people use figurative language? Psychological Science, 5(3):159-163.
Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Assionaandlanguuaee transformer without convolu- tion or region supervision. In Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, PMLR.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. 2022. High- resolution image synthesis with latent diffusion mod- els. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
pages
Diederik P. Kingma and Jimmy Ba. 2014. Adam: method for stochastic optimization.
Rossano Schifanella, Paloma de Juan, Joel R. Tetreault, and Liangliang Cao. 2016. Detecting sarcasm multimodal social platforms. CoRR, abs/16088 022289.
George Lakoff and Mark Johnson. 1980. Conceptual metaphor in everyday language. The Journal of Phi- lasophy, 77(8):55-4466
Dafna Shahaf, Eric Horvitz, and Robert Mankoff. 2015. Inside jokes: Identifying humorous cartoon captions. In Proceedings of the 21th ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Mining, KDD 11, page 1075,1074, New York, NY, USA. Association for Computing Machinery.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023a. Otter: A multi- modal model with in-context instruction tuning.
Jumnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. 2023b. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. arXiv preprint arXiv:2301. (tt::230../2999 /2597.
Vered Shwartz and Ido Dagan, 2019. Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition. Transactions of the Association for Computational Linguistics, 7:403-41
Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. International Conference on Ma- chine Learning, pages 12888-12900 PMLR.
Mohammad Soleymani, David Garcia, Brendan Jou, Bjöm Schuller, Shih-Fu Chang, and Maja Pantic. 2017. survey multimodal sentiment analysis. Image and Vision Computing, 65:3-14, Multimodal

---

Sentiment Analysis and Mining in the Wild Image and Vision Computing.
Phrase
the road
Shweta Yaday, Jainish Chauhan, Joy Prakash Sain, Kr- ishnaprasad Thirunarayan, Amit Sheth, and Jeremiah Schumm. 2020. Identifying depressive symptoms from tweets: Figurative language enabled multitask learning framework. In Proceedings of the 28th Inter- national Conference on Computational Linguistics. pages 696-709, Barcelona, Spain (Online). Interna- tional Committee Computational Linguisties.
Defnttooe
proce
begin troveling on automooble other
Figurotive
Higurotive
Ulera
Portisl lifeat
Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cho, and Lifu Huang. 2022. End-to-end multimodal fact-checking and explanation generation: chal- lenging dataset and models.
None
Jiahui Yu, Zirui Wang. Vijay Vasudevan, Legg Ye- ung, Mojtaba Seyedhosseini, and Yonghui Wu, 2022. Coca: Contrastive captioners are image-text founda- tion models. arXiv preprint arXiv:2205.01917.
Figure 6: The UI used annotate the automatic pipeline candidate images. Annotators need to choose the cat- egory that best describes the relationship between the idiom and the image.
Dongyu Zhang, Minghao Zhang, Heting Zhang, Liang Yang, and Hongfei Lin. 2021. MultiMET: multi- modal dataset for metaphor understanding. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the Inter- national Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 3214-322 4-222,,, Online, Association for Computational Linguisties.
A.2 Documents Filter
In an effort to minimize the number of images dominated by text, we filtered out images con- taining more than a few words, which accounted for 15% of the total. Despite this, certain images like documents, books, and contracts managed to bypass our OCR-based filters, representing 2% of the total images. To address this issue, we developed filter using the ViLT model (Kim al., 2021). This filter calculates an image's matching score with the prompts "a document", "a page of a book", or "a contract" and removes it if the total score surpasses a set "document" threshold. To find this threshold, we conducted grid search on 20 sampled images at each point in the distribution of -30,- -25, 220, -15, 10, -5, 0, 10, 15, 20, 25 "document" "non- document". The (20, 15) range showed the best results, so we conducted a more dense grid search within this range and found the best threshold to be 18.77 77 with a TPR of 100% and an FPR of 1%.
A Appendix
A.1 Annotation UI
la image Figurative B the
No
3 also Ulteral
is Lltaral 2
Partial Litural
Oves lowdly fie
Yes
No
Yes
No
Figurative Llterol
Figuraiive
Literal
is Pertial Lteen
.0 categorizing each
vhodeet
Yes
Ltteral
None
Figure 5: The scheme tree that was provided to annota- tors to illustrate the correct thinking process.
A.3 Literal Threshold
We conducted two grid searches on images that passed the OCR filters and had "phrase-image" score higher than the "search-query" score to find literal threshold. We sampled 20 im- ages from each point in the distribution of -10, -8 6, -4, 2, 0, 2, 4, 6, 8, 10, and anno- tated them as "literal" or "non-literal". This dis- tribution aligns with the normal distribution the images that stand the two criteria mentioned

---

Dear worker, this is your personal profile page. Below you will find your statistics and some uncommon annotation scenarios.
30 images from each point in the distribution of -5, -4, -2, -1, 0, 1, 2, 4, 5, and annotated them as "literal" or "non-literal". We chose the threshold of 1.150353 with a TPR of 86% and FPR of 18%.
Statistics
Numbor of Figuraiive Uierrl annotated
Numbar Figurativo
38
Numbar Lher annotated
Numoow of Portiel Userol annotsted 12
more crnecaated
Agreement
parentaga
annotttted
230
(by mojenbt amnototors.cr mome
Age
Rigurnt on heeal agrwermmn
Pigurat vo Agraamenn 91.9%
150
Liteeal eement
inneal Agnwemmnnt None Agrocmiont 86.9%
Spacial annotation cases
80
-10
Phrase pump iron
Figure 9; The distribution of the images that passed the OCR filters and had a "phrase-imas irrasee-maae"" score higher than the "search-query" score.
Dwi-toon
To weights
2 To reignge in wiiht morkigg
conctalos rigurative
We observed that when the pphrase-image score is high, high probability. However, the reverse is not true, there can be multiple "literal" images with very low literal score (Figure 10).
say that the image is literal with
You chosas Partial
Llteral
A.4 GenBench Evaluation Card
Figure 7: An example profile page includes worker's statistics and some handily picked examples where his choice was distant from a majority of four workers.
Motivation
Practical
Cognitive
Intrinsic
Fairness
O
Generalisation type
Compo- sitional
Cross
Cross Language
Cross
Robust- ness
Structural
Task
Domain
Mturk Leaderboard X
N0/10
Shift type
KOBBUX
476
Covariate
Label
Full
Assuned
12:0
812-
R000
KGG
Shift source
601
M13
Naturally
Partitioned
Generated shift
Fully generated
K0002
622
occuring
natural
7U8
9623
Shift locus
Train text
Finetune
Pretrain test
Figure 8: An example of the profile page includes the worker's statistics and some handily picked examples where his choice was distant from majority of four workers.
train test
Table 10: The GenBench evaluation card (Hupkes et al., 2023) for the IRFL Multimodal Figurative Language Detection Task and the Multimodal Figurative Language Retrieval Task.
above (Figure 9). We found the 2, range to result in the best thresholds, so we conducted a more dense grid search in this range. We sampled

---

press the panic button
To panic
Figure 10: Literal images of the idiom "Foam at the mouth" and the idiom "Take the bull by the horns". Both images have "phrase-image" score of -9.
A.5 Understanding Task Samples
shrinking violet
A very shy or timid person, who avoids contact possible
save someone's skin
seve somconc's
evennt undesirable occurrence
get boot
1o be dismissed from amployment
to be voted out, evicted, othersise mode to

---

heart of gold
jungle city
a night owl
The juice is as sweet as sugar
the car is rocket
The dog is as busy as a bee

---

The frog is as red as a tomato
The milk is as fresh as a daisy


Reviews:
[{"review_id": "Z704ISY6ol", "reviewer": "Reviewer_nddL", "summary": "", "questions": "There have been many open-source multimodal LLMs recently that have showcased impressive generalization capabilities. I am curious to know if there are any considerations to assess these models using this multimodal figurative understanding benchmark.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We are glad that you consider our experiments clear and robust, and that you recognize our work’s potential to advance multimodal models’ understanding of higher levels of abstraction.\n\n* “It would be better if you could provide further clarification on the training data”\n\nWe are sorry for the confusion. Line 295 refers only to idioms, for which we collected enough data to create both a training set and a test set. For metaphors and similes your understanding is correct. We will clarify this point in the manuscript, including more details about the training data.\n\n\n* “I am curious to know if there are any considerations to assess these models using this multimodal figurative understanding benchmark”\n\nWe are aware of the recent multimodal LLMs, and are quite excited about their potential. In the paper we tested models that were available a month before the submission; in follow-up work we are planning to assess the performance of models such as LLaVA, Llama-Adapter, InstructBLIP, etc., as well as release a collab notebook that will allow others to easily test their models on our benchmark.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The acceptance of this article could potentially catalyze the advancement of multimodal models that possess an enhanced understanding of figurative language. In contrast to existing multimodal datasets that primarily focus on factual question-answering and descriptive contexts, this particular dataset places its emphasis on the diversity, intricacy, and abstraction of expressive forms. Such an endeavor has the potential to stimulate the creation of multimodal models understanding higher levels of abstraction.\n2. This work presents clear and robust experiments that illuminate common weaknesses in the current state of VLMs in terms of their comprehension abilities.", "reasons_to_reject": "There is a mention of the utilization of training data in line 295. However, it is not clear to me where these training data come from. From my understanding, it seems that the data collected in Section 3 is quite limited and primarily constitutes a test-only dataset. It would be better if you could provide further clarification on the training data.", "questions_to_authors": "There have been many open-source multimodal LLMs recently that have showcased impressive generalization capabilities. I am curious to know if there are any considerations to assess these models using this multimodal figurative understanding benchmark.", "paper_topic_and_main_contributions": "1. Introduce a new question of figurative speech comprehension in a multimodal context. This work specifically explores metaphors, similes, and idioms understanding in a quantifiable manner, while previous works only focus on text modality or treat figurative language understanding as an implicit requirement.\n2. develop a novel benchmark for the assessment of the VLM's capability in comprehending multimodal  figurative language. This work also presents a comprehensive constructing pipeline and demonstrates its correctness with human evaluation.\n3. conduct detailed experiments on different VLMs to demonstrate that while current VLMs exhibit some degree of comprehension of figurative speech, they do not attain the level of human performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ftJfBD81nF", "reviewer": "Reviewer_Sk6i", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We are glad that you appreciate the quality of our dataset and  recognize its potential to drive the development of multimodal models with better figurative understanding. \n\n* “In my opinion, figurative language understanding should be in a specific context. it is not so valuable to guess the actual meaning of a figurative short phrase.”\n\nWe agree that understanding figurative language in context is an interesting task, but we argue that the standalone task is valuable and worth exploring in its own right (and indeed, has been studied before).\n\nFirst, we note that language models have seen figurative language in a myriad different contexts during training; in a sense, we are testing whether they managed to learn the _gist_ of each expression. We also note that figurative language is often used with very little context in real-world scenarios such as social media posts or advertising. \n\nWe believe that the fact that our task is easy for humans yet hard for models makes it a good stepping stone towards models that can better understand figurative language, whether with or without context. \n\n* “It is more like commonsense reasoning than a vision language understanding task”\n\nWhile commonsense is very important for understanding some forms of figurative language (e.g., open similes), the task goes beyond commonsense reasoning. For example, no amount of knowing about jumping and sharks will help one understand what the idiom “jump the shark” means.\n\nIn the case of similes and metaphors, the models also need to perform a mapping between the two modalities, understanding the links between them. \n\n* “Some figurative expressions are even very difficult for me to understand. It is because I do not have any relevant knowledge. I can only guess what the original words mean”\n\nAs non-native speakers, we can definitely relate. As you noted yourself, we made a significant effort to obtain clear examples of figurative language data through an extensive annotation process. We believe that the high success rate of humans on our benchmark demonstrates the quality of the data.\n(Also, as we noted, language models are likely to have seen those expressions in context, so they have more information than merely relying on guessing the meaning of words)\n\n\n* “you can test more multi-modal large language models like LLaVA and InstructBLIP”\n\nIn the paper we tested models that were available a month before the submission; in follow-up work, we are planning to assess the performance of models such as LLaVA, Llama-Adapter, InstructBLIP, etc., as well as release a collab notebook that will allow others to easily test their models on our benchmark.\n\n* “There are no suggestions or solutions for improving the ability to understand figurative language.”\n\nWe performed an error analysis, noting that in the vast majority of model failures the partially literal distractor was selected, and that training and fine-tuning significantly improve the results. We will gladly expand on this in the future work section, outlining some of our ideas for improving model performance even further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed dataset and benchmark will drive the development of multimodal models that better understand figurative language.\n2. A lot of processing and human labeling will give us a high-quality dataset.\n3. It is interesting to explore the ability of figurative recognition for visual language models.", "reasons_to_reject": "1. In my opinion, figurative language understanding should be in a specific context. it is not so valuable to guess the actual meaning of a figurative short phrase. It is more like commonsense reasoning than a vision language understanding task. My suggestion is that you can put the figurative text in a more complex context and then explain the meaning of it, possibly by retrieving related images. Without further context, a figurative expression can have multiple meanings.\n2. Some figurative expressions are even very difficult for me to understand. It is because I do not have any relevant knowledge. I can only guess what the original words mean. I think it is the same with vision-language models. As I noted above, commonsense is probably the key issue for your task, so you can test more multi-modal large language models like LLaVA and InstructBLIP.\n3. There are no suggestions or solutions for improving the ability to understand figurative language.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a new dataset and two tasks to evaluate the figurative recognition capabilities of common vision-language models. The authors also provide an extensive experiment of the dataset with several typical VMLs as well as human annotators. The results show that existing visual language models perform poorly on figurative recognition, which is not surprising.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "T1B9agOJdL", "reviewer": "Reviewer_W39D", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your useful feedback. We are glad you found our formulation clear and accurate.\n\n* “The proposed dataset and benchmark tasks may not adequately capture the complexity and diversity of real-world multimodal figurative language understanding scenarios.”\n\nWe completely agree that real-world multimodal figurative language is complex and diverse. While we have not covered the entire landscape, we believe that our dataset covers a significant (and interesting) portion of it: idioms, metaphors, and similes are some of the most common (and well-studied) types of figurative language. For each phrase, we collect multiple images that are related to the phrase in different ways (figurative interpretation, literal interpretation, etc.). \n\nWe will add a note to the Future Work section suggesting more types of figurative language to explore, and discuss the challenges they pose (e.g., allusions rely heavily on world knowledge and cultural references), as well as potential additional modalities (such as audio). We hope our work will invoke and catalyze further research in this emerging field.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This article accurately distinguishes the concepts of metaphor, simile, and idiom. Figure 5, the scheme tree, is very useful. In comparison to some of the dataset studies I have seen before, this article starts from the most fundamental elements and avoids sacrificing clear definitions for the sake of quantity (such as conflating metaphors and analogies).\n\n\nIn addition to the existing images matching figurative expressions, the paper explores whether generative models can create figurative images. Sampling 15 idioms from the IRFL dataset, they experiment with Dall-E and Stable Diffusion models. The results indicate that generative models struggle to generate figurative images for given idioms, producing literal images instead. Providing definitions as input improves the performance to some extent, but challenges remain in generating accurate figurative representations.\n\n\nThe paper sheds light on the difficulties in multimodal figurative language understanding, urging further research to develop models that better comprehend figurative language.", "reasons_to_reject": "The proposed dataset and benchmark tasks may not adequately capture the complexity and diversity of real-world multimodal figurative language understanding scenarios.", "questions_to_authors": "", "paper_topic_and_main_contributions": "\nThe paper proposes the \"Image Recognition of Figurative Language\" dataset to evaluate the understanding of figurative language by vision and language models. The authors leverage human annotation and an automatic pipeline to generate a multimodal dataset and introduce two novel tasks as a benchmark for multimodal figurative understanding. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an innovative new benchmark for assessing the comprehension of figurative language / images, and provides baseline performances of several models. The performance of models tested in the paper (sota at submission time) is far from human performance. It also shows that generative models like Dall-E struggle to generate figurative images.
The paper provides a useful problem description, distinguishing metaphors, similes, and idioms.
Decision: Accept-Findings