MemeCap: A Dataset for Captioning and Interpreting Memes
Authors: EunJeong Hwang
Vered Shwartz
Venue: EMNLP 2023 Main (2023)
Keywords: meme
captioning
image
large multimodal model

Abstract: Memes are a widely popular tool for web users to express their thoughts using visual metaphors. Understanding memes requires recognizing and interpreting visual metaphors with respect to the text inside or around the meme, often while employing background knowledge and reasoning abilities. We present the task of meme captioning and release a new dataset, MemeCap. Our dataset contains 6.3K memes along with the title of the post containing the meme, the meme captions, the literal image caption, and the visual metaphors. Despite the recent success of vision and language (VL) models on tasks such as image captioning and visual question answering, our extensive experiments using state-of-the-art VL models show that they still struggle with visual metaphors, and perform substantially worse than humans.

MEMECAP: A Dataset for Captioning and Interpreting Memes
EunJeonng Hwang4 and Vered Shwartz!.
University of British Columbia 2 Vector Institute for Al (ejhwang, (chwartz)@cs. ubc.ca
Abstract
Title: them my
Memes are widely popular tool for web users to express their thoughts using visual metaphors. Understanding memes requires recognizing and interpreting visual metaphors with respect to the text inside or around the meme, often while employing background knowledge and reasoning abilities. We present the task meme captioning and release a new dataset, МЕMЕCаР. Our dataset contains 6.3K memes along with the title of the post contain- ing the meme, the meme captions, the literal im- age caption, and the visual metaphors. Despite the recent success of vision and language (VL) models tasks such as image captioning and visual question answering, our extensive exper- iments using state-of-tee ofte-oof-hheart VL models show that they still struggle with visual metaphors, and perform substantially worse than humans.
Me and my 2 regular followers who always likes my posts
Meme poster appreciates their only two followers and one of them is their alternative account.
Caption:
1 Introduction
Figure I: A meme and its title. The caption describes what the meme poster was trying to convey.
Web users frequently communicate their thoughts and feelings online using memes (Buchel, 2012; Tanaka al., 2022). Memes are created by taking an existing widespread image and attaching new meaning to it by altering the text inside the image. For example, in Figure 1, Tom cat is a metaphor for the person who posted the meme and the cats he is shaking hands with represent his two regular fol- lowers who always like his posts. This incongruity between the image and the text makes memes hu- morous (Tanaka al., 2022).
accurate descriptions of images in both zero-shot and in-context setups. Such models are first pre- trained on language-only and vision-only datasets, and then trained on tasks such as image captioning and visual question answering, where the redun- dancy between the vision and language is used embed them shared space. For example, the majority of image captions existing datasets de- scribe what is depicted in the image, at most adding subjective interpretations or inferences about the story behind the image (Alikhani et al., 2020). In contrast, there is little work on visual metaphors to date (Zhang al., 2021; Chakrabarty et al., 2023).
Because of their complementary nature, inter- preting the meaning of a meme requires understand- ing both the visual and text modalities. Moreover, memes are often posted on social media platforms along with additional text, such as "one of them is my alt" in Fig. 1, which is further needed to understand the meme.
In this paper, models can successfully interpret memes. We pro- pose the task of meme captioning, in which models are presented with meme along with its title (e.g. the title the post containing the meme), and is tasked with generating a concise caption describing
investigating whether VL
Recently, there is surge of vision and language (VL. models (e.g. Alayrac et al., 2022; al.,
2023; OpenAI, 2023). VL models have shown re- markable capabilities generating detailed and

---

the meaning of the meme. This task goes beyond object recognition and language understanding. is challenging due to the metaphorical role of the visual content of the meme (Scott, 2021). For ex- ample, in Fig. I, the model needs to recognize that Tom cat is merely a metaphor for the meme poster, and that handshaking signals appreciation. The literal content of the image, such Tom or the handshake, should not be part of the meme cap- tion. Recognizing and interpreting such metaphors involve detecting facial expressions, the tone ex- pressed the texts, making commonsense infer- ences, and more Seitton-Guetae et al., 2023).
and more. Chakrabarty et al. (2023) tested image generation models on prompts involving a visual metaphor such as "My bedroom is pigsty". They found the unsatisfactory performance can be im- proved by using a large language model (LLM) to interpret the visual metaphors and add details to the prompt, such "messy bedroom".. Akula et al. (MetaCLUE; 2023) introduces set of tasks pertaining to visual metaphors in synthetic images, such as retrieval and captioning.
Finally, the Image Recognition of Figurative Language dataset (IRFL.; Yosef et al., 2023) presents an idiom, metaphor, or simile, along with 4 images, with the goal of selecting the image that matches the figurative expression. The distractors consist of an image that depicts the expression lit- erally, for example a picture of cheetah for the simile "as fast as a cheetah". This dataset is chal- lenging for state-of-the-an VL models.
To that end. we collected a meme captioning dataset MEMECAP, containing 6,384 memes along with their captions. Each meme is also annotated with the literal image description (e.g. "Tom cat is shaking hands with two small cats and smiling"), and the visual metaphors (e.g. Tom a metaphor for the meme poster).
We establish comprehensive baseline perfor- mances with recent large-scale VL models, in vari- ous training setups (e.g. zero-shot, few-shot, fine- tuning), and inputs (i.e. meme, title, literal image captions, and metaphors). Human evaluation of the generated captions shows that models are far from humans in captioning memes. In particular, models tend to ignore important visual or textual elements, and instead, repcat the text inside the meme or make up fake elements. Our findings merit future research on this task."
2.2 Memes
Recent work on memes focused on detecting hate- ful or harmful content in memes (Kiela et al., 2021; Qu et al., 2022: Sharma et al., 2023), classifying memes to humorous or not (Tanaka et al., 2022), and analyzing the sentiment of memes (Sharma et al., 2020). Earlier work automatically generated text go inside the meme (Wang and Wen, 2015), and the memes themselves (e.g. the ImgFlip575K dataset).?
Although MultiMET (Zhang et al., 2021) does not focus specifically on memes, the images were collected from range of sources including social media, which contains memes. The similar Met- Meme dataset (Xu et al., 2022) focuses memes. Differently from our work, both datasets contain annotations for visual metaphors while MEMECAP also contains meme captions.
2 Background
2.1 Metaphors
Most work on metaphors is limited to textual metaphors, and pertains to collecting resources (Dodge et al., 2015), detecting interpreting metaphorical expressions in context (Choi al., 2021; Chakrabarty et al., 2021a; Aghazadeh et al., 2022: Chakrabarty et al., 2022), and metaphor generation (Stowe et al., 2021; Chakrabarty et al., 2021b).
2.3 Other Image Datasets
The WHOOPS benchmark (Bitton-Guetta et al., 2023) consists of unconventional human-createe and machine-generater images that defy common- sense (c.g. an image of "Albert Einstein holding a smartphone"), along with their textual descrip- tions. It's meant to be used for image captioning, image-text matching, visual question answering, and explanation generation. In contrast, our work focuses memes, and tests models their ability to interpret real memes posted by web users.
Recently, there has been interest in visual metaphors. Visual metaphors occur when tar- get concept is compared another visual element (vehicle) (Forceville, 1996). MultiMET (Zhang et al., 2021) and Met-Meme (Xu et al., 2022) are two datasets of text-image pairs with annotations for the existence and types of metaphors, sentiment,
'Our code and data are available at: httpp://ghtt t/g.buub.. com/eu aewhwang/memeecam
Pht //gi:t/bbttt https///ggtttbo..occc/s///////// com/sschesa/mgg1ip55777 Dataset

---

#Memes
MMpCap
#I-Cap
#Mph
3.2 Captions
Train+Val
5,828
1.0
1.0
2.1
We conducted two rounds of annotations to obtain the captions. In the first round, collected the literal image descriptions, disregarding the text in the memes, while in the second round, we collected the meme caption along with the visual metaphors.
Test
559
3.4
1.0
3.1
Table I: The number of memes MEMECAP, and the average number of meme captions (M-Cap.), image captions (I-Cap.), and metaphorical keywords (Mph) per meme.
Literal Image Captions. We asked workers to caption the image, disregarding the text. For exam- plc, a suitable literal image caption for Figure 1 is "Tom cat is shaking hands with two small cats and smiling". To prevent biasing the workers with the text inside the meme, identified and removed the text in the meme using the LaMa inpainting tool (Suvorov et al., 2021). We collected one caption for each meme, which we manually verified.
Another multi-modal benchmark which is sim- ilar to ours is the New Yorker Cartoon Caption Contest (Hessel al., 2023). This benchmark in- volves tasks: matching caption to a cartoon, evaluating the quality of the caption, and explain- ing the joke. While both memes and cartoons use a combination of visual and textual elements to convey humor, memes are based recognizable images that are modified and repurposed to create new meanings based shared cultural knowledge. Cartoons, on the other hand, are originally drawn illustrations, often in the form of comic strips, that convey a more complex narrative. Further, while Hessel et al. (2023) focus on discriminative match- ing (i.e. selecting the more appropriate caption) and generating an explanation, in this paper we present generative task, i.e. generating a caption to describe meme.
Meme Captions. We showed a second set of an- notators the full meme, title, and literal image cap- tion, and asked them to provide a meme caption. This HIT included two steps. First, workers were asked to indicate for each term in the literal image caption whether it was used metaphorically, and if so, what was the target of the metaphor (e.g.. "Tom cat" is a metaphor for the meme poster). We then instructed the workers to write a concise caption describing the meaning that the meme poster was trying to convey, while excluding the metaphor ve- hicles (e.g., not mentioning Tom). We collected one caption for each meme in the training set, and 2 to 4 captions for memes in the test set.
3 The MEME Dataset
The overall data collection and annotation process is illustrated in Figure 2. We collected memes
(Sec 3.1) and crowdsourced their captions (Sec 3.2). We present the data splits and statisties in Sec 3.3.
Both rounds of annotations were conducted Amazon Mechanical Turk (MTurk). To ensure the quality of annotations, we required that workers were located English-speaking countrics (c.g. US, UK, Canada, Australia, and New Zealand), had an acceptance rate of at least 98% on 5,000 prior HITs, and passed a qualification test similar to the task.
3.1 Memes
We scraped memes from Reddit using the publicly available 3 particular, we focused on the subreddit /r /r/memes and collected posts that con- tained a meme with a post title. To ensure that the text and image are complementary, we manu- ally examined the memes and excluded memes that lacked any text contained an excessive number of characters. To exclude offensive content from the dataset, we filtered out memes with profanity in the text using the Google banned word list.* We also filtered out images with sexual content, for which the NudeNet Classifier returned an unsafe score higher than 5.9.5
We excluded from the dataset any memes that workers in each the rounds marked as offensive, sexual, hateful, or uninterpretable.
3.3 Final Dataset
We clustered the examples in the dataset based on the vector representation of their meme captions using OPT2.7b (Zhang et al., 2022). To ensure the diversity of topics both the training and test sets, we then sampled 10% of the memes from each cluster and assigned them to the test set, and the rest of the memes into the training and validation
Pht pottps:// hewttpp:/wwwrrrr www..reddii t. com/dev/ap.
"hctps://githb. cow/coffueeeeeeeeoooooo and pun/ profanity-ooooda Phttps: / t/t.bht..tttutete///////te... cww/notAI com/nootAT-tech//ueelle tech//ueeeh//

---

Memes
Meme Captions and Metaphors
Scraped from reddit/ /s/mome
Title: WWyythey gotta be like this
Manually filtered for quality and to exclude offensive content
Her: why doesn't he understand my signals? The signals:
L: intersection relationship between a man and a woman L tree of traffic lights the woman's complicated signals
Literal Image Captions
Remove text from meme
Crowdsource the image captions
The intersection the world has to be contralled by a tree of traffic lights
Women wonder why don't understand their signals when they are overly com complicated.
Figure 2: Overall process of collecting memes, literal image captions, visual metaphors, and meme captions.
Meme Type
Metaiphor Vehicle Type
Melaphor Target Type
Oonoler versey Tus Dommmntt
@ Persor oi Chancle
Appccattio Doscleg Plai
Ansthe Partor
Fecaal express of gentun
SAN
10
10%
10%
1916
ROOR
TORT
Figure 3: (I) Meme Type:
Percent of memes with no visual metaphors, and with metaphors that can be understood with the text alone, vision alone, or both (complementary). (2) Metaphor Vehicle Type: Types of visual elements used to convey metaphorical meaning. (3) Metaphor Target Type: The intended meanings of the metaphors.
set." Table I shows the statistics of our dataset.
used to convey the metaphorical meaning, as well as the target, i.c. the meaning itself.
3.4 Types of Metaphors
Metaphor Vehicle Type. The middle part of Fig 3 shows that the most common vehicle is a person character, followed by objects (such the trophy), facial expressions or gestures (such as the surprised look on the man's 's face), and actions.
We manually analyzed 28 memes along with their metaphor annotations.
Meme Type. First, following Zhang et al. (2021) and Xu al. (2022), we categorized the memes into three categories: text dominant and image dom- inant, where the text or the image respectively may be enough to understand the metaphor, and com- plementary, where both modalities are required. We added fourth category for memes that had no metaphor, i.c. whose meaning is conveyed explic- itly in the text. The left part of Figure 3 shows that the 44% of memes complementary, but each of the other categories is also prominent with 19%.
Metaphor Target Type. The types of targets are displayed in the right part of Fig 3. The major- ity of the metaphors describe either a behavior stance towards certain topic, or the meme poster themselves (with person vehicle, such as Drake). Other categories are an approach or a concept (for which the meme poster expresses a certain stance), another person, and "desire vs. reality" meme such as the drowning meme illustrated in Fig 3.
We then looked at the human annotations we obtained in Sec 3.2 for the metaphors in each meme. We looked at the vehicle, i.e. the visual element
4 Experimental Setup
We report the performance of various baselines on MEMECAP. All models are tasked with generating
'Note that our dataset doesn't 't contain duplicate memes.

---

right"
This is meme with the ditle "Every damn time". The image description is "A scary looking monster".
This is mome with the title "Why they gotta be like this". The image descripton is "The wort intersection in world controlled by The following text is wrinen inside the meme:
Ne: corl, why
N
following test written
meme:
tralfic lights".
"what time did you to bed last night Pas Met early, why" "
why doesn't understand my signals?fn
What is the menne poster Lry inng to coavey? meme poster horrible after sleeping right the night before
The signals".. What the meme poster trying to convey?
Figure 4: An example of the few-shot setup with the following inputs: meme, image description, and the text inside the meme. The tigure shows the last in-context meme and the target meme.
meme caption, and are based pre-trained VL or language models (Sec 4. but may differ by their inputs and number of training examples (Sec 4.2).
from LAION (Schuhmann al., 2022), Conceptual Captions (Sharma et al., 2018), and SBU (Ordonez et al., 2011).
4.1 Models
LLaMA LLaMA (Touvron et al., 2023) is a transformer based language model that was trained on trillions of tokens from exclusively publicly- available data. The LLaMA-13B model outper- forms GPT-3 (Brown et al., 2020) most bench- marks. We use the LLaMA-7B model, which achieves comparable performance the LLaMA- 13B model on most benchmarks. Since LLaMA is a language model rather than a VL model, its access to the visual content is through the image caption and the OCR text alone.
We experiment with two ofate-of-the-art VL models that can generate text conditioned on both text and images, as well as one language model.
Open Flamingo. Flamingo was initialized with a pre-trained LLM and a pre-trained vision model, and further trained on vision and language tasks, keeping the pre-trained models frozen. The interac- tion between the two modalities is facilitated with a gated cross-attention dense block. Since the orig- inal model is not publicly available, we use the open version, OpenFlamingo-998 (Awadalla et al., 2023). OpenFlamingo is built top of LLaMA 7B (Touvron al., 2023) and CLIP ViT/L-14 (Rad- ford et al., 2021). and was trained on 5M samples from the Multimodal C4 dataset (Zhu et al., 2023b) and 10M samples from LAION-2B (Schuhmann et al., 2022).
4.2 Evaluation Setup
Inputs. We test the models with different input settings. In the setup which is the most comparable to humans, we provide the models with the meme and title. We also experiment with setups that aid the model. One such input is the image caption, which can help the model focus on the language modality and ignore the image. The second such input is the text inside the meme, that we extracted using EasyOCR, 8 which helps the model focus on the visual aspects of the image and includes the text inside the image as part of the language input. We incrementally added each of these inputs.
MiniGPT4. MiniGPT4 (Zhu et al., 2023a) is sim- ilarly composed of frozen pre-trained language and vision models, and it employs a single pro- jection layer to align the visual and language fea- tures. Since GPT4's architecture and training data remain a mystery, we utilize MiniGPT4 as an al- ternative to GPT4 (OpenAI, 2023).? ? It has similar capabilities to GPT-4 in understanding and gen- erating the context (Zhu et al., 2023a). For its language model, MiniGPT4 uses Vicuna (Chiang al., 2023), which is built on top of L-aMA- 13B and performs on par with ChatGPT (OpenAI, 2023). For its vision component, it uses BLIP-2 (Li et al., 2023), which consists CLIP ViT-G/14 and a Q-Former architecture. Mi was trained on various multimodal datasets, including images
Learning Setups. We evaluate all models in a zero-shot setup. Flamingo and LLaMA enable in- context learning, we experiment with 8, and 12 shots. An example prompt (ineluding the meme, title, image caption, and text inside the meme) is illustrated in Figure 4. MiniGPT4 works in a chat format, so rather than in-context learning, we use it in either a zero-shot setup, or fine-tuned on our training set.
Lastly, motivated by Chakrabarty et al. (2023) and Zhang al. (2023), also tested models in
'The version of GPT-4 available through the Al API doesn't it support images.
"https: //gs:://iitt thub. www////idedl//aaaaaaeaaoaa

---

Chain of Thought (CoT) style prompting (Wei et al., 2022). In our case, we elicit multi-step reasoning from the LLM by providing the visual metaphors, using the following prompt:
shots but slightly decreases at 12 shots (see Ap- pendix A). MiniGPT4 achieved better performance in the zero-shot setup, while fine-tuning its last layer significantly decrease the performance. As we show in Sec 5.2, while the fine-tuned model learns to generate short captions, it tends to hallu- cinate more. We hypothesize that fine-tuning only the last layer is ineffective.
<image>This is is meme wi wh the title "(title)".
The image description is "(image caption]". The following text is written inside the meme: "(OCR texty".
what is the meme poster trying convey?
Rationale: "[keyword13" is me taphphor for "[meaning1)". "[keyword2)" is metaphor for "[meaning2]".
Inputs. the few-shot setups, the best perfor- mance achieved with as many of the inputs as possible, i.c. including both the image caption and the OCR text, despite the redundancy with the vi- sual inputs. This might be due to suboptimal cross- modal interaction in VL models. While prior work showed that explicitly stating the metaphors helps image generation models generate better images (Chakrabarty et al., 2023), we did not see similar gain in meme captioning.
Answer:
5 Results
We evaluated the performance of the various mod- els with both automatic metrics (Sec 5.1) and hu- man evaluation (See 5.2). We show that the vi- sion and language modalities complementary through ablation tests (Sec 5.3).
5.2 Human Evaluation
We focused on the models with the full set of in- puts except for the rationales (meme+tile++ing cap+OCR text) and evaluated the performance of all models (focusing on 4-shots for the few-shot setups), with respect to the following criteria:
5.1 Automatic Evaluation
To evaluate the quality of the generated cap-
tions, we use standard metrics for automatic evaluation of generative tasks: BLEU (Pa- pineni et al., 2002) ROUGE (Lin, 2004), and BERTScore (Zhang et al., 2020) (using microsoft wwwoofotddbbabtteeeeeeee -aleertttaxlarrmmmmi) BLEU and ROUGE are based on n-gram overlap between the generated captions and human-written reference captions, while BERTScore measures the semantic similarities between the two.
Does the caption correctly convey the meaning the meme poster wanted to convey?
Correctness:
Appropriate Length: Is the caption length ap- propriate for conveying the meaning (i.e. it is not too verbose)?
Visual Completeness: Does the caption describe all the important elements in the image?
Table 2 shows the performance of the various models and input setups in terms of these metrics.
For the few-shot setup, we show the best perfor- mance across (4, 8, and 12 shots). See Appendix A for the full results.
Textual Completeness: Does the caption de- scribe all the important elements in the text inside the meme and the title text?
Models. Flamingo dominates MiniGPT4 across all metrics, with a gap of 15, 12, and points in BLEU, ROUGE, and BertScore respectively for the best setups. This is likely due to the lengthy cap- tions generated by MiniGPT4, despite the prompt including the instruction to generate a single sen- tence. Finally, the LLaMA model is highly com-- petitive with Flamingo despite not having access to the image itself. It appears that the image captions and OCR text provide sufficient information.
Faithfulness: Are all the elements of the caption supported by either the visual or text elements (i.c. there are no made-up elements)?
We randomly sampled 30 memes along with their model-generated and human-written captions. The annotation was performed by students in the lab, and we took the majority vote across 3 annota- tors. Figure 5 shows the performance according to the human evaluation. All models perform signifi- cantly worse than humans, except for appropriate length criteria, with 36.6, 29.3, 24.5, and 18.4 point differences on correctness, textual completeness, visual completeness, and faithfulness respectively.
Learning Setups. The Flamingo performance significantly improves from the zero-shot to few- shot setting, and continues to improve from 4 to

---

Model
Setup
Inputs
BLEU-4
ROUGE-L
BERT-FI
meme+title memeting cap meme+title+ing cap
19.36
31.51
65.69
16.10
29.08
64.71
zero-shot
19.61
30.92
65.51
meme+title+iin cap+OCR
19.31
32.51
66.84
zero-shot CoTT
meme+titleting cap+OCR texi+rationale
2.49
15.89
58.23
Flamingo
meme+title
25.89
39.41
70.83
meme+ing cap
26.96
39.53
70.91
few-shot
meme+title+ing cap
26.44
39.42
71.04
meme+tilleting cap+OCR text
26.73
43.47
73.86
few-shot Co'T
meme+titlc+ing cap+OCR text+rationale
27.02
43,46
74.32
meme
22.20
63.31
meme+title
14.37
30.70
66.19
zero-shot
meme+ing cap
10.36
26.22
64.39
meme+tiflc+ing cap
12.49
28.51
65.81
MiniGPT4
meme+tilleting cap+OCR text
12.46
31.44
68 62
zero-shot CoT
meme+tifle++nng cap+OCR text+rationale
12.57
31.70
68.45
fine-tuned
meme+tilleting cap+OCR text
7.50
27.88
65.47
fine-tuned CoT
meme-titlc+ing cap+OCR text+rationale
7.25
26.68
65.86
title+ing
19.72
31.42
66.38
zero-shot
tilleiing cap+OCR text
20.77
36.48
69.67
zero-shot CoT
title+ing cap+OCR text4rationale titleting cap
6.72
20.56
61.38
LLaMA
26.41
38.70
70.01
few-shot
title+ing cap+OCR text
26.63
43.41
74.71
few-shot CoT
titleiing cap+OCR texitrationale
42.95
74.00
Table 2: Performance in terms of automatic metrics of the various models and learning setups (with shots for the few-shot setup). We report the full experimental results, including shots and 12 shots, in Appendix A.
96 7300
25.7
80
15.5
35.8
Virigera zero
Miniar14 4 Tru uane
60
667
Listle dianst
26.7
50.7
30
Llala zersesst
Flamings chot
20
20.0
Appropriate Length
Correctness
Iuthriunness Model
Textual Completeness
Visaal Completeress
Figure 5: Performance in terms of human evaluation.
Models. Model performances differ by criteria. Flamingo and LLaMA are more correct and faith- ful, while MiniGPT4 is more visually complete.
length of its training examples. Unfortunately, these captions are far worse than those of the zero- shot model in all criteria. The zero-shot version generates verbose captions that include lot of in- formation, often conveying the correct meaning along with irrelevant information such literal de- scriptions of the image. Conversely, the fine-tuned version adapts to the "correct" length but it often fails to focus on the relevant parts, leading to correct or incomplete captions. We hypothesize that the frozen language and vision model may not have enough information about interpreting memes, and simply fine-tuning the last projection layer of the model is not enough to produce high-qualit, captions. This conclusion is consistent with Zhou et al. (2023), according to which most knowledge in LLM is learned during the pre-training stage.
Learning Setups. For Flamingo, the few-shot models improve in textual and visual completeness upon the zero-shot model, but not terms cor- rectness and faithfulness. This may suggest that while access to examples improves the model's 's un- derstanding of the task, it might also confuse it with information irrelevant to the target meme. LLaMA doesn't 't gain any performance improvements from in-context examples, likely for the same reason. Without the visual features, it might struggle more to separate the text (title, image caption, and OCR) of the different examples.
MiniGPT4 zero-shot is very verbose, but the fine-tuned model learns to output captions in the

---

Error: unfaithful
SCHOOL
Title:
This is my character
EXCEL
Image caption: This poster of Game of throne from tower scene.
Human-writtee meme caption: Meme poster abandoned Microsoft Excel in school, but need to use it after they get their white collar job.
Model-generated caption: Meme poster trying convey that they successful life.
WHITE
MICROSOFT EXCEL
Error: visually incomplete (copying the text inside meme)
Me reading ige 153 comn ent entlong reneennnt that happened yoars
Title: Based on a true story
Image caption: Spongelob is eagerly watching TVV
Human-written meme cuption: Meme poster entertuining to read through long comment threads arguments that happened the past.
Model-generann meme caption: Meme poster trying to convey that they read a 153 comment long argument that happened 7 years ago.
Figure 6: Examples of incorrect captions generated by the few-shot Flamingo model.
Common Errors. Figure shows two examples of meme captions generated by Flamingo 4-shot along with the types of errors they exhibit. The top example demonstrates an unfaithful caption because neither the meme nor the title conveys anything about being successful in life. The bot- tom example illustrates a common error in which the model copies text from inside the meme while ignoring important visual elements. In this case, Spongebob's s smile indicates the meme poster's pos- itive attitude towards reading old and long forum threads, but the model-generated caption misses it. Another common error (not illustrated here) occurs when the model treats visual elements too literally, failing to interpret the metaphor. Finally, in some cases, the model might lack sufficient background knowledge to correctly interpret the meme.
Model
k
Inputs
ABL.
ARG
ABT
19.36
31.51
65.69
-title
-2.29
-1.35
-0.6
Flamingo
-meme
-1.49
-1.93
-1.71
full
25.89
39.41
70.83
4
-title
+0.35
+0.12
-0.19
-meme
-0.14
-0.85
-1.86
14.37
30.70
66.19
MiniGPT4
title
-8.2
-8.5
-2.88
meme
+3.5
-1.12
-2.21
full
19.72
31.42
66.38
-tille
-0.88
-0.93
-0.62
iimg cap
-1.85
-1.84
-2.4
LLaMA
26.41
38.70
70.01
-tile
-0.69
-0.73
-0.67
-img cap
-0.66
-0.14
-1.04
Table 3: Comparison models with both language and visual inputs (tille+ima cap for LLaMA, title+meme for VL models), compared modality. BL BLEU, RG = ROUGE, BT BERT, number of shots.
5.3 Ablation Tests
The analysis in See 3.4 shows that interpreting most memes in MEMECAP will require understanding both the visual and ested in the extent that models make use of each modality. To that end, we perform an ablation test to exclude each modality. Table 3 presents the results in terms of automatic metrics.
dicating that the model relies more on the visual modality than the information provided by the title. The same is true for LLaMA (in both settings), for which excluding the image caption yields worse performance. This expected since the title is typ- ically secondary in informativeness to the meme. In addition, Flamingo still has access to the text inside the meme via visual features.
modalities. We are inter-
In most cases, the best performance is achieved with both modalities. For Flamingo (zero-shot and few-shot), excluding the meme results in more de- crease in performance than excluding the title, in-
Conversely, MiniGPT4 exhibits higher depen- dency on textual modality, resulting in a signifi- cant decrease when the title is not provided. Since

---

MiniGPT4 shows higher textual and visual com- pleteness when the OCR text is provided ($5.2), we hypothesize that MiniGPT4 makes limited usage the visual modality.
described in Section 3, we found people have dif- ferent criteria for what they perceive as offensive, hateful, or sexual, and thus, such content may still exist in our data.
6 Conclusion
Data Collection We use Amazon Mechanical Turk to collect 6.3K image descriptions and 7.7K meme captions. We paid $0.03 for the image cap- tioning task and $0.16 for the meme captioning task. The annotators were compensated with an average hourly wage $13, which comparable to the US minimum wage. We did not collect any personal information from annotators.
We present MEMECAP, the first meme captioning dataset. MEMECAP is challenging for the existing VL models, as it requires recognizing and interpret- ing visual metaphors, and ignoring the literal visual elements. The experimental results using state-of- the-art VL models indeed show that such models are still far from human performance. In particular, they tend to treat visual elements too literally and copy text from inside the meme. Our work opens up interesting future research on recognizing visual metaphors, interpreting them with respect to tex- tual context, and generating meme captions that are complete with respect to both modalities without creating fake elements.
Models Our dataset may include some offensive content mild expletives and this can amplify po- tentially biased and unethical answers. addition, the large pre-trained VL models we used for the experiments are trained on large-scale publicly available web corpus and may also bring some bias when generating sentences.
A Acknowledgements
Limitations
This work was funded, in part, by the Vector Insti- tute for AI, Canada CIFAR AT Chairs program, an NSERC discovery grant, and a research gift from A12.
Quality of Metaphor Annotations. We put our best efforts into manually verifying the collected data, and indeed the human performance Sec- tion 5.2 shows the human- -rittenwritten captions are of high quality. With that said, we noticed that the quality of the visual metaphors is inconsistent. We believe that while people are capable of explaining meme, they don't ' always know to map the visual vehicles into textual targets. This likely explains why adding the metaphors as inputs didn't improve the performance.
References
Ehsan Aghazadeh, Mohsen Fayyaz, and Yadollah Yaghoobzadeh. 2022. Metaphors pre-trained lan- guage models: Probing and generalization across datasets and languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lonn Papers), pages 2037- 2050, Dublin, Ireland. Association for Computational Linguistics.
Subjectivity and Background Knowledge. The meme captioning task involves employing back- ground knowledge which may vary between anno- tators. To that end, we manually checked the meme captions to minimize the number of incorrect cap- tions in the dataset. In addition, there is some level of subjectivity with respect to the evaluation crite- ria for the meme caption quality. For this reason, we ensured a high quality of annotations by hav- ing in-house annotators that could ask clarification questions, but some subjectivity still remains.
Arjun Akula, Brendan Driscoll, Pradyumna Narayama, Soravit Changpinyo, Zhiwei Jia, Suyash Damle, Garima Pruthi, Sugato Basu, Leonidas Guibas, William T. Freeman, Yuanzhen Li, and Varun Jampani. 2023. Metaclue: Towards comprehensive visual metaphors research.
Jean- Mant-Bapistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Mal- colm Reynolds, Roman Ring. Eliza Rutherford. Serkan Cabi, Tengda Han, Zhitao Gong, Sina Saman- gooci, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Binkowski, Ricardo Bar- reira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022. Flamingo: visual language model for few-shot learning. In Advances in Neurul Infor- mation Processing Systems.
Ethics Statement
Data All the datasets used in our work are pub- licly available. Our dataset collected from Reddit and may contain offensive, hateful, or sexual con- tent. Despite our best efforts to filter them out as

---

Malihe Alikhani, Piyush Sharma, Shengjie Li, Radu Soricut, and Matthew Stone. 2020. mrosl co- herence modeling for caption generation. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6525-6533 On- line. Association for Computational Linguisties.
Stoica, and Eric P. Xing, 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90% chatgpt quality.
Minjin Choi, Sunkyung Lee, Eunseong Choi, Heesoo Park, Junhyuk Lee, Dongwon Lee, and Jongwuk Lee. 2021. MelBERT: Metaphor detection via contextual- ized late interaction using metaphorical identification theories. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1763-1773, Online. Association for Computational Linguistics.
Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hes- sel. Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023. Open- flamingo.
Nitzan Sitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. 2023. Breaking common sense: Whoops! vision-and- aaiion-nd-language language benchmark of syn- thetic and compositional images.
Ellen Dodge, Jisup Hong, and Elise Stickles. 2015. MetaNet: Deep semantic automatic metaphor anal- ysis. In Proceedings of the Third Workshop on Metaphor in NLP, pages 40-49. Denver. Colorado. Association for Computational Linguisties.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bernrr, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Charles Forceville. 1996. Pictorial metaphor in adver- rising. Psychology Press.
Jack Hessel, Ana Marasovic, Jena D. Hwang. Lillian Lee. Jeff Da, Rowan Zellers. Robert Mankoff, and Yejin Choi. 2023. Do androids laugh electric sheep? humor "understanding" benchmarks from the new yorker caption contest. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Lonng Papers), pages 688-714, Toronto, Canada. Association for Compu- tational Linguistics.
Branislav Buchel. 2012. Internet memes as means of communication. Brno: Masaryk University.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Casey A. Fitzpatrick, Peter Bull, Greg Lipstein, Tony Nelli, Ron Zhu, Niklas Muennighoff, Riza Velioglu, Jewgeni Rose, Phillip Lippe, Nithin Holla, Shantanu Chandra, San- thosh Rajamanickam, Georgios Antoniou, Ekaterina Shutova, Helen Yannakoudakis, Vlad Sandulescu, Umut Ozertem, Patrick Pantel, Lucia Specia, and Devi Parikh. 2021. The hateful memes challenge: Competition report. In Proceedings of the NeurIPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pages 144-360. PMLRR
Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. 2022. It's not rocket science: Interpreting figurative language narratives. Transactions of the Associa- tion for Computational Linguistics, 10:589-606.
Tuhin Chakrabarty, Debanjan Ghosh, Adam Poliak. and Smaranda Muresan. 2021a. Figurative language in recognizing textual entailment. In Findings of the Association for Computational Linguistics: ACI- IJCNLP 2021, pages
Online. Association for Computational Linguistics.
Tuhin Chakrabarty, Arkady Saakyan, Olivia Winn, Artemis Panagopoulou, Yue Yang. Marianna Apid- ianaki, and Smaranda Muresan. 2023, I spy a metaphor: Large language models and diffusion mod- els co-create visual metaphors. In Findings of ACI.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-imag pre- training with frozen image encoders and large lan- guage models.
Tuhin Chakrabarty, Xurui Zhang, Smaranda Muresan, and Nanyun Peng. 2021b. MERMAID: Metaphor generation with symbolism and discriminative de- coding. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4250-4261, Online, Association for Computational Linguistics.
Chin-Yes Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Ouf, pages 74-81, 81, Barcelona, Spain. Association for Computational Linguistics.
OpenAl. 2023. Gpt-4 technical report.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 201 Im2text: Describing images using million captioned photographs. In Advances in Neural In- formatiion Processing Systems, volume 24. Curran Associates, Inc.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguisties.
Kevin Stowe, Nils Beck, and Iryna Gurevych. 2021. Exploring metaphoric paraphrase generation. In Pro- ceedings of the 25th Conference on Computational Natural Language Learning, pages 323-336, Online. Association for Computational Linguisties.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong. Harshith Goka, Kiwoong Park, and Victor Lempitsky. 2021. Resolution-robust large mask inpainting with fourier convolutions. arXiv preprint arXiv:2109.07166
Jingnong Qu, Liunian Harold Li, Jieyu Zhao, Sunipa Dev, and Kai-Wei Chang. 2022. Disinfomeme: A multimodal dataset for detecting meme intentionally spreading out disinformation.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Nla Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR.
Kohtaro Tanaka, Hiroaki Yamane, Yusuke Mori, Yusuke Mukuta, and Tatsuya Harada. 2022. Learning to evaluate humor in memes based on the incongruity theory. In Proceedings of the Second Workshop on When Creative AT Meets Conversational AI, pages 81-93, Gyeongju, Republic of Korea. Association for Computational Linguisties.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022. Laion- 5b: An open large-scale dataset for training next generation image-text models.
William Yang Wang and Miaomiao Wen. 2015, can has cheezburger? a nonparanormal approach to com- bining textual and visual information for predicting and generating popular meme descriptions. In Pro- ceedings of the 2015 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 355-365, Denver, Colorado. Association for Compu- tational Linguistics.
Kate Scott. 2021. Memes as multimodal metaphors: relevance theory analysis. Pragmatics & Cognition, 28(2):277-298.
Chhavi Sharma, William Paka. Scott, Deepesh Bhageria, Amitava Das. Soujanya Poria, Tanmoy Chakraborty, and Bjöm Gambäck. 2020. Task Report: Memotion Analysis 1.0 SemEEval 2020: The Visuo-Lingual Metaphor! In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-200 2020).. Barcelona, Spain. Association for Computational Lin- guistics.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Fdd Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24837. Curran Associates, Inc.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypemymeed image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), pages 2556-2565. Melhourne, Australia. Association for Computational Linguistics.
Bo Xu, Tingting Li, Junzhe Zheng, Mehdi Naseriparsa, Zhehuan Zhao, Hongfei Lin, and Feng Xia. 2022. Met-meme: A multimodal meme dataset rich in metaphors. In Proceedings of the 45th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pages 2887-2899,
Ron Yosef, Yonatan Bitton, and Dafna Shahaf, 2023. Irfl: Image recognition of figurative language.
Shivam Sharma, Atharva Kulkarni, Tharun Suresh, Hi- manshi Mathur, Preslav Nakov, Md. Shad Akhtar, and Tanmoy Chakraborty. 2023. Characterizing the entities in harmful memes: Who is the hero, the villain, the victim? In Proceedings of the 17th Con- ference of the European Chapter of the Association for Computational Linguistics, pages 2149-2163, Dubrovnik, Croatia, Association for Computational Linguistics.
Dongyu Zhang, Minghao Zhang, Heting Zhang, Liang Yang, and Hongfei Lin. 2021. MultiMET: A multi- modal dataset for metaphor understanding. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 1 1ith Inte- national Joint Conference on Natural Language Pro- cessing (Volume Long Papers), pages 3214-3225, Online. Association for Computational Linguistics.

---

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with bert.
Zhuosheng Zhang, Aston Zhang, Mu Li. Hai Zhao, George Karypis, and Alex Smola. 2023. Multi- modal chain-of-thought reasoning in language mod- els. arXiv preprint arXiv:2302.000923
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis. Luke Zettlemoyer, and Omer Levy. 2023, Lima: Less more for alignment.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023a. Minigpt-4: Enhancing Vision-language understanding with advanced large language models.
Wanrong 7hu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge. Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. 2023b. Multimodal e4: An open, billion-scale corpus of images interleaved with text.
A Additional Experimental Results
We show the full experimental results in Table

---

Model
# Shots
Input
BLEU-4
ROUGE-L
BERT-F1
meme
17.07
30.16
meme+title
19.36
31.51
65.69
0-shot
meme+ing cap
16.10
29.08
64.71
meme+till+tnng cap
19.61
30,92
65.51
meme+tile+ingg cap+OCR text
19.31
32.51
66.84
0-shot CoT
meme+till+ing cap+OCK text+rationale
2.49
15.89
58.23
meme meme+tilee
26.24
39.53
70.62
25.89
39.41
70.83
4-shot
memeting cap
26.96
39.53
70.91
meme+title+ing cap
26.44
39.42
71.04
meme+tidde+ing cap+OCR text
26.73
43.47
73.86
Flamingo
4-shot Col
meme+tille+ing textaatonee
27.02
43.46
74.32
27.38
39.96
70,92
meme+tille
26.99
40.00
71.26
8-shot
memeting cap
28.11
40.32
71.24
meme+title+ing cap
27.30
40.00
71.32
memettill+ting cap+OCR text
28.70
43.54
74.33
8-shot Co'T
meme+title+ing cap+OCR texi+rationale
meme
26.74
38.89
70.20
meme+tile meme+img cap
27.32
40.13
70.86
12-shot
26.63
39.24
70.49
meme+title+ing cap
27.09
39.60
70.48
meme+tiilo+ing cap+OCR text
12-shot CoT
meme+title+ing cap+OCR texi+rationale
title img cap
17.87
29.58
63. 98
18.84
30.49
65.76
0-shot
title+ing cap
19.72
31.42
66.38
titleting cap+OCR text
20.77
36.48
69.67
0-shot CoT
titl+iing cap+OCR text+rationale
6.72
20.56
61.38
title
25.75
38.56
68.97
img cap
25.72
37.97
69.34
4-shot
titletimg cap titleting cap+OCR text
26.41
38.70
70.01
26.63
43.41
74.71
LLaMA
4-shot CoT
titl+iigg cap+OCR text=rationale
26.40
42.95
title
27.18
39.19
69.66
8-shot
img cap
27.25
38.61
title+ing cap
27.99
39.69
70.76
title+ing cap+OCR text
28.80
44.10
74.71
8-shot CoT
titl++ing cap+OCR text+rationale
26.32
42.06
73.95
title
25.71
37.15
68. 26
img cap
25.65
36.37
68.
12-shot
title+ing cap
26.63
38.57
69.96
titleting cap+OCR text
28.76
43.18
73.96
12-shot CoTT
title+ing cap+OCR text+rationale
meme
06.17
22.20
63.31
meme+tille meme+ing cap
14.37
30.70
66.19 19
0-shot
10.36
26.22
64.39
meme+ttile+ing cap
12.49
28.51
65.81
MiniGPT4
meme+title+ing CCR+ text
12.46
31.44
68.62
0-shot CoT
meme+till+tigg cap+OCR texi+rationale
12.57
31.70
68.45
finetuned
meme+tillc+ing cap+OCR text
7.50
27.88
65.47
FT CoT
meme-till+iing cap+OCR texi+rationale
7.25
26.68
Table 4: 0, 0,4, 8, 12 shot results with Flamingo, LLaMA, and MiniGPT4 models. indicaaes the model ran out of memory.


Reviews:
[{"review_id": "Qwc59hPK6e", "reviewer": "Reviewer_Bzu6", "summary": "", "questions": "1. The performance of \"MiniGPT4 fine-tuned\" is worse than \"MiniGPT4 zero-shot\". In L437~439, the authors think that it is because \"the frozen language and vision model may not have enough information about memes\". This claim is not exactly rational. This claim is more proper if the observation is that \"MiniGPT4 zero-shot\" is bad and \"MiniGPT4 fine-tuned\" is slightly better than \"MiniGPT4 zero-shot\". \n\n2. The most important contribution of this work is the dataset with both meme caption and visual metaphor, while there are already existing datasets for visual metaphor. I am still not so clear that whether the task of meme captioning is strongly required, it seems that there is a large overlap on the task of meme captioning and visual metaphor. \n\n3 (Minor, not important). The results/values with \"+\" in Table 3 (i.e., removing part of input can improve the performance) are not discussed. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback. We would love to address additional questions during the discussion period if anything is unclear.\n\n\n**Similar to the dataset proposed in \"Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest\".**\n\nThank you for the missing reference. We missed the arXiv version of  “Do Androids Laugh at Electric Sheep? Humor ‘Understanding’ Benchmarks from The New Yorker Caption Contest” and the ACL presentation was after the EMNLP submission deadline. We will make sure to cite it in the camera-ready version and elaborate on the similarities and differences between the papers, as we detail here.\n\nFirst, while both memes and cartoons use a combination of visual and text elements to convey humor, there are major differences between the two. Memes are based on recognizable images that are modified and repurposed to create new meanings based on shared cultural knowledge. Cartoons, on the other hand, are originally drawn illustrations, often in the form of comic strips, that convey a more complex narrative. Second, a more minor difference is that the cartoon dataset focuses on a discriminative matching setup, i.e. selecting the more appropriate caption (as well as generating an explanation), while our meme captioning task is generative, i.e. the task is to generate a caption. It would be interesting for future work that develops a model for one of the tasks, to test how transferable it is to the other; however, we believe that the existence of a cartoon captioning dataset doesn’t affect the significance and usefulness of a meme captioning dataset. \n\nGiven that this issue is easy to fix (i.e., we will cite the paper in the camera-ready version), we hope the reviewer will reconsider and not see this as a reason to reject the paper.     \n\n**Small samples for dataset analysis**\n\nThanks for raising this concern.\n\nAnalysis (section 3.4): please note that we manually verified all memes and captions during the data collection phase to guarantee the quality of our dataset. The purpose of the analysis is to provide a more in-depth look into the type of metaphors captured in the data. We have no concerns about the quality of the data. We will clarify this in the paper.   \n\nHuman evaluation (section 5.2): We chose to evaluate 30 memes based on the convention that a minimum of 30 observations is sufficient to conduct significant statistics. We note that 30 memes x 7 models x 5 evaluation criteria x 3 annotators yield 2,100 annotations in total, which is not trivial in terms of time and cost. Yet, to clear any concerns about the statistical meaningfulness of the evaluation, we will increase the number of evaluated memes for the camera-ready version. We don’t expect the results to change significantly.  \n\n**Clarification for \"The performance of 'MiniGPT4 fine-tuned' is worse than 'MiniGPT4 zero-shot'\" (In L437~439, \"the frozen language and vision model may not have enough information about memes\")**\n\nThanks for pointing this out, we were indeed not clear about this. What we meant is that it seems that the zero-shot version generates verbose captions that include a lot of information, often conveying the correct meaning along with irrelevant information such as literal descriptions of the image. The fine-tuned version adapts to the \"correct\" length but it often fails to focus on the relevant parts, leading to incorrect or incomplete captions. We will clarify this in the camera-ready version.\n\n**Different model size with MiniGPT4 based on LLaMa-13B (L288) and another with LLaMA-7B (L302).**\n\nThanks for pointing this out. While MiniGPT4 uses the LLaMA-13B model, the OpenFlamingo-9B model is based on LLaMA-7B model. Our initial experiments focused on the OpenFlamingo model considering that MiniGPT4 was released in late April 2023, one month before arxiv deadline, so we chose the LLaMA version accordingly. We will try to include the results from the LLaMA-13B model in the Appendix in the final version of the paper.\n\n**The results/values with \"+\" in Table 3 (i.e., removing part of input can improve the performance) are not discussed.**\n\nWe will include the discussion for the results with “+” in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. An interesting dataset with meme, title, literal image caption, visual metaphors, and meme caption. \n\n2. Both automatic evaluation and human evaluations on the task of meme captioning are provided.  ", "reasons_to_reject": "1. As a dataset collected by crowdsourcing, there is almost no quality analysis on the collected data (only in the part of human evaluation, using 30 memes, in Section 5.2, the results of \"human\"). \n\n2. There are about 6.3K memes in the datasets, but the authors only utilize 28 memes for the dataset analysis in Section 3.4, and 30 memes for the baseline evaluations in Section 5.2. The numbers of memes used in these evaluations are too small. The evaluations are not convincing enough. \n\n3. The dataset is somewhat similar to (and somewhat different from) the dataset proposed in \"Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest\", which gives an explanation (can be regarded as a caption) to a cartoon to understand the humor (potential meaning, similar to metaphor) of the cartoon (similar to meme). Although there are some differences, e.g., one is metaphor and the other is humor, it makes this work less exciting to me. \n\nOther minor comments:\n\n4. In the experiments, this paper uses MiniGPT4 based on LLaMa-13B (L288), but also uses LLaMA-7B (L302). Because the authors want to compare the settings of \"without accessing to image, LLaMa\" and \"with accessing to image, MiniGPT4\", it's better to utilize the same model size. ", "questions_to_authors": "1. The performance of \"MiniGPT4 fine-tuned\" is worse than \"MiniGPT4 zero-shot\". In L437~439, the authors think that it is because \"the frozen language and vision model may not have enough information about memes\". This claim is not exactly rational. This claim is more proper if the observation is that \"MiniGPT4 zero-shot\" is bad and \"MiniGPT4 fine-tuned\" is slightly better than \"MiniGPT4 zero-shot\". \n\n2. The most important contribution of this work is the dataset with both meme caption and visual metaphor, while there are already existing datasets for visual metaphor. I am still not so clear that whether the task of meme captioning is strongly required, it seems that there is a large overlap on the task of meme captioning and visual metaphor. \n\n3 (Minor, not important). The results/values with \"+\" in Table 3 (i.e., removing part of input can improve the performance) are not discussed. ", "paper_topic_and_main_contributions": "This work proposes a task of meme captioning and release a new dataset, MEMECAP. The dataset contains memes along with the title, the meme captions, the literal image captions, and the visual metaphors. It utilizes recent Visual-and-Language models to this dataset to analyze their performances on the task of meme captioning. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest, ACL 2023.  "}, "extra_text": {}}, {"review_id": "9or1vL6qm2", "reviewer": "Reviewer_sFDe", "summary": "", "questions": "A: Why not provide full-training setting on all models?\nB: It is interesting to see that Llama, a pure language model, can achieve similar or even better performance than VL models, shown in Table 2. Is there any further explorations on this part?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback.\n\n**Why not provide full-training setting on all models?**\n\nThe purpose of the evaluation is to provide baseline results for our new dataset, which is the main contribution of this paper. It was not meant to be comprehensive, but rather to show that the task is interesting and non-trivial. We choose several representative VL models that were publicly available and used them in the standard recommended way. Specifically, Flamingo is supposed to be used in a few-shot setup, as the paper title indicates: https://openreview.net/forum?id=EbMuimAbPbs. We leave it for future work to develop more advanced and/or computationally expensive models for this task.       \n\n\n**It is interesting to see that Llama, a pure language model, can achieve similar or even better performance than VL models, shown in Table 2. Is there any further explorations on this part?**\n\nNote that in Table 2, LLaMA always has access to the image caption, which is a (possibly incomplete) textual representation of the image. We hypothesize that in VL models, the vision component lags behind the language component. This claim is supported by prior work [1]. This would make the textual features more useful, which is also evident by the fact that VL models perform better when they are given the OCR text and image caption, which are technically redundant with the image itself. This means that a LM can still perform relatively well without access to visual features. We will clarify this in the camera-ready version.   \n\n[1] https://arxiv.org/abs/2102.10407", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "An novel meme caption dataset to help improve the humorous understanding ability of downstream large models.", "reasons_to_reject": "There's only the MiniGPT4 model be tested under fine-tuning setting. The other two models, Flamingo and Llama, are not tested under fine-tuning setting.", "questions_to_authors": "A: Why not provide full-training setting on all models?\nB: It is interesting to see that Llama, a pure language model, can achieve similar or even better performance than VL models, shown in Table 2. Is there any further explorations on this part?", "paper_topic_and_main_contributions": "The paper proposed a new vision-language dataset of meme pictures and corresponding captions. The scale of the dataset is 6.3K. For each sample, a meme picture, literal image captions, visual metaphors and meme captions are provided. In addition, the authors tested a series of SOTA vision-language and language models on the dataset, showing the lack of meme understanding ability of SOTA models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FvDY2GU1EB", "reviewer": "Reviewer_XU7o", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. We would love to address additional questions during the discussion period if anything is unclear.\n\n\n**Similar to the dataset proposed in \"Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest\". No citation of the aforementioned ACL paper**\n\nThank you for the missing reference. We missed the arXiv version of “Do Androids Laugh at Electric Sheep? Humor ‘Understanding’ Benchmarks from The New Yorker Caption Contest” and the ACL presentation was after the EMNLP submission deadline. We will make sure to cite it in the camera-ready version and elaborate on the similarities and differences between the papers, as we detail here.\n\nFirst, while both memes and cartoons use a combination of visual and text elements to convey humor, there are major differences between the two. Memes are based on recognizable images that are modified and repurposed to create new meanings based on shared cultural knowledge. Cartoons, on the other hand, are originally drawn illustrations, often in the form of comic strips, that convey a more complex narrative. Second, a more minor difference is that the cartoon dataset focuses on a discriminative matching setup, i.e. selecting the more appropriate caption (as well as generating an explanation), while our meme captioning task is generative, i.e. the task is to generate a caption. It would be interesting for future work that develops a model for one of the tasks, to test how transferable it is to the other; however, we believe that the existence of a cartoon captioning dataset doesn’t affect the significance and usefulness of a meme captioning dataset. \n\nGiven that this issue is easy to fix (i.e., we will cite the paper in the camera-ready version), we hope the reviewer will reconsider and not see this as a reason to reject the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces an innovative dataset and conducts exhaustive experiments to meticulously evaluate the performance of cutting-edge models, rendering it immensely valuable for future research endeavors.", "reasons_to_reject": "After reading this paper, it reminded me of one of ACL23's best paper: 'Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest.' The MemeCap dataset shares a similar concept with this influential ACL paper. However, a notable concern is that this paper fails to cite the aforementioned ACL paper, which is considered unacceptable in scholarly practices. Furthermore, the passage does not delve into the distinctions between these two works, leaving an important aspect unaddressed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the task of \"meme captioning\" which involves understanding memes and their visual metaphors while interpreting the text associated with the meme. The authors release a new dataset called MemeCap. The authors conduct extensive experiments using state-of-the-art Vision and Language (VL) models to evaluate their performance on the meme captioning task.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "*Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest* ACL23 best paper"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers found the dataset to be interesting, innovative, and valuable for future work. Reviewers generally found the experimentation to be thorough, although one reviewer pointed out that a few models were missing fine-tuning results. Two reviewers pointed out a missing citation to an ACL 2023 paper introducing a related (but sufficiently distinct) dataset, which the authors promised to add in the camera ready, addressing these concerns. One reviewer still had remaining concerns about possible data quality, given that the dataset was collected by crowdsourcing and a small number of data points were used in evaluation. However, in my opinion the manual checking of data quality by the authors and the human evaluations are likely sufficient to ensure that the dataset will be useful for future work.
Decision: Accept-Main