Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions
Authors: Zhe Hu
Tuo Liang
Jing Li
Yiren Lu
Yunlai Zhou
Yiran Qiao
Jing Ma
Yu Yin
Venue: NeurIPS 2024 oral (2024)
Keywords: comic narrative understanding
visual reasoning
multimodal benchmark
humor understanding
Primary area: natural_language_processing

Abstract: Recent advancements in large vision language models have demonstrated remarkable proficiency across a wide range of tasks. 
Yet, these models still struggle with understanding the nuances of human humor through juxtaposition, particularly when it involves nonlinear narratives that underpin many jokes and humor cues.  This paper investigates this challenge by focusing on comics with contradictory narratives, where each comic consists of two panels that create a humorous contradiction. We introduce the YesBut benchmark, which comprises tasks of varying difficulty aimed at assessing AI's capabilities in recognizing and interpreting these comics, ranging from literal content comprehension to deep narrative reasoning. Through extensive experimentation and analysis of recent commercial or open-sourced large vision language models, we assess their capability to comprehend the complex interplay of the narrative humor inherent in these comics. Our results show that even the state-of-the-art models still struggle with this task. Our findings offer insights into the current limitations and potential improvements for AI in understanding human creative expressions.

Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions
Zhe Hu'; Tuo Liang?;Jing Li', Yiren Lu", Yunlai Zhou', Yiran Qiao?, ,Jing Ma², Yu Yin?
DDepartment of Computing, The Hong Kong Polytechnic University *Department of Computer and Data Sciences, Case Western Reserve University
'zheederek. hwwconnect. h jing-amelia, 1i@polyu. edu
[[tx1859,xx135333 yx13538, y,205,,,,,22,,,,,,,, jmm yun.yu....................... @case. edu https://vulab-re gighlabbbii.ggtuu..... githgii.thuu.... io YESBUTT Homepage/
Abstract
Recent advancements in large multimodal language models have demonstrated re- markable proficiency across wide range of tasks. Yet, these models still struggle with understanding the nuances of human humor through juxtaposition, particu- larly when it involves nonlinear narratives that underpin many jokes and humor cues. This paper investigates this challenge by focusing comics with contradic- tory narratives, where each comic consists of two panels that create a humorous contradiction. We introduce the YESBUT benchmark, which comprises tasks of varying difficulty aimed at assessing Al's capabilities in recognizing and interpret- ing these comics, ranging from literal content comprehension to deep narrative reasoning. Through extensive experimentation and analysis of recent commercial open-sourced large (vision) language models, assess their capability to com- prehend the complex interplay of the narrative humor inherent in these comics. Our results show that even state-of-the-art models still lag behind human perfor- mance on this task. Our findings offer insights into the current limitations and potential improvements for AI in understanding human creative expressions.
1 Introduction
"The world is indeed comic, but the joke is on mankind."
Lovecraft
Comics a unique blend of visual art and narrative that encapsulate wide range of human periences and emotions. Understanding comics often requires significant social reasoning skills and cultural knowledge, as they heavily rely on context, cultural references, and visual metaphors. Furthermore, comics frequently employ nonlinear narratives [1, 2], demanding rigorous reasoning to grasp underlying ideas. Recent large (vision) language models have achieved impressive perfor- mance on various tasks [3-5], yet their ability to comprehend these complex human expressions remains insufficiently explored [6-8].
Examining AI models' ability to understand comics is essential for advancing their social and se- mantic comprehension. As significant part of human creative expression, comic offers valuable insights into human emotions and cultural contexts [9]. This understanding is crucial for developing socially intelligent systems and enhancing Al-related creativity, thereby improving user experience in applications such as recommendation systems and automated content creation tools.
Previous studies have applied vision language models (VLMs) to understand humor and deep se- mantics [7, 10]. However, these studies often focus on Lingle-pane comics and do not investigate
These authors contribute equally.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the more complex case of nonlinear narratives created through juxtaposition, a fundamental element in comics. Juxtaposition involves placing two contrasting elements together to provoke thought or evoke humor [11, 12]. This technique requires readers to pause and reassess the meaning, engaging in nonlinear thinking to reason about the relationships between panels for overall idea. [13-15].
In this work, we examine VLMs' ability to understand comics, specifically focusing on humor derived from juxtaposition. Our goal
Comic with two panels:
Yes,
But
Nasrratton Unseestanding
to determine if large models can accurately comprehend the complex and contradictory narratives present in comics. Such contradic- tions challenge conventional semantic inter- pretations and demand deeper analysis. For example, Figure 1 shows a comic with two panels: in the first, a driver stops for ducks to cross the road ("Yes"), and in the sec- ond, the driver enters a "Peking Duck" restau- rant (" ("But"), highlighting the contradiction in human-anima relationships through juxtapo- sition.
Pekiing
Remoriing
Narrative contradiction the drwer's depby compassion live ducks by stopping and watting for them to cress the road, yet this same respect does not extend to their trestment as food.
Underlying ing Philooophy Sclection
of highlights contradictory relationshio humans where value an animal's changes drastically based on context.
animals,
x comic sungests there no needs show ampothy the animals sinoe they will be eventually consumed as foods.
Matching
Understanding such juxtaposition in comics significant challenges for the models. First, requires deep comprehension of human norms, recognizing that people often have conflicting feelings, and identifying subtle so- cial cues and contexts tied to cultural back- grounds. Additionally,
Crosing Paths: From Compassion to Cuisine X Food Empothy vs. Animal Protection
Figure 1: We introduce YESBUT dataset for comic understanding of juxtaposed comic panels. Given two-panel comic with a contradictory narrative, we propose several tasks including narrative under- standing, underlying philosophy selection and title matching, tackling different levels of comic under- standing. (Comic by Anton Gudim).
demands nonlin- car reasoning to grasp the overall narra- tive, as the story is conveyed through the in- terplay of two panel elements, forming the core of the narrative beyond the literal mean- ing of each single panel. This type of juxtaposition necessitates critical thinking about similarities and differences, requiring in-depth reasoning. However, current models lack the ability to process information through nonlinear and deep thinking effectively, as the autoregressive paradigm of LMs limits their bidirectional reasoning capabilities [16-18]. By emphasizing these contradictions, we aim push Al models to develop more sophisticated semantic understanding, enriching their inter- pretative capabilities.
To this end, we collected and annotated new benchmark, YESBUT, for understanding comics with juxtaposition, focusing on contradictory narratives. Each comic is annotated with a literal descrip- tion. contradiction illustration, the underlying philosophy it reveals or satirizes, and title that summarizes the overall narrative, as shown in Figure 1. We then propose four tasks: (1) literal description writing, to produce surface description of the comic narrative; (2) contradiction gen- eration, where the model illustrates the narrative contradiction; (3) underlying philosophy selection, which targets at selecting the correct philosophy the comic reflects; and (4) title matching, where the model matches the comic with a proper title. These tasks jointly cover different levels of comic understanding, from literal content comprehension to more in-depth narrative reasoning, providing a thorough evaluation of comic understanding capabilities.
We conducted comprehensive experiments the YESBUT dataset, evaluating both commercial and open-sourced large (visual) language models. Both automatic and human evaluation results indicate that commercial VLMs outperform their open-sourced counterparts on most tasks. However, even the highest scores are far from perfect (e.g.. 84.1 % accuracy for underlying philosophy selection and 63.3% for title matching). underscoring the need for further advancements in this area. Addition- ally. our analysis reveals that augmenting models with oracle comic descriptions can significantly enhance performance, highlighting the considerable gap in current models' understanding of comic narratives. We release our annotations, code, and model results, aiming to provide valuable insights for future AI research on understanding human creative expression.
2

---

2 Related Work
Large Models and Evaluations. Recent large (vision) language models have demonstrated re- markable performance in following human instructions and performing various downstream tasks through zero-shot prompting [19-21, 29-21,4], 4]. Various benchmarks have been proposed to evaluate their performance, encompassing both language-only tasks [22-25] and vision-language tasks [26-30]. These tasks primarily focus on assessing the fundamental capabilities of large models. However, the ability of large models to perform in-depth social reasoning and accurately understand human contexts remains underexplored [6].
Computational Humor. Humor is vital component of human communication [31]. Our research is closely related to the computational understanding of humor. Previous studies have addressed humor recognition [32-34] and generation [35], with recent work expanding to multimodal data, such as visual humor prediction [36], humorous cartoon caption identification [37, 7, 38], and humor prediction in videos [39, 40J. Despite advancements, recent work shows that LLMs such ChatGPT has not fully solved computational humor yet [41]. In this work, we design tasks to evaluate large vision- Assionlanguage models their ability to understand humor through comic juxtapositions with contradictory narratives, requiring deep narrative comprehension. Through this study, we aim to provide insights into the capabilities of Al in processing and appreciating humor.
Interpretation of Human Creative Expressions. Visual artwork, encompassing mediums such as drawings, paintings, and sculptures, has been profound aspect of human culture and cognition. These creative expressions are not merely decorative; they are deeply entwined with the ways hu- mans perceive, interpret, and communicate their experiences and emotions [42]. Understanding these human creative expressions necessities valuable insights of human emotions, societal values, and cultural contexts, which is crucial for developing socially intelligent systems and enhancing Al-related creativity [43]. Previous research has explored AI interpretation of visual human cre- ative expressions in tasks such meme [44] and cartoon [45] understanding. Similar to work, studies like [7] and [10] apply Al models to comprehend comics. However, these studies primarily focus on single-pane comics, emphasizing humor and deep semantics. In contrast, work aims to investigate the significant feature of juxtaposition for understanding contradictory narratives.
Visual Reasoning. Our task is also related the visual reasoning ability, where the model requires in-depth reasoning to comprehend contradictions between two comic panels. Previous research has examined visual reasoning capabilities of large models in tasks involving commonsense reason- ing [46, 28, 47], visual question answering [48], disio-linguistic compositionality [49], and science question answering [50]. Unlike these studies, task involves nonlinear reasoning, which ne- cessitates AI to navigate multi-dimensional and complex information layers, often without explicit directives. While linear reasoning present their challenges, they usually exhibit clearer rules and structures, making them more accessible for AI to process with existing algorithms and models. Consequently, nonlinear reasoning represents more intricate task, demanding higher natural lan- guage processing and cognitive modeling capabilities from Al systems.
3
The YESBUT Dataset
Our benchmark consists of YESBUT comics featuring contradictory narratives. Specifically, each sample includes: (1) a two-panel comic that forms a narrative with inherent contradictions; (2) a literal description of the comic narratives; (3) an explanation that illustrates the contradiction within the narrative; (4) the deep philosophy or underlying message the comic aims to convey; and (5) a title of the comic. Based these components, construct various tasks for comic understanding.
3.1 Image Collection
Our dataset consists of captionless comics, primarily from Anton Gudim's "YES, BUT" series [51], each featuring two panels depicting contradictory everyday scenarios. We scraped the images from social media and conducted preprocessing, including deduplication, filtering out comics with more than two panels, and removing any inappropriate or offensive content. This process resulted in final dataset of 348 comics.
hnttps:/ttniter: and https: http://wwwwtntettcoom
3

---

Comic with two panels:
Step Narrative Description Writing
Yes,
But
Literal Description
② Contradiction
The comic shows a businessman with two different groph perspectives The left side deplyys close-up of chart with an upward spike, suggesting immediate success. While the right graph
The contradiction arises from selective disclasure, jantapesing
the immediate trumph against the broader contaxt defeat.
overall downward trend,
Step 2: Deep Contents Writing
@ Underlying Philosophies
cornic comments deceptive nature of presenting
(Neg1] portrays the lesson that pensonal confidence
remain unaltered by fluztuations (--)
Information without reminding us consider picture and not be swayed
(Neg2) The comic endorses the practice of cherry-
picking information
reduction
selective incomplete information.
(Neg3) poses criticism that ago can one econo mic realities (...
Cross Verification
Length control
Style consistency
+ Title
Readability
(Neg1) Graphs Don't People
(Pos) Selective Statistics; Tale Two Perspectives
(Neg2) Profit and Loss: The Dual Faces of Business
Step 3: Quality Check
(Neg3) Graphs
Businessman's Journey
Figure 2:
Overview of the data construction pipeline. Pos represents the positive options, and Neg stands for the negative options.
3.2 Data Annotation
For each comic, we annotate the corresponding literal description, contradiction explanation, under- lying philosophy and comic title [7, 10]. We primarily rely on human annotators to obtain gold- standard annotations. Hight human judges participated in the annotation process, all of whom are proficient English speakers based in English-speakin
countries and have at least a Bachelor's de- gree. Our annotation process included two stages: the progressive human-Al collaborative annota- tion stage and the quality check and cross-verification stage. The pipeline is illustrated in Figure 2.
Progressive Human-AI Collaborative Annotation. In this stage, we randomly assigned comic samples to each annotator, instructing them to first exclude any comics that may contain offensive, hateful, or sexual material before beginning the annotation process. To reduce human effort and costs associated with data annotation from scratch, we designed human-Al collaboration pipeline utilizing GPT-4 [52] for data annotation and component writing.
The pipeline operates through dialogue interactions with the GPT-4 model. Given comic image, first prompt GPT-4 to generate narrative descriptions, illustrating the comic's 's narrative and explain- ing the contradictory logic between the two panels. Human annotators then modify and annotate the contents to obtain a literal description and contradiction explanation.
After obtaining the gold-standard description, both the comic and the description used input to prompt GPT-4 deep content writing, including the underlying philosophy and an eye-catching comic title. The underlying philosophy aims to foster deep understanding of the comic and reveal the phenomenon it satirizes or the lesson it conveys; and the title is a more abstractive expression that reflects the overall narrative. Both components will be further checked by human annotators. Additionally, for the underlying philosophy and title understanding tasks, GPT-4 generates hard negative counterparts and distractions to design multiple-choice questions for our experiments. The prompts we used are provided in the Appendix A.
Our Auman-Al collaborative annotation pipeline is effective it leverages a progressive prompting strategy, annotating each component from easy to difficult. Understanding the underlying philoso- phy of the comic. for example, requires first understanding the literal narratives and contradictions. This approach reduces annotation costs and improves overall efficiency.
Quality Check with Cross Verification. To ensure the quality and accuracy of the components and reduce objective bias from different human annotators, we introduced cross verification stage. In this process, one annotator is assigned tor checks the annotated results to ensure all components are correct, unbiased, and appropriate. If any content is found to be of low quality judge to determine the final version. We exclude the comics with ambiguous or controversial nar- ratives. This process ensures the quality of the annotated components for benchmark construction.
inspector for each comic. The inspec-
ambiguous, third annotator is brought in as

---

After the cross-verification process, one of the authors re- views each sample to verify the validity and appropriate- ness of the components. Finally, we obtain dataset of 348 comics, each accompanied by high-quality components. The statistics of the components are shown in Table 1.
Components
#Num
Avg. Len.
Image
348
Literal Description
348
Contradiction Philosophy Title
348
31
1,392
24
Mitigating Annotation Bias. Our benchmark focuses on common interpretation of humor. However, the subjectivity of this task may introduce bias. To mitigate this issue, we have taken several steps in our annotation process: (1) Our annotators come from different genders and diverse cultural backgrounds, providing a range of perspectives: (2) Multi-
1.392
Table I: Data Statistics. Avg. Len. is the average number of words.
ple quality checks and verifications are incorporated to ensure consensus among different annotators, with controversial or potentially biased comics being filtered out; (3) Annotations are further vali- dated by cross-referencing social media comments for each comic to ensure alignment with widely accepted interpretations; (4) Recognizing that tasks such as generating titles and philosophical con- tents are inherently open-ended and involve subjective data annotation, we frame them as selection tasks, and ensure that the correct option is clearly and objectively superior than the negative options to mitigate subjectivity.
3.3 Task Design: Do Large Models Understand Humor in Juxtaposition?
We aim to evaluate the capabilities of recent large (visual) language models in understanding humor through contradictions. This is challenging because it requires both social reasoning about human events and nonlinear logical reasoning about the narratives, going beyond the literal understanding of the comic. We design series of tasks that require different levels of narrative understanding and reasoning abilities to evaluate the models' performance in reading comics.
1. Literal Description Writing. The first task is to generate the literal description of the comic narrative. We formulated this task as a text generation task: given an input comic, the model is required generate short description illustrating the narrative from the two panels of the comic This task is different from the traditional image captioning, which requires the model to illustrate the comic narrative instead of solely focusing image description.
2. Contradiction Generation evaluates whether the model can understand the contradiction within the narrative juxtaposition. Similarly, it is formulated as a text generation task.
Underlying Philosophy Selection. Understanding comics requires grasping not only the surface meaning of the images but also the underlying ideas the authors aim to convey. This task evaluates the model's s ability to recognize the comic's 's underlying philosophy. It is formulated as multiple- choice question answering (MCQ) task: given an input comic and four candidates of its underlying philosophy, the model must predict the correct option. The negative choices are crafted by annotators to be relevant to the comic, requiring reasoning to make the correct prediction.
Title Matching evaluates whether the models can identify the corresponding title, which chal- lenging because the title acts as an abstraction of the narrative and requires proper understanding of the comic's content. Similar to the underlying philosophy task, it is formulated as multiple-choice question answering task, where the most is asked to select the correct title from four options.
Experiments
4.1 Models and Settings
We evaluate the models' performance in a zero-shot manner using both recent VLMs and LLMs. For VLMs, the comic image and questions are provided inputs for output prediction. We include both commercial models such as GPT-4 [52] and Claude-3[53) as well as open open-sourced models including LLaVa [3, 54], CogVLM [55], Qwen- VL [56], mPLUG-Ow12 [57], and InstructBLIP [58].
For LLMs, since they cannot directly process images, we use the VL. 13B model generated literal descriptions as inputs due to its strong performance. We include ChatGPT, the Llama3 in- struction model [20], and the Mistral 7B instruction model [59]. More details of the models are included in Appendix B.
5

---

Literal Description
Contradiction
Philosophy
Title
Setting
Model
BERT
R-2
GPT
BERT
R-2
GPT
Accuracy
Accuracy
GPT-4
88.32
87.46
3.76
87.64
83.21
4.03
82.76
60.25
Claude-3
87.68
80.30
3.28
86.93
80.63
3.79
84.10
56.42
LLaVA-I.6-34B
86.45
67.67
2.86
86.04
75.95
3.51
78.83
63.31
LLaVA-1.6-1 1-BB13
81.34
75.95
2.96
86.48
80.96
3.36
69.16
55.08
LLaVA- 1-5VA-1.5--3B 13B-13B
78.77
58.21
2.51
86.48
67.67
3.36
69.73
48.75
VLMs
InstructBlip-13B
85.20
35.28
2.69
85.54
51.15
2.54
30.75
22.70
CogVLM
80.80
55.51
2.65
87.07
69.96
3.76
61.30
49.52
Qwen-VIe VL.-Chat
79.03
51.58
2.76
86.41
59.77
3.25
59.10
42.05
mPlug-Owl2
78.26
47.38
2.57
86.20
48.05
2.59
62.17
43.10
LLaVA-1.6-7B
80.71
70.36
2.79
86.58
75.36
3.24
47.41
37.07
InstructBlip-7B
76.02
38.02
2.60
86.32
66.29
2.85
25.86
26.44
ChatGPT
87.78
67.42
3.54
75.86
49.52
LLMs
Llama-3-8BBInsttuu
87.41
70.52
3.59
72.13
49.71
Mistral-7B 7B-nst--ttInttrutt
87.01
67.70
3.64
66.00
45.98
Table 2: Main results. For literal description and contradiction, report BERT score (recall), BLEURT (BLT), and GPT evaluation score. philosophy and title, we report accuracy (%). Best scores are bold and the second best ones are marked with underline.
Implementation Details. For GPT-4 and ChatGPT, set the temperature as 1. For other models, we use the default parameter settings during inference. To reduce variance across different task prompts, create three distinct prompts for each task and report the average scores from three runs with each prompt. The specific prompts and additional details are in Appendix B.
4.2 Evaluation Metrics
For the philosophy and title understanding tasks, which are formulated as multiple-choiee ques- tion answering, we use accuracy as the evaluation metric. For generation tasks including literal description and contradiction, we apply reference-ber -asee evaluation metrics commonly used in text generation studies [60], and report ROUGE-2 (recall) [61] and BERT Score (recall) [62]. Recent work shows GPT-based evaluation aligns well with human judgements [63-65], and also apply ChatGPT for evaluation The prompts for GPT-based evaluation are provided in Appendix B.
Due to the limitation of the automatic evaluations for text generation, we also include human evalua- tion assess the quality of the outputs for the literal description and contradiction generation tasks. We hire three human judges to rate each aspect a scale of (worst) to 5 (best). For literal de- scription, following [44]. we evaluate: (1) Correctness: Does the model output correctly convey the narrative of the comic? (2) Completeness: Does the model output cover all the important elements of the comic narrative? (3) Faithfulness: Can contents from the model output be supported by the comic image (i.e., there are no hallucinations)? For contradiction generation, we evaluate Correctness and Faithfulness. More details are provided in Appendix B.4.
5
Main Results
The main experimental results are shown in Table 2. For VLMs, the original image is directly used as input, while for LLMs, the generated comic description is used as input.
*We report recall scores considering the open-ended nature of the outputs, as there can be multiple valid expressions. Our focus is on evaluating whether the key points are covered by the model outputs, ensuring precise assessment of content coverage.
*We utilize different GPT variants for specific purposes: gpt4-turbo for data annotation, gps-i-vision- preview for experiments. and gpt-3.5- 1i--..-tuubb-0122 for GaT-bbaed evaluation in text generation tasks. This helps reduce potential evaluation bias toward GPT-4's generation. Further details in Section B.

---

GPT-4
Claude-3
LLaVA-1.6 6 13B
CogVLM
mPlug Owl2
4.0
3.75
3.86
3.63
3.0
2.97
2.19
2,66
2.5
2.62
2.81
2.31
2.82
3.1
2.48
2.2
3.22
2.0
1,29
1.1
1.21
1.0
0.0
Correctness
Completeness Literal Description
Falthfuiness
Conrectness
Falthfuiness
Contradiction
Figure Human Evaluation on literal description and contradiction generation.
5.1
Narrative Understanding Tasks
We evaluate the results of VLMs only for this task. We observe that the two commercial models generally outperform the smaller open-sourced models. Among these models, GPT44 achieves the highest scores. For the open-sourced models, the larger model variants (13B) consistently achieve better scores than their 7B counterparts, indicating that larger models have superior ability to understand the image and produce higher-quality literal descriptions.
Literal Description:
Contradiction Generation: A similar trend is observed where GPT-4 and Claude-3 achieve better results than other VI.M models. Notably, VLa variants outperform their counterparts in gen- erating contradiction descriptions. This is likely due to their improved reasoning ability and world knowledge [54], which are essential for understanding comic narratives and accurately capturing the relationship between the two panels. For LLMs, unlike VLMs, the Llama-3 and Mistral models achieve results comparable to ChatGPT. Another interesting observation is that Llama-3 and Mis- tral obtain similar better results for contradiction generation compared to open-sourced VLMs, despite not having access to the original comic images.
5.2 Deep Reasoning Tasks
The Underlying Philosophy Selection and Title Matching tasks require in-depth reasoning based on the comic narratives. seen in Table 2, for philosophy selection, Claude-3 achieves the best accuracy with 84.10%, while for title matching, the LLaVA-1.6 34B variant ranks the highest with 63.31% accuracy. One key observation is that larger models usually perform better in-depth un- derstanding of the comics, aligning with the findings that larger models typically exhibit superior reasoning abilities [66, 67].
Additionally, LLMs achieve performance comparable to open-sourced VL models. This can be attributed to the strong reasoning abilities of models like Llama-3 and Mistral [59, 20], which crucial for understanding narratives and performing nonlinear reasoning to grasp deep semantics. Further analysis the influence of descriptions for LLMs provided in Section 6.1.
Another observation is that model performance on title matching is consistently lower than on un- derlying philosophy selection. Titles are shorter and more abstract versions of the narrative and do not explicitly convey the underlying idea of the comic. Therefore, distinguishing the correct title from distractions requires deeper rigorous understanding and reasoning abilities, making it more challenging for models. Notably, the human evaluation results show a similar trend our proposed GPT-based evaluation, demonstraining its effectiveness.
5.3
Human Evaluations
We conduct human evaluations on 30 randomly selected samples assess the output quality of literal descriptions and contradiction generation. as shown in Figure 3. Similar trends are observed both human and automatic evaluations: commercial models generally outperform open-source models in producing both literal descriptions and contradictions, with GPT-4 achieving the highest scores in both tasks. Additionally, the scores for literal descriptions are consistently higher than those for contradictions across all models, suggesting that understanding narrative contradictions is
7

---

LLaWAL.5-7
Drucde
Phicoophtt (9)
Priosight locustoy (30
Irage Only
Image Dmade Decorel en
89.3
85
02.2
72.1
75.5
50
06.0
59.1
25.
63
85
67.1
73.5
65.5
Thee Accuraccy 241
60
Image Onty
Imrage Desoristion
55.8
63.3
80
16.0
40-7
40.5
37.1
2M4
Tile
Matrra-7B
LianaG-BB
ChenGHT
Ohen-VL
Figure 4: LLMs using different image descrip- tion as input.
Figure 5: VLMs with image only input and im- age + oracle description as inputs.
more challenging than generating literal descriptions, which requires in-depth reasoning to compare the various aspects of both panels.
comparison the scores for literal description and contradiction reveals a strong correlation between the two tasks: models that perform well literal descriptions also tend to achieve good results on contradictions. This indicates that understanding comic juxtaposition requires diverse set of skills, including image understanding, narrative comprehension, and reasoning abilities.
Analysis and Discussion
6.1 How Does Literal Understanding the Comic Influence Deep Reasoning?
We investigate whether the quality of surface-level literal descriptions influences subsequent deep reasoning tasks. For LLMs, provide different literal descriptions generated by LLaVA-1.6 7B and 13B variants, as well as oracle descriptions written by humans, as model inputs. The results shown in Figure 4. As the quality of literal descriptions improves, the prediction accuracy for both underlying philosophy and title selection also improves. This demonstrates a strong correlation between deep reasoning and literal narrative understanding. However, significant performance gap remains compared to when oracle descriptions are used.
We further examine the performance of VLMs by providing them with additional oracle descrip- tions. The results are shown in Figure 5. Compared to using only the comic image as input, aug- menting with human-writter literal descriptions significantly improves the deep reasoning results for all VLMs. This confirms that correctly reasoning about the underlying semantics a comic requires first accurately understanding its surface narrative. However, the performance gap indicates that current VLMs still lag in narrative understanding.
Additionally, an interesting observation from Figures 4 and 5 is that when the oracle literal de- scription is provided as (partial) input, LLMs tend to outperform their VLM counterparts in both philosophy and title selection. For example, LLaVA-1.6-7B employs Mistral-7B the language model backbone, yet its performance under oracle description is significantly worse than that of Mistral-7B One possible reason is that incorporating oracle descriptions makes VLM input much longer, thus making prediction more challenging. We provide further discussions in Section6.2.
6.2 Is Decomposing Literal Description Helpful for Deep Reasoning of VLMs?
VLMs typically predict results in an end-to-end fashion, requiring the model to perform image captioning, narrative understanding, and deep reasoning all at once. Here, we investigate whether decomposing the task into separate stages of narrative understanding and in-depth reasoning can improve model performance. Specifically, we first prompt the VLM to produce literal description of a comic; then the VLM predicts results based on both the comic image and the description. The results are shown in Table 3.

---

Human Reference
The contradiction less the expectation by the product's presentation as dippesble yot oresumably offective, versus the reality experienced the user, where the razon's Ived. performance disapocintnggy quality between
contradictin
GPT-4
from actuol inferior performance.
the disposable razor's expected thor-tarm utilit being contrasted with its depiction providing lffetime service.
situation fam lar many have
The comic flsttaates humorous contradiction where, an one site, the man
CogVLM
shown shaving off his beard. and on the other side he is shown with full beard. This plays on the idea that the razor is supposed to remove hair, but this comedic scenario, it soems to have the osposite effect.
RAZOR
Sample 1
Human Reference
The contradiction here that the bed percaised differently by two indiriitaass One parson sees it king sledd bed, which lange and well-tept, while the other sees quaenniieed bed, which is smaller and dhanggntteed This Hightights how people can have difierent intergratalonn of the situation which can lead misunderstanding disgreementss
highiights the comedic truth that no motter how large the bed, we aften end up using just small part of and becomes catch-ati balongingg.
LLVV
different pempactives penson sleeping in other side, the person sterping in small bed The contradiction lies the size of the bed, as the person depicted as sleeping both large and small bed simutaneousyy This creates a humorous and confusing visual effect.
reflag Owll
side, the person viepping comfortebly in king-sia bed,
Sample 2
I-.] This highlights that even "paper" disposable cups often have plastic lids and Inner inings that make them thai lenging to recycle and still contr bute plastic waste. The justapostion eaposes the misconcaption that choosing paper cup always
Human Reference
Gladde
contradictin between bartata's creative expression and the practical act lideing the drink for convenienze.
sustalnable option, when in fact, the plactic components can still have negative
envtronmental
On the iwfl, the churacter is unjoying the coffeu with the id off, symbolicing the beginning the experionce the coffee. On the right, the character is now comsuming coffee with the lid which signifies the and the aftermath of the cuffie experience, This jostapuetiin creates visuel metaphor for the fleeting relusatiin enjoyment.
LLaWi-13
Sample
of
Figure 6: Sample outputs of contradiction explanations generated by different vision language mod- els, along with human written references. We highlight different types of errors in model outputs.
As observed, decomposing the task and augmenting it with literal description does not necessarily improve perfor- mance. In fact, when descriptions are incorporated, per- formance across all models declines on the title selection task, which contrasts with previous findings [10]. One pos- sible explanation for this drop in performance is that the generated descriptions may contain errors, negatively im- pacting the model's 's deep understanding, Another explana- tion could be the length of the generated descriptions (c.g., the LLaVA-116 13B model's descriptions average around 170 words), leading to longer and more complex prompts that make prediction more challenging. We leave a more detailed investigation of this issue for future work.
Models
Philosophy
Title
I.LVA-1.6-11
69.16
55.08
4 w/ desp.
68.68
48.76
Qwen-VL-Chat
59.10
42.05
4 w/ desp.
59.58
37.55
mPlug-Ow12
62.17
43.10
4 w/ desp.
60.25
37.84
LLLVV--.6-7BB
47.41
37.07
( w/ desp.
53.07
34.96
Table 3: Decomposition model results
augmenting the predicted description.
6.3 Error Analysis and Future Directions
We present sample outputs of contradictions generated by vision language models (VLMs) in Fig- ure 6. VLMs can make various errors in contradiction understanding.
One type of error is visual misinterpretation, where the model incorrectly interprets the image con- tents. For example, in sample 1, VLM misinterprets the image by recognizing a person "shown with full beard." Similarly, in sample 2, LL. 13B misunderstands the image contents and generates incorrect content about "two individuals." " which is inconsistent with the comic. Such mis- interpretations can lead to incorrect understanding of the narrative. These observations align with our previous findings in Section 6. and Section 6.2. This highlights the need for future research to improve models' visual interpretation capabilities.
Models also struggle to conduct in-depth reasoning of the relationship between two panels by recognizing their differences and similarities. In sample 1, while the comic implies a compari- son between the expected disposable razor and its actual longevity, GPT-4 incorrectly explains the contradiction as being about the razor's quality. A similar error occurs with mPlug-Owi2 in sam- ple 2, where it incorrectly thinks the bed sizes are different the two panels, leading to wrong illustration focusing on the bed size. Future work might incorporate recent advanced reasoning

---

approaches (e.g., multi-agen debate [68], test-time compute scaling [69]) to further improve model performance.
Another common error hallucination and incorrect association. This is evident in sample 3. The original comic contrasts latte art before and after lidding the drink, but Claude-3 incorrectly ciates the narrative with environmental protection, focusing on the plastic lid. Meanwhile, LLaVA- 1.6 13B model suffers from hallucinations by interpreting the narrative being about relaxation and enjoyment, which is unsupported by the original comic. This suggests the need for improving world knowledge and social understanding abilities enhance model performance on this task. More sample outputs are in Appendix C.
7 Conclusion
In this work, we present YESBUT, the first benchmark dedicated to studying comic understanding through juxtaposition. YESBUT encompasses a variety of tasks that address both narrative compre- hension and deep reasoning. The results indicate that state-of-the-art vision and language models still struggle with these tasks. We also offer a comprehensive analysis and discussion of errors to evaluate model performance. Current models still struggle to accurately interpret the visual contents and conduct in-depth reasoning of the underlying narratives. Through this study, we aim to pro- vide insights for future research and advance the capabilities of AI models in understanding human context, ultimately contributing to more effective and culturally aware Al applications.
Limitations
We propose comprehensive data annotation process to annotate each component. However, due to the subjectivity of comic interpretation, especially regarding the underlying ideas, there might be potential ambiguity. While we acknowledge the relatively small size of images, we rigorously collect comics and annotate each component, ensuring their high-quality and reliability. We plan to expand the dataset with the inclusion of different types of narratives in future work.
Our proposed benchmark focuses predominantly on recognizing and interpreting visual humor via juxtaposition, and may not cover aspects of visual understanding required for more generalized AI upplications. In the future. intend to explore more deeply how Al can not only interpret but also creatively engage with content. This includes generating pivotal turning points from one per- spective and creating counterpoints to given scenarios, like generating "YES" image's counterpart.
9 Ethics Statement
All data samples collected soureed from publicly available content social media platforms. We ensure compliance with copyright by utilizing original links to comics without infringement. In addition, Liz Climo) to conduct our benchmark using these public images. Additionally, we commit to open- sourcing our annotated benchmark, providing corresponding links each comic image. We dili- gently review samples, filtering out potentially offensive or harmful content.
Copyright and License.
obtained permission from the author artist (e.g., Anton Gudim,
The Large Vision Language Models utilized in our experiments are pretrained using diverse web corpora, which may introduce biases in their outputs. We advise users to conscientiously evaluate the ethical implications of generated outputs when employing them in future research endeavors.
Data Annotation. Eight human judges are engaged in our annotation process. We compensate these judges with an average hourly wage of $11, ensuring fair remuneration for their contributions.
Acknowledgements
This work made use of the High Performance Computing Resource in the Core Facility for Advanced Research Computing at Case Western Reserve University, which supported by NSF award NSF. 2117439. We also thank the support from OpenAl Researcher Access grants #0000007745.
10

---

References
[1] Jessica Pressman. Digital modernism: Making it new in new media. Oxford University Press, USA, 2014.
[2] Alan D Manning. Understanding comics: The invisible art. 1998.
[3] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprini arXiv:2310.03744, 2023.
[4] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
[5] Jiayang Wu, Wensheng Gan, Zefeng Chen. Shicheng Wan, and S Yu Philip. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247-2256. IEEE, 2023.
[6] Zhiting Hu and Tianmin Shu. Language models, agent models, and world models: The law for machine reasoning and planning. arXiv preprint arXiv:2312.0522:: 2023.
[7] Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da Rowan Zellers, Robert Mankoff, and Yejin Choi. Do androids laugh at electric sheep? humor "understanding" bench- marks from the new yorker caption contest. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 688-714, Toronto, Canada, July 2023, Association for Computational Linguistics.
[8] Abu Rayhan, Rajan Rayhan, and Swajan Rayhan, Artificial general intelligence: Roadmap to achieving human-level capabilities, 2023.
[9] Randy Duncan and Matthew J Smith. The power of comics: History, form and culture. A&C Black, 2009.
[10] Yixin Yang, Zheng Li, Qingxiu Dong, Heming Xia, and Zhifang Sui. Can large multimodal models uncover deep semantics behind images? arXiv preprint arXiv:2402.1128) 2024.
[11] James O Young. Art and knowledge. Routledge, 2003.
[12] Thierry Groensteen. Comics and narration. Univ. Press of Mississippi, 2013.
[13] Eve Bearne, Rethinking literacy: Communication, representation and text. Reading, 37(3):98-
103, 2003.
[14] Jason Dittmer. Comic book visualities: methodological manifesto geography, montage and narration. Transactions of the Institute of British Geographers, 35()):22222 2010.
[15] Joshua Schechter. Juxtaposition: A new way to combine logics. The Review of Symbolic Logic, ((4):560-006 2011.
[16] Paul Kuttner, Marcus B Weaver-Hightower and Nick Sousanis. Comics-based research: The affordances of comics for research across disciplines. Qualitative Research, 21(2):19--114 2021.
[17) Yongqi Tong, Yifan Wang, Dawei Li, Sizhe Wang, Zi Lin, Simeng Han, and Jingbo Shang. Eliminating reasoming via inferring with planning: A new framework to guide Ilms' non-linear thinking. arXiv preprint arXiv:2310.12344 2023.
[18] Sébastien Bubeck. Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lec, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. /2712, 2023.
[19] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, el. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730-27744, 2022.
11

---

[20] Al@Meta. Llama 3 model card. 2024.
[21] Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024.
[22] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing. al. Judging Ilm-as-a-judge with mt- bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
[23] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy $ Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation frame- work for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
[24] Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang. Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. arXiv preprint S.GV::3366050883 2023.
[25] Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv. Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi -iuti-diseipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36, 2024.
[26] Kaining Ying. Fanqing Meng, Jin Wang. Zhiqian Li, Han Lin, Yue Yang. Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal bench- mark for evaluating large vision- vision-language models towards multitask agi. arXiv preprint ar04v:2404. 16006, 2024.
[27] Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schimdt. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use. arXiv preprini (i0i::2300.06599 2023.
[28] Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! a vision-and-languaae benchmark of synthetic and compositional images. In Proceedings of the IEEE/CVE Interna- tional Conference on Computer Vision, pages 2616-2627, 2023.
[29] Bohao Li, Rui Wang, Guangzhi Wang. Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal 1lms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
[30] Bohao Li. Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench-2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.1709. 2023.
[31] Jerry Palmer. Taking humour seriously. Routledge, 2003.
[32] Lei Chen and Chong MIn Lee. Predicting audience's laughter using convolutional neural network. arXiv preprint arXiv: 770.022584, 2017.
[33] Andrew Cattle and Xiaojuan Ma. Recognizing humour using word associations and humour anchor extraction. In Proceedings of the 27th international conference on computational lin- guistics, pages 1849-1858, 2018
[34] Diyi Yang, Alon Lavie, Chris Dyer, and Eduard Hovy. Humor recognition and humor anchor extraction. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2367-2376, 2015.
[35] Miriam Amin and Manuel Burghardt. A survey on approaches to computational humor gen- eration. In Stefania DeGaetano, Anna Kazantseva, Nils Reiter, and Stan Szpakowicz, editors, Proceedings of the 4th Joint SIGHUM Workshop Computational Linguistics for Cultural Herütage, Social Sciences, Humanities and Literature, pages 29-41, Online, December 2020. International Committee on Computational Linguistics.
12

---

[36] Arjun Chandrasekaran, Ashwin K Vijayakumar, Stanislaw Antol, Mohit Bansal, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. We are humor beings: Understanding and predicting visual humor. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 1603-461,, 2016.
[37] Dafna Shahaf, Eric Horvitz, and Robert Mankoff. Inside jokes: Identifying humorous cartoon captions. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1065-1074, 2015.
[38] Dragomir Radev, Amanda Stent, Joel Tetreault, Aasish Pappu, Aikaterini liakopoulou, Agustin Chanfreau, Paloma de Juan, Jordi Vallmitjana, Alejandro Jaimes, Rahul Jha, al. Humor in collective discourse: Unsupervised funniness detection in the new yorker cartoon caption contest. arXiv preprint (.Kiv:5506008122 2015.
[39] Yuta Kayatani, Zekun Yang. Mayu Otani, Noa Garcia, Chenhui Chu, Yuta Nakashima, and Haruo Takemura. The laughing machine: Predicting humor in video. Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 2073-2082, 2021.
[40] Yang Liu, Tongfei Shen, Dong Zhang, Qingying Sun, Shoushan Li, and Guodong Zhou. Comment-aided video-language alignment via contrastive pre-training for short-form video humor detection. arXiv preprini 2rXiv:2402.09055 2024.
[41] Sophie Jentzsch and Kristian Kersting. Chatgpt is fun, but it is not funny! humor is still challenging large language models. arXiv preprint arXiv:2306.05563 2023.
[42] Nicola De Pisapia, Francesca Bacci, Danielle Parrott, and David Melcher. Brain networks for visual creativity: a functional connectivity study of planning visual artwork. Scientific reports, 6(1):39185; 2016.
[43] Mika Koivisto and Simone Grassini. Best humans still outperform artificial intelligence in creative divergent thinking task. Scientific reports, 13(1):13601, 2023.
[44] EunJeong Hwang and Vered Shwartz. MemeCap: A dataset for captioning and interpreting memes. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Con- ference on Empirical Methods Natural Language Processing, pages 1433-1445, 1445, Singapore, December 2023. Association for Computational Linguistics.
[45] Dragomir Radev, Amanda Stent, Joel Tetreault, Aasish Pappu, Aikaterini Iliakopoulou, Agustin Chanfreau, Paloma de Juan, Jordi Vallmitjana, Alejandro Jaimes, Rahul Jha, and Robert Mankoff. Humor in collective discourse: Unsupervised funniness detection in the new yorker cartoon caption contest. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelník, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16), pages 475-479, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA).
[46] Yuqing Wang and Yun Zhao. Gemini in reasoning: Unveiling commonsense in multimodal large language models. arXiv preprint arXiv:2312. 2023.
[47] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6720-67311 2019.
[48] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual rea- soning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
[49] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguisiic com- positionality. In Proceedings of the IEEF/VV Conference on Computer Vision and Pattern Recognition, pages 5238 5238-5248, 2022.
13

---

[50] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
[51] "yes, but" series created by anton gudim. 2024.
Accessed:
[52] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[53] AI Anthropic. The claude model family: Opus, sonnet, haiku, Claude-3 Model Card, 2024.
[54] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[55] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:23 1.030799 2023.
(56) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vasion-language model with versatile abilities. arXiv preprint arXiv:2308. 129.66668..99,, 2023.
[57] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owi2: Revolutionizing multi-modal large language model with modality collaboration, 2023.
[58] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[59] Albert Q Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arX1v:2310.06825, 2023.
[60] Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation text generation: A survey. arXiv preprint XXXY:2006.11799, 2020.
[61] Chin-Yev Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81 Barcelona, Spain, July 2004. Association for Computational Linguistics.
[62] Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference Learning Representations, 2020.
[63] David Chan, Suzanne Petryk, Joseph Gonzalez, Trevor Darrell, and John Canny. CLAIR: Eval- uating image captions with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16638-1366,, Singapore, December 2023. Association for Computational Linguistics.
[64] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 2511-2522, Singapore, December 2023. Association for Computational Linguistics.
[65] Zhe Hu, Hou Pong Chan, and Yu Yin. Americano: Argument generation with discourse-driven decomposition and agent interaction. arXiv preprint arXiv:2310.2033 2023.
14

---

[66] Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajum Chen. Reasoning with language model prompting: survey. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pages 5368-5393, Toronto, Canada, July 2023. Association for Computational Linguistics.
[67] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206. 07662, 2022.
[68] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprini arXiv:2305. 14325, 2023.
[69] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint MrXiv:4488.033/4 2024.
15

---

A Data Annotation Details
Considering the workload of manually writing all components from scratch, we leverage Al-human collaborative pipeline for annotation. The prompts for generating each component are listed in Table 4. After producing each component, human annotators will verify and modify the outputs for the final components. We present a sample comic with all tasks in Figure
Tasks
Prompts
Literal Description
The given comic with two panels shows the same situation from two op- posite sides with contradictions. You need to first read and understand the comic. Generate a detailed description to illustrate the narrative of the comic and explain the contradiction of what makes the comic inter- esting or sarcastic.
& Contradiction
Underlying Philosophy
Write a brief description of the underlying moral of the narrative in one sentence, and include what phenomenon satirizing and what we can learn from the comic.
Title
Produce a short eye-catching title reflecting the narrative.
Negative Philosophy
Generate five contextualized, plausible, but ultimately incorrect criti- cisms and moral lessons we can learn from image, each in one sen- tence distracters. Keep the length and style the same the correct one.
Negative Title
Provide five seemingly reasonable, eye-catehing but incorrect titles.
Table 4: Prompts for data annotation
Literal Description:
The comic is divided into two panels, each presenting contradictory perspective of the same object-a mug. In the first panel, mug illusttrated as adorable fox with closed eyes, giving off a serene and cute vibe. It's an object that one would admire or find endearing. However,
panel reveals attempts to drink from the fox-shaped mug, but its design- design-featurion featuring protruding ears and head- awicwardly interferes, complicating the act of sipping comfortably.
Contradiction:
The comic illustrates a contradiction where mug designed as an adorable fax is charming to look at but proves impractical
awkwardly protruding ears head that hinder drinking.
Underlying Philosophy Selection:
A. The illustration critiques the clash between aesthetics and usability, emphasizing the need for balanced consideration of both ensure
practical experience any aspect
The comic shows that adding decorative elements object will enhance its value and enjoyment, when fact, the opposite true in this case.
initial charming appearance an always lead to positive experience, disregarding practical complications that arise later.
illustration suggests
D. The image shaws enduring incorvenience is worthwhilo sacrifice for the sake owning something that looks unique or cute.
Answer:
Title Matching:
Charming Design, Prickly Reality: The Fox Mug's Surprise
Toast to Vulpine Grace
Harmony
Enchanting Fox's
Answer:
Figure 7: Sample comic with all annotated tasks.
Analysis on Data Diversity. In order to show diversity our benchmark, we prompt ChatGPT to generate topical keywords for each comic based on its description, and then cluster these key- words. All these scenarios are presented in Figure 8. As we can see, the comics in our benchmark encompass diverse range of everyday life scenarios.
16

---

Road and Transportation
Urban and Humor
Diet and Control
Gifts and Romance
Effort and Simplicity
4.9%
4.9%
5.5%
9.8%
7.8%
Technology and Toys
4.3%
Digital Interfaces
3.2%
and Identity
4.3%
Global Views
Health and Fitness
6.0%
12.7%
10.7%
Rest and Dreams
Gender Perspectives
Daily Habits
Animals and Nature
Figure 8: The clusters comic topics covered by our benchmark.
B Experimental Details
B.1 Model Details
We include both commercial and open-sourced VLMS and LLMs in our experiments. For GPT- 4, we use spt-4-vvssoon-preeeeww version, and for ChatGPT we employ the gpt-3.5-turoo00122 model variant For Claude-3, we leverage the Claude Ops model updated on 29th Feb, 2024 For open- sourced models, we include 6L. (34B, 13B and 7B) [54], LLa 13B [3], InstructBlip 13B and 7B variants [58], CogVLM [55], Qwen-VL [56]. and mPLUG-Ow|2 [57]. For LLMs, we use the Llama3 instruction variant [20], and the Mistral 7B instruction model [59].
B.2 Implementation Details
All commercial models are accessed through their official API. For open-sourced models, im- plement the experiments using Hugging Face Transformers' For GPT-4 Claude-3 and ChatGPT, setting temperature 1.0. For other models, apply the default parameter setting greedy decoding during inference. The experiments are conducted on NVIDIA 4090 and A6000 GPUs.
For MCQ evaluation, we explicitly instruct the model to directly output the option in prompts, and use hard rules parse the answer. If none of the options can be parsed, we will assign a random option. For generation task evaluation, we apply rouge-score to compute ROUGE score, and calculate the BERT score using the official implementation" 9, For GPT based evaluations for literal description and contradiction, we use gpt-3.5 3pt-3.5-turbo-0125 version. The prompts we used are shown Figure 9.
B.3
Experiment Prompts
To reduce the biases from different prompts, we design three different prompts by different people to evaluate models and report the average results for all tasks. present the prompts used for Literal Description (Figure 13), Contradiction Generation (Figure 14), Underlying Philosophy Selection (Figure 15), and Title Matching (Figure 16).
"nccpe://platform.opoen openai. wom//ocss/oodels/
"nttps://ww.anthai www.aapic..ccooo.....o com/apppp.ccc//pppppp.
17

---

Prompts for Literal Description:
Candidate literal description: gen
Reference literal description: ref
Task: You need to determine how accurately the above candidate literal description matches the given reference literal deseription comic narrative.
Using a scale from to 5, rate the accuracy with which the candidate description matches the reference description, with being the least accurate and heing the most accurate.
Please directly output score by strictly following this format: [[score]], for example: Rating: [[3]].
Prompts for Contradiction:
Background: You impartial judge. You will be given literal description of comic that presents same situation from two opposing perspectives, highlighting contradictions. You will also be pro- vided with gold-standard illustration as reference that effectively demonstrates these narrative contra- dictions.
Your task is evaluate the quality of a generated illustration and determine whether it accurately depicts the narrative contradictions in the comic. Then, assign and the highest. based on its quality.
scale of to 5, where is the lowest
The literal description of the comic:description
The reference contradiction illustration:ref
The generated contradiction illustration:gen
Please directly output score by strictly following this format: [[score]], for example: Rating: [[3]].
Figure 9; Prompts for GPT based evaluations.
B.4
Human Evaluation Details
We present 30 random samples on each task for human evaluation. We anonymize the models and shuffle the outputs to the annotators. Following [44], include the following aspects:
Correctness:
Does the model output correctly convey the narrative of the comic?
Completeness: Does the model output cover all the important elements of the comic narrative?
Faithfulness: Can contents from the model output be supported by the comic image (i.e., there hallucinations)?
For Literal description, we evaluate on all three aspects. For contradiction, we evaluate on Correct- ness and Faithfulness.
C More Sample Outputs
Here, we present more randomly picked sample outputs on literal deseription and contradiction generation in Figure 10, Figure 1I, and Figure 12.
18

---

Human Reference
Literal Descritiio::
The left panel shows a musis player interface with song tiled Don't Care" by Li Big Money: The player has the typical features of muse spp with play, pause, nee, and previous buttons, and a progress har that shown the song being played The right panel menels the munitian, presunably Li Big Money, looking stressed and worner ss observes the sorg stats on chart that plummwinng. The chart labeled Don't Care SONG STATS, and the descending line parforming
Contradiction:
The contradiction that while the song and the artist's persona rodiote corefree sttitud, reality quite the uspusite to. Li Big Mhnonyy doss deuply about the parformance of shown the aneious
Claude-3 Generated Destription
The comic depicts man with facial hair and sunglesses in contrasting panels, the first panel, titled "I Don't Care" with the subtitle "Lil Big Money", the man appears disinterested while music controls are shown below. In the second panel, titled Dan't SDMG STATS", graph shows despite the man's supposedly carefree attitude portrayed in the first panel, he actually quite concerned about the performance and reception music as
decline, implying the
popularity quickly fading.
juxtaposition suggests that
by second panel showing song's diminishing stats.
Claude-3
Contradiction
The comic strip illustrates a satirical contradiction between the music artist's public persona and private attitude. the first panel, the artist promotes their new song Don't Care" with confident, indifferent expression. However, the second panel shows song stats plummeting, causing the artist This juxtapasition humorousiy highlights the discrepancy between the artist's prajected apsthetic image and their genuine fallure. The comic cleverly exposes how public figures may present facade while
distressed and concerned, directly contradicting
song's "I Don't Care"
emotional investment in the song's privately experiencing the ogposite sentiment.
CogVLM Generated Description
The comic portrays who initially indifferent song titled 'I Don't Care' by Money. However, upon seeing song's statistics, he becomes deeply engrossed and contemplates its popularity, even though he initially claimed not to care about it.
CogVLM Generated Contradiction
The comic illustrates humorous contradiction where the character claims not to care about song statistics, yet deeply engrossed observing them,
side, the character shown confidently listening song, while on the right, he seen contemplating the declining statistics, indicating sudden change his attitude.
Figure 10: Sample outputs of model generated literal description and contradiction.
Human Reference
Literal Descriation:
studio, first the
studio, filldd with disssic suppiles:
sculptures, the awalts the articts touch. However, sccond panel reveals punchildd artbt has chosen to create prece anime or margesllyle clasical environmentt and matarials surrounsing thim.
paints-all
tradittona
empty
sbands stark contrast
controdiction comic traditional artsetting propored for dassical creation the artist' unzxpected
contomporory onime stylc
Description
The comic depicts scene of artist's studio, with two panels showing the same space from oppasite perspectives. In the left surrounded by various
panel, is seen standing supplies. The studio well-itt and appears to professional workspace, the right panel, the same scene shown from the opposite side, but the artist is now sitting at the eesel, holding a smartphone, and the studio is now dlimly lit, suggesting different time
a canvas, holding paintbrush,
artist's moad activity. The contrast between the panels highlights the versatility artist's workspace and the different states of the artist's creative process.
change
LLaVA-166 13B Generated Contradiction
(..) The contradiction lies in the fact that the two sides represent two different stages of the creative process: one side is the beginning,
and other end, where artist completed their and likely to be evalusting contemplating the final product. This juxtapasition highlights the contrast between the initial stages of creation and the final stages reflection and completion.
are yet
mPlug-Owll Generated Description
The comic depicts man painting picture of woman in white dress. However, the image shown from two different wearing black dress and the other where she wearing white dress. This creates
perspectives, where the contradiction in the visual representation of the woman's is attire, making the comic visually intriguing and thought- provoking.
mPlug- Owi2 Generated Contradiction
In the comic, there is man painting picture of woman in studio. On one side, the man painting woman with a smilo, while with frown. This contradiction highlights difference emotions
on the other side, the expressions the artist trying to convey in the two paintings.
painting
Figure 11: Sample outputs of model generated literal description and contradiction.
19

---

Human Reference
Literal Description:
seemingly engrossed ballet, epitome high culture, poised to share her experience an social media with tags indicating her cultured taste appreciation for strips away the pretense to reveal the woman fast asleep during the performance.
The first panel depicts
the second panel
Contradiction:
contradiction lies between cultured persona she broadcasts online her true disinterest.
Description
The comic is into two panels, presenting a sequence of events with a humorous twist. the first panel, woman is seen upscale theater, smiling holding binoculars as
standing appears performance taking place. The ambiance depicted as refined and high culture emphasized by hashtags such "#cultural, "Mperformance; "#ballet." fine arts. The second panel switches perspective show the same head tilted back and eyes eyesel closed. The elegance implied the first frame juxtaposed with comical hashtag "#fart," indicating that
stage
"Eart" scattered around the image, suggesting sense sophistication and appreciation for the
sitting among audience, seemingly relaved with
experiencing loss than dignifled cultured grace and the reality of an awkward bodily function.
thereby creating comic disparity between the perception
GPT-4 Generated Contradiction
idnalized public image and realities, The comic plays on the juxtaposition of the refined setting of ballet performance with the mundane and natural bodily heshtags
This contradiction highlights often humarous disparity
less glamorous
function passing gas,
cultural
image.
Claude-3 Generated Description
side, performance categorized under "cultural" and 'art", with elegant ballerinas gracefully dancing stage. contrast, the right side depicts woman slouching in chair, suggesting a refined sophisticated setting. The hashtags imply this may represent some form "performance" the woman's 's casual, unrefined posture and dancers. The comic highlights the subjectivity labeling and perceiving different forms performance and art, calling into what truly constitutes "culture".
strip presents contradictory perspective performing arts.
stand in stark juxtaposition
poise and refinement the ballet
Claude-3 Generated Contradiction
The juxtapesition highlights and unremarkable, depending on the viewer's perspective and appreciation for the medium. points out the subjectivity in valuing interpreting various art forms.
expression
high brow, entertainment
Figure 12: Sample outputs of model generated literal description and contradiction.
Prompt1:
The given comic shows the same situation from two opposite sides with contradictions. Write one- paragraph literal description to describe the narrative of the comic.
Prompt2:
Please literally describe context image detail.
Prompt3:
Give me detailed literal description the image.
Figure 13: Prompts for Literal Description Generation in experiments.
20

---

Promptl:
The given comic shows the same situation from opposite sides with contradictions. Write short explanation to illustrate the contradiction of the two sides.
Prompt2:
Analyze the provided image, which is divided into two more panels, each illustrating contrasting views the same scenario. Describe elements visible each panel. Then concisely interpret how these elements convey contrasting perspectives in one or two sentences. Facus and only output the contradiction.
Prompt3: Given an image, the image is divided into two or more panels. There is the contrast relationship in the image through panels. Describe the elements visible in each panel. Give me the concise interpretation how these panels convey contrasting perspectives. which you only need output the contradiction in one or two sentences.
Figure 14: Prompts for Contradiction Generation in experiments.
Prompt1:
The given comic shows the same situation from two opposite sides with contradictions. Which of the following options best represents the underlying philosophy of the comic? (MCQ Options)
Just output the choice:
Prompt2:
You are presented with an image, which is divided into two more panels, each illustrating contrasting views of the scenario.
Which the following options best represents the philosophy the image provided?
{MCQ Options)
Select the correct option by typing the corresponding letter (A, B. C, or D).
Prompt3:
Given an image, which has two or more panels. There is contrast in these panels. Tell me the best option in the following options who represents the deep semantic of the image?
(MCQ Options) Just tell the correct option by outputing corresponding letter (A, B, C, D), explanation.
Figure 15: Prompts for Underlying Selection Task in experiments.
Promptl:
The given comic shows the same situation from two opposite sides contradictions. Which of the following titles are the most suitable for the comic? (MCQ Options)
Just output the choice:
Prompt2:
You are presented with an image, which is divided into two or more pancls, cach illustrating contrasting views of the same scenario. Which of the following title options best represents the image provided? (MCQ Options)
Select correct option by typing the corresponding letter (A, C, or D).
Prompt3:
Given image, the image divided into
panels. There is the contrast relationship in the
image through panels.
Tell me the best title in the following title options who represents the image?
[MCQ Options)
Just tell me the correct option by outputing corresponding letter (A, B. C. D), explanation.
Figure 16: Prompts for Title Matching Task in experiments.
21

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the work Section
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The experimental settings are thoroughly explained in Section and further detailed in Appendix Section B. Additionally, the code released, ensuring that the exper- iments can be replicated as described, supporting the main claims and conclusions of the paper.
Justification:
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the corresponding benchmark.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings are thoroughly explained in Section 4 and further detailed in Appendix Section
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Answer: [NA]
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All relevant details, such as the type of compute workers for open-sourced models and the API specifications for commercial models, are provided in Scction B.
9. Code Of Ethics
22

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.ce http:///cccppplcccceccccciiaoaaa
Question:
Answer: [Yes]
Justification: We have ensured that all ethical guidelines were followed throughout the study, with careful consideration fairness, transparency, and the responsible use of data and models, An Ethics statement is provided in Section 9.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to the Section 1, Section 2, Section 8, and Section 9.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creators and original owners of all assets used in the paper. including code, data, and models, properly credited. Additionally, the licenses and terms of use for the comic data are explicitly stated and fully respected, as outlined in Section 9.
Justification:
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: Section 3, Section and Section B for newly proposed GPT based evaluation.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: Section A
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
23


Reviews:
[{"review_id": "ldooAv2LYj", "reviewer": "Reviewer_2fnt", "summary": "This paper investigates the capability of large vision language models (VLMs) to understand humor in comics through narrative contradictions. The authors introduce the YESBUT benchmark, comprising tasks designed to evaluate AI’s ability to recognize and interpret contradictory narratives in comics. Experiments are conducted with both commercial and open-source VLMs to assess their performance on tasks ranging from literal content comprehension to deep narrative reasoning.", "questions": "How did you address the potential subjectivity and variability in human annotations, especially for tasks like underlying philosophy selection and title matching? Did you measure inter-annotator agreement, and if so, what were the results?\n\n--- \n\n\nHow did you ensure that the negative titles and philosophies were sufficiently challenging yet distinct from the correct ones? Can you provide examples where models struggled with these distinctions?\n\nFor example, I feel that the example in Figure 2, the negative title \"Graphs Don't Lie, People Do\" is also quite suitable to the comic. \n\n--- \n\nDid you assess the models for any cultural or contextual biases in their understanding of humor? If so, what were your findings, and how do you plan to address these biases in future work?\n\n--- \n\nCould you provide more details on why you chose BERT score, ROUGE-2, and GPT-based evaluations as your primary metrics? Have you considered using other evaluation metrics that might better capture the nuances of humor understanding?", "limitations": "It's discussed in Sec 8", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tInnovative Benchmark: The introduction of the YESBUT benchmark provides a structured approach to evaluating AI’s understanding of humorous contradictions in comics.\n\n2.\tHuman-AI Collaborative Annotation: The use of a human-AI collaborative pipeline for data annotation is innovative and helps in obtaining high-quality annotations efficiently.\n\n3.\tInsightful Analysis: The analysis of results, including error types and human evaluation, provides valuable insights into the challenges faced by current models in understanding humorous contradictions.", "weaknesses": "Novelty: The concept of using VLMs to understand humor in comics has been explored in various forms in previous research. The YESBUT benchmark, though focused on narrative contradictions, might be seen as a slight variation on existing benchmarks that evaluate multimodal understanding and humor recognition. \n\nLimitation: The benchmark focuses on a very specific type of reasoning—juxtaposition—which may not be broadly applicable to other forms of humor or narrative understanding. This narrow focus limits the benchmark’s utility in evaluating general AI capabilities in humor comprehension. The task of matching comics with titles involves a high degree of subjectivity. Different annotators might have varying opinions on what constitutes a suitable title, leading to inconsistent and potentially biased evaluations.  The dataset consists of only three hundreds of comics, which may limit the generalizability of the findings. Expanding the dataset could provide more robust insights.\n\nAnnotation: The annotation process, which relies heavily on human-AI collaboration, might introduce bias. Human annotators may inadvertently guide the AI’s outputs, leading to annotations that reflect human reasoning more than autonomous AI understanding.\n\nMetrics: The primary evaluation metrics include BERT score, ROUGE-2 (recall), and GPT-based evaluation scores for literal description and contradiction generation tasks. While these metrics are useful for assessing surface-level content generation, they may not fully capture the depth of understanding required for humor comprehension.\n\nSuitability:  Given the nature of the contribution, the work may be better suited for venues that specifically focus on datasets and benchmarks, such as the NeurIPS dataset/benchmark track and workshops or tracks dedicated to introducing new datasets. This setting would allow the authors to highlight the value of the YESBUT benchmark without the expectation of a significant theoretical or methodological breakthrough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and constructive comments. We appreciate your recognition of our work's innovative benchmarks, human-AI collaborative annotation, and insightful analysis. Below, we address your concerns and questions individually:\n\n### W1 & W5: Regarding Novelty and Suitability\n \nThank you for the feedback! However, we believe our work is uniquely novel and suitable for this venue. First, our benchmark formulates a new task to address humor understanding through narrative contradictions, a complex and underexplored aspect of AI research. This task challenges models to integrate **multiple fundamental abilities** including the comprehension of human norms, critical thinking about similarities and differences of elements, and nonlinear reasoning, which are critical for developing socially intelligent systems. Reviewers aFj6 and yTj4 have recognized these contributions, which we will clarify further in our revision. In addition, our work extends beyond a dataset introduction with **comprehensive evaluations and detailed analyses, providing valuable insights into AI's interpretive and reasoning capabilities**, making it highly relevant for broader AI and NLP communities. As a pilot study on this topic, we believe our work is suitable for the venue and can bring valuable insights for future research.\n\n---\n\n### W2: Limitation of benchmark\n\n> juxtaposition may not be broadly applicable\n\nThis is a good point. Juxtaposition is a common and sophisticated technique with applications in domains such as arts, literary, and mathematics [1]. As a pilot study, we consider it as a suitable entry point for evaluating complex narrative understanding for AI models.\n\n> The task of matching comics with titles involves a high degree of subjectivity\n\nThank you for this important question. Indeed, humor understanding can be subjective. To address this, we have formalized title matching as a **selection task rather than a generation task** (i.e., determining which title is \"objectively\" better). To ensure a \"common preference,\" we conduct multiple quality checks  and verifications during annotation, where each sample is reviewed by different annotators to reach an agreement (Line 165). We will clarify this in our revised version.\n\n> The dataset consists of only three hundreds of comics\n\nPlease see the Overall Response, point 1.\n\n[1] https://en.wikipedia.org/wiki/Juxtaposition\n\n---\n### W3 & Q1: Regarding bias and subjectivity in annotation\n\n> The annotation process might introduce bias and how did you address the subjectivity\n\nPlease see the Overall Response, point 2.\n\n> Did you measure inter-annotator agreement\n\nAs each component was annotated and verified by at least three annotators to reach agreement, we did not further measure inter-annotator agreement.\n\n---\n### W4 & Q4: Regarding the weakness and question of evaluation metrics\n\nPlease see the Overall Response, point 3.\n\n---\n### Q2: Regarding the quality of negative titles and philosophies\n\n> How did you ensure that the negative titles and philosophies were sufficiently challenging yet distinct from the correct ones\n\nThank you for raising this important question! The negative titles and philosophies are constructed by human annotators. For each sample, we first prompt GPT-4 to generate negative candidate options (as shown in Table 4). The annotators then revise and edit these options to **ensure they are sufficiently challenging yet distinct from the correct ones**. During the quality check stage, annotators will further verify the quality. As illustrated in Figure 7, all negative options are on-topic and relevant to the comic but may contain incorrect logic or do not accurately reflect the narrative. This process ensures quality of the negative options.\n\n\n> The negative title \"Graphs Don't Lie, People Do\" in the example of Figure 2\n\nThis is a very good point! The comic itself illustrates how the interpretation and presentation of data can be manipulated or selectively used to tell misleading stories, while the **graphs and data are factual and real**. Therefore, the title implying that people lie is not appropriate.\n\n---\n### Q3: Regarding the question of cultural or context bias\n\nThis is a very insightful question! In our current study, the annotations are based on  the **common interpretation** of humor, and our evaluations focus on how model performance aligns with such **average preferences**, without consideration of specific contextual or cultural information. However, we acknowledge the critical importance of considering cultural and contextual variations in humor understanding. We will explore this aspect in our future research. Additionally, we will include a discussion of these considerations in the Limitations section of our revised version.\n\n\n---\n### Regarding the raised ethical consideration\n\nThank you for highlighting these important concerns. To address them, we have included a comprehensive ethics statement in our paper (Section A in appendix). Below we outline the key points:\n\n> Data privacy, copyright, and consent\n\nAll data samples are sourced from publicly available content on social media platforms. We strictly adhere to copyright laws by using original links to the comics, thereby avoiding any infringement. We will provide the original links of each image when releasing the benchmark. \n\n> Discrimination, bias, and fairness\n\nWe have conducted a thorough review and rigorous quality checks of our samples to filter out potentially offensive or harmful content. Our annotators come from diverse cultural and gender backgrounds, and we conduct multiple quality checks and verification to minimize bias. \n\nWe will revise our paper to better clarify these points and provide reassurance regarding the ethical considerations of our work. We will also incorporate more detailed instructions to guide users in adhering to copyright regulations, and encourage users to carefully consider the ethical implications of the generated outputs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wcw278hWsz", "reviewer": "Reviewer_aFj6", "summary": "This paper presents a benchmark YESBUT, which contains pairs of images exhibiting a sense of humor via juxtaposition. For each pair, the authors employed human-AI collaboration methods to annotate detailed tasks to assess why the humor can be understood, including literal description writing, contradiction generation, underlying philosophy selection, and title matching. Based on the YESBUT benchmark, there are rich benchmark results presented, including the comprehensive comparison with varying LLMs and VLMs and in-depth analyses to provide insights.", "questions": "1.\tFor philosophy selection, why it is formulated as the MCQs instead of text generation like other tasks? \n2.\tHow to ensure the diversity of the benchmark data (so the image covers varying testing scenarios)?\n3.\tWould the prompts play crucial roles in models’ decision making? What will happen if different sets of prompts are used?\n4.\tWhy GPT-4 can engage in the annotation as well as the model comparison? Will that involve any bias?", "limitations": "The dataset size is relatively small, which hence cannot be used for model training. Also, it would be good if the authors can provide insight into how to further advance the techniques to better tackle the task.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe paper presents a new task of juxtaposition-based humor in multimodal settings. It can reflect the system's ability to interpret and generate nuanced, contextually appropriate responses, thereby improving its interaction with humans in more natural and engaging ways. The task is challenging because it requires models to make sense of the non-linear connections between two semantically related images and social reasoning skills to capture a sense of humor. The results also show the limitations of existing models in handling the task.\n2.\tThe YESBUT benchmark crafted to research the task is helpful. It contains rich human annotations reflecting varying perspectives for task evaluation. The benchmark can be beneficial for future research. \n3.\tThe paper presents rich experimental studies. The results and analyses offer a comprehensive view of the pros and cons of the cutting-edge VLM techniques and provide in-depth analyses to interpret where the challenges are and how to address them. \n4.\tThe paper is written very clearly with well-designed organizations, clear writing, and good presentations with cases, figures, tables, etc.", "weaknesses": "1.\tThe dataset is relatively small, with only 348 image pairs. Although I understand that the data is very hard to gather and the annotation is very labor intensive, a larger dataset will enable more sound evaluation results and allow the potential for model training (now it can only support model evaluation).\n2.\tIt would be good if the authors could also examine how the models can be further improved to tackle the task well. In the current version, most of the findings center on the limitations, but it would be good for the authors to also point out the directions of further studies in advancing the technology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable comments and suggestions! We are pleased to learn that you find our introduction of the YESBUT benchmark is helpful and beneficial to future research, and our rich experimental studies and in-depth analyses offer a comprehensive view. We address your questions and concerns below:\n\n### W1: Regarding data size\n\nThank you for the question! Please see point 1 in the Overall Response.\n\n---\n### W2: Regarding model improvements\n\nThank you for your insightful suggestion! We agree that highlighting directions for future research is crucial. Currently, we aim to uncover potential areas for improvement through results analyses. \n\nFirst, our analysis indicates that VLMs often struggle with accurately interpreting image content and may make errors in literal descriptions (Sec. 6.1 & 6.2). This suggests a need for future work to **enhance visual interpretation capabilities**. \n\nSecond, improving the **in-depth reasoning ability** of VLMs is essential for this task. For instance, LLaVA-1.6 significantly outperforms LLaVA-1.5, likely due to the advancements in reasoning abilities [1]. Future work might incorporate recent advanced reasoning approaches (e.g., multi-agent debate, refinement-based reasoning) to further improve model performance.\n\nFinally, our error analysis reveals that models tend to suffer from hallucination (Line 333), suggesting the need for **incorporating external knowledge** to enhance human understanding. To mitigate this problem, knowledge augmentation methods can be employed to enhance VLMs performance. \n\nWe will revise our paper to include a dedicated section that provides a more detailed discussion on these points and outlines future research directions. \n\n[1] Llava-next: Improved reasoning, ocr, and world knowledge\n\n---\n### Q1: Why philosophy selection is formulated as the MCQs\n\nThis is a good question! Compared to the literal description and contradiction illustration, philosophy and title are more open-ended and subjective, where there might exist multiple valid philosophies and titles for one comic. This characteristic makes the evaluation of these two tasks challenging if formulating them as text generation. Therefore, we follow previous work and formulate them as MCQs [1,2].\n\n[1] Can Large Multimodal Models Uncover Deep Semantics Behind Images? \\\n[2] Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest\n\n---\n### Q2: The diversity of the benchmark \n\nThank you for raising this good question! The comics in our benchmark encompass a diverse range of everyday life scenarios. To ensure and analyze the topic coverage, we prompted ChatGPT to generate topical keywords for each comic based on its description and then clustered these keywords. The complete clusters and their statistics are provided in the one-page PDF of the Overall Response. The results indicate the diversity of our benchmark. We will incorporate the analysis in our revised version.\n\n---\n### Q3: Would the prompts play crucial roles in models’ decision making?\n\nTo address the potential influence of prompts on model performance, in the experiments we created three distinct prompts for each task and reported the average scores from three runs with each prompt. The specific prompt sets are detailed in Appendix C.3. Our initial observations indicate that different VLMs exhibit varying degrees of sensitivity to prompts (for example, commercial models are less sensitive to the prompts than smaller VLMs); however, the overall performance gap across different prompts is not significant. We will revise the relevant section to clarify this point more effectively.\n\n---\n### Q4 : Why GPT-4 can engage in the annotation as well as the model comparison? Will that involve any bias?\n\nThis is a good question! To clarify, we use different GPT variants for distinct purposes: For data annotation, we leverage the gpt4-turbo variant; for experiments, we report results using gpt-4-vision-preview; and for GPT-based automatic evaluation on text generation tasks, we employ gpt-3.5-turbo-0125. While we do use different GPT models variants, there might still be some inherent bias since we do not know the backend model details. We will revise our paper to better clarify this distinction and acknowledge the potential for bias.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q2gK7ozqjx", "reviewer": "Reviewer_wVUt", "summary": "The paper proposes a new evaluation benchmark to evaluate how much current VLMs understand the humor and uses the new benchmark to compare various VLMs and LLMs.", "questions": "Could you provide some statistical significance analyses?\n\nWhich GPT you are using for evaluation? GPT-4? GPT 3.5 turbo?", "limitations": "The discussion of the weaknesses I mentioned could be added.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Although some previous studies such as [7] and [10] have proposed the humor benchmark for VLM, like author mentioned in 112-114, the proposed benchmark is the first one which has two input images in each sample and the focus is the relation between two images. As far as I know, this is indeed novel. \n\nBased on the dataset, the authors proposed several tasks, which allow researchers to conduct more in-depth analysis on LLM's performance.\n\nThe paper uses the benchmark to compare many VLMs and conduct some good analyses.", "weaknesses": "The dataset size is relatively small. Both [7] and [10] have > 1k samples. The small dataset size might create some instability of the scores. I guess that is why the paper does not provide statistical significance analyses. The subjectiveness of the task and the noise of automatic metrics could intensify this problem. We can see that although GPT-4 achieves the best performance in most metrics, the correlations between BERT, R-2, and GPT are not super high for some models. Although automatic evaluation using GPT has shown to have high correlation with human evaluation, prior studies (e.g., https://arxiv.org/html/2404.13076v1) show that it is biased toward their own generation and my experience is that BERT score or R-2 are both not reliable metrics for creative writing and GPT evaluation is only more reliable when the dataset size is large.\n\nThe paper does not provide human performance, which make interpreting the state-of-the-art performance more difficult. For example, are the scores of GPT-4 in Table 2 really good?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback! We are pleased to learn that you consider our benchmark novel, which includes two input images per sample to emphasize the relationship between the images. Additionally, we are grateful for your acknowledgment of our task settings and good analysis. Below, we address each of your concerns and questions in detail:\n\n### W1: The dataset size is relatively small, which might lead to instability of the scores\n\n> Regarding relative small dataset size\n\nThanks for raising this comment. Please see point 1 in the Overall Response.\n\n> Regarding potential instability of the scores and significance analyses\n\nThis is a good point. To enhance the stability and reduce potential bias, we create three distinct prompts for each task and report the average scores from three runs with each prompt (Line 213). Here, we include additional significance analyses for the description and contradiction generation tasks on VLMs. Specifically, we consider the results from all three prompts, resulting in each model producing 1,044 outputs per task. We use t-test [1] for statistical significance analysis:\n\n1. For description generation: GPT4 is significantly better than other baselines on all metrics (p<0.0001). Claude-3 achieves the second best result and is significantly better than all open-sourced VLMs (p<0.0001). \n\n2. For contradiction generation: Similarly, GPT4 output is significantly better than other baselines on all metrics (p<0.01). However, the results between Claude-3 and LLaVA-1.6-13B  on ROUGE-2 and between Claude-3 and CogVLM on BERT score are not significant. \n\nWe will incorporate the complete results in our revised version.\n\n[1] https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html\n\n---\n\n### W2: The subjectiveness of the task and the noise of automatic metrics.\n\n> Subjectiveness of the task\n\nThis is a good point! Indeed, humor understanding can be subjective. To address this, we have designed our tasks to minimize subjectivity. Specifically, we formalized **literal description and contradiction generation** as text generation tasks because they are **less subjective** and focus on specific descriptions and narrative illustrations of the comics. For the more abstract and subjective components, such as underlying philosophy and title selection, we formulated them as **selection tasks** (i.e., determining which title is “objectively” better), and evaluated these using straightforward accuracy metrics. We will better clarify this in our revision.\n\n> The noise of automatic metrics\n\nRegarding evaluation metrics, we acknowledge that evaluations of text generation tasks remain challenging, and current automatic metrics have limitations. To mitigate this issue: (1) For semantic-based evaluations like BERT score and Rouge, we report **recall scores** to measure how many key points of the reference are captured by the model, ensuring a more precise assessment of content coverage.\n(2) To mitigate bias in GPT-based evaluations, we use different GPT variants for different purposes: for experiments, we use the gpt-4-vision-preview version as a baseline, while for evaluation, we employ the gpt-3.5-turbo-0125 version. This approach helps **reduce potential bias towards the GPT4 model’s own generation**.\n\nWe will revise our paper to better clarify these points and discuss the potential limitations of automatic evaluations in more detail. Despite the inherent challenges, we believe our approach offers valuable insights into the model's capabilities.\n\n---\n### W3: Human performance\n\nThank you for this good suggestion! We have included human performance. Due to time constraints, we randomly select 50 samples and ask two participants to perform the underlying philosophy selection and title matching tasks. One participant is male and the other is female, with cultural backgrounds from East Asia and North America, respectively, to ensure a fair and diverse evaluation.\n\nThe results are shown as below. These results highlight the significant room for improvement for VLMs on these tasks. We will revise our paper to incorporate these discussions.\n\n| Acc(%) | LLaVA-1.6-34B | Claude-3 | GPT-4 | Human |\n|-----------|---------------------|---------------|----------|------------|\n| Philosophy | 82.00 | 85.55 | 77.33 | 94.00 |\n| Title | 66.00 |68.00 | 62.00 | 93.00 |\n\n---\n\n### Q1: Details of GPT used for evaluation\n\nWe employ the gpt-3.5-turbo-0125 variant for the automatic evaluation. We will better clarify this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m22YYbaATk", "reviewer": "Reviewer_yTj4", "summary": "This paper introduces YESBUT, a new benchmark for evaluating large vision-language models' ability to understand humor and contradictions in comics with juxtaposed panels. The benchmark consists of two-panel comics with contradictory narratives, along with annotations for literal descriptions, contradiction explanations, underlying philosophies, and titles. The authors design four tasks of increasing difficulty: literal description writing, contradiction generation, underlying philosophy selection, and title matching. They evaluate several commercial and open-source vision-language models on these tasks using both automatic and human evaluation. The results show that even state-of-the-art models struggle with these tasks, especially the deeper reasoning required for understanding contradictions and abstractions. The paper provides insights into current limitations of AI in comprehending complex human expressions and offers directions for improvement.", "questions": "1. How might the performance of these models change if tested on comics from different cultural contexts or with different styles of humor?\n\n2. Have the authors considered expanding the benchmark to include comics with more than two panels, to evaluate models' ability to understand more complex narrative structures?\n\n3. Could the authors provide more insight into why decomposing the task for VLMs (by first generating descriptions) doesn't consistently improve performance, especially for the title matching task?\n\n4. Can you provide more details on the types of biases that might have been introduced during the annotation process and how they were mitigated?\n\n5. How do the models' performances correlate with their training data or pre-training approaches? Could this provide insights into what types of pre-training might be most beneficial for these tasks? \n\n6. What are some specific strategies or architectural changes you propose for improving VLMs' understanding of contradictory narratives in comics?", "limitations": "The authors adequately address the limitations of their work, particularly acknowledging the relatively small dataset size and potential ambiguity in comic interpretation due to subjectivity. They also recognize that their benchmark may not cover all aspects of visual understanding required for more generalized AI applications.\n\nThe paper does not explicitly discuss potential negative societal impacts. While the focus on humor understanding is generally positive, it may be worth considering potential misuse cases (e.g., automated generation of misleading or offensive comics) or biases that could be amplified if such systems were deployed at scale. Suggestions for improvement include:\n\n1. Expanding the dataset to include a more diverse set of comics and narrative styles.\n2. Implementing bias detection and mitigation techniques in the annotation and model training processes.\n3. Considering the ethical implications of AI-generated content in real-world applications, ensuring that it respects cultural and social nuances.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper addresses a unique and underexplored area in AI research—understanding humor in comics through contradictory narratives. This is a novel problem formulation that pushes the boundaries of current Vision Language Model (VLM) capabilities.\n\nQuality: The data collection and annotation process are rigorous, involving multiple stages of human-AI collaboration and quality checks. The experimental design is comprehensive, evaluating multiple types of models on various aspects of comic understanding.\n\nClarity: The paper is well-structured and clearly explains the motivation, dataset creation, task designs, and experimental results. Figures and examples effectively illustrate the concepts.\n\nSignificance: Understanding humor and contradictory narratives in comics is significant for advancing AI's social and semantic comprehension. This research provides valuable insights into current AI limitations in this area and offers a pathway for future improvements, which is crucial for developing socially intelligent systems.", "weaknesses": "1. The dataset size is relatively small (348 comics), which may limit the generalizability of the findings. The authors acknowledge this limitation.\n\n2. The annotation process, while rigorous, relies heavily on human judges and GPT-4, which may introduce biases. The paper does not explore potential biases in the dataset, such as cultural specificity of the humor or potential annotator biases.  \n\n3. While the paper demonstrates that augmenting models with oracle descriptions improves performance, it does not thoroughly investigate why decomposing the task for vision-language models (by first generating descriptions) does not lead to consistent improvements.\n\n4. The paper highlights the limitations of current VLMs but does not provide concrete suggestions or experiments on how to overcome these limitations. Including some preliminary experiments with potential improvements could strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful suggestions! We appreciate your recognition of our problem formulation as novel and our experimental and evaluation methods as comprehensive. We are also encouraged that you think our work can provide insights for future research. We will revise our paper and incorporate additional discussions in Limitations. Below, we address your questions and concerns:\n\n\n### W1: The dataset size\n\nPlease see point 1 in the Overall Response.\n\n---\n\n### W2 & Q4: Bias mitigation in annotation\n\nThank you for raising this good point! Please see Overall Response point 2.\n\n---\n\n### W3 & Q3: Decomposition does not lead to consistent improvements\n\nThank you for this good question. Our manual analysis identifies two potential reasons. First, VLMs sometimes **misinterpret visual content**, leading to incorrect descriptions. This issue is also highlighted in Section 6.3. Such errors can cause **cascading errors** in the subsequent deep reasoning tasks. Second, the generated descriptions are often lengthy (over 100 words), resulting in more complex input prompts, which can complicate the reasoning process for VLMs. \n\nAdditionally, the decomposition is less beneficial for title matching than philosophy. Titles are more abstract and require more in-depth reasoning, whereas decomposition of surface-level description may not suffice. We will update our paper to include examples that better illustrate these potential causes.\n\n---\n\n### W4 & Q6: Concrete suggestions and proposal on future improvements \n\nThank you for your insightful suggestion! We agree that highlighting directions for future research is crucial. Currently, we aim to uncover potential areas for improvement through our results analyses. \n\nFirst, our analysis indicates that VLMs often struggle with accurately interpreting image content and may make errors in literal descriptions (Sec. 6.1 & 6.2). This suggests a need for future work to **enhance visual interpretation capabilities**. \n\nSecond, improving the **in-depth reasoning ability** of VLMs is essential. For instance, LLaVA-1.6 significantly outperforms LLaVA-1.5, likely due to the advancements in reasoning abilities [1]. Future work might incorporate recent advanced reasoning approaches (e.g., multi-agent debate, refinement-based reasoning) to further improve model performance.\n\nFinally, our error analysis reveals that models tend to suffer from hallucination (Line 333), suggesting the need for **incorporating external knowledge** to enhance human understanding. To mitigate this problem, knowledge augmentation methods can be employed to enhance VLMs performance. \n\nWe will revise our paper to include a dedicated section that provides a more detailed discussion on these points and outlines future research directions. \n\n---\n\n### Q1: Different cultural contexts or styles\n\nThis is a good question. Currently, we focus on humor understanding based on **common interpretation** rather than individual preferences with specific contextual information. However, we recognize that understanding humor and accounting for cultural and stylistic variations is crucial. In future work, we plan to delve deeper into these nuances by incorporating more contextual information into the predictions. We will also include a more detailed discussion on this part in our revised version.\n\n---\n\n### Q2: Expanding the benchmark\n\nYes, we plan to expand our benchmark to include comics with more than two panels. This will allow us to evaluate models' ability to understand more complex narrative structures. Additionally, we will incorporate comics with diverse narrative logic types beyond contradictions. We are committed to continuously updating our benchmark to foster future research in this area.\n\n---\n\n### Q5: Correlation with pre-training\n\nThis is a very insightful question! First, we believe that the reasoning ability and world knowledge of VLMs are highly correlated with their performance on our benchmark. Two observations support this hypothesis: (1) Larger models typically outperform smaller models, and it is widely recognized that larger models tend to have better reasoning abilities and world knowledge; (2) LLaVA-1.6 significantly outperforms LLaVA-1.5, likely due to advancements in these aspects [1].\n\nSecond, social understanding ability is also highly correlated with model performance. The comics in the YESBUT benchmark mainly focus on daily life concepts, and understanding these nuances requires a deep understanding of human norms. \n\nTherefore, improving models' abilities in reasoning, world knowledge, and social understanding during the pretraining stage will enhance their performance on this task. We will incorporate this discussion into our revised version.\n\n\n[1] Llava-next: Improved reasoning, ocr, and world knowledge\n\n\n---\n\n### Suggestions on Limitations\n\nThank you for your detailed and valuable suggestions! We acknowledge the raised limitations, and we will revise our paper and incorporate your raised points. We will discuss these aspects in greater depth to ensure that our work is aligned with responsible AI practices, respecting cultural and social nuances. Thank you again for the insightful suggestions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Overall Response to All Reviewers

We thank all reviewers for their valuable comments and suggestions. We are pleased to know that the reviewers consider our benchmark **novel** (reviewer wVUt, yTj4) and **innovative** (reviewer 2fnt), with a **rigorous** human-AI collaborative annotation process (reviewer 2fnt, yTj4). We are also excited to learn that our results and analyses are considered as **comprehensive** (reviewer aFj6), **insightful** (reviewer 2fnt, yTj4), and is **beneficial** for future research (reviewer aFj6). We also thank reviewer aFj6 and yTj4 for considering our paper **clearly written** and **well presented**.

Below we address some common concerns shared by the reviewers. 

### 1. Regarding data size

We acknowledge the relatively small size of our benchmark due to the challenges and costs associated with data collection and annotation. However, as a benchmark for a novel task, we have rigorously collected and annotated each comic to ensure high quality and reliability. Despite its size, YESBUT covers a broad range of domains and topics (please see our additional analysis on topic coverage in the attached PDF). We believe it serves as a pioneering benchmark to enhance future research in this area.

Meanwhile, we are committed to updating our benchmark with more samples and additional narrative logic types beyond contradictions. We will also explore synthesizing comics using image generation models in future work.



### 2. Regarding potential bias and subjectivity of annotation

Our benchmark focuses on **common interpretation** of humor. However, we recognize that the subjectivity of this task may introduce bias. To mitigate this issue, we have taken several steps in our annotation process: 
 
1. Diverse Annotator Backgrounds: Our annotators come from different genders and diverse cultural backgrounds, including North America and East Asia, providing a range of perspectives. This diversity helps to mitigate cultural and gender biases in humor interpretation.

2. Consensus Among Annotators: The annotation process incorporates multiple quality checks and verifications to ensure consensus among different annotators. Any comics with controversy and potential bias are filtered out. During the cross verification stages, annotations identified as biased by any annotator are properly modified. This process helps reduce biases stemming from individual perspectives.

3. Verification with Social Media Comments: We also verify our annotations by checking the comments on social media for each comic. This step ensures that our annotations align with the common interpretation of the comic.

Additionally, we recognize that subjectivity is an inherent aspect of data annotation, especially for more open-ended components such as title and philosophy. Therefore, we frame these two tasks as **selection tasks**, ensuring *the correct option is “objectively” better than the negative options*. Despite these efforts, we acknowledge that our annotations may still carry inherent biases. We will further clarify this and discuss the potential biases in the Limitation section for better guidance of future usage of our benchmark.


### 3. Regarding the evaluation metrics of generation tasks

Evaluations of text generation tasks remain challenging and current automatic metrics are not perfect. Therefore, we try to mitigate this challenge from several aspects:

1. Task formalization: We have strategically chosen to formalize **literal description and contradiction generation** as text generation tasks because they are **less open-ended** and more focused on specific descriptions and narrative illustrations of the comic. For the more abstract and subjective components, such as underlying philosophy and title selection, we formulated them as **selection tasks** (i.e., determining which option is “objectively” better), and evaluated these using straightforward accuracy metrics.

2. Evaluation metrics: For semantic-based evaluation, we report **recall scores** to measure how many key points of the reference are captured by the model. This ensures a more precise assessment of content coverage. Additionally, we employ a GPT-based metric alongside a gold standard reference, an approach that has shown strong alignment with human judgment in previous studies [1,2,3].

3. Human Evaluations: We incorporate human evaluations to provide a comprehensive assessment of model output quality. This human judgement helps to capture the nuances that automatic metrics might miss, especially in understanding complex and abstract content like humor.

We will revise our paper to better clarify these points and provide a more detailed discussion of the potential limitations of automatic evaluations in NLG. Despite the inherent challenges, we believe our multifaceted evaluation approach provides a more balanced and comprehensive assessment of the model's performance.

[1] Judging LLM-as-a-judge with MT-Bench and Chatbot Arena \
[2] CLAIR: Evaluating image captions with large language models. \
[3] Geval: NLG evaluation using gpt-4 with better human alignment.

--------

We also provide a one-page pdf that includes the additional analysis on the following content:

(1) Analysis on the diversity of comic scenarios covered by our benchmark.

(2) The results of human performance compared with VLMs on 50 randomly sampled comics.


Once again, we appreciate all reviewers for your insightful comments and valuable suggestions. We will revise our paper to clarify the unclear parts and incorporate the suggestions. If our responses have addressed your concerns, please kindly consider increasing your scores. We sincerely appreciate your consideration.

Best regards,

Authors

Meta-review: Summary (taken from yTj4's excellent recap)
==============
This paper introduces YESBUT, a new benchmark for evaluating large vision-language models' ability to understand humor and contradictions in comics with juxtaposed panels. The benchmark consists of two-panel comics with contradictory narratives, along with annotations for literal descriptions, contradiction explanations, underlying philosophies, and titles. The authors design four tasks of increasing difficulty: literal description writing, contradiction generation, underlying philosophy selection, and title matching. They evaluate several commercial and open-source vision-language models on these tasks using both automatic and human evaluation. The results show that even state-of-the-art models struggle with these tasks, especially the deeper reasoning required for understanding contradictions and abstractions. The paper provides insights into current limitations of AI in comprehending complex human expressions and offers directions for improvement.

Metareview
===============

All but one of the reviews of this paper are strong accepts, and I agree with that consensus.  This is a simple but clever idea executed well.  While there are technical criticisms noted in the reviews (using BLEU for equivalence judgement) these are relatively minor and well-addressed by the author response.

I cannot put it in my Metareview easily, but I encourage the SAC to look at Figure 6 in the paper.  It's a good encapsulation of the goal of the paper and what we learn about the limitations of multimodal models.  This is backed up by the paper's thorough analysis.  In the end, I think this is the best recommendation of the paper: I found it interesting, I cannot think of anyone who has done something like this, and I learned something.

I think this paper would be a valuable addition to the NeurIPS program, so I am strongly arguing for its acceptance.

However, I would *strongly* encourage the authors to more deeply engage with copyright and fair use in the final version.  This is not just scraping from Twitter, this is scraping from a single artist's Twitter feed.  In an ideal world, you should reach out to them and explicitly get their permission to use this as a part of your dataset.
Decision: Accept (oral)