YesBut: A High-Quality Annotated Multimodal Dataset for evaluating
  Satire Comprehension capability of Vision-Language Models
Authors: name: Abhilash Nandy; name: Yash Agarwal; name: Ashish Patwa; name: Millon Madhur Das; name: Aman Bansal; name: Ankit Raj; name: Pawan Goyal; name: Niloy Ganguly
Published: 2024-09-20 15:45:29

Abstract: Understanding satire and humor is a challenging task for even current
Vision-Language models. In this paper, we propose the challenging tasks of
Satirical Image Detection (detecting whether an image is satirical),
Understanding (generating the reason behind the image being satirical), and
Completion (given one half of the image, selecting the other half from 2 given
options, such that the complete image is satirical) and release a high-quality
dataset YesBut, consisting of 2547 images, 1084 satirical and 1463
non-satirical, containing different artistic styles, to evaluate those tasks.
Each satirical image in the dataset depicts a normal scenario, along with a
conflicting scenario which is funny or ironic. Despite the success of current
Vision-Language Models on multimodal tasks such as Visual QA and Image
Captioning, our benchmarking experiments show that such models perform poorly
on the proposed tasks on the YesBut Dataset in Zero-Shot Settings w.r.t both
automated as well as human evaluation. Additionally, we release a dataset of
119 real, satirical photographs for further research. The dataset and code are
available at https://github.com/abhi1nandy2/yesbut_dataset.