Title: Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!
Benchmark: UNPIE
Summary: This paper introduces the UNPIE benchmark to evaluate multimodal understanding in machines using puns. By pairing puns with explanatory images, the study tests models on tasks like grounding, disambiguation, and reconstruction, showing that visual context significantly enhances performance over text-only approaches.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: visual understanding
Phenomenon defined by authors: Yes
Phenomenon definition: Humans possess multimodal literacy, allowing them to actively integrate information from various modalities to form reasoning. Faced with challenges like lexical ambiguity in text, we supplement this with other modalities, such as thumbnail images or textbook illustrations. Is it possible for machines to achieve a similar multimodal understanding capability?
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: (1) pun grounding: to identify the specific phrase in a sentence that forms a pun
(2) pun disambiguation: to choose the translation that best matches the image provided as a pun disambiguator.
(3) pun reconstruction: to recreate the original English pun sentence using a translated version with potentially no ambiguity.
Task definition detail: Three different tasks are proposed
Task source: Expert-crafted task examples (e.g. hand-written examples), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: To generate pun explanation images, the expert annotators prompt the DALLE 3 model to create images; The paper also designs a cooperative framework between machines and humans for pun translation
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1000
Dataset metadata available: Yes
Dataset metadata detail: language
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: image
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: The report scores for different tasks (subsets) separately
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Grounding / 
New taxonomy: General Capability / Grounding