Title: SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality
Benchmark: SugarCrepe
Summary: We introduce SUGARCREPE, a new benchmark for vision-language compositionality evaluation. 

We employ large language models, instead of rule-based templates used in previous benchmarks, to generate fluent and sensical hard negatives, and utilize an adversarial refinement mechanism to maximally reduce biases.' (abstract)
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Compositional understanding of vision-language models
Phenomenon defined by authors: Yes
Phenomenon definition: Through compositional reasoning, humans can comprehend new scenes and describe those scenes by composing known atoms. 

For instance, compositionality allows people to differentiate between a photo of “a girl in white facing a man in black” and “a girl in black facing a man in white”. 

Vision-language research has sought to develop models that can similarly comprehend scenes and express them through compositional language.' (page 1)
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: This is formulated as an image-to-text retrieval task. 

It evaluates a vision-language model’s ability to distinguish the correct caption for an image from a closely matched, compositionally altered hard negative, thereby testing its compositional understanding of visual scenes.

Task item definition: An image paired with two captions: one positive caption that correctly describes the image, one hard negative caption that is compositionally similar but incorrect. 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 7512
Dataset metadata available: Yes
Dataset metadata detail: It provides types of compositional perturbations applied to generate hard negatives. These include REPLACE-OBJ, REPLACE-ATT, REPLACE-REL, SWAP-OBJ, SWAP-ATT, ADD-OBJ, and ADD-ATT, which specify the type of atomic concept (object, attribute, relation) and the operation used (replace, swap, add)​. 
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice
Response format detail: Multiple choice as each image presents two captions (a positive and a hard negative), and the model choose which one correctly describes the image. 
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric subscores: Yes
Metric subscores detail: Seven fine-grained hard negative types under three categories of REPLACE, SWAP, and ADD:

REPLACE: REPLACE-OBJ (object substitutions), REPLACE-ATT (attribute substitutions); REPLACE-REL (relation substitutions);

SWAP: SWAP-OBJ (object swaps), SWAP-ATT (attribute swaps);

ADD: ADD-OBJ (adding an object), ADD-ATT (adding an attribute).
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The benchmark was proposed to address the construct validity failures in prior benchmarks (e.g., ARO, CREPE) by showing that a blind model can even succeed without using visual input.

They address it in their benchmark by improving hard negative generation using ChatGPT, followed by adversarial refinement, to ensure captions differed only in compositional meaning and remove any biases. 
Taxonomy: Reasoning / Compositional
New taxonomy: General Capability / Compositional Reasoning