Title: SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality
Benchmark: SugarCrepe (Synthetic yet Unbiased Generation with Adversarially Refined Compositional REPresentation Evaluation)
Summary: SugarCrepe is a multimodal benchmark for multimodal compositional understanding. The benchmark specifically ensures that all hard negative (incorrect) descriptions in multiple-choice image-to-text retrieval tasks are fluent and plausible. The benchmark is publicly available, and the data was manually reviewed for quality control. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: compositionality, compositional understanding, compositional reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: Compositionality is the fundamental presupposition characterizing uhman perception and linguistic processing, that enables humans to comprehend new scenes and describe those scenes by composing known atoms. 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: SugarCrepe isolates three domains of hard negative types: Replace, Swap, and Add. Each domain has sub-types. Replace hard negatives are Replace-Obj, Replace-Att, and Replace-Rel. Swap hard negatives are Swap-Obj and Swap-Att. Add hard negatives are Add-Obj and Add-Att. 
Task definition: Models are given an image, and two descriptions, and must choose the correct description and avoid choosing the incorrect description, termed a "hard negative." 
Task item definition: A single item would contain the image, the correct description, and the hard negative
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: SugarCrepe uses image description pairs from COCO. It then generates sensical and fluent hard negatives using an LLM (ChatGPT), filters incorrect hard negatives with human validation,  and then de-biases the dataset with adversarial refinement. 
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The task may be representative or real task, but the paper does not present any literature grounding the validity of the task beyond its use in current compositionally benchmarks. 
Dataset size: 7512
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Multiple choice
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: The paper defines two custom metrics, where they use two blind models Vera and Grammar. The Vera score gap is the score difference between the positive and hard negative texts: V(T^p) - Vera(T^n). The Grammar score gap is defined by Grammar(T^p) - Grammar(T^n). The paper refers to the Vera score gap as the commonsense score as well. 
Metric aggregation: Simple Mean
Metric statistics: Reports average scores for commonsense Vera score gap and Grammar score gap. The paper also reports the pairwise better ratio between SugarCrepe and ARO+CREPE. 
Metric subscores: Yes
Metric subscores detail: Replace, Swap, Add
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: The Vera and Grammar models may be well-established and commonly used in compositionality or linguistic tasks, but it is not apparent in the paper. No justification is provided for the use of Vera and Grammar.
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: The papers justify the improvement of the task design displayed in their benchmark, but not the choice of the task itself.
Author validity detail: The authors highlight that current image-to-text compositionality benchmarks are biased in using implausible or non-fluent hard negatives, allowing blind/language-only models to pass the multimodal task. However, the authors do not justify the choice of image-to-text retrieval task formulation, besides its use in current compositionality benchmarks. 
Taxonomy: Reasoning / Compositional
New taxonomy: General Capability / Compositional Reasoning