Title: ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs
Benchmark: ConMe (Confuse Me)
Summary: ConMe is a multimodal compositional reasoning benchmark that presents a novel automatic hard negative generation pipeline using VLMs. It is publicly-available, manually verification, and presents a complementary automatic analysis and error categorization pipeline. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: compositional reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: Compositional reasoning "is the ability of the VLM to recognize and
attend to the language concepts beyond objects (i.e., nouns), such as attributes, relations, finegrained object alternatives, and more, in both the image and text of a VL pair" (1).
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Task definition: A model is given an image, a correct caption, and a hard negative caption, and must choose the correct caption. 
Task item definition: A single item contains the image, the generated correct caption, and the generated hard negative caption. 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: ConMe modifies the SugarCrepe examples with their own novel text generation pipeline for VLM-created hard negatives. First, GPT-4V generates "a detailed description of the input image," used as the "ground truth" (4). Then, 4 downstream VLMs (LLaVA 1.6-7b, LLaVA 1.5-7b, InstructBLIP Flan-T5, InstructBLIP Vicuna-7b) then generate descriptions of the image. Next, GPT-4V receives the description it produced, and the descriptions of the VLMs, and is prompted "to generate multiple challenging compositional reasoning questions based on the generated descriptions" from the VLMS (4). The models then cycle through disregarding the GPT-4V generated samples that all VLM models correctly answer, and improving the kept samples to become more challenging. The authors manually verified a random subsample of 1000 samples from ConMe. 
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 24347
Dataset size detail: ConMe uses three partitions used in SugarCrepe: replace-att, replace-obj, and replace-rel hard negatives. ConMe uses the same 3846 images as SugarCrepe, but generates more examples per image across the three partitions. 
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric definition detail: The benchmark evaluates sample accuracy and perplexity. 
Metric aggregation: Simple Mean
Metric statistics: Simple mean/accuracy
Metric subscores: No
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors highlight that image-to-text matching tasks rely on "LLM only negative text generation pipeline(s)" that produces "improbable" or "outlier" captions for the given image (1). As a result, the authors claim that using VLMs in the hard negative generation pipeline is required for a more accurate benchmark of compositional reasoning in VLMS. 
Taxonomy: Reasoning / Compositional
New taxonomy: General Capability / Compositional Reasoning