Title: ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs	
Summary: We introduce ConMe – a compositional reasoning (CR) benchmark and a novel data generation pipeline leveraging VLMs to produce ‘hard CR Q&A’.

Our pipeline autonomously generates, evaluates, and selects challenging compositional reasoning questions, establishing a robust CR benchmark. 

Our benchmark provokes a noteworthy, up to 33%, decrease in CR performance compared to preceding benchmarks, reinstating the CR challenge even for state-of-the-art VLMs.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Compositional reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: Compositionality Reasoning (CR) is the ability of the VLM to recognize and attend to the language concepts beyond objects (i.e., nouns), such as attributes, relations, finegrained object alternatives, and more, in both the image and text of a visual-language pair. 
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: Both integrated and sub-elements are measured
Task definition: On the collected text-image pairs, grouped by the presence of certain CR concepts, such as relations, attributes, etc., randomly “flip” the present CR concept in the positive text to form a “negative alternative” text (having the CR concept wrong). The VLM’s preference for the resulting negative is then compared to the true positive source text thus testing the VLM’s ability to entail the correct text from the image. The questions are framed as a binary multiple-choice selection.
Task item definition: A text-image pair where the text describes certain CR concepts, such as relations, attributes
Task source: LLM- and VLM- generated task examples
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 24347
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: Accuracy scores on three data partitions: Replace-Attribute, Replace-Object, Replace-Relation.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Compositional
New taxonomy: General Capability / Compositional Reasoning