Title: Cola: A Benchmark for Compositional Text-to-image Retrieval
Benchmark: COLA
Summary: This paper looks at compositional visual reasoning in LLMs, presenting the COLA benchmark which targets text-to-image retrieval to compose objects with localized attributes. Strategies to adapt pre-trained vision-language models for compositional reasoning are assessed, and the authors find training with multimodal layers to be highly promising. COLA is compared to the CREPE benchmark, demonstrating greater difficulty than this contemporary counterpart.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: compositional reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: "Compositionality is a fundamental characteristic of human intelligence, allowing us to elicit 'the meaning of the whole [as] a function of the meanings of its parts'. In vision, the whole is an image made up of visual elements like objects and attributes. Recent work has consistently identified that this type of compositionality—that between objects and their attributes—is something existing vision-language models struggle to represent."
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Given a query and set of objects, associate the objects in the query with the correct attributes and ignore difficult distractor compositions where the query attributes are attached to distractor objects.
Task item definition: In the multi-object setting, each sample is represented as a pair of images and captions. In the single-object setting, samples are represented by an image and a dictionary of objects in the image and relevant attributes. Additional 0/1 labels indicate whether each of 320 label classes is present in the image, along with similar labels indicating whether the image is counted within a difficult set for each label class.
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: Unclear how to compute based on description in the text
Dataset size notes: Unclear how to compute based on description in the text
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: data source, single-object compounds, multi-object compounds
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Compositional
New taxonomy: General Capability / Compositional Reasoning