Title: When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?
Benchmark: ARO
Summary: This paper creates the Attribution, Relation, and Order (ARO) benchmark to systematically evaluate the ability of VLMs to understand different types of relationships, attributes, and order information. They demonstrate that VLMs can perform well on image-text retrieval over existing datasets without using the composition and order information.
Contribution: The authors propose a simple finetuning method that improves model understanding of attributes and relations by introducing two types of composition-aware hard negatives: visually similar images to emphasize fine-grained differences, and captions with scrambled word order to enforce sensitivity to syntax. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Compositional understanding in VLMs
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Not defined
Task definition: ARO consists of Visual Genome Attribution, to test the understanding of objects’ properties; Visual Genome Relation, to test for relational understanding; and COCO-Order & Flickr30k-Order, to test for order sensitivity in VLMs.
Task item definition: A sample would be an image, 1 true and 1 false statement about the image, the two objects presented in the image, the attributes of the objects
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 28,700
Dataset metadata available: No
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: macro-accuracy
Metric subscores: No
Metric subscores detail: Stratification based on the four introduced tasks: 1) Visual Genome Attributions, 2) Visual Genome Relations, 3) COCO Order and 4) Flickr30k Order
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Compositional
New taxonomy: General Capability / Compositional Reasoning