Title: An Examination of the Compositionality of Large Generative Vision-Language Models
Benchmark: SyntActically DE-biased benchmark (SADE)
Summary: The authors explore the different failure modes across evaluation methods for image composition understanding in multimodal models. They find via ablation that a popular metric, VisualGPTScore, is biased towards syntactical correctness in the caption over image contents. They compose a new benchmark, SADE, by combining debiased subsets of existing composition understanding benchmarks.
Contribution: Exemplary paper for construct validity—the contribution consists in debiasing the evaluation of a popular task to better match the desired phenomenon.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: understanding multimodal compositionality
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given an image and 2-3 reference sentences, rank the appropriate sentence as the most likely image caption.
Task item definition: An image, multiple candidate captions for the image in English, and an index for which caption is correct.
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragarph), Log-likelihood of a given free response
Metric definition: Distribution (perplexity, calibration, correlation), recall@1
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: Conceptual task subsets: "Comprehensive, Relation, Attribute, Atomic, Negate, Content." Partially corresponding to the existing benchmarks that the authors sample (e.g. "Comprehensive" is just Winoground's group score, VL-CheckList and ARO both have "Relation" and "Attribute" subsets, etc.).
Metric access: Model access required (e.g. logits)
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: "we identify the syntactical bias that exists in current datasets for GVLMs, and define the bias with SyntaxBias Score quantitatively. We then pro- pose a SADE benchmark that mitigates the syntacti- cal bias and provides a better content understanding evaluation for GVLMs" (700)
Taxonomy: Reasoning / Compositional
New taxonomy: General Capability / Compositional Reasoning