Title: Are Diffusion Models Vision-And-Language Reasoners?
Benchmark: GDBench (Generative-Discriminative Evaluation Benchmark)
Summary: GDBench is a benchmark designed to assess vision-and-language reasoning in diffusion-based models using image-text matching. GDBench aggregates 8 existing datasets/benchmarks to measure text retrieval, image retrieval, and bias towards religious groups, national identity, and sexual orientation. The code and benchmark setup are publicly available. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: compositional reasoning, image-text matching, vision-and-language reasoning
Phenomenon defined by authors: No
Phenomenon definition: Vision-and-language reasoning is implicitly defined as the "fine-tuned understanding of vision and language," and the ability for models "to understand how various objects and parts compose together" (1). 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: "We present our method Diffusion Image-Text Matching (ITM). Our goal is to assign a score to an image(x)-text(w) pair (x, w) which is broadly useful for downstream applications. We provide (x, w) to the diffusion model and task it to “edit” the image according to the text. Our main intuition is if the image is not described by the text, a lot of edits are needed to fit the text, in which case it gets a low score, and vice-versa" (4). The benchmark includes "7 ability-centric" ITM tasks, and one bias task (6). 
Task item definition: A single item would be an image, and a text caption, which can either be a hard negative or a positive caption. 
Task definition detail: Broadly, the tasks of GDBench are split into image retrieval tasks and text retrieval tasks. The benchmark is designed primarily for diffusion models. 
Task source: Modified from another benchmark (e.g. translation into another language)
Task source detail: The 8 distinct tasks are based on 8 existing datasets (Flickr30k, Winoground, Aro, ImageCoDe, SVO, CLEVR, Pets) instead of the creation of an entirely new dataset. The GDBench then reports scores on the sub-elements per sub-part of the subsumed benchmarks. 
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Generated image
Response format detail: The model must edit the provided image until it matches the provided positive or negative caption. 
Metric definition: Distribution (perplexity, calibration, correlation)
Metric definition detail: The paper defines a custom metric for the normalized image retrieval error on page 5. The metric is intended to measure the "relative difference of how much easier or harder it becomes to denoise the image with a given text relative to when no text is given" (5). The metric is intended for diffusion models. The paper also measures the religious, nationality, and sexual orientation biases in the image outputs using effect size, or the defined "normalized association score" on page 7. 
Metric statistics: Image retrieval error, effect score bias
Metric subscores: Yes
Metric subscores detail: SVO: Verb, Subj, Obj; ImageCoDe: Static, Video; ARO: VG Attr., VG Rel., COCO Ord., Flickr Ord. Bias: Religion, Nationality, Sexuality
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors highlight that image-text-matching (ITM) tasks have "become a standard paradigm for diagnostic vision-and-language datasets" because they enable "interpretable evaluation on many downstream skills" (6). Thus, the authors implicitly position the dataset as a valid construct to measure vision-and-language reasoning. 
Taxonomy: Grounding / 
New taxonomy: General Capability / Grounding
Other notes: Though the benchmark aggregates existing datasets, it is included due to its contribution in creating a more extensive and comprehensive benchmark and its extension of image-text matching tasks to diffusion models.