Title: NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples
Benchmark: NaturalBench
Summary: This paper introduces NaturalBench to evaluate vision-language models on their natural adversarial samples, i.e. samples that challenge models significantly more than humans. NaturalBench offers comprehensive skill tags to assess compositional reasoning abilities and highlights model biases in VLMs.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: visual reasoning on adversarial samples
Phenomenon defined by authors: Yes
Phenomenon definition: For discriminative tasks like visual recognition, adversarial samples are images that models misclassify. For generative VLMs trained on tasks like VQA, we define their adversarial samples as image-question pairs that humans can easily answer but models cannot.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: NaturalBench requires diverse visio-linguistic skills, such as attribute bindings, spatial/action/part relations, and advanced reasoning, including comparison
and logic. They tag each sample with all applicable skills from a defined taxonomy of 27 skills.
Task definition: They incorporate two tasks: 
- binary VQA: yes or no 
- multiple-choice VQA
Task item definition: Each sample includes two images, two questions based on these images, and corresponding gold answers that are intentionally contradictory. It also contains metadata describing the type of skill or reasoning required to answer the questions. It also contains the source dataset from which the images were taken.
Task source: Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: The authors performed a human evaluation/verification of the generated VQA pairs. 
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size notes: 1,900
Dataset metadata available: Yes
Dataset metadata detail: The samples are tagged based on a skill taxonomy the authors crafted that contains: 8 types of objects, 8 types of attributes, 3 types of relations (with spatial relation further divided into 4 subtypes), and 5 types of reasoning.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: They introduce three aggregated metrics. “question accuracy” (Q-Acc) metric to award a point only if a model correctly answers a question for both images. “image accuracy” (I-Acc) metric awards a point when a model correctly answers both questions for an image. “group accuracy” (G-Acc) metric awards one point when a model correctly answers all four pairs. 
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Language Modelling / Robustness
New taxonomy: General Capability / Reliability