Title: Large Language Models as Automated Aligners for benchmarking Vision-Language Models
Benchmark: Auto-Bench
Summary: The authors utilise LLMs to produce question-answer-reasoning triplets from COCO images. The result is Auto-Bench, a general multi-modal capability and value alignment benchmark dataset.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: "human capacities and values"
Phenomenon defined by authors: Yes
Phenomenon definition: "We adopt a capacity-oriented perspective to generate visual questions, covering a broad spectrum of perception, reasoning, planning, and value alignment" (4)

Perception: "Perception-oriented questions evaluate the model’s proficiency in comprehending, interpreting, and engaging with the objects or scenes in an image"
Reasoning: "Visual reasoning involves the ability to provide logical responses based on a holistic understanding of visual information"
Planning: "goal-directed questions that require VLMs to perceive objects in an image, understand the function of each object, and integrate the rich knowledge inherent in LLMs to achieve target goals"
Value Alignment: "aligning model behaviors with human values and preventing unintended harm or deviation from expected outcomes" (5)
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Answer a general question about an image, either multiple choice or free-form. Questions can concern compositional aspects of the image, spatial reasoning and planning, etc., or can be "unethical" requests that should be refused.
Task item definition: An image, a question with potential multiple choice options, an answer, and some given rationale for the answer.
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 28.5K
Dataset size detail: task_dataset_size_extra is the training set, task_dataset_size reports the human annotator-validated validation set
Dataset size notes: 3.504M
Dataset metadata available: Yes
Dataset metadata detail: capacity, skill, sub-skill, rationale for correct answer
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: sub-skill (e.g. counting, counterfactual reasoning, physics/biology/chemistry, privacy compliance, ...)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: No, no link is provided
Task face validity: No
Metric face validity: No
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: "To verify the rationality of our curated data, we adopt human verification for assessment... The results indicate that the data generated by Auto-Bench largely meets human acceptance in terms of both the rationality of alignment across different dimensions" (4)
Taxonomy: Alignment / Alignment
New taxonomy: General Capability / Alignment