Title: VHELM: A Holistic Evaluation of Vision Language Models
Benchmark: VHELM
Summary: The paper's main contributions are three-fold. First, the authors identify the aspects that are both applicable to VLMs and important to evaluate from either a technological or societal perspective. Second, they assemble 21 existing VLM benchmark datasets, which are sets of image-text prompts and expected output, and map to the aspects to ensure complete coverage. Third, they standardize the evaluation procedures so that apple-to-apple comparisons can be made across the models.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: visual understanding
Phenomenon defined by authors: Yes
Phenomenon definition: This benchmark measures multiple visual phenomena in VLMs and more specifically: bias, fairness, knowledge, multilinguality, reasoning, robustness, toxicity, and visual perception
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: A scenario represents a use case for a VLM and is identified by a task (e.g., question answering, code generation, and captioning) and a usage category such as the domain, origin, language, or subject. An example scenario is “visual question answering on medical images” where the task is visual question answering and the usage category is medical images.
Task item definition: the aspect of the task, a prompt, an image, a gold response and the corresponding metric reference for this task
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 9,000
Dataset metadata available: Yes
Dataset metadata detail: aspect of the task and the respective metrics to be used
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), LLM post-processing (extracting answers, reformatting for automated scoring), Win rate
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: The results are stratified based on the used benchmarks
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: VQA / 
New taxonomy: General Capability / Visual Understanding