Title: Are We on the Right Way for Evaluating Large Vision-Language Models?
Benchmark: MMStar
Summary: The authors review general multi-modal capability benchmarks and finds problems to do with data leakage and questions answerable without any visual input. They automatically and then manually filter instances from these benchmarks, resulting in MMStar, a "vision-indispensible" multi-modal benchmark. Evaluation and ablation studies show that MMStar mitigates leakage better than existing benchmarks.
Contribution: Second instance I've seen of a benchmark that is novel for being a filtration of existing benchmarks (after SADE (maExaminationCompositionalityLarge2024)). Both seem to prioritise construct validity in a helpful way.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: general multi-modal capabilities
Phenomenon defined by authors: Yes
Phenomenon definition: "The core capabilities consist of two perception-related dimensions, Coarse Perception (CP) and Fine-grained Perception (FP), two reasoning-related dimensions, Instance Reasoning (IR) and Logical Reasoning (LR), and two knowledge-related dimensions, Science & Technology (ST) and Math- ematics (MA)." (15)
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given an image and a question in text about that image, answer a multiple choice question. There are many possible topics, including mathematics, emotion perception, geography,  etc., and questions may require multi-step reasoning or simply visual information retrieval.
Task item definition: An image, a multiple-choice question, a correct answer, and the original benchmark from which the instance is sourced.
Task source: Modified from another benchmark (e.g. translation into another language)
Task source detail: Filtered from 6 other benchmarks
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1,500
Dataset metadata available: Yes
Dataset metadata detail: capability type, capability subtype, benchmark of origin
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall), Custom metrics: multi-modal gain, multi-modal leakage
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum, plus comparisons to scores from the base LLMs comprising the multi-modal models (called "multi-modal gain" and "multi-modal leakage" statistics)
Metric subscores: Yes
Metric subscores detail: Core capability type
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: "The evaluation samples for constructing the MMStar benchmark should meet three fundamental criteria: 1) Visual dependency. The collected samples can be correctly an- swered only based on understanding the visual content; 2) Minimal data leakage. The collected samples should minimize the risk of unintentional inclusion in LLMs’ training corpus, or be effec- tively transformed from uni-modal to multi-modal formats to prevent LLMs from ”recalling” the correct answers; 3) Requiring advanced multi-modal capabilities for resolution. In addition to ensuring fairness and reliability by adhering to the above criteria, we also aim for samples to cover various difficulty levels. We expect to comprehensively capture LVLMs’ multi-modal capabilities with succinct high-quality samples" (6)
Taxonomy: VQA / 
New taxonomy: General Capability / Visual Understanding