Title: MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
Benchmark: MM-Vet
Summary: MM-Vet (MultiModal Veterinarian) is a general benchmark for vision-language capabilities, emphasising the integration of multiple capabilities per problem. Questions are sourced from "various online sources" and authors-hand annotate most of the answers. As questions are often open-ended and diverse, the benchmark uses LLM-scoring with GPT-4 as a judge.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Six core capabilities: "recognition, optical character recognition, knowledge, language generation, spatial awareness, and math" (2) and their interactions for more complex tasks
Phenomenon defined by authors: Yes
Phenomenon definition: Long bulleted definitions on pp. 3-4, but very broad, like "Recognition refers to the general visual recognition capability, including recognizing scenes, objects, object attributes (color, material, shape, etc), counting, and various other high-level vi- sual recognition tasks"
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: 16 tasks employing multiple vision-language capabilities, each a free response question. E.g., a question like "How many gallons of supreme gasoline can I get with $50?" with a corresponding image qualifies as a question employing both optical character recognition and math.
Task item definition: An image, e.g. of a sign for gas, a question, like "How many gallons of supreme gasoline can I get with $50?", and possibly multiple accepted answers, like "13.6 OR 13.7."
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 205
Dataset metadata available: Yes
Dataset metadata detail: capabilities tested
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: The authors use LLM-as-a-judge to evaluate all questions, including those with numerical answers that could be exactly matched, to "[allow] any style of model outputs to be evaluated with a unified consistent metric" (5)
Metric aggregation: Simple Mean
Metric statistics: mean and variance
Metric subscores: Yes
Metric subscores detail: capabilities tested
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: VQA / 
New taxonomy: General Capability / Visual Understanding