Title: VisIT-Bench: A Dynamic Benchmark for Evaluating Instruction-Following Vision-and-Language Models
Benchmark: VisIT-Bench
Summary: VisIT-Bench (Visual InsTruction Benchmark) is a benchmark for evaluating instruction-following vision-language models for real-world use. The authors curated 70 “instruction families” that they believe instruction-tuned vision-language models should be able to address.  We conduct a large-scale empirical comparison of multimodal instruction-following models using their benchmark.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Instruction following capabilities
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The format of the questions is either MCQ or binary QA (Yes/No).
They provide a collection of 70 different open-generation tasks, like reasoning over plots, object recognition, location understanding etc.
Task item definition: Each instance contains an instruction, input image(s), an instruction-conditioned caption (a human-crafted caption for the image(s)/instruction), and a human-verified reference.  Instructions are image-contextual imperative requests or questions, e.g., for an image of pancakes, a user asks “How can I cook this in a healthy way?”.
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 592
Dataset metadata available: Yes
Dataset metadata detail: instruction type, image source
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice
Metric definition: n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Elo ratings, Win rate
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: stratification of results based on the instruction category
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Instruction Following / 
New taxonomy: General Capability / Instruction Following