Title: Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy
Benchmark: OVQA
Summary: This paper propose a novel VQA benchmark based on well-known visual classification datasets which allows a granular evaluation of text-generative vision-language models and their comparison with discriminative vision-language models. To improve the assessment of coarse answers on fine-grained classification tasks, the authors suggest using the semantic hierarchy of the label space to ask automatically generated follow-up questions about the ground-truth category. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Open-ended visual understanding
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Authors' description is unclear
Definition integrity detail: Not applicable
Phenomenon contested: Not defined
Task definition: Open-ended Visual Question Answering (oVQA), which tests vision-language models (VLMs) on their visual understanding by asking questions via natural language. Unlike multiple choice VQA, where answers can be chosen from a predefined set of options, oVQA requires the model to generate the answer rather than simply choosing the option with the highest score.
Task item definition: Image, question and a golden answer
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 95,864
Dataset metadata available: Yes
Dataset metadata detail: attribute type
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), - Contains: A less restrictive option is to consider a response correct if the prediction contains the true class name after preprocessing - ClipMatch: matching the prediction and label using cosine similarity in a vector embedding space
Metric aggregation: Simple Mean
Metric statistics: simple mean and variance
Metric subscores: Yes
Metric subscores detail: Stratification of results based on the used VQA benchmarks
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: There is no specified phenomenon besides the models' ability to answer open-ended questions.
Metric face validity: Yes
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: VQA / 
New taxonomy: General Capability / Visual Understanding