Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy
Authors: Simon Ging
Maria Alejandra Bravo
Thomas Brox
Venue: ICLR 2024 spotlight (2024)
Keywords: Open-ended VQA
benchmark
Vision-Language
VL
Vision-Text
VLM
Vision-Language models
Image classification
Visual question answering
Text-generating VLM
Primary area: datasets and benchmarks
TL;DR: We evaluate Vision-Language models by asking them open-ended questions about existing datasets like ImageNet.

Abstract: The evaluation of text-generative vision-language models is a challenging yet crucial endeavor. By addressing the limitations of existing Visual Question Answering (VQA) benchmarks and proposing innovative evaluation methodologies, our research seeks to advance our understanding of these models’ capabilities. We propose a novel VQA benchmark based on well-known visual classification datasets which allows a granular evaluation of text-generative vision-language models and their comparison with discriminative vision-language models. To improve the assessment of coarse answers on fine-grained classification tasks, we suggest using the semantic hierarchy of the label space to ask automatically generated follow-up questions about the ground-truth category. Finally, we compare traditional NLP and LLM-based metrics for the problem of evaluating model predictions given ground-truth answers. We perform a human evaluation study upon which we base our decision on the final metric. We apply our benchmark to a suite of vision-language models and show a detailed comparison of their abilities on object, action, and attribute classification. Our contributions aim to lay the foundation for more precise and meaningful assessments, facilitating targeted progress in the exciting field of vision-language modeling.

Published as a conference paper at ICLR 2024
OPEN-ENDED VQA BENCHMARKING OF VISION- LANGUAGE MODELS BY EXPLOITING CLASSIFICATION DATASETS AND THEIR SEMANTIC HIERARCHY
Simon Ging
María A. Bravo
Thomas Brox
{gings, , bravoma, brox) uni mmi--reiibugg.e University of Freiburg, Germany
/ttps://github... wwt..commmmmmmmmmm... f frei burg/ovqa
ABSTRACT
The evaluation of text-generative vision-language models is challenging yet crucial endeavor. By addressing the limitations existing Visual Question An- swering (VQA) benchmarks and proposing innovative evaluation methodologies, our research seeks to advance our understanding of these models' capabilities. We propose novel VQA benchmark based well-known visual classification datasets which allows a granular evaluation of text-generative vision-language models and their comparison with discriminative vision-language models. To im- prove the assessment of coarse answers on fine-grained classification tasks, we suggest using the semantic hierarchy of the label space ask automatically gen- erated follow-up questions about the ground-truth category. Finally, we compare traditional NLP and LLM-based metrics for the problem of evaluating model pre- dictions given ground-true answers. We perform human evaluation study upon which we base our decision on the final metric. We apply our benchmark to a suite of vision-language models and show a detailed comparison of their abili- ties on object, action, and attribute classification. Our contributions aim to lay the foundation for more precise and meaningful assessments, facilitating targeted progress in the exciting field of vision-language modeling.
INTRODUCTION
Image-text generative models have advanced artificial intelligence by connecting visual and lan- guage domains. Recently, such models have been scaled up successfully in terms of model size and training data, showing impressive capabilities across various applications. One such capability is open-ended Visual Question Answering (oVQA), which tests vision-language models (VLMs) on their visual understanding by asking questions via natural language. Unlike multiple-choice VQA, where answers can be chosen from a predefined set of options, oVQA requires the model gen- erate the answer rather than simply choosing the option with the highest score. We refer to such text-generative VLMs as Text-VLLe VLMs.
The unconstrained nature of oVQA presents a significant challenge during evaluation. Firstly, natu- ral language has a wide variety of possibilities to express the same content. There exists an inherent difficulty in comparing the semantic similarity of two language expressions, rendering it hard to say whether model's answer is right or wrong. Current automatic metrics do not capture the richness of model responses, especially when multiple valid answers, synonyms, paraphrasing, explanations are possible (Chen et al., 2020 Khashabi et al. (022). They especially penalize longer responses, leading to a bias towards methods that produce shorter answers (Risch et al. (021).
nuanced
Secondly, there is typically ambiguity in a question, which depends on the context provided and the expected scope of the response, resulting in multiple valid answers that may not match the annotated ground truth. For example consider, when asking about the type of animal in a picture, model may answer dog. but the ground truth expects the more detailed newfoundland dog as the answer. A natural way of solving this ambiguity is by following up with the model's first response
'These authors contributed equally this work.

---

Published as a conference paper at ICLR 2024
What's this?
A black dog standing in the water
Parent Hierarchy
dog domestic dog
canine / canid
camiivore
placental
mammal mammalian
entity
What's this?
(Full image)
What type of dog is this
A tree with no leaves
(Cropped image) porcupine
Newfoundland dog
(a) Cropping the biggest object guides the model to the location the question.
(b) Asking follow-up question using the concept hierarchy helps to resolve ambiguities.
Our proposed framework for building open-ended VQA benchmark using visual guid- ance for question context and probing for details with follow-up questions.
Figure I:
and asking what type of dog it is. This follow-up mechanism is closely related to Image-text dialog in which systems engage in conversations with users about images (Das et al. 2017). It allows interleaving text and image inputs to create visual conversations and enables in in-context few-shot learning (Alayrac al., 2022). However, these systems currently do not seek clarification about missing information in the conversation,
human would do when a question is unclear.
Finally, current VQA benchmarks only provide a single aggregated performance metric as out- put and coarse division into question and answer types based on heuristics. While this format convenient for comparing different methods, aggregated metries conceal information about the true strengths and limitations of models. This problem is amplified by biases in the distribution of question-answer pairs that lead methods to use simple statistical shortcuts, e.g. image-blind algo- rithms often perform well in VQA (Kafle et al. 2019). As result, classical VQA benchmarks have reached a certain saturation where models achieve high overall performances, with minor differences between them. Nonetheless, find that models still have clear deficits, as seen in a qualitative in- spection of the results, especially in out-of-distribution scenarios. These deficits are hard to explain with single performance metric, and qualitative comparison on thousands of images is impractical.
In this work, we introduce ways to approach these challenges. To provide more details about the performance of model different aspects of VQA, we introduce dedicated subbenchmarks for questions about objects, actions, and attributes, corresponding to nouns, verbs, and adjectives. To this end, we leverage existing classification benchmarks, from which we automatically generate questions and ground truth answers. Additionally, this allows us to compare Text-VLMs with dis- criminative VLMs that directly classify the image without generating text.
We propose follow-up procedure to reduce question ambiguity and guide models toward the level of detail expected by the ground truth. When adapting classification benchmarks for oVQA, we address this ambiguity by initially cropping the image to provide the model with the appropriate visual context for the target. Subsequently, after receiving an initial answer from the model, we generate a follow-up question using concept hierarchies to enable the model to generate answer with the desired granularity.
To select an appropriate metric for evaluation, we compare various metrics from the VQA and NLP communities that consider paraphrasing and synonyms as correct answers. We validate the selected metrics by numan-judgmen study of the answers as gold standard metric. Finally, we evaluate several state-of- the-art VLMs on our extended VQA benchmark and provide an in-depth analysis of the strengths and weaknesses identified.
2

---

Published as a conference paper at ICLR 2024
100
d BLIP BEX+-VLMUI InstructBLIP BBBLIP-2EE LLaVA Upper Bound
80
Accuracy
60
40
20
VQAv2
GQA
Attribute- VQA Object-oVQA (OVAD)
Object-oVQA Activity-oVQA (ImageNet) (ActivityNet)
(COCO)
Datasets
Figure 2: Comparison of Models on open-ended VQA datasets, testing traditional VQA, Object- oVQA, Act, and AttributeooQQA. Generic pretrained models like BLIP-2 are stronger on predicting objects and activities with high semantic granularity while exhibiting lower perfor- mances for Attribute-oVQA and classical VQA tasks. Contrary, VQA pretrained models are stronger on predicting attribute concepts and answering generic VQA questions but perform poorly for fine- granularity of nouns and activities.
2 RELATED WORK
Large VLMs have substantially advanced in various multimodal tasks, including visual question answering and multimodal conversation. Recent studies have introduced more efficient vision-text interactions, enhanced training techniques, and incorporated instruction tuning into VLMs, such as Llama-adapter v2 (Gao et al. (003)) and LLaVA (Liu et al.] 2023). Other notable VLMs in- clude Flamingo (Alayrac al. [2022), which integrates visual features into Large Language Models (LLMs) through cross-attention layers, and BLIP-2 (Li et al. (2023)), which pre-trains a module for zero-shot image-to-text generation, achieving improved performance image captioning and VQA.
Evaluating the performance of such VLMs poses significant challenge, with no uniform standard for testing. For instance, [Liu et al. (003)) assess VLMs by generating predictions for few manually curated image-question pairs and comparing them to a reference prediction generated by GPT-4. Bai et al. (023) focus on evaluating conversational and visual storytelling abilities, employing a manually created visual dialogue dataset. Their evaluation methodology compares model outputs to detailed image annotations, revealing insights into model performance relative to human judg- ments. However, these evaluation criteria are not standardized. A quantitative comparison between a pretrained Text- VLM like BLIP-2, instruction tuned models like LLaVA, and retrieval models like CLIP is not feasible since benchmarks are only applicable to the respective tasks designed for such types of models. Improvements in this area would ease model selection for applications and enable more targeted research to overcome models' previous weaknesses.
great interest has been shown in the VQA task. Antol et al. (2015) create the first free-form and open-ended Visual Question Answering dataset by asking human workers create questions that smart robot probably can not answer and then collect ten human answers per question. The follow- up work VQAv2 (Goyal al., 2017 balances the previous dataset to reduce statistical biases in the answer distribution. Opposed to leveraging human workers, the GQA dataset (Hudson & Manning, 2019) uses scene graph structures from Visual Genome (Krishna et al. 2017) generate question- answer pairs. These graph structures enable the authors to balance the answer distributions for each question, reducing the efficacy of depending on answer statistics. Both datasets remain widely used today (Dai et al. 2023 Zeng al. 2022 Alayrac al. 2022).
In contrast to Text-VLMs, assessing LLMs has been a focal point of research within the Natu- ral Language Processing (NLP) community. Diverse methods, including LLaMA (Touvron et al.) 2023a), GPT-3 (Brown et al. 2020), Chinchilla (Hoffmann et al. 2022). and PaLM (Chowdhery | 2022), trained on large amounts of text and evaluated in zero-shot few-shot scenarios. These evaluations traverse commonsense reasoning, question answering, reading comprehension, mathematical reasoning, and code generation. Comparing candidate and reference text has been
3

---

Published as a conference paper at ICLR 2024
extensively studied in the context of machine translation (Mathur et al. 2020 Koemi et al., 2021
and image captioning (Cui et al. 2018). Traditional automated metrics like BLEU (Papineni et al.
2002), ROUGE (Lin 2004), METEOR (Banerjee & LLvie 2005), CIDEr VVedantam et al.) 2015
and SPICE (Anderson et al. 2016) aim to approximate human judgments, using heuristics in the text space. Some follow-up works like BLEURT (Sellam al.. 2020), LERC (Chen et al.) 2020) and BEM (Bulian et al. 2022) propose learning metrics in a supervised manner on human ratings, while others like BertScore (Zhang et al., 2020) use feature spaces from transformers to compare text. However, these evaluation efforts mostly apply to comparing longer sentences. As we show in this work, they do not improve over simple text comparison when presented with very short ground truth answers and mixed length model predictions.
recent focus has been assessing LLMs' capabilities in human-like dialogues like chatbots. Chiang et al. (2023) evaluate Vicuna, their chatbot, by pinning it against another assistant and having GPT-4 compare and rank both model outputs. Zheng et al. (2023) show the agreement between GPT-4 and humans to be comparable to the inter-human agreement on their MT-bench data, which consists of 80 multi-turn questions. Despite these extensive evaluation efforts, challenges such as overfitting to benchmarks and training distribution biases, capturing correctly semantic similarity between synonyms and paraphrases remain partially addressed.
OPEN-ENDED VQA BENCHMARK
3.1 TRANSFORMING CLASSIFICATION DATASETS TO VQA
The main goal of this work is to assess TeXI-VLMs' performance in an open-ended VQA (oVQA) setup. To achieve this, we transform traditional image classification datasets into a VQA format to create new, fine-gre fine-grained VQA subbenchmarks. We consider two types of image classifica- tion datasets, multi-class classification and binary classification, to build our oVQA benchmark. Supp. Fig. D shows an example for the proposed oVQA datasets and 2 classical VQA datasets.
Multi-class classification. In multi-class classification, the goal is to predict the correct class index t E (1...., C}} from a set of classes, given an input image Similar to other works that test VLMs for classification, such as Radford et al.) (202T), we use the class names to enable zero-shot classification. For every image-class pair (I, e), we generate a question such that Text-VLM can produce the corresponding class name.
Binary classification. In binary classification, the task is to predict whether specific categories e apply to each image not. To address this task in oVQA, consider only the positive categories for each image I to generate relevant questions for the TLM.
Visual guidance for question context. Consider the example depicted in Fig. Da from the ImageNet dataset. Imagine you want a EeMt-VLM to identify the ImageNet category by asking a very general question, such as "What's this?" Unfortunately, the task becomes ambiguous if given such broad question and the entire image as context. A Text-VIA might provide a very general response, such as "a tree with no leaves." Although it is not a wrong prediction, it is incorrect given the "porcupine" label. To resolve this ambiguity, we crop the object of interest given the ground-true box.
Metries. To evaluate the correctness of an answer, it is required to compare a model's output with a class name. We begin with the simplest evaluation metric, called ExactMatch (EM). It involves preprocessing the model's output p. a free-format text that resembles natural language with human- like fluency, and the corresponding class name c by removing non-al aon-alphanumeric characters, applying lowercase conversion, and normalizing spaces. For EM a prediction is considered correct only if the model's output exactly matches the correct class name. A less restrictive option is to consider response correct if the prediction contains the true class name after preprocessing (Xu et al. 2023). named as Contains (Cont). We use both metrics to evaluate Text-VLMs and extend both metrics in the presence of category synonyms, considering the prediction as correct if any one of the synonyms is correct under each of the metrics. We denote these extensions as EM Syn and Cont Syn.
For multi-class classification, an image belongs to only one of the categories C. Therefore, we adopt the evaluation introduced by Conti et al. (2023). We report matching the prediction and label using cosine similarity in a vector embedding space and name this metric as ClipMatch (ClipM). For this, we embed the model prediction and each class name using EVA-Clip (Fang et al. 2023
4

---

Published as a conference paper at ICLR 2024
Subsequently, we consider the prediction as correct if the vector corresponding to the correct class exhibits the highest similarity (ClipM@1 1) or is within the top-5 similarities (ClipM@S5)
Probing for details with follow-up question. The expected specificity of the task is typically unclear when asking in an open-ender way. Take, for example, Fig. When asking "What's this?", the model might respond with "A black dog standing in the water," which is an accurate answer, but not the specific category in the ground truth. Therefore, we propose a follow-up procedure enhance the precision of model responses in oVQA, especially for fine-grained questions. We assume a hierarchy of parents for each class as shown in Fig. Db. First, we consider all data points where the model response judged as incorrect by the ClipM @ / metric. Then, we collect all parent nodes in the hierarchy of the ground truth label and calculate their similarity to the model prediction using EVA-Clip. We define the parent with the highest similarity as the candidate for the follow-up. In the example, we compare the prediction and the parent class of "Newfoundland dog". We find the prediction to be sufficiently close to the parent "dog" and ask the follow-up question "What type of dog is this?"
Our goal to only ask follow-up questions about concepts that the model already described. There- fore, we only ask a follow-up question about the parent if the similarity with the prediction is at least When the similarity is lower than 6, we instead define the parent as generic term like "object" or "activity", depending
the task, in order to avoid disclosing part of the ground truth information. This systematic procedure, rooted in lexical and semantic similarity assessment, refines the model's responses in oVQA enhancing precision and contextuality.
Upper bound. In order to compare ELXs with discriminative VLMs and infer how far away they are from each other, we evaluate discriminative VLMs in the same multi-class classification VQA datasets using zero-shot retrieval. It is important to note that generating the correct class name given the visual input in the language space considerably harder than generating good visual embedding vector representing the visual concepts. Similar to Conti et al. (2023), we consider the retrieval performance as an upper bound for our evaluation.
3.2 DATASETS AND EXPERIMENTAL SETUP
Object-o VQA. For the task of object classification oVQA, we consider two well-known datasets: MS COCO for the common categories with coarse granularity and ImageNet for the fine granularity.
The COCO (Common Objects in Context) (Lin et al. 2014) dataset is a widely used dataset designed for object recognition. contains 80 object categories easily recognizable by a 4-year-old, such as "dog", "person", "boat", etc. We crop each of the 36,781 objects from the validation 2017 set and consider them individually. Since the object categories already have a coarse granularity, we do not ask follow-up question for the COCO-oVQA task.
ImageNet (Deng et al. 2009) is a standard benchmark for image classification (Russakovsky et al, 2013). Its validation set contains 50000 images from 1000 classes, including simple labels such as "broom" and fine-grained labels like "Lakeland terrier" and 22 other types of "terrier". For each image, we crop the biggest bounding box. We define follow-up questions using the WordNet (Miller) 1994) hierarchy and ask "What type of object is this?" by replacing "object" with the closest parent category.
ACt. We use the ActivityNet (Caba Heilbron et al., 2015) dataset and the activity cat- egory hierarchy provided by the original paper. consists of 4,926 untrimmed validation videos belonging to 200 activity classes. To build the ActivityNet-o VQA task, we use the annotated video segments (in which the activity is happening) from the validation set and extract the middle frame. This way, we obtain 7,654 frames within 200 activity categories in a multi-class classification setup. This dataset considered as having fine-grained categories since the activity categories are very spe- cific. Therefore, we follow similar procedure in ImageNet-oVQQ. general question and create follow-up questions when the first answer is not hitting the exact class label. We select the parent node most similar to the model's prediction to form the follow-up ques- tion. Specifically, we ask, "What type of activity is this?" by replacing the word "activity" with the closest parent category.
Initially, ask the model
Attribute-oQQA. For the attribute classification, we use the OVAD dataset, introduced by Bravo al. 2023), which focuses on common adjective concepts like color, material, length, and others.
5

---

Published as a conference paper at ICLR 2024
Table 1: Zero-shot retrieval results for the evaluated multi-class classification datasets using discrimina- tive VLMs on ImageNet (cropped), COCO objects and ActivityNet.
Table 2: OVAD-oVQA (cropped object bounding boxes) accuracy. Results over three different questions depending on the attribute type. See Supp. Fig. for the spe- cific questions.
INet-Crop
COCO obj
ActivityNet
Model
R@1
R@5
R@1
R@5
R@1
R@5
Method
Cont Syn +
EM Syn 1
BLIP wis
45.70
5.46
36.99
5.00
CLIP
ViT-L-114
77.90
95.30
55.20
85.35
68.76
89.27
xL B
43.01
7.24
37.26
6.68
X--VLMap
44.56
6.33
39.13
5.62
CLIP EVA 01-g-14
81.60
96.62
62.81
89.67
71.14
89.98
BLIP-2 OPT
21.89
1.41
5.23
0.52
BLIP-2 TS
33,45
0.94
15.73
3.69
BLIP-2
69.20
89.70
57.16
89.04
61.03
82.27
InstructBLIP TS
43.29
0.53
38.88
2.28
BLIP-2
InstructBIIIP V
42.92
6.33
40.42
5.81
(COCO ft)
65.21
87.05
62.98
89.04
59.59
81.36
LLaVA
36.02
1.29
0.00
0.00
The OVAD dataset contains object-level attribute annotations for 117 attribute categories and 80 object classes over 2000 images. Attributes are organized in a two-level hierarchy such that every attribute is under a parent attribute type. We crop every object using the ground-truth bounding box as in COCO-oVQA and consider it an independent visual input. To formulate questions for the VQA task, we employ the parent attribute types within the hierarchy to inquire about specific attributes effectively. For instance, if the ground-truth annotation of person specifies "green" under the "clothes color" attribute type, we construct the question, "What colors are the clothes the person wearing?"' A complete list of questions is included in the supplementary. We employ this approach to generate questions for each positive attribute annotated in the OVAD dataset. The final OVAD- oVQA dataset consists of 122,997 object-attribute--uesiio tuples. During evaluation, we consider the synonyms for every positive attribute category as correct answers. Since the OVAD-oVQA consists of binary classification problem, we employ the metrics EM Syn and Cont Syn.
Classical VQA. To link traditional VQA evaluation with our proposed benchmark, we evaluate on two existing VQA datasets. The VQAv2 dataset (Goyal et al. 2017) aims to test systems for detailed image understanding and complex reasoning. We evaluate the validation split with 214K questions and the test-dev split with 104K questions. For test-dev, the ground truth answers reserved, and evaluation is done by submitting model predictions online. For each question, the dataset contains 10 human-annotated answers. We follow the authors and calculate our metries as follows. Given model prediction P and human answers h2 E 1..... 10, the final score for the prediction min (1, 0.3 Σ, metric(P, h₂)) e.g. for ExactMatch (EM), if humans agree with the model, the score is 90%. We utilize the same text preprocessing as the VQAv2 dataset work. The GQA dataset (Hudson & Manning 2019) uses scene graph structures from Visual Genome (Krishna et al., (017) to generate question-answer pairs. Evaluating all models on the "balanced-testder" split, our analysis encompasses 13k questions across 400 images. The preprocessing mirrors that of the VQAv2 dataset. Opposed to VQAv2, only one ground truth answer exists per question, report EM and Cont metrics between this answer and the model prediction.
3.3 TESTED VLMS
We compare state-of-the-ar models in classical VQA and recent conversational models. We selected two models fine-tuned on VQAv2, BLIP (Li et al. 2022) and X2 X2- (Zeng et al. 2022); one multi- purpose model evaluated in a shoo manner BLIP- (Li et al. 2023b); and two conversational models InstructBIIIP Dai et al. 2023) and LLAVA (Lu et al. 2023). All models consist of an image encoder and text encoder based on the transformer architecture as well as fusion model. They use pretrained models for the unimodal encoders and train the fusion model using image-text data. BLIP and X3-VLM fine-tune the encoders together with the fusion module using common losses such as Image- Text Contrastive Loss (ITC), Image- Text Matching Loss (ITM), and image- conditional Language Modeling Loss (LM). BLIP-2, InstructBLIP, and LLaVA keep the unimodal encoders frozen and only train the fusion module. Part of the training data of InstructBLIP includes the VQAv2 dataset. We report the models' details in the Supp. Appendix A.I

---

Published as a conference paper at ICLR 2024
Table 3: ImageNet-oVQA (cropped bounding box) object classification accuracy. Top: Results for
asking the questions "What can be seen in the image?", "What is in the image?" and "What's this?" Bottom: Results for asking follow-up question "What type of object is this?" response to the output of the first question. We highlight the first, second and third best method.
Method
ClipM@1
=
ClipM @5 +
Cont +
EM +
BLIP (vqa finetumed)
24.51
0.95
43.83
1.70
10.22
0.85
9.24
1.29
X²- B
21.17
1.83
40.55
2.89
8.24
1.42
7.80
1.57
1s Question
XLg L
23.67
2.29
43.64
3.40
9.42
1.59
8.84
1.80
BLIP-2 OPT
57.10
2.08
77.24
2.03
35.49
2.04
0.87
0.37
BLIP-2 TS
54.50
2.35
75.71
2.26
30.05
2.89
1.77
1.59
InstructBI. JIP T5
41.12
3.76
60.90
3.86
20.87
2.81
19.59
3.02
InstructBLIP V
39.69
3.34
59.59
3.59
20.03
2.90
19.30
2.82
LLaVA
45.07
0.76
72.11
0.87
23.18
0.63
0.00
0.00
BLIP (vqa finetuned)
34.68
0.20
53.06
0.17
15.00
0.47
13.85
0.92
X B
29.24
0.13
47.43
0.01
11.64
0.52
11.07
0.69
1st Question
VL L.
33.50
0.37
52.54
0.16
13.69
0.65
12.92
0.87
BLIP-2 OPT
67.22
0.33
83.54
0.14
40.31
0.90
2.54
0.27
BLIP-2 TS
64.71
0.43
80.83
0.20
34.43
1.49
4.81
0.62
InstructBLIP TS
55.41
0.72
73.34
0.34
27.92
1.13
26.48
1.42
InstructBLIP V
55.10
1.27
73.11
0.68
27.38
1.78
26.40
1.73
LLaVA
57.12
0.29
79.13
0.03
27.59
0.31
0.00
0.00
RESULTS
We evaluate the models from Sec. 3.3 on our benchmark, reporting the performance on objects, activities. attributes, and classical VQA. Fig. 2 summarizes the results. To reduce the dependency on single prompt, we ask three questions for each visual input and report the mean and standard deviation.
Retrieval Upper Bound. Tab. D shows the upper bound performance, i.e., when all allowed classes are tested explicitly by a prompt and the maximum score yields the decision (multiple-chool set- ting). Recent works like EVA improve over the original CLIP on object and action classification. Despite impressive results caption retrieval (Li et al. 2023b), BLIP-2 performs worse than CLIP on fine-grained (ImageNet) object retrieval but comparable on coarse classes (COCO obj).
ImageNet-oVQA. We report the performance for every model in Tab. 3 by first asking a general question (upper half) and then a more specific follow-up question (lower half). We observe that BLIP-2 outperforms both instruction-tuned models like InstructBLIP and LLaVA, as well as VQA- finetuned models. These results highlight that models pretrained on large, generic data collections perform well in zero-shot task with fine-grained classes. Additionally, ness of our experimental framework in the Supplementary (see Supp. Fig. D. In terms of metrics, find that ClipM is able to rank the correct label most similar among possible labels, even if the prediction does not contain the exact label text. For example, using ClipM, the model answer correct; however, it does not match the label "cougar" under the
illustrate the effective-
"a mountain lion" is marked metrics EM and Cont (see Supp. Fig. 44). We observe that follow-up questions allow for a fair com- parison between models and improve the performance by 32% on average across tested models. See Supp. Fig. 4b for qualitative examples.
COCO-oVQA. For this task, we show the results in Tab. 4 InstructBlip models show higher performance for the ClipM and EM metrics with a high sensitivity w.r.f. the question being asked (high standard deviation). The method produces very short answers in most of the cases. Other methods like LLaVA and BLIP-2 show competitive performance when measuring ClipM metric but are highly penalized by EM mainly due to producing long answers. As compared to any of the other datasets, the gap to the retrieval upper bound in Tab. i is much closer, showing that for are reasonably good classifiers. See Supp. Fig. 6 for
a coarse granularity of categories some qualitative examples.
ActivityNNetooVAA. Similar to ImageNet-oVQA, we evaluate models for activity classification (Tab. in two setups: First asking general question (upper half) and then posing specific
7

---

Published as a conference paper at ICLR 2024
Table 4: COCO-oVQA (cropped bounding box) object classification accuracy. Results for asking the questions "What can be seen in the image?", "What is in the image?" and "What's this?"
Method
ClipM@1 +
ClipM@5
EM +
Cont +
BLIP uga
46.58
3.35
65.54
4.05
21.74
3.87
26.04
1.11
X - B
48.22
7.46
63.72
5.51
23.53
1.23
25.76
1.34
X
51.09
4.79
67.82
2.50
22.50
2.41
25.00
1.30
BLIP-2 OPT
47.62
0.90
70.93
4.76
0.46
0.19
33.59
3.20
BLIP-2 T5
51.97
2.25
76.21
3.10
0.10
0.14
32.50
0.86
InstructBLIP T5
59.38
5.88
72.80
3.61
25.07
1.55
26.32
1.77
InstructBLIP V
59.58
8.22
73.32
4.92
26,50
3.56
27.52
3,96
LLaVA
51.13
1.17
81.04
0.32
0.00
0.00
45.08
2.33
Table 5: AetivityNetoVVQA (center frame) activity classification accuracy. Top: Results for ask- ing the questions "What activity this?", "What is happening in the image?" and "What is this?" Bottom: Results for asking follow-up question: "What type of activity is this?"
Method
ClipM@ 1 +
ClipM@5 +
EM +
Cont +
BLIP
23.73
1.28
43.04
1.98
6.70
4.92
7.30
4.70
X B
21.02
1.59
42.27
3.56
5.41
4.18
5.86
4.14
1 Question
x²-VLM.ga L
24.21
2.08
46.27
3.73
6.34
4.17
6.92
4.00
BLIP-2 OPT
55.08
0.35
74.32
0.70
7.10
7.28
14.66
5.03
BLIP-2 T5
53.39
3.02
74.71
2.59
7.07
9.59
15.70
3.78
InstructBI.ng JIP T5
40.18
0.71
61.82
0.92
10.87
3.83
11.64
4.10
InstructBLIP
40.64
2.67
62.35
3.41
12.51
1.87
13.61
2.95
LLaVA
46.75
0.47
72.26
0.61
0.00
0.00
8.76
1.08
Follow-up Question
BLIP
30.16
2.28
44.14
2.48
8.05
5.04
8.65
4.86
X2-VLM B
25.74
2.91
42.17
4.42
6.52
4.01
6.92
4.00
X L
30.35
2.94
45.64
4.48
7.65
3.89
8.24
3.76
BLIP-2 OPT
60.96
1.28
73.90
0.96
7.68
6.93
16.14
4.54
BLIP-2 T5
62.02
2.61
75.13
0.95
8.84
8.99
18.09
3.36
InstructBLIP T5
50.23
2.84
66.11
2.95
13.31
4.56
14.08
4.89
InstructBLIP
49.18
3.49
66.34
3.26
13.99
1.99
15.09
3.14
LLaVA
54.67
1.68
73.94
0.63
0.00
0.00
10.68
1.08
follow-up question (lower half), Activity questions may be the most out-of-distribution questions with respect to the classical VQA datasets, which might be the reason for the very poor performance of finetuned models on VQA (BLIP, x²- X2.VLM). On this task, BLIP-2 models outperform all other models for both ClipM and Cont metrics. Follow-up questions consistently improve the performance of all models but remain far from the upper bound (Tab. 1. See Supp. Fig. for some qualitative examples.
OVAD-oVQA. For the attribute classification task (see Tab. 2). we observe a similar behavior as in the classical VQA case (Tab, 6): Models finetuned instruction-tunel on VQA (BLIP, X2-MLM, InstructBLIP) perform well, while BLIP-2 and LLaVA show a very poor performance. This is likely due the imbalanced pretraining dataset where attributes often lie in the tail of the distribution, skewing such models more towards nouns and verbs. From qualitative analysis, we observe that the BLIP-2 OPT model tends to answer with objects regardlesss of the attribute-relate question, and LLaVA hallucinates long answers, ignoring the image when asked some contextual questions like the number of objects the image. These could potentially explain their lower performance on the OVAD-oVQA task. See Supp. Fig. 6 for some qualitative examples.
Classical VQA. Tab. 6 shows the results on classical VQA datasets. Methods finetuned on VQAv2 outperform zero-shot models on both VQA datasets; however, the zero-shot methods are reasonably competitive. We find that EM is unsuitable for evaluating instruction-tuned models like LLaVA, reporting 0% accuracy in Tab. [ even for reasonable answers (illustrated in Supp. Fig. 8 Cont a better choice to evaluate such models. This finding aligns with the results from the user study described in the following paragraph. However, in some cases Cont may overestimate model per-
8

---

Published as a conference paper at ICLR 2024
Table 7: Pearson correlation to human
Table 6: Classical VQA. Zero-shot methods (top) and models finetuned on VQA (bottom). 9: Authors use rank-based evaluation with a fixed vocabulary, we use generative evaluation. Italic: Evaluated data is part of training.
judgement scores on the VQAv2 dataset.
Metric Model
Medeil
Both
BLIP-2
LLaVA
GPT-410-sho
0,972
0.979
0,965
GPT-4
0.947
0,964
0,929
Llama25-ahe
0.919
0,943
0.894
VQAv2 val
GQA testdev
Llama20-she
0.798
0.733
0.865
Model
test-dev
balanced
Cont
0.906
0,917
0.894
EM
EM
Cont
EM
Cont
EM
0.525
0.837
0.000
BLIP-2 OPT
51.32 51,51 52.66
32.21
32.93
BertScore
0,223
0.357
-0.081
BLIP-2 T5
62.80
63.13 64.80
44.03
44.88
BLEURT-20
0.554
0.636
0.573
LLaVA
0.00 0.00 55.66
0.00
41.58
BEM
0.627
0.747
0.526
BLIP vgs
77.50 83.69 85.81
47. 07.04
49.20
LERC
0.827
0,926
0.720
V
79.64 83.05 84.00
52.54
53.62
BLEU-1
0.637
0.898
0.693
X--VLM.m L
81.34 85.33 86.29
52.80
53,90
BLEU-4
0.023
0,027
0,014
InstructBLIP TS
74.75
73.22 74.29
47.61
48.46
CIDEr
0,597
0,841
0.435
InstructBLIP
77.54
76.12 76.92
48.34
48.97
METEOR
0.689
0.895
0.830
ROUGE
0.717
0.913
0.756
formance. Using Cont, the model's prediction is deemed completely correct even if the model predicts several answers, out which only one is correct (see example in Supp. Fig. 8b).
User Study: Comparing automatic metrics to human judgment. In order to evaluate the ef- fectiveness of metrics for comparing a model prediction and ground truth answer, we collect human judgment scores the VQAv2 dataset. Tab. 7 shows the Pearson correlation between evaluated metrics and our manually curated human gold standard for model predictions from BLIP-2 T5 with short answers and LLaVA with long answers. For more details, pendix C We find that depending on the budget and size of the evaluation, GPT-4 (OpenAl 2023a) strong choice. LLaMA-2 (Touvron et al.) (003b) is a good option that does not depend on an external service but is still computationally expensive with 70B parameters. For large-scale com- parisons, Cont more reliable choice than EM but might pose risk of degrading in terms of correlation with human judgment when optimizing hyperparameters for it.
refer the reader Supp. Ap-
5 CONCLUSION
In summary, our work has several significant implications: We establish complete framework for transforming classification benchmarks into VQA benchmarks. By removing ambiguities from the VQA evaluation setup using concept hierarchies to follow up with the model, we enable a compa- rable and fair evaluation of VLMS on open-ended VQA. Based on this framework, we create stratified benchmark for diagnosis of Text-VLMs answering questions about objects, actions, and attributes.
Our user study shows that none of the existing NLP metrics improve VQA evaluation over simple text comparison metric, with the notable exception of using GPT-4 or LLaMA-2. This finding encourages the use of LLMss gold standard metric reduce the cost of human annotation and motivates research to improve cost-effective Text-QA and VQA metrics. Our selection of metrics enables evaluating Image-dialog models without unfairly penalizing the use of synonyms or longer answers. Together, these contributions enable direct comparison of classical Text-VLMs, Image- dialog models, and discriminative VLMs on the same datasets.
Finally, we show an imbalance in model strengths as highlighted in Fig. 2 Pretrained generic mod- els are stronger in classifying fine-grained objects and activities, while models trained or finetuned for VQA are stronger in classifying coarse concepts and answering common VQA questions, high- lighting the possible directions of improvement.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was funded by the German Research Foundation (DFG) 401269959, 417962828, and 499552394 (SFB 1597 Small Data). This work was supported by the German Academic Ex- change Service (DAAD) - -07440941 Research Grants Doctoral Programmes in Germany, 2019/20. The authors acknowledge support from the state Baden Württemberg through bwHPC. We thank Max Argus and Sudhanshu Mittal for their critical discussions, support, and feedback on the pa- per. We thank annotators Max Argus, Sudhanshu Mittal, Simon Schrodi, Johannes Dienert, Philipp Schröppel, and Leonhard Sommer for their help, time, and effort.
REFERENCES
Jean-Bag Bap-ittt Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 2022.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic proposi-
tional image caption evaluation. In European Conference of Computer Vision, 2016.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on
Computer Vision, 2015.
Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou. Touchstone: Evaluating vision-language models by language models. arXiv preprini arXiv:2308.168P0 2023.
Satanjeev Banerjee and Alon Lavie. Meteor:
An automatic metric for mt evaluation with improved correlation with human judgments. In Association for Computational Linguistics, 2005.
Maria A Bravo, Sudhanshu Mittal, Simon Ging, and Thomas Brox. Open-vocabulary attribute detection. In IEEE Conference on Computer Vision and Puttern Recognition, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Jannis Bulian, Christian Buck, Wojeiech Gajewski, Benjamin Börschinger, and Tal Schuster. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2022. doi: 10.11865//1/22222 htt/22222..mmmmmaaia main.20. URL
http:://x.doi.comm
wew..mmmnlppmmm.mmm 20
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Mocha: A dataset for training and evaluating generative reading comprehension metrics. Conference Empirical Methods in Natural Language Processing. 2020.
Wei-Lin Chiang, Zhuohan Li, 7i Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. URL https://
Lmsyss.org//loo/22233 30133300vviuna..
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02::3 2022.
Alessandro Conti, Enrico Fini, Massimiliano Mancini, Paolo Rota, Yiming Wang, and Elisa Ricci. Vocabulary-free image classification. arXiv preprint wrXiv:2306.00917. 2023.
10

---

Published as a conference paper at ICLR 2024
Yin Cui, Guandao Yang, Andreas Veit, Xun Huang. and Serge Belongie. Learning to evaluate image captioning. IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li. Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpon purpose vision-language models with instruction tuning. arXiv preprint vrXiv:2305.0650 2023.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In IEEE Conference Computer Vision and Pattern Recognition, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. IEEE Conference on Computer Vision and Pattern Recognition, 2023.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304. 15000, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stur Dhruy Batra, and Devi Parikh. Making the V in
VQA matter: Elevating the role of image understanding in Visual Question Answering. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Andrew F. Hayes and Klaus Krippendorff. Answering the call for standard reliability measure for coding data. Communication Methods and Measures, 2007.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv: 15556 2022.
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reason- ing and compositional question answering. IEEE Conference Computer Vision and Pattern Recognition, 2019.
Kushal Kafle, Robik Shrestha, and Christopher Kanan, Challenges and prospects in vision and language research. Frontiers in Artificial Intelligence, 2019.
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel Weld. Genie: Toward reproducible and standardized human evalu- ation for text generation. In Conference Empirical Methods in Natural Language P Processing, 2021.
Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmen Hitokazu Matsushita, and Arul Menezes. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference Machine Translation, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsoureed dense image annotations. International Journal of Computer Vision, 2017.
Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C.H. Hoi. LAVIS: A one-stop library for language-vision intelligence. In Association for Computational Linguistics, 2023a.
11

---

Published as a conference paper at ICLR 2024
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping lunuuage-image pre- training for unified lasoon-anguage understanding and generation. In International Conference Machine Learning. 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- pre- pre-training with frozen image encoders and large language models. In International Conference Machine Learning. 2023b.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Association for Computational Linguistics, 2004.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference of Computer Vision, 2014.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0845 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, 2019.
Edward Loper and Steven Bird, Nitk: The natural language toolkit. arXiv preprint arXiv:0205028, 2002.
Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondrej Bojar. Results of the WMT20 metrics shared task. In Conference Machine Translation, 2020.
George A. Miller WordNet: A lexical database for English. In Human Language Technology:
Proceedings of Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994. URL https: ptttps://aclanthin ogy. org/ 1111
OpenAl. Gpt-4 technical report, 2023a.
OpenAl. Pricing, 2023b. URL httptpss///peeaa.commppptt//////
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Association for Computational Linguistics, 2002.
Zhiliang Peng, Dong, Hangho Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208. 06366, 2022.
Amy Pu, Hyung Won Chung, Ankur P Parikh, Sebastian Gehrmann, and Thibault Sellam. Learning compact metrics for mt. In Empirical Methods in Natural Language Processing, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. Semantic answer similarity for evalu- ating question answering models. arXiv preprint arXiv:2108.06 06130,1000000000 2021.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 2015. doi: 10.1007/5 1263-015-0816-y
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text gener- ation. In Association for Computational Linguistics, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
12

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In IEEE Conference Computer Vision and Pullern Recognition, 2015.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. LVLM-eHub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.. 0920000996,... 2023.
Yan Zeng, Xinsong Zhang, Hang Li, Jiawei Wang. Jipeng Zhang, and Wangchunshu Zhou, X All-in-one pre-trained model for vision-language tasks. arXiv preprint arXiv:2211. 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Eval- uating text generation with BERT. In International Conference on Learning Representations, 2020.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing. Hao Zhang, Joseph E Gonzalez, and lon Stoica. Judging LLM-aa-a-judge with MT-Bench and Chatbot Arena. arXiv preprint arXiv:2306 05685, 2023.
13

---

Published as a conference paper at ICLR 2024
A EVALUATION PROTOCOL
A.1 MODEL DETAILS
In this section and Supp. Tab. 8 we provide additional information about the models we evaluate.
BLIP (Li al. 2022) is a Languala model trained jointly with three objectives: Image- Text Contrastive Loss (ITC) aligning visual and textual feature spaces, Image-Text Matching Loss (ITM), a binary classification loss for positive or negative image-text pairs, and Language Mod- eling Loss (LM) generating textual descriptions conditioned images. The pre-trained model is then fine-tuned for various downstream tasks, particular VQA. Additionally, BLIP generates synthetic captions and filters noisy captions from web-collected image-text pairs to improve data quality and vision-language learning.
BLIP-2 (Li et al. 2023b a VLM that leverages frozen pre-trained image encoders and LLMS by only training intermediate Q-former, 12-layer Transformer encoder to join the multimodal data. They used ITC, LM, and ITM losses in the first stage to train the model using only the image encoder and the Q-former. In the second stage, they integrate the pre-trained language model and only use the LM loss. The VQA task is evaluated a zero-shot manner. When evaluating using the VQA datasets (VQAv2 and GQA) in our work, we the suggested VQA hyperparameters from the authors: We limit the maximum number of tokens to 10 and set length penalty of -1 during beam search enforce shorter answers.
InstructBLIP (Dai et al., 2023) builds instruction tuming framework based the pre- traieed BLIP-2. It uses 26 publicly available datasets grouped into task categories and introduces instruction-aware visual feature extraction, allowing flexible feature extraction based given in- structions. For the frozen LLM, InstructBLLIP uses FlanTT5 or Vicuna.
LLaVA (Liu et al., 2023) is an open-source Large Language-ang Vision Assistant. It connects a pre-trained CLIP ViT-L/14 visual encoder and a large language model LLaMA using simple projection matrix (linear layer). considers a two-stage instruction-tume stage, the authors pre-train for Feature Alignment and update only the projection matrix based on a subset of the CC3M dataset. In the second stage, the authors fine-tune the projection matrix and LLM end-to-end for multimodal instruction-followin data (Visual Chat) and multimodal reasoning dataset for the science domain (Science QA).
procedure: In the first
X2-VLM (Zeng et al. 2022) is a VLM trained to aligm visual-text data more localized way. In comparison with other models, it learns from four types of data: box-object and noun labels, region and text annotations, image-caption data, and video-text data. X2-VLM consists of vision, text, and multimodal cross-attention module. The vision encoder is initialized with BETT-2 |engg et al. (2022) while the text encoder is initialized with BERT Devlin et al. (2018). X2-VLM is trained using the three objectives: ITC loss, ITM loss, and masked language modeling (MLM) loss.
Table 8: Model details. In the prompt, "[}" is replaced by the question, QA: "Question: { Short answer:". *; LLaVA prompt (see end Supp. Sec. A.T). Inference generation hyperparameters: Maximum number of generated tokens and number of beams in beam search. We mark frozen weights for methods where the authors only train the Fusion Module.
Image Res. (px)
Visual Encoder
Text Encoder
Fusion Module
Model Size
Max Tokens
Method
Prompt
Beams
BLIP aga
480
0
ViT-B/16 (ImageNet)
BERT buse
0.361B
20
X B
768
[
12I. ViT (BeiT-2)
BERT True
6 Trans. 0,334B
20
XL-VLM,, L
768
[]
24L VIT (BeiT-2)
BERT base
6 L Trans. 0.724B
20
BLIP-2 OPT
224
QA
VIT-g/1 -VA-CLIP
unsup. OPT 12L Trans. 3.745B
30
BLIP-2 T5
224
QA
ViT-g/1 EVA-CLIP
inst. FlanTS XL 12L Trans. 3.942B
30
InstructBLIP T5
224
QA
VIT-g/1 EVA-CLIP
inst. FlanT5 XL 12L Trans. 4.023B
256
InstructBLIP V
224
QA
ViT-g/1 EVA-CLIP
Vicuna-7B 12L Trans. 7.913B
256
LLaVA
224
VIT-L/14 CLIP
LLaMA 7B
Linear 6.743B
1024
14

---

Published as a conference paper at ICLR 2024
LLaVA prompt. "You are LLaVA, large language and vision assistant trained by UW Madison WAIV Lab. You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language. Follow the instructions carefully and explain your answers in detail.
Human: Hi!
Assistant: Hi there! How can I help you today?
Human: 00"
Handling long model outputs. The EVA-CLIP model used by the ClipM metric has a maximum input length of 77 tokens. Also, the Cont metric might become unreliable for very long outputs since it rates ground truth label as correct if contained in the output, regardless of the output length. Therefore, we reduce all model predictions on all datasets to a maximum of about 45 words. We use the nlt (Loper & Bird. 2002) English tokenizer to split model predictions into sentences and try cut off the prediction at a sentence boundary to total length of 40-50 words. If there no sentence boundary in that range, we cut off after 45 words. This only affects the LLaVA model, which tends to produce long paragraphs of answers. On ImageNet-oVQA output, this procedure reduces the average length of LLaVA outputs from 76 to 44 and affects 30% of the imput.
Cut-off points. We observe that BLIP-2 OPT often generates outputs in the form of "(answer) Long answer: (reformulated answer)" after being prompted with "Question: (question) Short answer: ". We consider this a signal that the model has finished answering and only evaluate all text that appears before the model generates "Long answer: or "Short answer: On ImageNet-oVQA output, this procedure affects 25% of the input.
Follow-up. After determining the candidate parent class for the follow-up, we calculate the simi- larity between the model prediction and all parent synonyms using EVA-CLIP. Since our goal is to only ask follow-up questions about concepts that the model already described, we only ask follow- up question if the most similar synonym has similarity of at least 5. We set the hyperparameter 8 0.37 based on qualitative evaluation. For ImageNet, we create the parent embeddings using prompt templates described in Supp. Sec. A.2 For ActivityNet, we do not use prompt templates.
A.2 DATASET-SPECIPE DETAILS
We report statistics over all evaluated datasets in Supp. Tab. 9 We also provide an overview of all questions ask in Supp. Tab. I0 and Supp. Fig. 3
COCO-oVQA. COCO contains bounding boxes for each object instance among the 80 object cate- gories. We use the "validation 2017" set to build the COCO-oVQA task, which consists of 36,781 object instances. For each instance, we crop the object using the ground-truth bounding box consid- ering minimum side of 40 pixels leaving margin of 4 pixels each side. We three different questions to ask the VLMs about the object category: "What can be seen in the image?", "What is in the image?" and "What's this?". For the ClipMatch metric, we obtain single class embedding by averaging the embeddings of 80 different prompt templates such as A photo of {label} and A good photo of the {label} done by Radford et al. (2021).
imageNet-oVQA. ImageNet provides bounding boxes for all objects corresponding to the ground truth class label of an image. For each image, we select the box with the biggest area and increase it to minimum size of 64 pixels on each side. Since all models we test require squared inputs, we convert the rectangle boxes to squares by increasing the size of the smaller side. We reuse the same questions as for COCO and the same prompt templates to obtain the embeddings. To build the label hierarchy from WordNet, we only consider parents with more than one child to simplify the hierarchy. We also exclude the root node "entity".
ActivityNNt-oVQA. We do not use any prompt templates for creating the class embeddings. For asking the follow-up question, use the label hierarchy provided by the dataset authors and remove the root node "activity""
A.3 CODE
We build our codebase top of LAVIS (Li et al. 2023a
15

---

Published as a conference paper at ICLR 2024
cleanliness
hair type
How would you describe the object's appearance terms of cleanliness?
How would you describe the object's hair texture?
What kind of hair texture does the object have? What is the nature of the object's hair?
How would you describe the object in terms of cleanliness?
length
What the state the object terms of clcanliness?
Describe the object's length. What is the length of the object?
clothes color
What colors are the object's clothes in the image?
Describe the length of the object. material
What colors are the clothes the object is wearing?
What colors you notice in their attire?
What material is the object made of? What is the material the object? Describe the material of the object. maturity
clothes pattern
What patterns are the clothes the object is wearing?
What type of pattern is on the object's 's attire?
How would you describe the design the object's clothing?
What is the object's 'ss maturity level?
What age category does the object helong to? Which is the maturity of the object?
color quantity
The number of colors the object has is
optical property
How many colors does the object have?
How would you describe the object's ability transmit light?
How many distinct colors do you notice on the object?
color
What is optical property of the object? What is the object's optical property? order
What colors are present in the object?
Describe the color of the object.
Identify the color of the object.
Does the object exhibit systematic arrangement more disordered appearance? What is the state of organization the object? How would you describe the arrangement of the object?
cooked
What's 's the condition the object's cooking process? How would you describe the condition of the object terms of cooking?
How cooked the object?
pattern
face expression
Which the pattern the object?
Which is the face expression of the ohject?
Which pattern does the ohject have? What pattern is the object?
What is the face expression of the object?
What is the object's facial expression?
position
gender
What is position of the object? Describe the position of the object.
Describe the gender of the object. What is the gender of the object?
What position is the object in? size
Which gender is the object?
group
How would you describe the size the object?
How many objects are present the image?
Describe the object's size.
How many objects are visible?
What size is the object? state
Are we looking at individual object or group?
hair color
Which the state of the object?
Which is the hair color of object?
What is state the object? Describe the state of the object. texture
What is the hair color of the object?
Descrihe the color of the object's 's hair. hair length
Describe the texture of the object. What is the feel of the object's surface? Describe the object's texture. tone
What the hair length of the object? How would you classify the hair length of the object?
Describe the hair length of the object. hair tone
How would you describe the object's color intensity? What is the lightness darkness of the object?
How would you describe the brightness of the object's hair?
Describe the tone of the object.
Can you specify the lightness or darkness the object's hair?
What intensity is the object's hair color?
Figure 3: OVAD-oVQA questions. For every attribute type we build three different questions to evaluate the Text-VLMs We show an example of the three different questions for every type of attribute. The word object is replaced by the noun category for every question.
16

---

Published as a conference paper at ICLR 2024
Table 9: Dataset details. We report details for the balanced-testdev set on GQA and for the val set on all other datasets. For ActivityNet we report the number of videos and segments instead of images and crops. During evaluation on ActivityNet we consider the center frame of segment as input for the model. For OVAD we evaluate 123k attribute annotations with 3 questions each. Q: question, A: answer, len: length (number of words), Cls: classes
Imgs (k)
Crops (k)
Uniq Q
A
Uniq
Dataset
+
Cls
(k)
len
(k)
len
A
ImageNet
50.0
50.0
150.0
5.00
2.00
3
150.0
1.67
0.74
1000
1000
COCO
5.0
36.8
110.3
5.00
1.63
3
110.3
1.10
0.30
80
80
Net
4.9
7.7
23.0
4.33
1.25
3
23.0
2.02
0.84
200
200
OVAD
2.0
14.3
369.0
7.61
1.77
3099
1229.9
1.08
0.27
249
117
VQAv2
40.5
214.4
214.4
6,23
1.96
81272
2143.5
1.17
0.57
90805
GQA
0.4
12.6
12.6
8.53
3.23
11484
12.6
1.06
0.24
722
Table 10: Overview of questions. When asking follow-up questions, replace the {placeholder) with the corresponding object or activity to ask about e.g. "What type of dog is this?"
Dataset
Question
Follow-up question
What can be seen in the image?
ImageNet
What in the image?
What type of (object) is this?
What's this?
What can be seen in the image?
COCO
What in the image?
n/a
What's 's this?
What activity is this?
ActivityNet
What happening in the image?
What type of (activity) is this?
What this?
OVAD
See Supp. Fig.
n/a
B ADDITIONAL RESULTS
We report all results on ImageNet without cropping using retrieval models in Supp. Tab. I and using VQA models in Supp. Tab. 12
We compare the hyperparameters of BLIP-2 both in the default and VQA settings. The default setting is shown in Supp. Tab. 3 For VQA, Li et al. 2023b) apply exponential length penalty = -1 that promotes generating shorter answers and limits the maximum number of tokens to 10 in order to match better the EM metric used in VQAv2 and GQA. Supp. Tab. 13 shows that the hyperparameters optimized for VQA benchmarks produce significantly shorter answers and worse results ImageNet-oVQAA benchmark.
B. QUALITATIVE EXAMPLES
We show qualitative examples of our oVQA benchmark for ImageNet in Supp. Fig 4 ActivityNet in Supp. Fig. 5 OVAD Supp. Fig. ( and for COCO Supp. Fig. [7] Additionally, we show classical VQA examples evaluated with the different metrics in Supp. Fig. Finally, show qualitative overview of all evaluated datasets in Supp. Fig. 9
17

---

Published as a conference paper at ICLR 2024
Table 11: Zero-shot retrieval results for ImageNet (not cropped).
Model
R@
R@5
CLIP ViT-L-14
75.52
94.57
CLIP EVA01-g-14
78.52
95.50
BLIP-2
63.59
87.72
BLIP-2 (COCO ft)
62.26
85.31
Table 12: ImageNet (not cropped) classification accuracy using VQA models. Top: Results for asking the questions "What can be seen in the image?", "What in the image?" and "What's this?" Bottom: Results for asking "What type of object is this?" response to the output of the first question. We highlight the first, second and third best method.
Method
ClipM@I
ClipM@S
Cont 1
EM
BLIP (yqa finetuned)
23.71
0.88
42.94
1.66
9.40
0.90
8.27
1.37
X²-VLMap B
20.30
1.86
39.47
3.00
7.60
1.37
7.10
1.48
1 Question
X²-VLM L
22.77
2.59
42.31
3.93
8.71
1.70
7.97
1.94
BLIP-2 OPT
49.02
1.43
69.02
1.04
27.89
1.17
0.83
0.20
BLIP-2 T5
45.53
1.42
65.37
1.44
23.66
1.76
1.55
1.38
InstructBLIP T5
33.48
4.07
52.78
4.78
15.24
2.78
14.64
2.79
InstructBLIP V
32.03
2.48
51.51
2.92
14.77
2.59
14.14
2.54
LLaVA
42.84
0.58
69.91
0.93
21.24
0.54
0.00
0.00
BLIP (vqa finetuned)
34.44
0.22
53.02
0.20
14.48
0.60
13.07
1.08
X²-VLMap
29.03
0.29
47.76
0.21
11.27
0.53
10.59
0.66
1 Question
X"-VLMaa L
33.42
0.46
52.57
0.29
13.43
0.70
12.47
0.93
BLIP-2 OPT
60.59
1.29
78.22
0.97
34.02
0.21
2.83
0.54
BLIP-2
58.69
0.55
75.89
29.71
1.05
5.58
0.88
InstructBLIP T5
49.53
0.98
67.60
0.58
23.18
1.04
22.41
1.08
InstructBLIP V
48.20
0.89
66.03
0.60
22.30
1.72
21.47
1.69
LLaVA
54.92
0.21
77.67
0.12
25.51
0.22
0.00
0.00
Table 13: ImageNet (cropped) classification comparing different hyperparameter setups for BLIP-2.
Method
ClipM @ +
ClipM@5 +
Cont +
EM +
Len =
OPT vqu
35.32
13.99
50.31
18.92
18.67
8.10
6.67
1.76
1.96
0.37
OPT
57.10
1.70
77.24
1.66
35.49
1.67
0.87
0.31
7.94
0.94
T-5
43.02
2.51
62.75
2.51
21.76
2.22
13.96
4.09
1.56
0.24
T-5
54.50
1.92
75.71
1.85
30.05
2,36
1.77
1.30
4.21
0.52
OPT vqu
50.93
8.77
66.26
9.96
26.95
5.05
12.01
1.09
1.94
0.07
Foll. Q.
OPT
67.22
0.27
83.54
0.12
40.31
0.74
2.54
0.22
6.53
0.38
T-5
55,51
1.32
72.26
1.05
27.84
1.01
19.96
3.81
1.32
0.08
T-5
64.71
0.35
80.83
0.16
34.43
1.22
4.81
0.51
3.17
0.20
18

---

Published as a conference paper at ICLR 2024
Question:
What's this?
Question: What's this?
Label: cougar
Label: Dalmatian
BLIP-2 T5 output: a mountain lion
BLIP-2 OPT output: it's dalmatian
LLaVA output: The image features a large black and white dog laying down on the floor, possibly on carpet.
ClipM Top-5 similarities:
cougar (0.792)
lion (0.682)
Follow-up Question: What type of dog this? LLaVA oulput: The dog in the image is Dal- matian.
snow leopard (0.636)
lynx (0.527)
leopard (0.5/1)
24
27
RAL
30
35
Question:
What's this?
Label: magnetic compass
BLIP-2 T5 output: radio compass
Question: What's this?
ClipM Top-5 similarities: magnetic compass (0. (817
Label: slip-on shoe
X LLM L output: shoe
radio (0.701)
Follow-up Question: What type of shoe is this?
barometer (0.629)
X X2M L follow-up output: black
odometer (0.517)
stopwatch (0.513)
(a) ClipM allows evaluating predictions as correct which text-matching metrics like EM and Cont fail.
(b) Top: Follow-up questions improve the faimess of the evaluation. since they reduce the chance of punish- ing reasonable answers. Bottom: If model cannot produce required level of detail, will still fail evaluation.
Figure 4: Qualitative results for ImageNet-oVQQ. Only the bounding box crop is considered as input image for the model prediction. Coloring: Answers are considered correct / wrong under ClipM metric.
19

---

Published as a conference paper at ICLR 2024
Question: What activity is this?
Question: What activity is this?
Label: playing drums
Label: beach volleyball
BLIP vqa output: playing music
BLIP-2 OPT output: volleyball
Hierarchy: playing musical instruments
Hierarchy: playing volleyball
Follow-up Question: What type playing
Follow-up Question: What type of playing
musical instruments is this?
volleyball is this?
BLIP vqa follow-up output: drums
BLIP-2 OPT follow-uy up outpur: beach volleyball
(b)
rst draw to 16
st
TO
2T1
Question: What is this?
Question: What is this?
Label: playing blackjack
Label: dodgeball
BLIP voa output: business
BLIP vga output: tennis
BLIP-2 OPT output: it's blackjack
BLIP-2 OPT output: dodgeball
InstructBLIP TS outpur: blackjack
InstructBLIP T5 output: frisbee
LLaVA output: The image shows person sit- ting at a table, shuffling a deck of cards...
LLaVA output: This is a group activity taking place in a gym where people are playing with basketballs and volleyballs.
X--VMM L output: table
X2 L output: soccer
(c)
(d)
Figure 5: Qualitative examples for the Activity oVQA task. Figures (a) & (b) show how the follow- up question makes use of the hierarchical parent to find the exact answer. Figures (c) & (d) show the outputs of different VLM models for the Activity-oVQA task. Coloring: Answers are considered /orrect wrong under ClipM metric.
20

---

Published as a conference paper at ICLR 2024
Question: Which texture does the dining table have?
Label: sleek / smooth
BLIP vga output: smooth
BLIP-2 OPT output: it feels like wood, but it's
Question: What is the position of the person?
not wood, it's laminate
Label: standing / upright / vertical
InstructBLII T5 output: wood
BLIP vqa output: standing
LLaVA output: The dining table's surface is made of wood, giving it a warm and natural ap- pearance.
BLIP-2 OPT output: 1st base
InstructBLIP TS output: batter
LLaVA output: The person in the position of
X2V L output: smooth
batter, standing at the home plate...
x VLM L. oulput: standing
(a)
Question: How many people present in the image?
Question: How many surfboards present in the image?
Label: individual / one / single 1 sole alone
BLIP vqa output: one
Label: individual one single 1 sole alone BLIP vga output:
BLIP-2 OPT outpur: None.
InstructBLIIP TS output: 2
BLIP-2 OPT output: One.
LLaVA output: There are two people present in the image.
InstructBLIP T5 output:
LLaVA output: There are two surfboards in the image. x² X2-VLMug output:
X--VLMaqa L output: one
(c)
(d)
Figure 6: Qualitative examples for the Attribute-oVQA task. BLIP-2 OPT tends output nouns/objects for many attribute-type questions (see (a) & (b)). We observe that LLaVA tends hallucinate long answers ignoring the image for contextual questions like quantity of objects in the image (see (c) & (d)). Only the bounding box crop is considered as input image for the model prediction. Coloring: Answers are considered correct wrong under Cont Syn metric.
21

---

Published as a conference paper at ICLR 2024
Question: What is in the image?
Question: What's this?
Label: dog
Label: surfboard
BLIP vya output: puppy
BLIP vqa output: dog
BLIP-2 OPT output: A puppy playing with a toy
BLIP-2 T5 output: photo of man holding surfboard
in the bathroom
InstructBLIP TS output: dog
InstructBLIP T5 output: surfboard
LLaVA output: The image features large tan dog sitting on the floor, with its back facing the camera (...)
LLaVA VA output: The image is a blurry photo of person holding surfboard (...) The surfboard is white and extends across the image. (...)
vg L. output: puppy
X2 X2VLM output: man
(b)
Question:
What can be seen in the image?
Question: What can be seen in the image?
Label: person
Label: cup
group of people BLIP-2 OPT output: man, woman, dog, cat, dog, a cat, a cat, a cat, a cat, cat
BLIP vga output:
BLIP vqu oulput: coffee cup and saucer
BLIP-2 OPT outpul: plate, a cup, knife, fork, spoon, a knife, fork
group of people LLaVA output: In the image, person is walk- ing down the street (...) The person is wearing heavy winter coat, gloves, and a hat. X2.VLM,gu L output: people walking
InstructBLIP T5 output:
InstrucIBLIP T5 output: coffee cup
walking down a street
LLaVA output: In the image, there two cof- fee cups sitting on a table, with one of them being white coffee mug with heart shape on X2-VLMyq L output: mug
(d)
(c)
Figure 7: Qualitative examples for the COCO vVQA task. We observe that ClipMatch can consider synonyms and longer texts positive matches (see (a) for puppy & (b) for container). In (c) and (d), we observe how BLIP-2 OPT tends to repeat many nouns instead of simply the correct one and how LLaVA generates detailed answers but sometimes hallucinates. Only the bounding box crop is considered as input image for the model prediction. Coloring: Answers are considered correct wrong under Cont Syn metric.
22

---

Published as a conference paper at ICLR 2024
Question: What is the zebra eating? Label: grass (x9), dry grass (x1)
BLIP-2 TS output: grass
LLaVA output: The zebra cating grass in the field.
Output
Label
EM
Cont
LLaMA-2
GPT-4
grass
grass
1.00
1.00
1.00
1.00
The zebra is eating grass in the field.
grass
0.00
1.00
1.00
1.00
grass
dry grass
0.00
0.00
0.75
0.75
The zebra eating grass in the field.
dry grass
0.00
0.00
0.75
0.75
VQAv2 with 10 human answers. Metrics are evaluated for each answer and then aggregated as described in See. 3.2 Row 1: All metrics give full points if the model output matches exactly. Row 2: EM cannot properly judge the output of LLaVA while Cont and both LLM-based metrics successfully evaluate the output as correct. Rows 3, 4: Both text-based metrics cannot judge either output correct, since the label is longer than the prediction and therefore can neither match nor be contained in the prediction. The LLM-based metrics still succeed in judging the output.
Question: What are the vegetables the left of the bowl that is to the left of the cookies?
Label: carrots
BLIP-2 T5 oulpul: carrots
LLaVA VA output: The vegetables to the left of the bowl are carrots and green beans.
Output
Label
EM
Cont
LLaMA-2
GPT-4
carrots
carrots
1.00
1.00
1.00
1.00
The vegetables to the left of the bowl are carrots and green beans.
carrots
0.00
1.00
1.00
0.25
GQA with label. Row 6: Cont and LLaMA-2 overestimate the performance of LLaVA and do not penalize the incorrect "green beans" part of the answer, while GPT-4 penalizes it.
Figure 8: Qualitative examples for different metrics. Metric scores range from 0.00 (worst) to 1.00 (best).
23

---

Published as a conference paper at ICLR 2024
Dataset: ImageNet
Dataset: COCO
Question: What's this?
Question: What's this?
Label: cougar
Label: elephant
Dataset: ActivityNet
Dataset: OVAD
Question: What activity is this?
Question: What is the position of the person?
Label: playing drums
Label: standing upright / vertical
Dataset: VQAv2
Dataset: GQA
Question:
Where is the cat?
Question: What is the spoon made of? Label: metal
Label: on desk (x4), desk (x3), center of picture, at home, on table
Figure 9: Qualitative comparison between all datasets used in the benchmark.
24

---

Published as a conference paper at ICLR 2024
C HUMAN EVALUATION
We base our human study design on Chen et al. (2020) and ask humans to rate model predictions from (completely wrong) (completely correct).
Data selection. We use the VQAv2 validation dataset for our human evaluation and select questions from each of the three subsets: 50% "other", 25% "number" and 25% "yes/no". To avoid very noisy questions, we analyze the 10 human answers per question and only select questions with at least 4 of 10 humans agreeing for "other" and "number" questions and least 9 of 10 humans agreeing for "yes/no" questions. We keep the top answer for each question as the ground truth answer. In total, we select 1000 questions.
Choice of model predictions. A good text semantic metric should work well on both short and long answers. Therefore, we select the two models BLIP-2 T5 with 1.6 words on average per answer and LLaVA with 12.6 words on average per answer for our study.
Processing. For metrics based on text heuristics (Cont, EM and last 5 rows of Tab. 7 we preprocess the model predictions as the authors of the dataset VQAv2, described in Sec. 3.2 To reduce the human workload, we automatically label all predictions that are correct under EM as "5" and all empty model outputs "1". When giving predictions to humans or deep learning based models, we do not preprocess the outputs since we hypothesize that those judges can use certain punctuation model-specific cues, e-g- preserving upper case letters. Given score in the range from 1 to 5, we convert it into the range from 0 to I as Sporm = (s 1)/4
Instructional examples. To create examples of the task for human workers and GPT-4, we sample a second set of data in the same way. We manually judged 100 examples on the second set and commented on our reason for the rating on 20 of them.
Human evaluation. We prompt the human workers with the instructions shown in Supp. Fig. 10 present them the 20 commented examples, which contain the fields "Question", "Correct answer", "Predicted answer", "Correct rating" and "Reason". We run the evaluation internally. Each datapoint annotated by workers.
Majority voting and inter-annotator agreement. Same as Chen et al. (2020) we calculate inter- annotator agreement using Krippendorff's s Sipha-Reliabiligy (Hayes & Krippendorff (007) and ob- tain 0.93, indicating strong agreement. We use majority voting to receive the final gold standard annotation as done by Bai et al. (2023). For the 4.6% of cases where all 3 annotators disagree, we average and round the votes.
GPT-4 evaluation. We use the manually judged 100 examples as in-context few-shot prompts for GPT-4. To construct the input prompt, experimentally find upper limit on the input length where GPT-4 starts to produce inconsistent outputs and find the output to be consistent for 10 few- shot examples and 30 datapoints in one prompt. An output is considered consistent if it matches the exact output schema defined in the prompt in Supp. Fig. I1 Inputting new datapoints in a dialogue style without repeating the instructions resulted in GPT-4 forgetting the task and producing inconsistent outputs, so we simply restart the dialogue for each new group of 30 datapoints. At each iteration, we randomly sample 30 datapoints and 10 few-shot examples and input them into GPT-4 as described until all datapoints are annotated. In the 0-shot experiment, we remove the "Example input:" and "Example output:" sections from the input prompt.
GPT-4 monetary evaluation cost. To approximate a lower bound on the cost of such an evaluation, we calculate the average number of words in the input and output. We ignore the JSON formatting characters since we assume a cost-optimal evaluation would choose a format with less such char- acters. OpenAT (2023b) reports 1K tokens as representing about 750 words and sets the price of their API at $0.03 $0.06 per 1K input / output tokens, respectively. On average, we have 1720 / 970 input output tokens for a batch of 30 predictions, resulting in cost of about $3.66 for 1000 predictions.
LLaMA-2 evaluation. For each model output to rate, we randomly select 5 in-context few-shot examples from the manually judged 100 examples and format them as described in Supp. Fig. 12 We use the 70B version and generate one single token with greedy sampling. If the token is not an integer, we set the score to 1. If the token is integer above 5 or below 1, set the score to 5 or
25

---

Published as a conference paper at ICLR 2024
Read the question, the correct answer and the predicted answer.
2. Select the score that best reflects how closely the predicted answer captures the same information as the correct answer.
Note: Please try to not think about what the image could have looked like. This is a text- only task. All you know about the image is the content of the reference. In the rare cases where the question does not make sense, simply compare the predicted answer to the correct answer and ignore the question.
Scores:
1: completely wrong
2: mostly wrong
3: half right
4: mostly right
completely right
Figure 10: Instructions for the human metric annotation task the VQAv2 dataset.
1 respectively. However, in 99% of cases, L.aMA-2 produces integer between and 5. In the O-shot experiment, we remove the 5 in-context examples from the input prompt.
Results. GPT44 performs up par with individual workers in both the short and long prediction case. However, it depends on an external proprietary service and with $3.66 for 1000 predictions, evaluation quickly becomes costly when evaluating several models and datasets. LLaMA-2 is good option that does not depend on an external service but is still computationally expensive with 70B parameters. Notably, LLaMA-2 requires in-context examples perform well. EM as the de facto standard metric for VQA performs reasonably well in the short answer case. However, EM completely fails in evaluating the long answers from LL. We find Cont to the best choice for metric that can be evaluated inexpensively when evaluating short ground truth answers and mixed length model answers. However, it may overestimate the performance of models that list several guesses. We test the extreme case and find that listing the top 20 training set answers for every question achieves 59.91% Cont on VQAv2 (val set).
We also consider n-gram based metrics that used traditionally for evaluating translation models: BLEU, CIDEr, METEOR, and ROUGE. These metrics match n-tuples of words between model prediction and true answer. However, potentially due the ground truth answers in VQAv2 being very short, they do not improve over the more simple EM and Cont. We also observe interesting cases, as in METEOR, where the correlation human judgment is rather high when evaluating only short or long answers but drops when evaluating mixed lengths of answers. This behavior indicates bias to the answer length which only diminishes the correlation once the answer length varies.
Finally, we compare with 3 learned metrics. BertScore (Zhang et al. 2020) computes token simi- larity between embeddings created with RoBERTa (Liu et al.) 2019) while BLEURT-20 (Pu al., 2021) and LERC (Chen et al. 2020) directly learn to output human judgmen scores via regression. However, none of these models beat the Cont baseline. In conclusion, depending on the budget and size of the evaluation, GPT-4 and LLaMA-2 are strong LLM-based choices. For more efficient evaluation, Cont more dependable option compared to EM however is potentially vulnerable against hyperparameter optimization.
26

---

Published as a conference paper at ICLR 2024
User:
You are an annotator that judges the output of QA system. Instructions:
(instructions for the dataset)
Please output JSON format and not output anything else.
Example input:
: "Is the skateboard in the air?", "correct_answer"; "no", "predicted answer": "Yes, the skateboard is in the air, and the girl is sitting on the ground with her feet on the skateboard.") "), (...)
Example output:
If (same fields as input), "score": 1h,
(...)
Real input:
(actual inputs, same format as example inputs)
Real output:
GPT-4: (actual oulpuls, same format example oulputs)
Figure 11: Input prompt for GPT-4 as evaluator.
User:
You are an annotator that judges the output of a QA system. Instructions:
(instructions for the dataset)
Question: the skateboard in the air?
Candidate: Yes, the skateboard is in the air, and the girl is sitting on the ground with her feet on the skateboard.
Reference:
Vote:
(4 more in-context examples)
Question: What type of donut en the top right? Candidate: chocolate glazed donut Reference: chocolate iced glazed
Vote:
LLaMA-2: 4
Figure 12: Input prompt for LLaMA-2 as evaluator. In this example, LLaMA-2 rates the candidate as 4 of 5 (i.e. 80% accuracy).
27

---

Published as a conference paper at ICLR 2024
Variation of Accuracy depending Frame Selection
55
50
45
BLIP
BLIP-2 OPT
40
BLIP-2 T5
Accuracy
InstructBlip
InstructBlip
35
LLavA
X-VLM
30
X--LLM
0.8
Frame Selection
Figure 13: Accuracy of models on ActivityNct-oVQA dataset when varying the frame selection.
D VIDEO FRAME SELECTION
To determine the effectiveness of selecting the middle frame among all others, we conduct an abla- tion study, testing all models over nine frame selections by dividing the segment into nine parts and selecting the closest frame. The results are shown in Supp. Fig. [3 We measure accuracy as the average response the same three initial questions across all models. The results suggest that frame selection does not significantly impact performance, especially since frames are selected from short segments in which the relevant action occurs.
E LIMITATIONS
Our open-ended VQA evaluation still provides room for improvement. Supp. Fig. 14 shows ex- amples of failure cases. Supp. Fig. 14 a) shows the weakness of the Contains metric when listing all possible answers, while Supp. Fig- T and d) show that the ClipM metric can deal with such scenarios by marking the answers wrong. Supp. Fig. [4/b) shows how the label space of the COCO dataset unfairly penalizes correct answers such as "green beans" for the question "What can be seen in the image?". Supp. Fig. (4(c) shows how ClipM can produce sub-optimal similarities. The model answer describes a red slide that should match the activity "fun sliding down", but ClipM gives higher weight to other playground activities.
Finally, Supp. Fig. d) shows two interesting cases: The output "a fossilized leaf" is associated with animals that often occur fossils and therefore shows high similarity to the marine animal "trilobite". On the contrary, the term "fossil" seems to be confused with the digital watch brand of the same name and therefore produces wrong similarities.
28

---

Published as a conference paper at ICLR 2024
Dataset:
OVAD-oVQA
Dataset: COCO-oVQA
Question: What colors do you notice their
Question: What can be seen in the image?
attire?
Label: bowl
Label: white
BLIP vga output: green beans
BLIP-2 OPT output: Black, white, gray, brown, green, blue, purple, red, orange, yellow (...)
LLaVA output: The image shows tray bowl filled with green beans (...)
a
(b)
Dataset: ActivityNet-oVQA
Dataset: ImageNet-oVQA
Question: What is happening in the image?
Question: What can be seen in the image? Label: trilobite
Label: Fun sliding down
LLaVA output: In the image, woman is helping a child get onto a red slide at park, ensuring the child's safety and guiding them they climb onto the playground equipment.
BLIP-2 T5 output: a fossilized leaf ClipM Top-3 similarities: trilobite (0.55l) flatworm (0.539)
ClipM Top-3 similarities:
sea slug (0.502)
Swinging at the playground (0.495)
InstructBLIP T5 output: fossil
Using the monkey bar (0.379)
ClipM Top-1 similarities:
Fun sliding down (0.349)
digital watch (0.322)
purse (0.322)
(c)
fig (0.310)
(d)
Figure 14: Examples of the limitations of the oVQA benchmark. Only the bounding box crop is considered as input image for the model prediction. Coloring: Answers considered correct / wrong under Cont Syn metric. Refer to the text for the discussion the examples.
29


Reviews:
[{"review_id": "wFTGhjKpB3", "reviewer": "Reviewer_bBwP", "summary": "The paper proposes a dataset to granularly evaluate the text-generative vision-language models. The authors base the evaluation benchmark on classification datasets where the labels semantic hierarchy are present. The semantic hierarchy is used as the source of generating the follow-up questions.", "questions": "Please comment on the weakness part.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) propose a new topic for evaluating VQA models that assess how well the VQA models can classify or recognize fine-grained objects, activity, and attributes\n\n(2) A intuitive solution to constructing a benchmark that contains the class semantic hierarchy based on classification model.\n\n(3) comprehensive evaluation on lots of open sourced VQA systems.", "weaknesses": "(1) I believe assessing how granular a VQA system is in good and necessary, however, when evaluating, the current benchmark put additional constrains on the image space, cropping the image to certain objects and using imagenet which is object centric. These constrains largely limited the scope of VQA that is supposed to work on natural use case, for example, natural image QA (VQA v2, vizwiz, etc). \n\n(2) The Cropping activity operations seems very risky the action needs a few frames to evaluate? Like sit down and stand up?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dsQASn1WLm", "reviewer": "Reviewer_3kX8", "summary": "Evaluating text-generative vision-language models remains an intricate undertaking due to the inherent limitations of current Visual Question Answering (VQA) benchmarks. Motivated by the need for a more comprehensive assessment mechanism, this paper aims to address the ambiguity and specificity issues rampant in open-ended VQA (oVQA) tasks. These challenges emerge, in part, from the diverse ways natural language can express similar ideas and the constraints of current evaluation metrics that often favor shorter responses. To navigate these obstacles, this research introduces a novel VQA benchmark, enriched with subbenchmarks focusing on objects, actions, and attributes. It leverages a follow-up procedure, drawing from existing classification benchmarks, to generate contextually apt questions and expects models to provide answers with the requisite level of detail. Additionally, the study explores various metrics from the VQA and NLP spheres to find an optimal evaluation criterion, validated via human judgment. This holistic approach paves the way for an intricate analysis of leading vision-language models, unearthing their specific strengths and vulnerabilities.", "questions": "Please also refer to the previous section.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative Evaluation Methodologies: The research addresses the inadequacies of existing Visual Question Answering (VQA) benchmarks by proposing a new VQA benchmark. This benchmark, derived from well-established visual classification datasets, facilitates a more detailed evaluation of text-generative vision-language models and their comparative performance against discriminative vision-language models.\n\n2. Addressing Coarse Answer Challenges: Recognizing the challenges in evaluating coarse answers in fine-grained tasks, the research introduces a method using the semantic hierarchy of labels. By doing so, it can automatically generate pertinent follow-up questions about the true category, pushing for more accurate and detailed model responses.\n\n3. Enhanced Ambiguity Management: To better handle the inherent ambiguities in open-ended visual questions, a unique follow-up procedure is proposed. By adapting classification benchmarks for oVQA, the model is first provided with an apt visual context, and then, based on its initial response, a further clarifying question is posed using concept hierarchies. This ensures answers with the desired detail and precision.\n\n4. Comprehensive Metric Evaluation: The research undertakes a rigorous examination of various metrics from both the VQA and NLP domains, emphasizing those that treat paraphrasing and synonyms as valid answers. The eventual metric is grounded in a human evaluation study, ensuring that it aligns closely with human judgments and provides a reliable measure for assessing model performances.", "weaknesses": "The paper omits certain statistical details regarding the oVQA dataset, such as the distribution of question/answer lengths, the number of entity class labels, and the formats/types of the questions. Given the dataset's general and expansive nature, there is a concern that, despite the introduction of subbenchmarks, it might introduce new challenges that could affect the quality of evaluations, such as issues related to imbalance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6GqEGkxzzS", "reviewer": "Reviewer_GTBt", "summary": "The work proposes a more principled evaluation of vision-language models by taking advantage of the semantic hierarchy of textual labels. The core problem the work addresses is that the exact match scoring systems used by VQA benchmarks can be ambiguous, and penalize models for essentially correct answers. To fix this, the authors crop images so to the object of interest, allow coarse answers, and ask follow up questions to clarify the fine-grained answer. Finally, they also measure performance with multiple metrics, such as BertScore and GPT-4 based evaluation.", "questions": "I have no questions.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "While a substantial body of work exists on designing better benchmarks for VQA, the idea of using the semantic hierarchy to ask follow up questions is original. VQA is hard to evaluate, yet is the most fundamental task in vision-language (other than perhaps, image-text matching). One of the biggest problems in VQA is that questions can be underspecified or ambiguous and there can be multiple correct answers to questions. \n\nThe method presented here is a neat way to deal with these ambiguities and inconsistencies in the evaluation process. \n\nAnother useful contribution of the paper is the empirical evaluation of vision-language models on different aspects of vision-language, such as attribute recognition and coarse vs fine-grained recognition. This is useful. \n\nFinally, Table 7 is also very useful, especially because it helps resolve questions about the appropriateness of different metrics for open-ended VQA scoring.", "weaknesses": "There are no substantial weaknesses. \n\nA figure showing the differences between a VQA question + label and examples from the proposed datasets would be useful. Fig. 2 would be easier to read if it said \"attribute\", \"object\" etc instead of OVAD, COCO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)