Trust but Verify: Programmatic VLM Evaluation in the Wild

## Abstract

Abstract Vision-Language Models (VLMs) often generate plausible but incorrect responses to visual queries. However, reliably quantifying the effect of such hallucinations in free-form responses to open-ended queries is challenging as it requires visually verifying each claim within the response. We propose Pro grammatic V LM E valuation ( PROVE ), a new benchmarking paradigm for evaluating VLM responses to open-ended queries. To construct PROVE , we provide a large language model (LLM) with a high-fidelity scene-graph representation constructed from a hyper-detailed image caption, and prompt it to generate diverse question-answer (QA) pairs, as well as programs that can be executed over the scene graph object to verify each QA pair. We thus construct a benchmark of 10.5k challenging but visually grounded QA pairs. Next, to evaluate free-form model responses to queries in PROVE , we propose a programmatic evaluation strategy that measures both the helpfulness and truthfulness of a response within a unified scene graph-based framework. We benchmark the helpfulness-truthfulness trade-offs of a range of VLMs on PROVE , finding that very few are in-fact able to achieve a good balance between the two. Project page: https://prove-explorer.netlify.app/ .

## 1 Introduction

Vision-language models (VLMs) have emerged as an effective solution for generating responses to queries about visual content. However, despite impressive progress (and much like their LLM-counterparts), VLMs are still known to hallucinate – to generate plausible but incorrect answers that are either inconsistent or unverifiable against the provided visual context (^1^11A few LLM-focused works also consider responses that contradict *world knowledge* as hallucinations, but we exclude these from our scope.). This crucial shortcoming has the potential to erode trust in such systems and has already begun to attract significant research (Yu et al., ; Liu et al., ; Gunjal et al., ; Huang et al., ) and regulatory (Biden, ) interest, particularly as using such models as the “foundation” of various high-stakes applications becomes imminent (Bommasani et al., ).

This has led to a flurry of research on *reliably* benchmarking VLM performance (Liu et al., ), by measuring not just the helpfulness but also the *truthfulness* of their responses. Existing benchmarks fall into two categories – *discriminative* (Hu et al., ; Lovenia et al., ; Li et al., ), which evaluate the model’s responses to close-ended, existence-based queries (“Is there a man in this image?”), and *generative* (Rohrbach et al., ; Sun et al., ; Liu et al., ; ; Gunjal et al., ), which evaluate responses to free-form, open-ended questions (“Describe this image.”). While discriminative benchmarks ease evaluation, they do not realistically simulate in-the-wild usage. On the other hand, generative benchmarks, while realistic, are *extremely* challenging to reliably evaluate, as they require verifying both that the model response fully answers the question (*i.e.* is helpful) and does not make any false claims (*i.e.* is truthful).

Evaluating such free-form responses typically relies on external models (usually, a proprietary LLM) to score responses given some image context (typically ground-truth annotations). However, we find that in several such benchmarks, the context provided is completely insufficient to judge if the response contains hallucinations. Consider Fig. 1: a VLM may respond to the query “How many puppies are in the image?” (correct answer = “four”), with “There are four labradoodle puppies”. Evaluating the truthfulness of this statement requires verifying multiple claims about the puppies (<count == four> and <breed == labradoodle>); however, an LLM judge provided only with a brief image caption as context (“four puppies placed on a light blue rug”) will be unable to do so! Further, the absence of a clear scoring rubric coupled with the sensitivity of LLMs to minor prompt differences often leads to inconsistent and arbitrary scores in such cases. In Fig. 2, we provide real examples from existing benchmarks that illustrate these challenges.

We propose Programmatic VLM Evaluation (PROVE), a new evaluation paradigm that performs reliable and interpretable *programmatic* evaluation of free-form VLM responses to challenging, diverse, and grounded questions. To build this dataset, we first use hyper-detailed image captions to construct a high-recall scene graph image representation. We then use an LLM to generate a diverse set of open-ended question-answer (QA) pairs along with accompanying *verification programs*.
While the QA pairs are meant to test a range of model capabilities under real-world use, the verification programs can be executed over a given scene graph object to verify the correctness and groundedness of its corresponding QA pair. We only retain the QA pairs that we can programmatically verify and construct a benchmark of 10.5k  diverse and challenging examples which are visually grounded by design, that we can use to reliably benchmark VLM responses.

Next, we benchmark VLM responses to queries in PROVE by comparing scene graph representations. First, we measure the helpfulness of a response by computing its scene graph-based *recall* against the ground truth answer. Next, we measure response truthfulness as its scene graph-based *precision* against both the scene-graph constructed from the full caption or the image itself. We benchmark a range of VLM responses using this approach, and study their respective trade-offs between helpfulness and truthfulness. Our findings suggest that much of the recent progress in training “better” VLMs also translate to improved helpfulness on our benchmark, but not necessarily to higher truthfulness.

Figure: Figure 1: Top. Existing VLM benchmarks either limit query-types to easy-to-evaluate but restrictive binary questions, or use external LLMs to generate open-ended questions (without verifying their validity) and score answers (often without complete image context or a clear scoring rubric). Bottom. We propose PROVE, a new benchmark that constructs high-fidelity scene-graph representations from hyper-detailed image captions, that are queried via an LLM-generated program to verify a free-form generated question-answer pair. At test-time, we perform an interpretable programmatic evaluation of the helpfulness and truthfulness of free-form VLM responses by comparing scene-graphs.
Refer to caption: x1.png

## 2 Related work

Benchmarking VLM hallucination. Existing benchmarks fall into one of two groups (see Fig. 2):

$\triangleright$ Discriminative benchmarks generate a series of binary questions to verify the presence (or absence) of various entities (or distractors) in the image. Early benchmarks like POPE (Li et al., ) limited their scope to object entities annotated by humans or external off-the-shelf models (Zou et al., ), whereas follow-up works additionally evaluate responses to *negative presence* queries (Lovenia et al., ), which stress-test the model’s abstention capabilities on questions about entities *absent* from the image, or use an LLM to generate a broader range of existence-based questions covering objects and their attributes (Hu et al., ). However, while the binary questions that typify such benchmarks simplify evaluation, they do not realistically simulate in-the-wild use.

$\triangleright$ Generative benchmarks instead evaluate model hallucinations in response to free-form questions. CHAIR (Rohrbach et al., ) measures the precision and recall of entities mentioned in a generated image description against the ground truth. HaELM (Wang et al., ) additionally uses a large language model (LLM) to judge generations, whereas M-HalDetect (Gunjal et al., ) has humans annotate hallucinations in model generated descriptions are used to train a predictive model. Recently, AMBER (Wang et al., ) combines a POPE style evaluation with a generative evaluation over an open-ended split. While these benchmarks are indeed more realistic, they still restrict the query instruction to image captioning-style templates (“Describe this image in detail.”).

Most recently, a few benchmarks with truly open-ended queries have been proposed (Sun et al., ; Liu et al., ; Jing et al., ; Liu et al., ), which either hand-design or use an LLM to generate free-form questions, and use external models to judge the corresponding responses. However, these too have limitations: MMHal (Sun et al., ) and HallusionBench (Liu et al., ) rely on a series of off-the-shelf models at various stages which introduce noise (see Fig. 2, col 3). GAVIE’s (Liu et al., ) reliance on dense captions and bounding boxes leads to a majority of questions querying localized image regions and spatial relationships, many of which have unnatural-sounding responses (*eg.* mentioning image coordinates, see Fig. 2, col 4). Finally, GPT-4-based evaluation is both expensive and inherits the model’s own limitations.

Figure: Figure 2: Top. Existing VLM hallucination evaluation benchmarks either measure VLM performance on object existence queries (“discriminative” (Li et al., )) or object precision/recall in generated image captions (“generative, templated” (Rohrbach et al., )), neither of which realistically simulate in-the-wild usage. Some recent benchmarks contain open-ended queries (“generative, free-form” (Sun et al., )), which are more realistic but also harder to both generate (*e.g.* see unnatural QA-pair from GAVIE (Liu et al., ) – first from right), and evaluate with an LLM-as-judge (*e.g.* see GPT-4 penalizing a correct response that includes details absent from the ground truth in MMHal-Bench (Sun et al., ) – second from right). Bottom. We propose PROVE , a benchmark of challenging but verifiable open-ended questions that we use to jointly evaluate both the truthfulness and helpfulness of free-form model responses.
Refer to caption: x2.png

Understanding and mitigating VLM hallucination. Several works have sought to better understand *why* VLMs hallucinate. One prevalent theory is the model learning spurious correlations between the input and the output: either due to overly strong text priors learned by the LLM backbone (Huang et al., ; Leng et al., ), or due to distilling synthetic outputs generated by stronger models (such as GPT-4V) that may themselves contain confabulation (Liu et al., ). This is often exacerbated by the predominant training recipe (Liu et al., ; Abdin et al., ) that learns a shallow projection from the visual input to the text embedding space which limits the expressivity of the model to learn visually grounded representations.

Recent work has proposed training-based and training-free strategies for mitigating hallucinations. The former involves finetuning (Liu et al., ) or preference optimization (Yu et al., ; Sun et al., ) of “preferred” ground truth responses against dis-preferred synthetically generated “hallucinations”. Training-free methods instead focus on specialized decoding strategies (Huang et al., ; Kim et al., ; Leng et al., ) that seek to correct for potential statistical bias that may lead to hallucination.

However, developing better understanding and mitigation strategies are both contingent on the availability of reliable evaluation benchmarks. In this work, we introduce such a benchmark of challenging but verifiable open-ended visual questions that we use to jointly evaluate both the truthfulness and helpfulness of free-form model responses.

## 3 Approach

Figure: Figure 3: The PROVE dataset. For each image-caption pair, we generate a high-fidelity scene graph representation with which we prompt an LLM to generate challenging QA pairs and their verification programs. We only retain QA pairs that we can programmatically verify, ensuring diverse but reliable evaluation data that is grounded by design.
Refer to caption: x3.png

Vision-language models are trained to respond to a question $\mathcal{Q}$  about an image $\mathcal{I}$  with a ground-truth answer $\mathcal{A}$. Let $\mathrm{m}_{\theta}$(.) denote a VLM model trained on a large dataset of such ($\mathcal{I}$, $\mathcal{Q}$, $\mathcal{A}$) triplets. At test time, we wish to evaluate the model response $\hat{\mathcal{A}}$=$\mathrm{m}_{\theta}$($\mathcal{Q}$, $\mathcal{I}$). Specifically, while prior work typically evaluates either the response’s correctness (is $\hat{\mathcal{A}}$$=$$\mathcal{A}$) or truthfulness (is $p$($\hat{\mathcal{A}}$|$\mathcal{I}$) $>$ threshold), we propose a unified framework that jointly evaluates both.

### 3.1 Generating verifiable Visual Question-Answers

To build PROVE , we first download image-caption pairs ($\mathcal{I}$, $\mathcal{C}$) from the test set of the recently proposed DOCCI (Onoe et al., ) dataset, containing 5k manually curated images with comprehensive human-annotated descriptions. DOCCI is particularly well-suited for VLM evaluation because: i) its captions are extremely detailed, with a higher median caption length than competing datasets, which correlates with high image recall ii) its comprehensive and rigorous 3-stage human annotation protocol leads to high-fidelity captions that are suitable to test a range of image understanding challenges including spatial reasoning, counting, text rendering, and compositionality, and iii) its images are newly curated and so more likely to be truly held-out for existing models.

Building a robust scene-graph representation. Scene-graphs are comprised of entity (<entity>), attribute (<entity, attribute>), and relationship (<entity_1, attribute, entity_2>) tuples that describe a scene. We use the tuples included with the DOCCI test set that were automatically extracted from its captions using an LLM (Cho et al., ), and use it to construct a scene graph representation $\mathrm{g}$($\mathcal{C}$) as a directed graph with attributed entities as nodes and relationships as edges. The scene graph is implemented as a Python class with methods to query the graph for its entities, attributes, and relationships, as well as to extract and describe subgraphs in natural language (full API in Lst. 1).

Generating open-ended questions with verifiable answers. Next, for each image, we prompt a pre-trained LLM to generate 10-15 challenging, diverse, and unambiguous question-answer (QA) pairs given a caption and scene graph, along with an accompanying Python program that accepts the scene graph as input and can be executed to verify the generated QA pair (Gupta & Kembhavi, ; Surís et al., ). We include a few in-context examples of such scene-graph and QA+program input/output pairs in the prompt (see Fig. 8). We repeat this procedure to generate a large dataset of open-ended image+QA pairs $\{(\mathcal{I}_{i},\mathcal{Q}_{i},\mathcal{A}_{i})\}_{i=1}^{N}$ and their verification programs.

Filtering QA pairs. Next, we perform two rounds of filtering:

1. Programmatic: First, we execute the generated program with the scene graph as input to verify the QA pair. We discard pairs for which the program either fails or returns an answer that is semantically different (Reimers, ) from the ground truth answer.

2. Text-based: Next, we perform a few additional post-processing steps to exclude low-quality QA pairs which are i) trivial, ungrammatical, ambiguous, or incomplete (using an LLM, see Fig. 9), ii) not entailed by the image (using a visual entailment model (Cai et al., )), iii) include one or more words from a manually curated list of taboo words that we find to result in low-quality questions, or iv) semantic duplicates for the same image (using SemDeDup (Abbas et al., )). Our final dataset after filtering contains  10.5k  high-quality visual question answers – see Fig. 3.

Dataset statistics. We now present some statistics about PROVE, which comprises of 10.5k QA pairs generated from 5k image-caption pairs from the DOCCI test set. These are obtained after applying both programmatic filtering *i.e.* either the unit test fails (18.3%) or returns the wrong answer (9.8%), and text-based filtering ($\sim$ 50% of the total from the previous stage). Note that we opt to filter out such a large percentage of QA pairs in the interest of ensuring high-quality evaluation data. Further, our benchmark curation process is fully automatic and so can be readily scaled to a larger image-caption source. Questions in PROVE average 10.3 words in length whereas answers average 13.4 words (see Fig. 6, *right*). In Fig. 6, *left* we present a sunburst visualization of the first 4 words in the questions, highlighting the diversity of questions in our benchmark.

### 3.2 Programmatic VLM Evaluation ( PROVE )

After ensuring the validity of the generated QA pairs, we proceed to evaluating free-form VLM responses to the same $\hat{\mathcal{A}}$=$\mathrm{m}_{\theta}$($\mathcal{Q}$, $\mathcal{I}$). We first extract tuples from $\hat{\mathcal{A}}$ (using an LLM (Dubey et al., ) with in-context prompting), that we use to build a scene graph representation $\mathrm{g}$($\hat{\mathcal{A}}$). We also build a similar scene graph from the ground truth answer tuples after excluding “premise” tuples included in the question $\mathrm{g}(\mathcal{A})-\mathrm{g}(\mathcal{Q})$. We then measure response helpfulness $\mathsf{hscore}$(.) based on *recall* of this scene graph, *i.e.* the fraction of tuples (nodes, attributes, and relationships) in $\mathrm{g}(\mathcal{A})-\mathrm{g}(\mathcal{Q})$ that are recovered by $\mathrm{g}$($\hat{\mathcal{A}}$).
Concretely, we compute average cosine similarity between each ground truth tuple and its closest response tuple in embedding (Reimers, ) space.

$$ $\mathsf{hscore}(\hat{\mathcal{A}})=\frac{\sum_{t\in\mathrm{g}(\mathcal{A})- \mathrm{g}(\mathcal{Q})}\max_{t^{\prime}\in\mathrm{g}(\hat{\mathcal{A}})}\text {sim}(t,t^{\prime})}{|\mathrm{g}(\mathcal{A})-\mathrm{g}(\mathcal{Q})|};\;\;\;$ (1) $$

Next, we compute $\mathsf{tscore}$(.) as the *precision* of the response *i.e.* the fraction of response tuples that are consistent with either the original scene graph *or* the image itself(^2^22This reduces false-positive hallucination detections, as no caption can capture every aspect of an image.). We define:

$$ $\mathsf{tscore}(\hat{\mathcal{A}})=\frac{\sum_{t^{\prime}\in\mathrm{g}(\hat{ \mathcal{A}})}\text{max}\big{(}\max_{t\in\mathrm{g}(\mathcal{C})}\text{sim}(t^ {\prime},t),p(\mathcal{I}\models t^{\prime})\big{)}}{|\mathrm{g}(\hat{\mathcal {A}})|};\;\;\;$ (2) $$

where $\models$ denotes visual entailment, and $p(\mathcal{I}\models t^{\prime})$ is approximated using a visual entailment model (Cai et al., ). Note that $\mathsf{hscore}$ and $\mathsf{tscore}$ are not necessarily correlated – a response can be helpful (by answering the query) but not entirely truthful (might contain hallucinations), and vice versa. Naturally, different models may achieve different trade-offs between the two – an aspect that PROVE is uniquely suited to analyze.

## 4 Experiments

**Table 1: Benchmarking VLMs on PROVE (*=LLaMA-3.1 (Dubey et al., ) backbone, ^†=closed-source). For each model, we report helpfulness ($\mathsf{hscore}$), truthfulness ($\mathsf{tscore}$), and their average. We find larger and more recent models achieve higher $\mathsf{hscore}$ but not necessarily higher $\mathsf{tscore}$ .**
| Method | #params | $\mathsf{hscore}$ ($\uparrow$) | $\mathsf{tscore}$ ($\uparrow$) | average ($\uparrow$) |
| --- | --- | --- | --- | --- |
| Qwen2-VL (Bai et al., ) | 2B | 69.36 | 80.64 | 75.00 |
| InternVL2 (Chen et al., ) | 2B | 73.96 | 79.51 | 76.74 |
| Phi-3.5-Vision (Abdin et al., ) | 4B | 73.35 | 82.27 | 77.81 |
| LLaVA-1.5 (Liu et al., ) | 7B | 72.67 | 82.58 | 77.62 |
| LLaVA-Next (Liu et al., ) | 7B | 74.28 | 80.03 | 77.15 |
| InternVL2 (Chen et al., ) | 8B | 74.55 | 80.56 | 77.56 |
| Pixtral (Mistral, ) | 12B | 73.34 | 82.43 | 77.88 |
| LLaVA-1.5 (Liu et al., ) | 13B | 72.46 | 82.40 | 77.43 |
| InternVL2 (Chen et al., ) | 26B | 74.63 | 79.23 | 76.93 |
| Claude-3.5-Sonnet^† (Anthropic, ) | - | 71.06 | 77.31 | 74.19 |
| GPT-4o-mini^† (Achiam et al., ) | - | 73.18 | 79.24 | 76.21 |
| Gemini-1.5-Flash^† (Team et al., ) | - | 72.73 | 81.74 | 77.23 |
| GPT-4o^† (Achiam et al., ) | - | 76.53 | 80.92 | 78.72 |
| Oracle* | - | 82.84 | 85.59 | 84.22 |

We now present our benchmarking experiments on PROVE. We include a broad set of models spanning a range of sizes and learning strategies and extensively analyze their performance, including their performance trade-offs. We also conduct a human study to validate both the quality of our benchmark and how well our proposed metrics correlate with human judgement.

### 4.1 Setup

Baselines. We include VLMs of three sizes – small (<5B parameters), medium (5-10B parameters), and large (>10B parameters) – and include both open-source and proprietary models. We also benchmark an additional LLM-based “oracle” model as an upper bound (a blind model that is provided with the ground truth caption image). For the model, we use a LLaMA-3.1-8B backbone (Dubey et al., ) with in-context prompting.

Data. PROVE  is constructed from images, tuples, and captions released under a CC by 4.0 license as the test split of the DOCCI (Onoe et al., ) dataset. DOCCI images were reviewed both by human and automatic methods to remove or obfuscate PII (faces, phone numbers, and URLs) and unsafe content. Images underwent a rigorous 3-stage human annotation phase resulting in hyper-detailed and high-recall captions averaging 136 words.

Implementation details. We use GPT-4o (Achiam et al., ) for generating structured question, answers, and verification programs using the batch API and prompting it with a detailed task description, examples, and a Python definition of the SceneGraph class. We also use GPT-4o for the first round of text-based post-processing described in Sec. 3.1. We use OFA (Cai et al., ) fine-tuned for visual entailment for both post-processing and measuring image-tuple entailment (Eq. 2), and Sentence-Bert  (Reimers, ) to extract text embeddings.

Figure: Figure 4: We plot $\mathsf{hscore}$ and $\mathsf{tscore}$ for VLMs on PROVE – as seen, models with higher helpfulness tend to lag behind on truthfulness, with very few striking a good trade-off between the two. Averaged across models, we observe a weak linear correlation of 0.03 between $\mathsf{hscore}$ and $\mathsf{tscore}$ .
Refer to caption: extracted/5932966/figures/tradeoffs.png

### 4.2 Results

Table 1 and Figure 4 present evaluation results. We find that:

$\triangleright$ Few models strike a good balance between helpfulness and truthfulness. As Fig. 4 (left) shows, models tend to exhibit a range of trade-offs between helpfulness and truthfulness, with very few from the subset that we study (GPT-4o, Phi-3.5-Vision, Pixtral) managing to strike a good balance between the two. In fact, we find that many recent models that rank highly on perception and reasoning-focused aggregate benchmarks (Lu et al., ), such as Claude-3.5-Sonnet (Anthropic, ) and Intern-VL2 (26B) (Chen et al., ) *do not* necessarily translate to high truthfulness on PROVE, lagging behind simpler and smaller models like LLaVA-1.5 (Liu et al., ) in $\mathsf{tscore}$. In fact, we find the LLaVA-1.5 model series to obtain the best $\mathsf{tscore}$ overall. Overall, we observe a weak linear correlation of 0.03 between $\mathsf{hscore}$ and $\mathsf{tscore}$ averaged across models, suggesting that the impressive recent gains in model helpfulness have not necessarily translated to higher truthfulness.

Table 1 shows a more detailed breakdown of performance and includes additional baselines, categorized by parameter count. As seen, the oracle model
does considerably better than other models, indicating that there is still significant room for improvement in the current generation of VLMs.

$\triangleright$ Increasing model size improves $\mathsf{hscore}$  but not necessarily $\mathsf{tscore}$. Across both model families that we benchmark at multiple sizes – InternVL2 (Chen et al., ) (2B, 8B, and 26B), and LLaVA (Liu et al., ) (1.5-7B, Next-7B, and 1.5-13B), we find that larger or more recent variants tend to outperform smaller ones in terms of helpfulness but not necessarily truthfulness.

Figure: Figure 5: Example responses from two VLMs that achieve high $\mathsf{hscore}$ (GPT-4o) and $\mathsf{tscore}$ (LLaVA-1.5 (7B)) respectively. While both models struggle with sub-tasks such as OCR, counting, and reading an analog clock, GPT-4o’s errors tend to be less egregious which leads to a higher $\mathsf{hscore}$.
Refer to caption: x4.png

$\triangleright$ Models fail in different ways. In Fig. 5 we provide example responses from two models with high $\mathsf{hscore}$ (GPT-4o) and $\mathsf{tscore}$ (LLaVA-1.5-7B) respectively. We find that while both models struggle with subtasks such as OCR, counting, and reading an analog clock, GPT-4o’s errors tend to be less egregious (*e.g.* reading 3/6 letters of the graffiti correctly, while LLaVA only gets 1/6). Further, GPT-4o tends to generate more descriptive answers (*e.g.* correctly identifying that while the wall in the first image is white, the bricks at the bottom are gray), which boost its $\mathsf{hscore}$. In Fig. 7(a), we include a fine-grained analysis of GPT-4o performance across different question types.

Human evaluation of PROVE  and proposed metrics. Finally, we conduct two human studies of our benchmark. We first ask human annotators (3 per example) to evaluate the question relevance and answer correctness of QA pairs generated from the qual-test split of the DOCCI dataset (100 images, 170 generated QA pairs) that is specifically set aside for human evaluation. After majority voting, annotators judge 163/170 questions to be relevant (95.9%) and 167/170 answers to be correct (98.2%). We manually inspect the small number of examples judged as irrelevant or incorrect in and find most to be either particularly challenging or subjective, rather than irrelevant or incorrect.

In the second study, we ask subjects (3 per example) to *rate* responses from four models – GPT-4o, LLaVA-1.5-7B, LLaVA-Next-7B, and GPT-4o-mini – on the same set of 170 QA pairs based on their helpfulness (0=unhelpful, 1=helpful) and truthfulness (0=fully false, 0.5=partially false, 1.0=fully true), and average. We then automatically compute $\mathsf{hscore}$ and $\mathsf{tscore}$ for the same set of responses and measure the Pearson correlation between the two, observing a strong correlation of 0.81 for $\mathsf{hscore}$ and a modest correlation of 0.45 for $\mathsf{tscore}$, supporting the validity of these metrics.

## 5 Discussion

Our work takes a step towards reliably evaluating the helpfulness-truthfulness trade-offs of vision-language models. Our design leverages an LLM prompted with a robust scene graph representation and API to construct “in-the-wild” visual question answer pairs that are *grounded by design*. Further, these QA pairs lend themselves to programmatic evaluation via comparing scene-graph representations. The reliability of our benchmark comes from three factors: i) high-recall human-annotated image captions that seed the scene graphs, which make it possible to (almost) exhaustively validate the veracity of any claim ii) programmatic verification of the generated QA pairs that ensure that both the question and answer are indeed grounded in the visual input, and iii) evaluation metrics that are both holistic (*i.e.* consider all the provided context) and interpretable (*i.e.* provide a concrete scoring rubric based on scene graph-based matching).

Limitations. While we hope that PROVE will serve as a useful test-bed for reliable VLM evaluation and spur future research on the topic, it is not without limitations. While we try to ensure high precision in QA pairs retained in our benchmark (via programmatic verification), this naturally comes at some cost to recall (*i.e.* some hard-to-verify question types may be excluded from the benchmark). Next, even high-recall image captions may not capture every aspect of an image, and so our evaluation may not be able to catch all model hallucinations. Further, our evaluation relies on off-the-shelf models for computing text-embeddings, scene graph tuples, and image-text entailment, and so almost certainly inherits some of their limitations. Finally, we hope future work will study the effectiveness of recent fine-tuning (Liu et al., ), preference-tuning (Yu et al., ; Sun et al., ), and training-free (Huang et al., ; Kim et al., ; Leng et al., ) hallucination mitigation strategies on PROVE, as well as agentic models that can plan (Surís et al., ; Gupta & Kembhavi, ), reason, and self-reflect (Valmeekam et al., ), towards the elusive goal of achieving Pareto improvements in both helpfulness and truthfulness.

Acknowledgements. We would like to thank Jieyu Zhang and Artemis Panagopoulou for valuable feedback and discussions, Juan-Carlos Niebles, Gabriel Bernadette Shapiro, and Silvio Savarese for feedback on our draft, and Srinath Reddy Meadusani for help with compute infrastructure.

## Appendix A Appendix

Figure: (a) Sunburst visualization of the first 4 question words in PROVE.
Refer to caption: extracted/5932966/figures/prove_v1.0_sunburst.png

### A.1 Additional dataset details

Fig. 6(a), *left* presents a sunburst visualization of the first 4 words in the questions within the PROVE dataset. As seen, the questions are diverse and span a wide range of question types. Further, while nearly 50% of the questions begin with “What”, even this subset spans a range of topics testing numerous model capabilities – see Fig. 5. Fig. 6, *right* shows the distribution of question and answer lengths in PROVE. Questions in the dataset average 10.3 words in length, whereas answers average 13.4 words, with both following a normal distribution spread.

### A.2 Additional implementation details

Lst. 1 provides a Python implementation of the SceneGraph class used to represent scene graphs in PROVE. The class provides methods to generate subgraphs, describe subgraphs in natural language, and query entities, attributes, and relationships. We include the prompts used for generating verifiable question-answer pairs as well as the post-processing prompt used to filter out low-quality QA pairs in Figs. 8- 9.

### A.3 Additional performance analysis

Fig. 7(a) presents a fine-grained performance analysis of GPT-4o on PROVE. We break down helpfulness and truthfulness scores by question type, and display the top-10 most common question types sorted by performance. As seen, the model performs particularly well on questions that require reasoning about spatial relationships (where are/is), object attributes (what color), and generating image descriptions.

Fig. 7(b) shows a word cloud of the most commonly hallucinated objects in answers to questions from PROVE across all models. As seen, models commonly hallucinate common objects such as “tree”, “building’, “wall”, and “sign”.

Figure: (a) GPT-4o fine-grained performance analysis
Refer to caption: extracted/5932966/figures/scores.png

Figure: Listing 1: Python API for the SceneGraph class

Figure: Figure 8: LLM prompt for generating visual question-answer pairs along-with verification programs.

Figure: Figure 9: LLM text-based post-processing prompt.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See Weakness
soundness: 2
strengths: I do like how this method could evaluate VLM compositionally with a prepared set of programs, instead of all in a whole with a LLM. 
The design of helpfulness and truthfulness is interesting. It is interesting to find a way to evaluate hallucination in VLMs.
summary: Programmatic VLM Evaluation (PROVE) introduces a novel benchmark for assessing VLMs. Normally, we evaluate image captioning with the generated caption and the gold caption as two whole paragraphs. Building upon DOCCI, which is a new dataset that came out this year, PROVE collects a robust set of 10.5k visually grounded question-answer (QA) pairs by using a detailed scene-graph approach that evaluates image captions compositionally. It provides a programmatic evaluation strategy that measures both the helpfulness and truthfulness of a free-form model responses.
weaknesses: 1. The presentation is poor. It took me a while to finally realize that this paper presents a way to evaluate image-captioning, through breaking the captioning task into VQA tasks, and the answers are evaluated by a program generated by GPT based on gold scene graph, instead of evaluated by GPT based on the gold caption. 

2. The results in Table1 and example outputs in Figure 5 are very confusing. Why are the performance of all models look similar? From my personal experience, GPT-4o should be better than other models, especially the open-source models by a lot. But they seem to have same performance as in Table1. From the original DOCCI paper, different models also score very differently. From Figure 5 in the first example's top question, I don't understand why the hscore for GPT-4o and LLaVA are both so high -- none of them gave the correct answer. In that same example's bottom question, I don't understand why the hscore for LLaVA is so low, given that it answers the question perfectly. 

3.  All the questions are generated by LLM. This could potentially only include easy questions, and might explain why the performance in Table 1 are all similar. 

4. The Oracle setting result is way too low -- only 4 % higher than all other models. Isn't the Oracle setting the same setting you applied when generating the QA dataset? Shouldn't this setting then achieve 100% in accuracy? 

5. The average cosine similarity score in (1) and (2) is not very convincing. From Figure 5 in the first example's top question, I don't understand why the hscore for GPT-4o and LLaVA are both so high -- none of them gave the correct answer. In that same example's bottom question, I don't understand why the hscore for LLaVA is so low, given that it answers the question perfectly.


===

Thanks for the clarifications. I've adjusted my rating given the additional scores and explanations.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Questions on the human study:
- Are human ratings on helpfulness continuous or discrete?
- The correlation score with the helpfulness score is 0.54 and quite low – can the authors provide insights into why this is the case? Is there anything humans pay more attention to for helpfulness that the hscore doesn’t capture?
soundness: 2
strengths: - The paper is well-motivated and tackles an important research problem in VLMs evaluation. The inclusion of truthfulness in addition to helpfulness is thoughtful and often neglected. 
- The paper is generally well-written with clear definitions of the helpfulness and truthfulness metrics, and helpful illustrations like figure 4. 
- The evaluation covers a broad range of models. 
- The authors perform multiple data filtering steps to ensure the correctness of the programs and high quality of the QA pairs.
summary: This paper proposes a new benchmark which the authors constructed by turning detailed captions into scene graph representations, and generating QA pairs as well as the corresponding verification programs based on the scene graphs. It also proposes a programmatic evaluation of the VLMs’ responses on both helpfulness and truthfulness by comparing the predicted scene graphs and ground-truth scene graphs.
weaknesses: - The reviewer is mostly concerned about the use of models in multiple parts of the dataset generation, filtering, and evaluation pipeline, especially in extracting the scene graphs from captions. 
- For example, the scene graphs are not guaranteed to be completely accurate, as they are automatically extracted from the captions in the DOCCI dataset by an LLM without any human verification or filtering. 
- Similarly, as the authors mentioned, the sentence BERT model and visual entailment model OFA are used for metrics calculation, which means the evaluation accuracy is limited by these models’ accuracies, making this a much less rigorous evaluation benchmark. Have the authors analyzed how errors in these models might propagate through the evaluation?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness.
soundness: 3
strengths: 1. The definition of the two metrics, i.e. helpfulness and trustfulness, based on the scene graphs, is interesting.
2. The writing is clear and easy to follow.
summary: The paper proposes a new evaluation paradigm, named PROVE, for evaluating the “helpfulness” and “trustfulness” of VLM. The evaluation is based on the newly proposed dataset, where the images are paired with LLM generated dense scene graphs using detailed descriptions and the QA pairs, together with an executable program to derive/verify the answer, are generated using LLMs. Then a “helpfulness” score (measuring the recall of entities) an “trustfulness” score (measuring the precision) are defined based on the program and the scene graph. Two limited human evaluations are provided, verifying the correctness of the eval paradigm. Multiple VLMs are evaluated, showing that these models can hardly balance the two metrics.
weaknesses: 1. Generalizability of the proposed evaluation paradigm is limited. The evaluation requires a dense scene graph and an executable program, which limits its usage to only the proposed dataset. The evaluation can be hardly generalized to images/questions without detailed annotations. Moreover, the evaluation’s effectiveness is bounded by the quality of the dense scene graph/detailed caption. Anything that is not in the scene graph cannot be evaluated. This is not exactly a “in-the-wild” evaluation as the paper claimed.
2. What is the advantage of the proposed method, over the claim-based evaluation method, where the model’s answer is decomposed into several claims, then LLMs verify each of the claims directly? The advantage of the latter includes that it is more flexible (does not require scene graph/program, thus can be applied to open world images), and more simple (thus is easier to apply).
3. The human study shows Pearson correlation of 0.81 for helpfulness and 0.45 for trustfulness, which is not super-high (especially for trustfulness). Any analysis on this? Another human verification can be conducted in a side-by-side manner: given 2 model responses for the same question, let human raters annotate which one is better (in terms of helpfulness and trustfulness), then use this side-by-side human rating to check the evaluation scores.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - The citation for OFA seems to be wrong; it should be Wang et al., https://arxiv.org/abs/2202.03052
soundness: 2
strengths: - (1) Introduction of new hallucination benchmark for VLM based on programmatic questions generated from scene graph.
summary: The authors propose PROVE, a new evaluation benchmark for VLMs’ hallucination. PROVE is built on top of the DSG scene graphs of DOCCI dataset, by generating question-answer pairs from the scene graphs and pragmatic/text-based filtering of wrong question-answer pairs. The question/GT answer/verification programs are generated with GPT4o, and the answers generated by VLMs are evaluated with Sentence-BERT and OFA. PROVE has two metrics — hscore (helpfulness) and tscore (truthfulness), based on recall and precision with respect to scene graphs, respectively. The authors evaluate different VLMs in PROVE, and show that different models have different balance between helpfulness/truthfulness. The authors also provide human evaluation showing that PROVE questions are mostly relevant and correct.
weaknesses: - (1) **Methodological contribution is weak.** The authors generate question-answer pairs from the existing Davidsonian scene graphs (DSG; Cho et al., 2024) already shared by DOCCI (Onoe et al., 2024) authors. DSG paper already provides a question-answering generation pipeline from the scene graphs, and it is not clear how the proposed question-answering generation pipeline is more novel or more helpful.

- (2) **No comparison with existing metrics/benchmarks.** The authors mention many VLM hallucination benchmarks in the related work but do not show qualitatively or quantitatively how PROVE is better than existing benchmarks.

Meta-review: The authors present Programmatic VLM Evaluation (PROVE), which introduces a new benchmark for assessing VLMs. The AC appreciates the authors' efforts and improvements. However, explicit qualitative and quantitative comparisons between DSG and PROVE are essential to establish the necessity and effectiveness of the proposed framework. The low correlation (0.45) between truthfulness and human judgment also weakens the evaluation. Addressing these issues would significantly strengthen the submission. The AC shares the lack of enthusiasm with all four reviewers, but would encourage the authors to revise and improve.

Decision: Reject