Trust but Verify: Programmatic VLM Evaluation in the Wild

Authors: Viraj Uday Prabhu, Senthil Purushwalkam, An Yan, Caiming Xiong, Ran Xu
Primary area: datasets and benchmarks
Keywords: vision-language models, evaluation, hallucinations

Abstract:
Vision-Language Models (VLMs) often generate plausible but incorrect responses to visual queries. However, reliably quantifying the effect of such hallucinations in free-form responses to open-ended queries is challenging as it requires visually verifying each claim within the response. We propose Programmatic VLM Evaluation (PROVE), a new benchmarking paradigm for evaluating VLM responses to open-ended queries. To construct PROVE, we provide a large language model with a high-fidelity scene-graph representation constructed from a hyper-detailed image caption, and prompt it to generate diverse question-answer (QA) pairs, as well as programs that can be executed over the scene graph object to _verify_ each QA pair. We thus construct a benchmark of 10.5k challenging but grounded visual QA pairs. Next, to evaluate free-form model responses to queries in PROVE, we propose a _programmatic_ evaluation strategy that measures both the helpfulness and truthfulness of a response within a unified scene graph-based framework. We benchmark the helpfulness-truthfulness trade-offs of a range of VLMs on PROVE, finding that very few are in-fact able to achieve a good balance between the two.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729975358353

thread_type:
official_review

<thread_object>
creator: reviewer_42b9

<content>
summary:
The authors propose PROVE, a new evaluation benchmark for VLMs’ hallucination. PROVE is built on top of the DSG scene graphs of DOCCI dataset, by generating question-answer pairs from the scene graphs and pragmatic/text-based filtering of wrong question-answer pairs. The question/GT answer/verification programs are generated with GPT4o, and the answers generated by VLMs are evaluated with Sentence-BERT and OFA. PROVE has two metrics — hscore (helpfulness) and tscore (truthfulness), based on recall and precision with respect to scene graphs, respectively. The authors evaluate different VLMs in PROVE, and show that different models have different balance between helpfulness/truthfulness. The authors also provide human evaluation showing that PROVE questions are mostly relevant and correct.

strengths:
- (1) Introduction of new hallucination benchmark for VLM based on programmatic questions generated from scene graph.

weaknesses:
- (1) **Methodological contribution is weak.** The authors generate question-answer pairs from the existing Davidsonian scene graphs (DSG; Cho et al., 2024) already shared by DOCCI (Onoe et al., 2024) authors. DSG paper already provides a question-answering generation pipeline from the scene graphs, and it is not clear how the proposed question-answering generation pipeline is more novel or more helpful.

- (2) **No comparison with existing metrics/benchmarks.** The authors mention many VLM hallucination benchmarks in the related work but do not show qualitatively or quantitatively how PROVE is better than existing benchmarks.

questions:
- The citation for OFA seems to be wrong; it should be Wang et al., https://arxiv.org/abs/2202.03052
</content>
</thread_object>

</thread>
<thread>
creation date:
1730313440800

thread_type:
official_review

<thread_object>
creator: reviewer_gdu3

<content>
summary:
The paper proposes a new evaluation paradigm, named PROVE, for evaluating the “helpfulness” and “trustfulness” of VLM. The evaluation is based on the newly proposed dataset, where the images are paired with LLM generated dense scene graphs using detailed descriptions and the QA pairs, together with an executable program to derive/verify the answer, are generated using LLMs. Then a “helpfulness” score (measuring the recall of entities) an “trustfulness” score (measuring the precision) are defined based on the program and the scene graph. Two limited human evaluations are provided, verifying the correctness of the eval paradigm. Multiple VLMs are evaluated, showing that these models can hardly balance the two metrics.

strengths:
1. The definition of the two metrics, i.e. helpfulness and trustfulness, based on the scene graphs, is interesting.
2. The writing is clear and easy to follow.

weaknesses:
1. Generalizability of the proposed evaluation paradigm is limited. The evaluation requires a dense scene graph and an executable program, which limits its usage to only the proposed dataset. The evaluation can be hardly generalized to images/questions without detailed annotations. Moreover, the evaluation’s effectiveness is bounded by the quality of the dense scene graph/detailed caption. Anything that is not in the scene graph cannot be evaluated. This is not exactly a “in-the-wild” evaluation as the paper claimed.
2. What is the advantage of the proposed method, over the claim-based evaluation method, where the model’s answer is decomposed into several claims, then LLMs verify each of the claims directly? The advantage of the latter includes that it is more flexible (does not require scene graph/program, thus can be applied to open world images), and more simple (thus is easier to apply).
3. The human study shows Pearson correlation of 0.81 for helpfulness and 0.45 for trustfulness, which is not super-high (especially for trustfulness). Any analysis on this? Another human verification can be conducted in a side-by-side manner: given 2 model responses for the same question, let human raters annotate which one is better (in terms of helpfulness and trustfulness), then use this side-by-side human rating to check the evaluation scores.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621898955

thread_type:
official_review

<thread_object>
creator: reviewer_ubc1

<content>
summary:
This paper proposes a new benchmark which the authors constructed by turning detailed captions into scene graph representations, and generating QA pairs as well as the corresponding verification programs based on the scene graphs. It also proposes a programmatic evaluation of the VLMs’ responses on both helpfulness and truthfulness by comparing the predicted scene graphs and ground-truth scene graphs.

strengths:
- The paper is well-motivated and tackles an important research problem in VLMs evaluation. The inclusion of truthfulness in addition to helpfulness is thoughtful and often neglected. 
- The paper is generally well-written with clear definitions of the helpfulness and truthfulness metrics, and helpful illustrations like figure 4. 
- The evaluation covers a broad range of models. 
- The authors perform multiple data filtering steps to ensure the correctness of the programs and high quality of the QA pairs.

weaknesses:
- The reviewer is mostly concerned about the use of models in multiple parts of the dataset generation, filtering, and evaluation pipeline, especially in extracting the scene graphs from captions. 
- For example, the scene graphs are not guaranteed to be completely accurate, as they are automatically extracted from the captions in the DOCCI dataset by an LLM without any human verification or filtering. 
- Similarly, as the authors mentioned, the sentence BERT model and visual entailment model OFA are used for metrics calculation, which means the evaluation accuracy is limited by these models’ accuracies, making this a much less rigorous evaluation benchmark. Have the authors analyzed how errors in these models might propagate through the evaluation?

questions:
Questions on the human study:
- Are human ratings on helpfulness continuous or discrete?
- The correlation score with the helpfulness score is 0.54 and quite low – can the authors provide insights into why this is the case? Is there anything humans pay more attention to for helpfulness that the hscore doesn’t capture?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697303801

thread_type:
official_review

<thread_object>
creator: reviewer_wmz9

<content>
summary:
Programmatic VLM Evaluation (PROVE) introduces a novel benchmark for assessing VLMs. Normally, we evaluate image captioning with the generated caption and the gold caption as two whole paragraphs. Building upon DOCCI, which is a new dataset that came out this year, PROVE collects a robust set of 10.5k visually grounded question-answer (QA) pairs by using a detailed scene-graph approach that evaluates image captions compositionally. It provides a programmatic evaluation strategy that measures both the helpfulness and truthfulness of a free-form model responses.

strengths:
I do like how this method could evaluate VLM compositionally with a prepared set of programs, instead of all in a whole with a LLM. 
The design of helpfulness and truthfulness is interesting. It is interesting to find a way to evaluate hallucination in VLMs.

weaknesses:
1. The presentation is poor. It took me a while to finally realize that this paper presents a way to evaluate image-captioning, through breaking the captioning task into VQA tasks, and the answers are evaluated by a program generated by GPT based on gold scene graph, instead of evaluated by GPT based on the gold caption. 

2. The results in Table1 and example outputs in Figure 5 are very confusing. Why are the performance of all models look similar? From my personal experience, GPT-4o should be better than other models, especially the open-source models by a lot. But they seem to have same performance as in Table1. From the original DOCCI paper, different models also score very differently. From Figure 5 in the first example's top question, I don't understand why the hscore for GPT-4o and LLaVA are both so high -- none of them gave the correct answer. In that same example's bottom question, I don't understand why the hscore for LLaVA is so low, given that it answers the question perfectly. 

3.  All the questions are generated by LLM. This could potentially only include easy questions, and might explain why the performance in Table 1 are all similar. 

4. The Oracle setting result is way too low -- only 4 % higher than all other models. Isn't the Oracle setting the same setting you applied when generating the QA dataset? Shouldn't this setting then achieve 100% in accuracy? 

5. The average cosine similarity score in (1) and (2) is not very convincing. From Figure 5 in the first example's top question, I don't understand why the hscore for GPT-4o and LLaVA are both so high -- none of them gave the correct answer. In that same example's bottom question, I don't understand why the hscore for LLaVA is so low, given that it answers the question perfectly.


===

Thanks for the clarifications. I've adjusted my rating given the additional scores and explanations.

questions:
See Weakness
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors present Programmatic VLM Evaluation (PROVE), which introduces a new benchmark for assessing VLMs. The AC appreciates the authors' efforts and improvements. However, explicit qualitative and quantitative comparisons between DSG and PROVE are essential to establish the necessity and effectiveness of the proposed framework. The low correlation (0.45) between truthfulness and human judgment also weakens the evaluation. Addressing these issues would significantly strengthen the submission. The AC shares the lack of enthusiasm with all four reviewers, but would encourage the authors to revise and improve.