Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction
Authors: Koki Maeda, Shuhei Kurita, Taiki Miyanishi, Naoaki Okazaki


Reviewer paper summaries: This paper introduces a reference-free measure VisCE2 for evaluating image caption quality. It consists of a two stage approach. In the first stage (VLM-EXT), a large vision and language model is used to extract a "visual context" including objects, attributes, and relationships between objects from the image. In the second phase (VLM-EVAL), the candidate caption is evaluated by a vision-language model using a prompt which combines the reference image, the candidate caption, and the visual context and asks the model to predict a quality score ranging from 0 to 100.  The evaluations show that VisCE2 correlates significantly better with human judgments of caption "precision" on the THUMB dataset, and correlates well with human judgements of caption quality on the Flickr8k and Composite datasets.
Summary of strengths: This paper tackles an interesting area in reference-free evaluation of image captioning models. The paper is clearly organized, and it is clear from the paper both (1) how the method is implemented, and (2) the implementation details of the method. The approach is fairly novel, and the evaluation and discussion is quite interesting, particularly 5.1, which looks at the effects of the underlying components, and shows some interesting patterns in how different underlying models affect the measure. This paper presents a fairly simple, and direct approach for reference-free evaluation of image captions.
Summary of weaknesses: While the proposed metric has many strengths, it also has some weaknesses:

- **Metric Variance**: It's not immediately clear from the paper what the variance of the measure is (or even if the measure is deterministic). Given that large VLMs can produce many outputs (esp. given that the sampling parameters are not specified in the paper), the underlying metric may have some variance as well. This induces a question for the authors: what are the sampling parameters that were used for the measure in both stages, and how do these impact the evaluation of the measure?
- **Model Baselines**: While the VisCE2 paper is quite simple already in terms of evaluation, I'm curious if an even simpler (and perhaps more controversial) approach of just asking the VLM to evaluate caption quality would perform as well as the proposed method. Similar to CLAIR, which simply asks for caption quality estimations, would a VLM-version of that measure perform as well as the proposed approach? This may be the None (Vanilla) line of Table 3, but it's unclear to me if that's actually the case. Similarly, in section 4.2 the paper mentions that an ensemble of LLMs is effective for the CLAIR metric - would it be possible/reasonable to apply a similar ensembling technique here to increase quality/variance?
- **Ref-VisCE2**:  While many measures have a reference-based version, this isn't present in the paper. While it may be out of scope for the current work, it would be *quite simple* to extend the VisCE2 metric to be reference based, and doing so would help compare against some of the other reference-optional metrics such as CLIP-S, or PAC-s. 
- **Reliance of the measure on Vision/Language Models**: I am somewhat concerned about the procedure of using a vision/language model to extract visual context used for the evaluation of other vision/language models, particularly given the likelihood of hallucination in the visual context extractor. While it provides a method for evaluating captioning models without references, without anchoring the evaluation in human evaluations at any point in the visual evaluation process it could lead to significant issues in the evaluation of higher quality models, or when models evaluate themselves. It would be good to analyze this in the limitations section at least, or perform some qualitative evaluations to determine how frequently this is an issue (if it is at all).
- **Performance on THUMB 1.0 recall**: It's very clear that this approach is significantly worse than other methods in terms of recall (i.e. evaluating the coverage). While this is discussed briefly in section 4.2, it's still not clear to me *why* this measure is so much worse. I believe that some additional quantitative/qualitative evaluation would be nice to investigate this effect - are there samples for which VisCE2 are much worse than other measures, or much better? What phenomena are leading to this inability to understand coverage? These are interesting questions that I think need to be discussed or answered in the paper. 
- **Performance on PASCAL-50S**: While the results are in the appendix (it would be good to have these in the main paper), it's surprising how poorly the model performs on PASCAL-50S, particularly with respect to models such as Ref-CLIPs and InfoMetIC. It would be nice to understand why this is the case.
- **Score design**: The paper uses 0-100 as a potential score (similar to CLAIR), however that may not be reasonable for a LLM based on human judgement (for example, 0 to 10 may be more in distribution). Was the score design ablated at any point/does this impact the overall performance of the measure?
- **System-level correlation**: While the approach does seem to rank individual samples better than existing approaches, it's not clear that it changes the overall ordering of the methods. It would be good to add an evaluation of system-level correlations to demonstrate that the measure is picking up on things that other measures are not at a global evaluation level. 
- **Related Work - Hallucination Evaluation**: While this paper does discuss related work in caption evaluation, there is some similar work in hallucination evaluation that might be worth discussing such as ALOHa (https://aclanthology.org/2024.naacl-short.30/) which constructs parsed object sets from image captions, similarly CHAIR (https://aclanthology.org/D18-1437/) extracts fixed object sets and compares those, and Dai et al. do something similar for open-domain objects sets (https://aclanthology.org/2023.eacl-main.156/). 

Some minor thoughts:
- The results for GPT-4o are interesting, but are not included in the main tables  - is this because of the poor performance on Flickr8K? It seems to outperform the standard version on every other result.
- It would be interesting to see the performance for other closed-source models such as GPT-4V, Gemini, and Claude. 
- Could VisCE be made explainable similar to CLAIR? Would those explanations make more sense?
Comments, suggestions and typos: See weaknesses. It would be good to see some more detailed baselines, and a discussion of some of the associated related work and limitations. 

Overall, while I think that this metric is an interesting extension to the field of reference-free measures, I would like to see the discussion improved in the paper, primarily with respect to some of the baselines, and some of the questionable performance on THUMB 1.0 recall and PASCAL-50S. I think these things are addressable, but may require additional revision and experiments. As it stands, it seems to me like this paper has significant potential, but the approach could be better evaluated and discussed.

**Comments after author discussion**: I have read the author response, and adjusted my score accordingly. See response comments for deatils.
Overall assessment scores: 2.5