Vision Language Model Based Caption Evaluation Method Leveraging Visual Context Extraction

## Abstract

Abstract Given the accelerating progress of vision and language modeling, accurate evaluation of machine-generated image captions remains critical. In order to evaluate captions more closely to human preferences, metrics need to discriminate between captions of varying quality and content. However, conventional metrics fail short of comparing beyond superficial matches of words or embedding similarities; thus, they still need improvement. This paper presents VisCE 2 2 {}^{2} start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT , a vision language model-based caption evaluation method. Our method focuses on visual context, which refers to the detailed content of images, including objects, attributes, and relationships. By extracting and organizing them into a structured format, we replace the human-written references with visual contexts and help VLMs better understand the image, enhancing evaluation performance. Through meta-evaluation on multiple datasets, we validated that VisCE 2 2 {}^{2} start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT outperforms the conventional pre-trained metrics in capturing caption quality and demonstrates superior consistency with human judgment.

## 1 Introduction

The evaluation of the machine-generated image caption is a core research topic to illustrate models’ ability to describe their visual observation in textual forms and shape the branch of vision and language modeling studies into meaningful and grounded directions. In the early stage of neural image captioning research, such as neural image caption generator Vinyals et al. (), attention-based Xu et al. (), and sentinel and spatial attention Lu et al. (), models had enabled more and more detailed and accurate descriptions. Hence, they achieved better and better performance on reference-based automatic evaluation metrics such as BLEU Papineni et al. () and CIDEr Vedantam et al. (). SPICE Anderson et al. () was also proposed to access better correspondences of reference and generated captions in grammatical aspects. More recently, BERTScore Zhang* et al. () and CLIPScore Hessel et al. () have been introduced and are utilized to measure the similarity of embeddings of the generated text and references.

In very recent advances in vision and language models (VLMs), however, the model ability often seems to exceed the evaluation metrics themselves or even the coverage of the set of annotated references. Both InstructBLIP Dai et al. () and LLaVA Liu et al. () follow textual instructions and generate tailored descriptions that are not even similar to references but are of high quality.
We, hereby, go back to the basics of image captioning: first capture the contents of the images, then rearrange them to describe and compose a phrase. As an auto-evaluation metric, this corresponds to accessing different abilities: coverage of image contents, accuracy for describing them, and writing composition. In this line of evaluation metrics, InfoMetIC Hu et al. () relies on the matching of image regions and words in the captions. While it is quite sensitive to the alignments of objects that appear in images and captions, it becomes less sensitive to supportive facts such as attributes or interactions of objects. Unfortunately, it is also a learning-based metrics and relies on fine-tuned with multiple captioning datasets.

This paper concentrates on reference-free image caption evaluation and proposes a new Vision Language Model-based Caption Evaluation Method leveraging Visual Context Extraction (VisCE${}^{2}$). In this context, “visual context” refers to information about objects, attributes, and their relationships, including those in the background and inconspicuous objects. This visual context is given to the model in the evaluation process with an image and a candidate caption. Explicitly providing the visual context in a structured format, rather than reference images or hand-written captions, helps the VLM better understand the images’ content and expose how accurately the candidate caption describes the parts of the image or which parts are missing.

We investigated the quality of the proposed evaluation method on several image caption datasets, THumB Kasai et al. (), Flickr8k-Expert Hodosh et al. (), Composite Aditya et al. (), and Pascal-50S Vedantam et al. (). Our method, VisCE${}^{2}$, outperformed conventional metrics and was highly correlated with human judgments. Furthermore, meta-evaluation experiments uncovered that the evaluation scores of VisCE${}^{2}$ strongly correlate with the accuracy of candidate captions. In ablation studies, we verified that the proposed method is effective with the state-of-the-art VLM. Moreover, we revealed that Applying VisCE${}^{2}$ to LLaVA-v1.5 Liu et al. () improves evaluation performance comparable to that of GPT-4V OpenAI et al. () without visual context.

## 2 Related Work

Figure: Figure 1: Overview of automatic caption quality evaluation by VisCE${}^{2}$ and example of the input/output. First, VLM extracts the visual context from the image, organized in a bullet list format, presenting objects, object features, and relationships between objects. Then, VLM evaluates the caption using the obtained visual context along with the image and candidate caption.
Refer to caption: x1.png

### 2.1 Evaluation Method for Image Captioning

#### Text-only (reference-based) methods

Image captioning has been evaluated using a combination of several metrics. While some have been adapted from metrics used in other NLP tasks such as machine translation and summarization (BLEU Papineni et al. (), METEOR Denkowski and Lavie (), ROUGE Lin ()) and others proposed for image captioning (CIDEr Vedantam et al. (), SPICE Anderson et al. ()), all of them are mainly based on *n*-gram matches with the reference caption.
Following these classical approaches, an evaluation metric was proposed that exploits the versatility of pre-trained models: BERTScore Zhang* et al. () measures the similarity of embeddings output by BERT Devlin et al. () for each reference and candidate caption. In BERTScore++ Yi et al. (), they fine-tuned BERT to the task using the image caption dataset.
More recently, CLAIR Chan et al. () utilized large language models for evaluation.
This ensembles ChatGPT OpenAI (), Claude Bai et al. (), and PaLM Chowdhery et al. () and achieves high evaluation performance by providing only reference sentences and instructions for the task.

#### Crossmodal methods

Covering the rich visual information in an image with only predefined reference captions is difficult. To alleviate this loss of information, evaluation methods based on vision and language models (VLMs), which leverage features of the images, have been proposed. TIGEr Jiang et al. () uses a pre-trained SCAN model Lee et al. (), fine-tuned on the COCO dataset Chen et al. (), and calculates how much the candidate caption is grounded in the image. ViLBERTScore Lee et al. () extracts features of the images and calculates the score as with BERTScore with a pre-trained ViLBERT model Lu et al. (). FAIEr Wang et al. () connects images and texts via scene graphs and computes scores according to their overlap.

#### Image-only (reference-free) methods

To simultaneously improve evaluation performance and reduce the cost of annotation for references, several methods have been proposed for image caption evaluation that apply quality estimators or pre-training models trained on large-scale web-crawled datasets.
CLIP-S Hessel et al. () attaches the score by simply calculating the modified cosine similarity between an embedding of an image and that of a candidate caption with the CLIP Radford et al. (). PAC-S Sarto et al. () further refined the pre-training method of CLIP with data augmentation using an image captioner Li et al. () and image generator Rombach et al. (), resulting in improved evaluation performance. UMIC Lee et al. () fine-tunes UNITER Chen et al. () via contrastive loss for gold and automatically perturbated caption pair. InfoMetIC Hu et al. () fuses image and language modalities by stacking CLIP and Transformer Vaswani et al. (), fine-tuned with large image-caption datasets Young et al. (); Aditya et al. ().

Unlike existing reference-free methods, our method utilizes visual context to detail the structure of the image content for caption evaluation.
In our evaluation protocol, the VLM extracts the visual context to encapsulate the comprehensive image content and feeds it to the VLM itself. By doing so, it is possible to reference information from both the vision and language sides and hence is expected to improve the accuracy of quality estimation.

### 2.2 Recent Vision-Language Models

The recent vision and language model proposal has progressed the fusion understanding of image and language modalities. One of the most important milestones is CLIP Radford et al. (). Contrastive learning on a large-scale image-text dataset constructed by web crawling has significantly improved zero-shot performance on various vision and language tasks. Following this, the development of various models accelerated progress in the V&L domain. BLIP Li et al. () refined pre-trained methods by enhancing the quality of image and text data by the image captioner. OFA Wang et al. () integrated various V&L tasks by modifying the input-output architecture. Several other studies also presented remarkable performance, which are still being upgraded Bai et al. (); Zhu et al. (); Chen et al. (); Li et al. (). In addition, InstructBLIP Dai et al. () and LLaVA Liu et al. () allow us to perform various tailored tasks according to instructions. Furthermore, state-of-the-art VLMs such as GPT-4V OpenAI et al. () and Gemini Google () are now available via APIs.

We focus on the fidelity to instructions and hence applied our VisCE${}^{2}$ to the LLaVA-v1.5 Liu et al. (), the improved version of open-source LLaVA, to ensure transparency and reproducibility in our experiments. We also tested the evaluation performance on commercial GPT-4V to verify whether our method is effective with other VLMs and to measure the upper bound of the VLM-based evaluation methods.

## 3 Proposed Method: VisCE 2 2 {}^{2} start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT

We introduce a reference-free automatic caption evaluation method VisCE${}^{2}$ with VLM.
Figure 1 shows the overview of VisCE${}^{2}$.
Our method takes an image and a caption as inputs and predicts a rating score for how accurately the caption describes the image.
This assessment is conducted via two key components: visual context extraction and VLM-based caption evaluation.

#### Visual Context Extraction

VisCE${}^{2}$ initially extracts a detailed visual context from an input image using VLM.
We define *visual context* as the content of an image classified into objects, object attributes (including color, shape, and size), and relationships between objects following the similar notion of scene graphs Xu et al. ().
To extract relevant image details, we instruct the VLM to articulate the visual context in a bullet list format by category rather than employing the structured graphical representation due to its complexity and the difficulties associated with textual representation.
We believe that providing the successive VLM module with detailed visual context in a structured format helps bridge the gap between the reference image and the candidate text.
This facilitates the model to evaluate both consistently and comprehensively.

#### Vision-LM Caption Evaluation

In the next step, VisCE${}^{2}$ evaluates the candidate caption based on the extracted visual context and an input image.
As with many other LLM-based tasks, VisCE${}^{2}$ treats caption evaluation as a multimodal text completion task.
The VLM is given a prompt combining a reference image, visual context, and a candidate caption as input and then generates an output sentence that encapsulates the overall quality scores ranging from 0 to 100.
In the preliminary experiments, we found that the VLM occasionally disregards instructions to generate only evaluation scores, instead producing sentences that include these scores (e.g., *“The score is X out of 100”*.)
We eliminated these canonical phrases in the postprocessing phase and designated the first integer value in the output sentences as the evaluation score.
In contrast to previous embedding-based methods, such as CLIP-S Hessel et al. (), which cannot determine the quality of scores without multiple examples, our approach can provide absolute scores close to human intuition, allowing for determining the caption’s quality by just looking at a single caption.

## 4 Experiment

This section demonstrates the effectiveness of our VisCE${}^{2}$ and conducts a meta-evaluation of the automatic evaluation methods.

### 4.1 Experimental Settings

#### Implemntation details

To ensure the transparency and reproducibility of experimental results, we use
LLaVA-v1.5-13B(^1^11[https://huggingface.co/liuhaotian/llava-v1.5-13b](https://huggingface.co/liuhaotian/llava-v1.5-13b)), the best-performing model among the publicly available VLMs with the default hyperparameter settings and report the results of a single run.
Note that VisCE${}^{2}$ allows for the use of any VLM without restrictions.
Hence, we perform a meta-evaluation of GPT-4V in Sec.  5.1 to validate this and explore the upper bounds on the evaluation performance of current VLMs.

#### Baseline metrics

We compared the evaluation performance against five of the most common metrics in the automatic evaluation of image captioning: BLEU Papineni et al. (), ROUGE Lin (), METEOR Denkowski and Lavie (), CIDEr Vedantam et al. (), and SPICE Anderson et al. ().
These metrics evaluate the overlap of *n*-grams with reference captions.
Specifically, SPICE generates semantic scene graphs from the candidate captions using dependency parse trees, assessing matches based on objects, attributes, and their relationships.
Furthermore, we employed more recent reference-based measure BERTScore++ Yi et al. () and modern reference-free metrics, CLIP-S Hessel et al. (), PAC-S Sarto et al. ().
Among reference-free metrics, InfoMetIC Hu et al. () relies on a quality estimator that is fine-tuned with the training splits of Flickr-30k Young et al. () and MS-COCO Chen et al. ().
In contrast, the other metrics, including our VisCE${}^{2}$, do not depend on such data-specific fine-tuning.
Furthermore, while our method uses a publicly available model, CLAIR Chan et al. () exploits the zero-shot evaluation results of proprietary LLMs such as ChatGPT, Claude, and PaLM.
While we include their results for comparison in the subsequent experiments, it should be noted that their results are not directly comparable without careful consideration.

#### Measures for meta-evaluation

In our meta-evaluation experiments, we utilized three different indicators for each evaluation dataset, following previous studies:
Pearson correlation for measuring the linear correlation between two sets of data,
Kendall’s Tau for measuring the ordinal association between two measured quantities and
the accuracy as of the percentage of the correct pairwise ranking between two candidate captions.

**Table 1: The effect of textual resources given to the model on evaluation performance on multiple image caption datasets. Correlations with human ratings are measured by Pearson’s $\rho$, Kendall’s $\tau$, and agreement with human preference by accuracy. Bold means the best scores.**
| Method | Ref-<br>free | THumB w/o human<br>Pearson’s $\rho$ | THumB w/ human<br>Pearson’s $\rho$ | Flickr8k-Exp.<br>Kendall’s $\tau$ | Composite<br>Kendall’s $\tau$ | Pascal-50S<br>Accuracy (%) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| P | R | Total | P | R | Total | Mean |  |  |  |  |
| Vanilla | ✓ | .44 | .08 | .38 | .41 | .08 | .34 | 55.9 | 52.4 | 80.5 |
| Reference | ✗ | .42 | .12 | .40 | .40 | .14 | .38 | 54.6 | 54.5 | 79.2 |
| Description | ✓ | .53 | .08 | .44 | .48 | .07 | .39 | 57.4 | 52.5 | 77.5 |
| VisCE${}^{2}$ (Ours) | ✓ | .54<br>(+.10) | .08<br>($\pm$.00) | .45<br>(+.07) | .49<br>(+.08) | .07<br>(-.01) | .39<br>(+.05) | 59.0<br>(+3.1) | 55.0<br>(+2.5) | 80.8<br>(+0.3) |

Figure: Figure 2: Heatmaps of human rating and automatic evaluation scores on THumB (left), Flickr8k-Expert (mid) and Composite (right). Normalized for each human evaluation score (i.e., rows). The human evaluation of THumB is referenced to the total score.
Refer to caption: extracted/5431000/figures/heatmap_thumb_Vanilla.png

### 4.2 Datasets

We conduct a meta-evaluation of automatic evaluation metrics across four image captioning datasets: THumB 1.0 Kasai et al. (), Flickr8k-Expert Hodosh et al. (), Composite Aditya et al. (), and Pascal-50S Vedantam et al. ().

#### THumB 1.0 Kasai et al. ( 2022 )

consists of 500 images sourced from MSCOCO, each assigned one human-written caption and four automatic-generated captions.
The evaluation of candidate captions involves manual assessment based on three criteria: precision (how precisely the caption describes the image), recall (how well the caption covers the salient information in the image), and total (the overall quality of the caption, including fluency, inclusive language, and conciseness).

#### Flickr8k-Expert Hodosh et al. ( 2013 )

contains 5,644 pairs of images collected from Flickr and automatically generated captions, each evaluated by three experts. A score of 1 indicates that the caption is unrelated to the image, and a score of 4 indicates that it accurately describes the image.

#### Composite Aditya et al. ( 2015 )

includes 2,007 images sourced from MSCOCO Chen et al. (), 997 images from Flickr8k Hodosh et al. (), and 991 images from Flickr30K Young et al. ().
Each image is assigned two automatically generated captions and one human-written caption.
Candidate captions are evaluated on a scale from 1 (irrelevant) to 5 (ideally related).

#### Pascal-50S Vedantam et al. ( 2015 )

comprises 4,000 images from the UIUC Pascal sentence dataset Rashtchian et al. (), each paired with candidate captions: one written by a human and the others automatically generated using five different methods.
Annotators were asked to determine which caption was more similar to the reference.

### 4.3 Effectiveness of Visual Context

We initially investigated the impact of incorporating visual context into the auto-evaluation method.
In addition to the candidate captions and images as VLM input, we compare the following four types of text: (i) *Vanilla* uses only the task instruction,
(ii) *Reference* employs references provided within datasets,
(iii) *Description* incorporates detailed captions generated by LLaVA-1.5,
(iv) *VisCE${}^{2}$* (*Ours*) leverages visual contexts extracted during the initial step of VisCE${}^{2}$. See Appendix A for the prompts in each setting.

Table 1 presents the results for different textual inputs of VLM. Our VisCE${}^{2}$ demonstrated superior evaluation performance across all datasets, suggesting that incorporating visual context enhances overall evaluation performance, except recall. In contrast, other methods, such as *Reference* and *Description*, showed no improvement or even degradation from the *Vanilla* on some datasets. This suggests that the visual context’s structured and comprehensive nature enables consistent caption evaluations.

To conduct further analysis on VisCE${}^{2}$, we compared the distribution of scores assigned to captions on three different datasets in the Vanilla and VisCE${}^{2}$ settings.
The heatmap presented in Figure 2 plots the distribution of human and automatic evaluation scores in THumB, Flickr8k-Expert, and Composite, which comprise Likert scale human evaluations. Heatmaps are normalized by human rating points to remove sample size bias in the data set. Thus, each row sums to 1, and the heatmap rows represent the ratio of the automatic rating score to a caption of a specific human rating.
We observed that the score distribution in the Vanilla setting has two prominent peaks, with captions classified as good (70+) and bad (0-10), whereas VisCE${}^{2}$ exhibits three distant peaks: good (70-80), bad (0-10), and fair (40-50).
Furthermore, in THumB and Composite, the VisCE${}^{2}$ heatmap shows a decrease in the value of the upper right corner when compared to the Vanilla heatmap, indicating a reduction in the percentage of overrated captions.
These distribution changes suggest that introducing visual context contributes to a closer evaluation of the human impression.

**Table 2: Correlation (Pearson’s $\rho$) between baseline metrics and human judgment on THumB 1.0. “w/o” means discarding human annotated samples. Bold fonts for best score among reference- and tune- free models. $\dagger$: The scores reported in previous works.**
| Method | Ref-<br>free | Tune-<br>free | THumB w/o | THumB w/ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| P | R | Total | P | R | Total |  |  |  |
| BLEU-4 | ✗ | ✓ | .21 | .13 | .25 | .15 | .04 | .13 |
| ROUGE | ✗ | ✓ | .26 | .17 | .31 | .18 | .07 | .18 |
| CIDEr | ✗ | ✓ | .27 | .18 | .33 | .21 | .11 | .23 |
| SPICE | ✗ | ✓ | .26 | .15 | .30 | .20 | .09 | .21 |
| RefCLIP-S | ✗ | ✓ | .34 | .27 | .44 | .31 | .26 | .41 |
| ${}^{\dagger}$InfoMetIC | ✓ | ✗ | .22 | .30 | .37 | .21 | .32 | .38 |
| CLIP-S | ✓ | ✓ | .18 | .27 | .32 | .17 | .28 | .32 |
| VisCE${}^{2}$ | ✓ | ✓ | .54 | .08 | .45 | .49 | .06 | .39 |

### 4.4 Correlation with Human Judgment

We analyze the proposed VisCE${}^{2}$ metrics on the THumB1.0 Kasai et al. () dataset by assessing the correlation between automatic evaluation scores and human ratings in three aspects: precision, recall, and the total score.
Following previous studies, we used Pearson’s correlation coefficient as the meta-evaluation index.

The results presented in Table 2 highlight VisCE${}^{2}$’s outstanding performance in terms of correlation with precision, surpassing all other metrics.
It exceeded RefCLIP-S, the previous state-of-the-art metric, by 0.20 and 0.18 points in settings with and without human-written captions, respectively. This suggests that VisCE${}^{2}$ accurately evaluated precise captions.
On the other hand, VisCE${}^{2}$’s performance in recall presented a contrast, exhibiting minimal correlation.
In the THumB dataset, recall scores reflect the extent to which the caption encompasses the salient information in the image.
Meanwhile, the evaluation scores of our method were calculated by considering all objects, attributes, and their relationships within the image.
The difference between focusing on the salient objects and focusing on all objects may have led to the low correlation with recall scores.
Despite its imbalanced nature, our method is highly correlated with the total scores, indicating the overall quality of captions.

**Table 3: Correlation between human judgement and metrics on Flickr8k-Expert and Composite.**
| Method | Ref-<br>free | Tune-<br>free | Flickr8k-Exp.<br>Kendall’s $\tau$ | Composite<br>Kendall’s $\tau$ |
| --- | --- | --- | --- | --- |
| BLEU-4 | ✗ | ✓ | 30.6 | 28.3 |
| ROUGE | ✗ | ✓ | 32.1 | 30.0 |
| METEOR | ✗ | ✓ | 41.5 | 36.0 |
| CIDEr | ✗ | ✓ | 43.6 | 34.9 |
| SPICE | ✗ | ✓ | 51.7 | 38.8 |
| ${}^{\dagger}$BERTScore++ | ✗ | ✗ | 48.1 | 42.3 |
| RefCLIP-S | ✗ | ✓ | 52.6 | 51.2 |
| ${}^{\dagger}$RefPAC-S | ✗ | ✓ | 55.5 | 51.5 |
| ${}^{\dagger}$CLAIR${}_{\mathrm{Claude}}$ | ✗ | ✓ | 56.2 | 54.2 |
| ${}^{\dagger}$CLAIR${}_{\mathrm{E}}$ | ✗ | ✓ | 62.7 | 59.2 |
| ${}^{\dagger}$InfoMetIC | ✓ | ✗ | 54.2 | 59.2 |
| CLIP-S | ✓ | ✓ | 51.1 | 49.8 |
| ${}^{\dagger}$PAC-S | ✓ | ✓ | 53.9 | .51.5 |
| VisCE${}^{2}$ | ✓ | ✓ | 59.0 | 55.0 |

**Table 4: Comparison VLM-based caption evaluation methods on correlation (Pearson’s $\rho$) with human judgement on THumB 1.0.**
| VLM | Method | THumB w/o<br>Pearson’s $\rho$ | THumB w/<br>Pearson’s $\rho$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| P | R | Total | P | R | Total |  |  |
| LLaVA-v1.5 | Vanilla | .44 | .08 | .38 | .41 | .08 | .34 |
| LLaVA-v1.5 | VisCE${}^{2}$ | .54<br>(+.10) | .08<br>($\pm$0) | .45<br>(+.07) | .49<br>(+.08) | .07<br>(-.01) | .39<br>(+.05) |
| GPT-4V | Vanilla | .53 | .03 | .41 | .50 | .05 | .38 |
| GPT-4V | VisCE${}^{2}$ | .58<br>(+.05) | .06<br>(+.03) | .46<br>(+.05) | .55<br>(+.05) | .08<br>(+.03) | .44<br>(+.06) |

Figure: Figure 3: Comparison between evaluation scores of VisCE${}^{2}$, that of CLIP-S, and human ratings for candidate caption for images from the Composite dataset.
Refer to caption: extracted/5431000/figures/qual_1_COCO_val2014_000000408774.jpg

We also test the caption evaluation capability of VisCE${}^{2}$ using Flickr8k-Expert Hodosh et al. () and Composite Aditya et al. () dataset.
Kendall’s rank correlation coefficient was used as a meta-evaluation index to measure the correlation between automatic and human evaluation.

VisCE${}^{2}$ performed at a high level compared to the other automatic evaluation metrics in its correlation with human evaluation (Table 3). VisCE${}^{2}$ in the Flickr8k-Expert achieved state-of-the-art performance by a significant margin of more than 4.8 pts compared to other reference-free metrics, demonstrating that our method is highly capable of estimating the quality of image captions aligned with human judgments.
A similar trend was observed in the Composite. Compared to CLIP-S and PAC-S, VisCE${}^{2}$ surpassed them by over 3pts. It was found that our method fell short of InfoMetIC’s performance. This degradation is assumed to originate from the training method of InfoMetIC, a fine-tuning quality estimator on the three datasets that comprise the Composite dataset.
The result also offers insight into the quality estimation based on large-scale models.
While CLAIR${}_{\mathrm{E}}$, which utilizes an ensemble of non-public LLMs such as ChatGPT, PaLM, and Claude, achieved significant results in the benchmarks, VisCE${}^{2}$ outperformed the outcome of CLAIR${}_{\mathrm{Claude}}$, which solely exploits Claude. This distinction highlights the efficacy of our method, considering its reliance on publicly available and comparatively smaller models. Such performance showcases the potential of more accessible models to achieve high-quality benchmarks.

## 5 Analysis

### 5.1 Evaluation Performance of GPT-4V

We explore the proposed method’s current upper bounds and effectiveness using GPT-4V.
GPT-4 exhibits superior performance on many vision and language tasks Yang et al. (), and it has also been quantitatively validated to outperform LLaVA-v1.5 in several tasks Li et al. (); Cui et al. ().
Due to budgetary constraints, we selected the relatively small yet finely-annotated dataset, THumB, used in Section 4.4, to compare the open-source VLM model, LLaVA-v1.5, with GPT-4V using the different prompt settings (Vanilla and VisCE${}^{2}$).
We used the gpt-4-vision-preview as the GPT-4V model through the Azure OpenAI API.

As shown in Table 4, the performance of GPT-4V outperformed that of LLaVA-v1.5 for precision and total score at each prompt setting.
Similar to the LLaVA-v1.5 model, applying VisCE${}^{2}$ to GPT-4 enhances the evaluation performance, indicating that the proposed method is effective even with more advanced VLMs.
In addition, GPT-4V with VisCE${}^{2}$ achieved state-of-the-art performance in terms of correlations with precision and total score.
Furthermore, the evaluation performance of the LLaVA-v1.5 model with VisCE${}^{2}$ applied was improved to the same extent as the performance of GPT-4V in the Vanilla setting.
The effectiveness of our auto-evaluation method using this open-source VLM suggests that it can contribute to reducing evaluation costs.

Figure: Figure 4: Comparison between evaluation scores of VisCE${}^{2}$ and classic reference-based metrics. The image and the references are from the MS-COCO dataset. Both captions are automatically generated by GPT-4V.
Refer to caption: extracted/5431000/figures/COCO_val2014_000000521509.jpg

### 5.2 Qualitative Analysis

To clarify the differences in trends among other reference-free metrics, we compared the evaluation scores assigned to the images by VisCE${}^{2}$ with those of CLIP-S, a typical reference-free metric.
Figure 3 presents examples of human ratings and scores of the auto-eval methods for image-caption pairs.
These qualitative examples indicate the discriminative ability of VisCE${}^{2}$ to the accuracy of captions.
CLIP-S tends to overestimate captions describing the presence of objects in the image.
For example, CLIP-S assigns a relatively high score to the incorrect caption *“a baseball player swinging a bat at a ball.”* to the image in the first row.
However, although “the baseball player” is present in the image, the event of “swinging the bat at the ball” has not occurred.
Humans can detect such contradiction accurately, whereas CLIP-S often ignores it.
In the second example, CLIP-S also overestimated incorrect image descriptions containing the salient object, an umbrella, in the image.
A previous study Ahmadi and Agrawal () has also pointed out that CLIP-S tends to be affected by the presence or absence of descriptions of salient objects in the image, confirmed in these examples.
The third example shows that CLIP-S overlooks mistakes in combining objects and their colors.
In contrast, VisCE${}^{2}$ can identify the discrepancy using the visual context and accurately assess image-caption pairs.

### 5.3 Comparison to Reference-based Metrics

We present another qualitative example to shed light on the effectiveness of our proposed method as well as the limitations of existing reference-based methods.
Figure 4 provides an image, five annotated reference captions drawn from the MS-COCO dataset, and two generated caption candidates with the recent image captioning model of GPT-4V with automatic evaluation scores.
The first candidate is a caption generated using the straightforward prompt, *“Generate a detailed description for the given image.”* This caption accurately describes the image’s content in quite a detailed manner. (^2^22CLIP-S and RefCLIP-S cannot handle such extended captions correctly due to the context length.)
Indeed, its detailedness overwhelms those of five human-written references, and hence, all of the reference-based metrics undervalued the caption.
The other candidate’s caption follows the style of references, providing accurate descriptions of some objects, although it falls short in describing overall details.
This candidate caption, however, achieves higher metrics in some auto-evaluated scores.
We assume this is because the sophisticatedness of the recent image captioning models exceeds those of the reference-based metrics, and hence reference-free metrics, such as VisCE${}^{2}$, have a clear merit for evaluating current overwhelmingly detailed image captions.

## 6 Conclusion

We have proposed VisCE${}^{2}$, a prompting method for automatic VLM-based evaluation of image captions.
VisCE${}^{2}$ deviates from the traditional method of evaluating similarity using only reference text and images, weaving visual context into the evaluation framework with a compositional form. This technique allows the VLMs to understand the detailed visual dependencies of images better and validate them based on more exhaustive content than those that the reference captions provide. Meta-evaluation experiments revealed that scores output by VisCE${}^{2}$ have excellent consistency with human judgments, especially in caption accuracy, which outperforms existing evaluation metrics.
Future work includes providing VLM-based automatic evaluation along various perspectives, which would provide even more fine-grained information for humans.

## Limitations

#### Evaluation cost

Our proposed method, which utilizes VLMs for evaluation, requires a higher computational cost than previous approaches. In our experiments, we employed an NVIDIA A100 40GB GPU, and extracting visual context took about 10 seconds per sample and caption evaluation 100 milliseconds, respectively.
This limitation significantly depends on the inference speed of VLMs, suggesting that future improvements in VLMs could mitigate this issue.

#### Sensitivity to the prompt

Salinas and Morstatter () points out that Large Language Models (LLMs) performance varies based on the provided prompts.
Although this is an issue for LLMs, it is natural to consider it for VLMs, even if the impact is not quantified yet. Understanding how VLMs respond to different prompts is key to making them more reliable, which remains for future work.

## Ethics Statement

As our proposed method VisCE${}^{2}$ is utilized for providing scores to captions, we do not anticipate the negative impacts of this work. However, as with other machine learning methods, we recommend exercising caution.

## Acknowledgements

This work was supported by JSPS KAKENHI Grant Number JP22K17983, and by JST PRESTO Grant Number JPMJPR20C2.

## Appendix A Prompts

We provide the prompts used in our experiments in Table 5.

**Table 5: The prompts used in the experiment. {caption}, {references}, and {context} indicate the place to insert. The image is given at the beginning of each prompt.**
| Method | Prompt |
| --- | --- |
| Vanilla | On a precise scale from 0 to 100, rate whether the candidate caption is appropriate for the given image.<br>Candidate caption: {caption}<br>Your rating must be a single digit between 0 and 100. |
| Reference | On a precise scale from 0 to 100, rate whether the candidate caption is appropriate for the given image.<br>Candidate caption: {caption}<br>Use the image and the following reference to evaluate the candidate caption:<br>Reference: {references}<br>Your final rating must be a single digit between 0 and 100. |
| Description - Step 1 | Generate a detailed description for the given image. |
| VisCE${}^{2}$- Step 1 | Analyze the uploaded image and provide a structured output focusing on the objects, their features, and the relationships between them.<br>Select up to five of the most important elements. The output should be organized as follows:<br>List of Important Objects (up to five):<br>- Object 1: [Brief description]<br>- Object 2: [Brief description]<br>- (Continue as necessary, up to five objects)<br>Features (Specific characteristics and attributes of each object, such as color, shape, size, and texture):<br>- Features of Object 1: [Detailed description of features]<br>- Features of Object 2: [Detailed description of features]<br>- (Continue as necessary for each selected object)<br>Relationships (The way objects interact or are positioned relative to each other, without using specific object names or symbols):<br>- Description of a relationship: [General description]<br>- Another relationship: [General description]<br>- (Continue as necessary for each relevant relationship)<br>Focus on providing unique and detailed insights into the features and relationships of the selected objects up to five objects. |
| Description - Step 2<br>VisCE${}^{2}$- Step 2 | On a precise scale from 0 to 100, rate whether the candidate caption is appropriate for the given image.<br>Candidate caption: {caption}<br>Use the image and following visual context to evaluate the candidate caption:<br>Visual context: {context}<br>Your final rating must be a single digit between 0 and 100. |

**Table 6: Accuracy on the Pascal-50S dataset. Bold fonts for best score among reference- and tune- free models. $\dagger$: The scores reported in previous works.**
| Method | Reference-free | Tune-free | PASCAL-50S |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| HC | HI | HM | MM | Mean |  |  |  |
| BLEU-4 | ✗ | ✓ | 53.0 | 92.4 | 86.7 | 59.4 | 72.8 |
| ROUGE | ✗ | ✓ | 51.5 | 94.5 | 92.5 | 57.7 | 74.0 |
| METEOR | ✗ | ✓ | 56.7 | 97.6 | 94.2 | 63.4 | 77.9 |
| CIDEr | ✗ | ✓ | 53.0 | 98.0 | 91.5 | 64.5 | 76.7 |
| SPICE | ✗ | ✓ | 52.6 | 93.9 | 83.6 | 48.1 | 69.5 |
| ${}^{\dagger}$BERTScore++ | ✗ | ✗ | 65.4 | 98.1 | 96.4 | 60.3 | 80.1 |
| RefCLIP-S | ✗ | ✓ | 64.9 | 99.5 | 95.5 | 73.3 | 83.3 |
| ${}^{\dagger}$RefPAC-S | ✗ | ✓ | 67.7 | 99.6 | 96.0 | 75.6 | 84.7 |
| ${}^{\dagger}$CLAIR${}_{\mathrm{Claude}}$ | ✗ | ✓ | 57.9 | 98.5 | 91.3 | 62.9 | 77.6 |
| ${}^{\dagger}$CLAIR${}_{\mathrm{E}}$ | ✗ | ✓ | 57.7 | 99.8 | 94.6 | 75.6 | 81.9 |
| ${}^{\dagger}$InfoMetIC | ✓ | ✗ | 69.0 | 99.8 | 94.0 | 78.3 | 85.3 |
| CLIP-S | ✓ | ✓ | 55.9 | 99.3 | 96.5 | 72.0 | 80.9 |
| ${}^{\dagger}$PAC-S | ✓ | ✓ | 60.6 | 99.3 | 96.9 | 72.9 | 82.4 |
| VisCE${}^{2}$ | ✓ | ✓ | 60.7 | 99.6 | 93.6 | 69.3 | 80.8 |

## Appendix B Caption Pairwise Ranking

To further verify how accurately our VisCE${}^{2}$ determines the relative preference of captions, we conducted a meta-evaluation with the Pascal-50S dataset Vedantam et al. () that comprises human preference judgments indicating which of the two captions for an image is more appropriate.
The preference judgments for pairs of candidate captions are classified into the following four categories;

- 1.
HC: pairs of human-written captions, both of which correctly represent the image’s content;
- 2.
HI: pairs of human-written captions where one caption is correct and the other is incorrect;
- 3.
HM: pairs of correct captions, one of which is human-written and the other machine-generated;
- 4.
MM: pairs of machine-generated captions, both of which correctly represent the image’s content.

For each category, we measured accuracy, the judgment agreement between the human preferences, and the results of the automatic evaluation method.

The pairwise ranking agreement result shown in Table 6 depicts that our method outperformed CLIP-S and PAC-S in the HC and HI categories. On the other hand, relatively low performance was observed in HM and MM compared to other metrics. This degraded performance, especially noticeable for MM, can be attributed to the nature of the candidate captions. The automatically generated captions in the Pascal-50S dataset are made with classical captioning methods, such as Midge Mitchell et al. () and Babytalk Kulkarni et al. (), often containing errors. Therefore, both are judged as poor-quality captions, causing the model’s performance to deteriorate.
Another contributing factor might be the disagreement in objectives between human annotation and automatic quality estimation. In the annotation process for the Pascal-50S dataset, the worker selected captions that are more similar to randomly selected references, which does not necessarily equate to superiority or adequacy as a caption.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - What are the main differences between the proposed method and SPICE/InfoMetIC? What unique innovations does this paper offer?
- Why choose THumB instead of MSCOCO for evaluation?
- The meanings of the numbers should be stated more clearly. For example, what do 5/5 and 80/100 mean？
soundness: 2
strengths: - The paper highlights the urgent need for developing new metrics, considering the fact that model generations have become so detailed that they often exceed the capability of the automatic evaluation metrics.

- The paper is easy to follow.
summary: The paper presents VisCE2, a vision-language model-based caption evaluation method designed to evaluate captions in a manner that aligns more closely with human preferences. VisCE2 focuses on visual context, which refers to the detailed content of images, including objects, attributes, and relationships. Experiments are conducted on several datasets.
weaknesses: - The literature review should be more accurate. For example, SPICE (Anderson et al., 2016) is mainly based on scene graphs rather than n-grams.

- The novelty of this paper is limited. The proposed evaluation method consists two stages: visual context extraction and VLM-based caption evaluation. The first stage analyzes images based on scene graphs, similar to SPICE (Anderson et al., 2016).  The second stage evaluates captions with VLMs, which is not new given existing works such as InfoMetIC (Hu et al., 2023) and CLIPScore (Hessel et al., 2021). While  the combination of these two stages may be new, it may not meet the innovation standards expected for ICLR submissions.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please see weaknesses.

I will be happy to raise my score if authors address my concerns.
soundness: 3
strengths: 1. A novel reference-free image caption evaluation method with VLMs.
2. This paper is well-written and easy to follow.
4.  This paper proposes a visual context extraction module to describe the image as sentences, which also can be seen as a pseudo reference with abundant details.
4. The authors conduct comprehensive experiments across multiple datasets.
summary: This paper introduces a reference-free image captioning evaluation metric, called VisCE$^2$. Specifically, VisCE$^2$ leverages pre-trained Vision-Language models (VLMs) to realize two-stage measurements for candidate captions. The first is Visual Context Extraction which uses VLM to obtain detailed descriptions including objects, object attributes and relationships. The second is Vision-LM Caption Evaluation which takes visual context, image and candidate captions as inputs to obtain an evaluation score. Experimental results demonstrate the superiority of this reference-image free method against other metrics.
weaknesses: 1. Figure 1 is not comprehensive. For the left part, RefCLIP-S[1] and RefPAC-S[2] can also accomplish the same measurement. On the other hand, better evaluation performances of VisCE$^2$ than BLEU-4, ROUGE, SPICE and CIDEr are not enough. While for the right part, authors should compare with PAC-S[2] to illustrate the superiority of this work.

2. Line 49 - Line 51 describes the disadvantages about InfoMetIC, but evidence is lacked and can therefore be listed in Figure 1.

3. It is suggested to evaluate the VisCE$^2$ and other reference-free metrics within different *image captioning methods* such as InstructBLIP, LLaVA and even GPT-4, as mentioned in Line 42-Line 44. This is a key step to comprehensively measure the effectiveness of VisCE$^2$. The authors can refer to Table 7 in PAC-S paper[2].

4.  Although this paper focuses on reference-free evaluation, it is also recommended to report the results of VisCE$^2$ when the reference captions are provided.  

5.  An example of visual context given the image should be added into appendix. For instance, authors can list all the objects, object attributes and relationships about the image in Figure 2. 

6. In Table 2, it seems that authors only report the values of Kendall’s $\tau_b$ on Flickr8k-Expert and Composite datasets. Kendall’s  $\tau_c$ should also be included. 

7. It is a little bit confusing to read Table 3 about ablation experiments. The first two settings are to prove the effectiveness of each component with the same backbone VLM (LLaVA-v1.5-13B). Then the current model (VisCE$^2$ ours) achieves the best scores across all datasets. But for the last two settings, authors aim to explore the influences of different backbone models or model sizes. From Table 3, GPT-4o can achieve **59.0** score on Composite dataset, higher than VisCE$^2$(**56.0**). THumB and Pascal-50S observe similar phenomenon.  Hence, it would be better to split Table 3 into two small tables.

[1] Hessel, Jack, et al. "Clipscore: A reference-free evaluation metric for image captioning." arXiv preprint arXiv:2104.08718 (2021).

[2] Sarto, Sara, et al. "Positive-augmented contrastive learning for image and video captioning evaluation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to the questions in the weakness.
soundness: 2
strengths: - The proposed method is easy to understand.
- The proposed method shows favorable performance compared to existing evaluation methods.
summary: Given the accelerating progress of vision and language modeling, accurate evaluation of machine-generated image captions remains critical. 
In order to evaluate captions more closely to human preferences, metrics need to discriminate between captions of varying quality and content. 
However, conventional metrics fall short of comparing beyond superficial matches of words or embedding similarities; thus, they still need improvement. 
This paper presents VisCE2, a vision language model-based caption evaluation method. 
The authors’ method focuses on visual context, which refers to the detailed content of images, including objects, attributes, and relationships. 
By extracting and organizing them into a structured format, the authors replace the human-written references with visual contexts and help VLMs better understand the image, enhancing evaluation performance.
weaknesses: - The novelty of the proposed method is weak. The only idea in this paper is to use a language model instead of CLIP to evaluate image captioning where the sentence generation performance of the VLMs is imperfect, unlike CLIP’s image-caption alignment performance. The authors suggest using an image captioning model to evaluate image captioning models. How can we evaluate the models that perform better than LLaVA? Using the proposed metric instead of CIDEr or CLIPS scores for future image captioning research is not convincing.

- The discussion on design choice is also weak. In Table 3, the only discussions are on what VLM to use and what kind of visual context to use. However, there are other design choices to be considered. For example, when using language models, a proper prompt is essential. However, the authors didn’t analyze the choice of prompts for the language model. Moreover, whether the visual context extractors (object, attribute, relation) have the best design choice isn't justified. Therefore, it is not clear whether the proposed metric is the best possible method.

- This paper lacks experimental analysis. When suggesting a new evaluation metric, it would be better to evaluate popular image captioning models, such as BLIP2, and analyze the tendency of the performances to understand the unique characteristics of the proposed metric. Also, it would be better to evaluate the proposed metric in different settings, such as FOIL hallucination detection, as CLIPS did.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the questions in Weaknesses.
soundness: 2
strengths: The paper proposed a new method to evaluate the generated captions considering the objects, attributes, and relations within the images. And the paper makes great efforts to demonstrate the reliability of the evaluation method by comparing with human judgement. The results indicate this method is better consistent with human rating compared to other metrics.
summary: The paper proposes a method that uses the visual concepts extracted by MLLMs to help evaluate image captions, which makes the evaluation results more consistent with human ratings.
weaknesses: The evaluation process heavily relies on the use of MLLMs in the following ways: 1. It utilizes MLLMs to extract visual concepts from images; 2. It employs MLLMs to generate evaluation scores for these image captions. If the candidate captions are generated by a same MLLM, the evaluation method may fail to provide a fair evaluation.

It seems that the evaluation time is significantly longer than the time required by other metrics, due to the use of MLLMs in two stages. How long does it take to evaluate one caption based on an image? Please provide concrete timing comparisons between the proposed method and existing metrics. Additionally, why is the image necessary in the Vision-LM Caption Evaluation stage? If the visual concepts are sufficient to represent an image, the evaluation could potentially be conducted without using the image, which might speed up the evaluation process. The paper should include an ablation study comparing performance with and without the image in the Vision-LM Caption Evaluation stage.

Also, the paper should add ablation studies on the used prompts, particularly regarding the maximum number of objects. According to the prompts shown in Table 4, the maximum number of objects extracted by MLLM is set to 5.  How could this choice affect the reliability of the evaluation method?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. When simply constructing the initial prompt (Vanilla) in a more refined way, e.g. by adding a chain-of-thought (CoT) prompt, would better assessment results also be achieved?

2. Can the authors provide a comparison of runtime with existing evaluation methods?

3. The visual context extracted in the first phase will contain some hallucinations, does this have an impact on the evaluation results?
soundness: 3
strengths: 1.	The paper is well-organized and clearly written.
2.	The proposed VisCE2 is intuitive. And evaluation experiments on multiple datasets demonstrate that the method outperforms existing evaluation metrics and meets human judgments.
summary: This paper proposes a VLM-based image caption evaluation method called VisCE2. The proposed method first obtains structured visual context by prompting the VLM, and then evaluates candidate captions based on the extracted visual context and input image. Extensive evaluation experiments show that VisCE2 outputs scores that have good agreement with human judgment and outperform existing evaluation metrics.
weaknesses: 1. The paper is somewhat weak on innovation. The method is simply based on two rounds of prompts, which makes the VLM automatically evaluate image captions, and its core is based on the in-context learning ability of the VLM. Assuming that only one round of prompt is used and combined with the chain-of-thought (CoT) method to make the VLM automatically mine the visual context, while setting the last sentence generated by the VLM as the evaluation result, can this also lead to a good image caption evaluation performance?

2. Since two rounds of prompts are required for the VLM to evaluating the image caption, resulting in a high time complexity of this evaluation method, which is not conducive to real-time evaluation. Can the authors provide a comparison of runtime with existing evaluation methods? 

3. Based on Table 3 of the ablation experiment, the enhancement brought by visual context does not seem to be particularly significant compared to the original prompt (Vanilla). Can the authors further analyze the reasons for this condition?

Meta-review: The paper introduces a vision-language model-based caption evaluation method. 
Reviewers have raised major concerns on inaccurate literature review, limited novelty, insufficient comparisons, ablations, and analysis. All five reviewers recommended rejection. No rebuttal was provided.

Decision: Reject