Unblocking Fine-Grained Evaluation of Detailed Captions: An Explaining AutoRater and Critic-and-Revise Pipeline

## Abstract

Abstract Large Vision-Language Models (VLMs) now generate highly detailed, paragraph-length image captions, yet evaluating their factual accuracy remains challenging. Current methods often miss fine-grained errors, being designed for shorter texts or lacking datasets with verified inaccuracies. We introduce DOCCI-Critique , a benchmark with 1,400 VLM-generated paragraph captions (100 images, 14 VLMs) featuring over 10,216 sentence-level human annotations of factual correctness and explanatory rationales for errors, all within paragraph context. Building on this, we develop VNLI-Critique , a model for automated sentence-level factuality classification and critique generation. We highlight three key applications:
(1) VNLI-Critique demonstrates robust generalization, validated by state-of-the-art performance on the M-HalDetect benchmark and strong results in CHOCOLATE claim verification.
(2) The VNLI-Critique driven AutoRater for DOCCI-Critique provides reliable VLM rankings, showing excellent alignment with human factuality judgments (e.g., 0.98 Spearman).
(3) An innovative Critic-and-Revise pipeline, where critiques from VNLI-Critique guide LLM-based corrections, achieves substantial improvements in caption factuality (e.g., a 46% gain on DetailCaps-4870).
Our work offers a crucial benchmark alongside practical tools, designed to significantly elevate the standards for fine-grained evaluation and foster the improvement of VLM image understanding.

## 1 Introduction

Automatic descriptive image captioning, a prominent vision-language research area [33], has evolved from short highlights [30] to detailed, paragraph-length descriptions, thanks to powerful Large Vision-Language Models (LVLMs) [4, 7, 20, 34, 41, 45]. Evaluating these complex captions remains challenging; current metrics, often for short texts, miss subtle, fine-grained details and typically assess sentences in isolation, lacking crucial paragraph context for resolving ambiguities and co-references. While some studies address full paragraphs [9], granular sentence-level assessment is still difficult.

Evaluating VLM-generated text factuality has led to specialized benchmarks. However, many existing error/hallucination detection benchmarks target short sentences or QA tasks, inadequately addressing paragraph-length descriptions. While datasets like M-HalDetect [14] and CHOCOLATE [18] offer valuable sentence-level factuality annotations, they may lack full error diversity from long-form VLM outputs or consistent, detailed textual explanations for inaccuracies—vital for developing effective, fine-grained automated evaluation. Response-level evaluations (e.g., CAPTURE [9]) score entire paragraphs but miss sentence-level granularity. Thus, a benchmark with comprehensive, context-aware sentence-level factuality annotations, including explanatory rationales for errors across diverse VLM-generated paragraphs, is critically needed.

To address this, we introduce DOCCI-Critique, a novel benchmark for fine-grained evaluation of detailed image descriptions. It comprises 1,400 paragraph-length captions (14 VLMs, 100 images), with its core value in 10,216 sentence-level human annotations. Each sentence’s factuality was judged by five annotators, with detailed textual rationales for every identified error. This multi-perspective annotation offers a new resource for in-depth VLM analysis, providing multiply-verified sentence-level judgments and error explanations for long captions, distinct from existing datasets.

Building on DOCCI-Critique, we developed VNLI-Critique, a model for automated sentence-level factuality classification (using paragraph context) and explanatory critique generation. This dual capability enables a novel Critic-and-Revise pipeline (Fig. [1](https://arxiv.org/html/2506.07631v1#S0.F1)): VNLI-Critique evaluates and generates a critique for an incorrect VLM-generated sentence, guiding an LLM to revise it. The utility of VNLI-Critique and this pipeline is shown via key results:
(1) VNLI-Critique achieves state-of-the-art performance on external benchmarks M-HalDetect (0.76 Macro-F1) and competitive results on CHOCOLATE [18], demonstrating strong generalization.
(2) On our benchmark, the VNLI-Critique powered DOCCI-Critique AutoRater shows VLM rankings with strong correlation to human judgments ([Table 3](https://arxiv.org/html/2506.07631v1#S4.T3)).
(3) The Critic-and-Revise pipeline significantly boosts factuality of incorrect sentences (e.g., by 46% on DetailCaps-4870 [9], 51% on PixelProse [32]), confirmed by human evaluation.
Collectively, these contributions deliver an essential benchmark and powerful methodologies, enabling more precise fine-grained assessment and demonstrably enhancing the factual accuracy of detailed image understanding by VLMs.

## 2 Related Work

Our work intersects with advancements in Vision-Language Models (VLMs) for detailed captioning, the development of image captioning datasets, and methodologies for evaluating caption quality, especially factual accuracy and fine-grained detail.

#### Vision-Language Models

Recent VLMs [34, 26, 1, 20, 24, 41, 8, 46, 3, 21, 7, 49, 35] have achieved remarkable SOTA performance in multimodal tasks . Typically, they combine a visual encoder [28, 48, 37, 10] with an LLM [43, 6, 5, 36], often using a connector module to bridge visual features with textual tokens. Training usually involves pre-training the visual encoder and then fine-tuning the LLM, with objectives like masked language modeling or image-text matching. While end-to-end training is explored, this two-stage approach is common, balancing dataset scale, computational resources, and evaluation.

#### Image Captioning Datasets

Image understanding and captioning datasets are crucial for supporting advancements in current image captioning techniques. Early datasets provided positive image-text pairs with short sentences, primarily focusing on main objects and scenes, such as COCO [22], Flickr8k [16], and Flickr30k [47]. More recent image captioning datasets offer image-text pairs with longer, more detailed descriptions. The Densely Captioned Images (DCI) dataset [38], for instance, introduces long, mask-aligned descriptions to specifically evaluate VLM understanding of distinct image regions. PixelProse [32] offers a large scale dataset of 16.9 million synthetically generated captions using Gemini-1.0-Pro-Vision [35]; however, the correctness of these descriptions is not guaranteed. The IIW [11] dataset utilizes a VLM to generate initial captions and then employs a human-in-the-loop framework to ensure high-quality positive image-text pairs. M-HalDetect [14] and CHOCOLATE [18] caption images and charts, respectively, using various VLMs, and employ sentence-level human annotation to assess the correctness of generated captions. DetailCaps [9] leverages GPT-4 [26] to assign a quality score (ranging from 0 to 5) to synthetically generated captions from existing datasets. The DOCCI dataset [25] stands out as a particularly strong choice for research due to its high-resolution images of diverse real-life scenes paired with detailed, fully human-written descriptions, offering a valuable resource for training and evaluating VLMs. It provides high-resolution images of diverse real-life scenes, each paired with carefully crafted, long, human-annotated descriptions.

#### Evaluation of Detailed Captions

Traditional metrics (e.g., BLEU [27], METEOR [2], CIDEr [39]) compare generated captions to references using n-gram overlap, often missing semantic nuance crucial for paragraph-length text. Embedding-based methods like CLIPScore [15] and SIGLip [48] offer better semantic assessment but typically evaluate sentences in isolation, lacking the paragraph context needed to resolve ambiguities or co-references in detailed descriptions. QA-based metrics (e.g., VQAScore [23], TIFA [17], $\text{VQ}^{2}$ [44], GECKO [42]) assess understanding via question-answering but face scalability challenges for long narratives. Recent work such as Mismatch Quest [13] has focused on providing detailed textual and visual feedback for misalignments, primarily in the text-to-image domain and often for shorter, direct textual inputs, by automatically generating explanations. While response-level evaluations like Dong et al. [9] score entire paragraphs, they don’t offer sentence-level factuality details for descriptive captions. Our work addresses the need for fine-grained, context-aware sentence-level factuality evaluation with rich, human-verified explanatory feedback specifically for detailed, paragraph-length image captions

## 3 The DOCCI-Critique Benchmark: Construction, Annotation, and Analysis

DOCCI-Critique is a novel benchmark for fine-grained factuality assessment of paragraph-level image descriptions. Its primary purpose is twofold: (1) providing a robust platform to assess SOTA captioning models’ descriptive capabilities and factual accuracy, and (2) serving as a challenging testbed for automated image understanding and fact-checking systems (auto-raters).

Construction began with 100 diverse, high-resolution images from the DOCCI dataset’s ‘qual-dev’ split [25]. For each, 14 SOTA Large Vision-Language Models (Table [2](https://arxiv.org/html/2506.07631v1#S3.T2)) generated detailed, paragraph-length descriptions, yielding 1,400 model-generated descriptions. This corpus deliberately captures a wide spectrum of stylistic variations, detail levels, and factual accuracies, from concise and factual to more verbose accounts that might introduce subtle inconsistencies (e.g., object misidentification).

The core of DOCCI-Critique is its rich human annotation layer. Following Steiner et al. [34] (see Appendix for annotation template), five human annotators independently evaluated each sentence within the 1,400 descriptions for factual correctness against the image, assigning labels: ‘Entailment’ (factually supported), ‘Neutral’ (not verifiable/contradicted), ‘Contradiction’ (factually contradicted), or ‘Nothing to assess’ (e.g., filler). A sentence is classified as ‘Non-entailment’ if a majority labeled it ‘Neutral’ or ‘Contradiction’. Crucially, annotators provided detailed textual rationales for each non-entailed judgment. Thus, a single non-entailed sentence often has multiple rationales, capturing diverse perspectives on the inaccuracy and offering insights into VLM error types (e.g., object misidentification, attribute/spatial errors, hallucination).

**Table 1: Illustrative example from the DOCCI-Critique benchmark, detailing sentence-level annotations for fine-grained factuality assessment, including rater judgments and explanatory rationales.**
| Image |  |  |  |
| --- | --- | --- | --- |
| Description<br>Sentence | “… Behind the car, there is a large mural or poster on the wall …” | “… The mural features a Formula 1 racing car, also red, with the number 16 prominently displayed on the side. …” | “… The background of the mural includes a racing track with the colors of the French flag (blue, white, and red) and a checkered flag, indicating a racing theme …” |
| Does the sentence include a claim about the image? (Answers from 5 raters) | \scalerel<br>*○, \scalerel*○, \scalerel*○, \scalerel*○, \scalerel*○ | \scalerel<br>*○, \scalerel*○, \scalerel*○, \scalerel*○, \scalerel*○ | \scalerel<br>*○, \scalerel*○, \scalerel*○, \scalerel*○, \scalerel*○ |
| Is the sentence factual? (Answers from 5 raters) | \scalerel<br>*○, \scalerel*○, \scalerel*○, \scalerel*○, \scalerel*○ | \scalerel<br>*○, \scalerel*○, \scalerel*○, \scalerel*○, \scalerel*○ | \scalerel<br>*○, \scalerel*○, \scalerel*○, \scalerel*○, \scalerel*○ |
| Rationales | - | •<br>The 16 number is not on the side of the racing car, but in the front of it.<br>•<br>The mural does feature a red Formula 1 race car, but the number 16 is painted on the front not the side | •<br>There is no checkered flag visible in the image.<br>•<br>There’s no checkered flag in the poster/mural.<br>•<br>The background mural does feature the colors blue, white and red, but there is no checkered flag |

[Table 1](https://arxiv.org/html/2506.07631v1#S3.T1) illustrates this structure, showing per-sentence annotations: five independent factuality judgments (\scalerel*○ for Entailment, \scalerel*○ for Contradiction/Neutral) and image content reliance. For non-entailed votes, collected textual rationales explain the specific error, as exemplified by the car mural’s number placement or the non-existent checkered flag. Multiple rationales for one incorrect sentence reflect diverse annotator viewpoints.

This comprehensive annotation resulted in 10,216 sentence-level judgments. The dataset—with its diverse VLM outputs, fine-grained majority-vote factuality labels, and multiple rich explanatory rationales per error—is an invaluable resource. It enables rigorous VLM evaluation beyond surface-level similarity, allowing deeper probes into model understanding and descriptive fidelity.

Table [2](https://arxiv.org/html/2506.07631v1#S3.T2) details per-model statistics from DOCCI-Critique, including description and sentence lengths, factual accuracy, and lexical diversity. These internal statistics reveal quantitative and qualitative VLM behavioral differences. For instance, high-accuracy models like GPT-4o contrast with Gemini models that produce more sentences with comparable accuracy, suggesting a verbosity/detail vs. error-risk trade-off. Paragraphs in DOCCI-Critique average 752.7 characters in length. This is considerably longer than in other contemporary datasets used for factual error analysis, such as M-HalDetect (averaging 456.2 characters), CHOCOLATE (577.6), and DetailCaps-4870 (612.9). This emphasis on longer, more intricate descriptions, combined with patterns like common error types (discernible from rationales, though not directly in Table [1](https://arxiv.org/html/2506.07631v1#S3.T1)), highlights DOCCI-Critique’s utility for nuanced comparative studies of VLM generation strategies and visual faithfulness, underscoring its role in evaluating diverse VLM behaviors in detailed captioning.

**Table 2: DOCCI-Critique statistics, detailing paragraph-level metrics and lexical diversity (unique 2-grams) for each LLM’s generated image descriptions.**
|  | Description<br>Length avg. | # Sentences<br>Avg | Sentence<br>Length Avg | % Correct Sentence<br>in Description | Uni. 2-gram |
| --- | --- | --- | --- | --- | --- |
| MiniGPT-4 [49] | 483.5 | 5.6 | 84.8 | 45.6 | 4,695 |
| mPLUG-Owl2-7B [46] | 458.8 | 4.4 | 102.1 | 52.7 | 4,038 |
| LLaVa-1.5-7B [24] | 395.4 | 4.2 | 91.5 | 60.0 | 3,081 |
| InstructBLIP [7] | 509.8 | 4.0 | 195.4 | 61.3 | 3,260 |
| PALI-5B [3] | 1098.9 | 10.9 | 69.5 | 68.0 | 1,881 |
| VILA [21] | 870.7 | 8.6 | 100.4 | 78.1 | 6,841 |
| mPLUG-Owl3-7B [45] | 118.0 | 2.0 | 65.2 | 80.4 | 700 |
| LLaVA-Onevision-7B [20] | 672.0 | 6.4 | 107.7 | 81.8 | 5,878 |
| Molmo-7B-D [8] | 747.6 | 6.6 | 111.9 | 82.7 | 6,788 |
| LLaVA-Onevision-7B-Chat [20] | 1091.6 | 9.5 | 113.1 | 85.7 | 8,550 |
| Qwen2-VL-7B-Instruct [40] | 1022.6 | 9.8 | 102.9 | 87.6 | 8,250 |
| Gemini-1.5-Pro [35] | 1326.9 | 12.0 | 109.3 | 95.1 | 11,705 |
| Gemini-1.5-Flash [35] | 1199.0 | 11.8 | 100.0 | 96.1 | 10,186 |
| GPT-4o [2024-08-06] [26] | 583.5 | 6.2 | 94.2 | 97.1 | 6,160 |
| TOTAL | 752.7 | 7.3 | 103.8 | 76.5 | 40,444 |

## 4 VNLI-Critique: Development and Evaluation

### 4.1 VNLI-Critique Model Development

We developed VNLI-Critique by fine-tuning the 10B parameter PaliGemma-2 architecture [34] (details in Appendix) for automated sentence-level factuality assessment and critique generation. This required a specialized training dataset of VLM-generated captions, distinct from DOCCI-Critique, annotated for factuality and error critiques. To create this diverse training data, we first generated paragraph-length captions using over 70 PaliGemma-2 variants (fine-tuned with varied configurations on DOCCI training data [25]) to capture a wide spectrum of generation styles and potential errors. These synthetic captions were then human-annotated per the protocol in [Section 3](https://arxiv.org/html/2506.07631v1#S3) (majority vote for labels; longest rationale for non-entailed sentences as critique target).

VNLI-Critique was fine-tuned on this curated data for two tasks using specific prompts incorporating paragraph context (<PREFIX>Claim-Prefix</PREFIX>), vital for accurate assessment of potentially ambiguous standalone sentences. For Factuality Classification, the prompt was: “Given the image and the prompt prefix <PREFIX>Claim-Prefix</PREFIX>, does the following text align with the image: <TARGET>Target-Claim</TARGET>?”, requiring a "Yes"/"No" prediction. For Critique Generation, the prompt was: “Given the image and the prompt prefix <PREFIX>Claim-Prefix</PREFIX>, the text <TARGET>Target-Claim</TARGET> is considered inaccurate. Explain the misalignments and factual inaccuracies that make it inaccurate.”. This dual-task strategy enables VNLI-Critique to identify discrepancies and articulate their reasons.

### 4.2 Factuality Classification: Benchmarking and Generalization Results

This section details the performance of VNLI-Critique in its factuality classification task, presenting key results from its application as an automated benchmarking tool on DOCCI-Critique and its generalization capabilities when tested on diverse external datasets.

#### The DOCCI-Critique AutoRater: Automated VLM Benchmarking Results.

A primary application of VNLI-Critique’s classification capability is to serve as an AutoRater for establishing an automated leaderboard that ranks Vision-Language Models (VLMs) based on their factual accuracy when describing images from the DOCCI-Critique benchmark. The objective is to provide a scalable and reliable alternative to extensive human evaluation for this task. To assess its viability, we evaluated VNLI-Critique alongside other VLM-based methods as potential automated rankers. We compared how well their automated assessments correlated with human judgments across three distinct factuality criteria: (1) Response-Level Correctness (whether the entire generated paragraph was factually accurate), (2) Percentage of Correct Sentences Overall (total correct sentences across all generated descriptions for a model), and (3) Average Percentage of Correct Sentences per Description. The detailed leaderboards showing the rankings of VLMs for each of these criteria, as determined by both human evaluation and the automated methods, are provided in Appendix. The correlation results, using Spearman’s $\rho$ (Sp $\rho$) and Kendall’s $\tau$ (Kd $\tau$), are presented in Table [3](https://arxiv.org/html/2506.07631v1#S4.T3). VNLI-Critique demonstrates exceptional performance as an AutoRater, achieving the highest Spearman correlation with human rankings on Response-Level Correctness (Sp $\rho$ = 0.981) and Percentage of Correct Sentences Overall (Sp $\rho$ = 0.979), and a very high correlation for Average Percentage of Correct Sentences per Description (Sp $\rho$ = 0.968). Its strong performance across these different evaluation granularities, significantly aligning with human assessments, validates its effectiveness as a reliable tool for automatically benchmarking VLM factuality on the DOCCI-Critique dataset.

**Table 3: Evaluating Automated Methods as AutoRaters. Correlation (Spearman’s $\rho$, Kendall’s $\tau$) between model-based rankings and human judgments of VLM factuality on DOCCI-Critique across three accuracy metrics. Bold indicates the best score, and underline indicates the second best.**
|  | % Response Correct | % Sentences Overall | % Sentences per Description |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Ranking Method (Model) | Sp $\rho$ | Kd $\tau$ | Sp $\rho$ | Kd $\tau$ | Sp $\rho$ | Kd $\tau$ |
| Emu3-Chat | -0.192 | -0.167 | 0.059 | 0.011 | 0.007 | -0.055 |
| InstructBLIP [Vicuna-7B] | -0.059 | -0.046 | 0.367 | 0.187 | 0.354 | 0.143 |
| Qwen2.5-VL-7B-Instruct | 0.290 | 0.249 | 0.692 | 0.516 | 0.697 | 0.495 |
| Janus-Pro-7B | 0.294 | 0.211 | 0.521 | 0.341 | 0.578 | 0.407 |
| mPLUG-Owl3-7B | 0.734 | 0.573 | 0.741 | 0.582 | 0.798 | 0.648 |
| LLaVa-OneVision[Qwen2-7B] | 0.889 | 0.760 | 0.855 | 0.758 | 0.851 | 0.736 |
| GPT-4o | 0.920 | 0.818 | 0.975 | 0.911 | 0.987 | 0.934 |
| Gemini-2.0-Flash | 0.972 | 0.884 | 0.976 | 0.911 | 0.956 | 0.890 |
| VNLI-Critique (Ours) | 0.981 | 0.928 | 0.979 | 0.912 | 0.968 | 0.906 |

#### Evaluating Broader Applicability on External Benchmarks.

To assess VNLI-Critique’s capabilities beyond our specific benchmark, we evaluate its performance on two established external datasets: M-HalDetect [14], a benchmark for detecting hallucinations in VLM descriptions of diverse images, and CHOCOLATE [18], which focuses on descriptions of charts and plots. We compare VNLI-Critique against various baselines, including other VLM-based classifiers and embedding-similarity methods, using two standard meta-evaluation metrics: ROC-AUC and Macro-F1.  [Figure 2](https://arxiv.org/html/2506.07631v1#S0.F2) provides a qualitative example of these sentence-level classification comparisons.

The performance metrics reported in Table [4](https://arxiv.org/html/2506.07631v1#S4.T4) were generated based on each model’s output type. For models trained to classify via specific output tokens (e.g., ‘Yes’ and ‘No’) – this includes our VNLI-Critique and other VLM-based classifiers (where scores are derived via a 5-sample strategy) – metrics reflect both confidence and prediction. For all such VLM classifiers, the entailment score for ROC-AUC calculation is obtained by applying a softmax function to the confidence scores associated with the positive and negative classification outputs (yielding a normalized positive probability). The binary classification (‘Accurate’ or ‘Inaccurate’) for Macro-F1 is determined by selecting the label with the higher confidence score. In contrast, for methods like CLIPScore [15], SigLIP [48], and TIFA [17], which output numerical similarity scores, only ROC-AUC is reported, as it directly applies to such scores without requiring an arbitrary threshold.

As shown in Table [4](https://arxiv.org/html/2506.07631v1#S4.T4), VNLI-Critique achieves state-of-the-art (SOTA) performance on M-HalDetect. Furthermore, its highly competitive performance on the CHOCOLATE dataset demonstrates notable adaptability and robust reasoning capabilities, even when evaluating descriptions of charts and graphs without specific training on such visual data. These strong findings across different benchmarks underscore the general utility of our fine-tuned model for factual verification tasks.

**Table 4: Evaluating VNLI-Critique’s factuality classification: Comparison with baselines on in-distribution (DOCCI-Critique) and external (M-HalDetect, CHOCOLATE) datasets. Key results include SOTA on M-HalDetect and strong generalization to CHOCOLATE.**
|  | DOCCI-Critique | M-HalDetect | CHOCOLATE |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | ROC-AUC | Macro-F1 | ROC-AUC | Macro-F1 | ROC-AUC | Macro-F1 |
| CLIPScore | 0.48 | - | 0.59 | - | 0.56 | - |
| VQAScore [CLIP-FlanT5] | 0.73 | - | 0.79 | - | 0.71 | - |
| VQAScore [GPT-4o] | 0.88 | - | 0.85 | - | 0.84 | - |
| SigLIP | 0.50 | - | 0.63 | - | 0.56 | - |
| TIFA | 0.61 | - | 0.70 | - | 0.57 | - |
| PaliGemma2 [9B-448res] | 0.51 | 0.23 | 0.61 | 0.39 | 0.53 | 0.00 |
| Qwen2.5-VL-7B-Instruct | 0.65 | 0.36 | 0.81 | 0.75 | 0.81 | 0.74 |
| InstructBLIP [Vicuna-7B] | 0.50 | 0.45 | 0.45 | 0.40 | 0.53 | 0.37 |
| Emu3-Chat | 0.51 | 0.50 | 0.52 | 0.42 | 0.50 | 0.37 |
| Janus-Pro-7B | 0.67 | 0.58 | 0.72 | 0.59 | 0.65 | 0.47 |
| LLaVa-OneVision[Qwen2-7B] | 0.76 | 0.58 | 0.82 | 0.60 | 0.75 | 0.44 |
| mPLUG-Owl3-7B | 0.73 | 0.65 | 0.76 | 0.68 | 0.68 | 0.54 |
| Gemini-2.0-Flash | 0.73 | 0.74 | 0.74 | 0.74 | 0.81 | 0.79 |
| GPT-4o | - | 0.74 | - | 0.69 | - | 0.70 |
| VNLI-Critique (Ours) | 0.93 | 0.83 | 0.86 | 0.76 | 0.73 | 0.68 |

### 4.3 Evaluating Critique Generation

Beyond classifying sentences’ correctness, a key capability of VNLI-Critique is generating textual critiques that explain why a sentence is factually inaccurate. To assess the quality and correctness of these generated explanations, we conducted a dedicated human evaluation study.
The evaluation proceeded as follows: First, we sampled a set of sentences previously identified by human annotators as factually incorrect from both our DOCCI-Critique benchmark and M-HalDetect dataset. For each sampled incorrect sentence, we prompted VNLI-Critique as well as several competitive VLMs (listed in [Table 5](https://arxiv.org/html/2506.07631v1#S4.T5)) to generate an explanation detailing the specific factual inaccuracies or misalignments, using the critique generation prompt format described in [Section 4.1](https://arxiv.org/html/2506.07631v1#S4.SS1).
Human annotators were then presented with evaluation instances, each containing: (1) the original image, (2) the specific factually incorrect sentence, and (3) the critique generated by the model being evaluated. The annotators’ task was to judge the quality of the critique itself – specifically, whether it accurately and relevantly identified the factual error(s) present in the sentence when compared against the visual evidence in the image. [Table 5](https://arxiv.org/html/2506.07631v1#S4.T5) displays the results of this human evaluation, reporting the percentage of generated critiques deemed correct and relevant by the annotators.
The results show that VNLI-Critique is highly effective at generating useful critiques. On critiques for DOCCI-Critique sentences, VNLI-Critique achieves the highest score (73.39%), slightly surpassing GPT-4o (73.1%). On critiques for M-HalDetect sentences, VNLI-Critique performs very strongly (79.33%), slightly behind Gemini-2.0-Flash (79.89%) but ahead of GPT-4o (78.77%). Notably, VNLI-Critique significantly outperforms several other capable VLMs like Janus-Pro-7B, Qwen-2.5-VL-Instruct, and LLaVA-OV on both datasets. This demonstrates that VNLI-Critique consistently generates high-quality, accurate explanations for factual errors, a crucial capability for providing interpretable feedback on caption quality and enabling downstream correction tasks.

**Table 5: Human evaluation of critique quality. Percentage of generated explanations judged as correct and relevant for incorrect sentences sampled from DOCCI-Critique and M-HalDetect.**
|  | DOCCI-Critique | M-HalDetect |
| --- | --- | --- |
| LLaVA-OV | 35.96 | 48.04 |
| Qwen-2.5-VL-Instruct | 45.03 | 58.1 |
| Janus-Pro-7B | 44.15 | 62.57 |
| Gemini-2.0-Flash | 64.91 | 79.89 |
| GPT-4o | 73.1 | 78.77 |
| VNLI-Critique (Ours) | 73.39 | 79.33 |

## 5 Critic-and-Revise

Many vision-language tasks, from image captioning to text-to-image generation, rely heavily on large-scale datasets of image-text pairs, often utilizing VLM-generated captions for training or as part of their data. For example, large synthetic caption datasets like PixelProse [32] are used to train captioning models, and datasets pairing images with descriptive text are fundamental for training text-to-image synthesis models (e.g., leveraging datasets like LAION [31]). However, the factual accuracy and visual alignment of these automatically generated or web-crawled captions can vary, potentially introducing noise or inaccuracies into downstream model training. Improving the quality and factual alignment of such datasets is therefore crucial for advancing these fields.
Leveraging the critique generation capability of VNLI-Critique, we introduce and evaluate a novel Critic-and-Revise pipeline. This pipeline is designed not only to correct individual factual inaccuracies in image captions but also offers a pathway to enhance the overall quality of image-text training datasets, thereby potentially improving the performance of models trained on them. This section first outlines the pipeline’s methodology ([Section 5.1](https://arxiv.org/html/2506.07631v1#S5.SS1)). We then evaluate its applicability in correcting synthetically generated captions from large-scale datasets ([Section 5.2](https://arxiv.org/html/2506.07631v1#S5.SS2)).

### 5.1 Pipeline Methodology

The Critic-and-Revise pipeline, illustrated in [Figure 1](https://arxiv.org/html/2506.07631v1#S0.F1), operates in two main steps. First, in the Critic step, VNLI-Critique analyzes each sentence of a given caption using its classification function; sentences identified as factually inaccurate trigger the generation of a textual critique explaining the specific error based on the image content. Subsequently, in the Revise step, the original inaccurate sentence and its corresponding critique from VNLI-Critique are used to instruct a separate Large Language Model (LLM) to fix the inaccurate description. For our experiments, we utilized Gemini-2.0-Flash(^1^11Accessed via the Vertex AI API: [https://cloud.google.com/vertex-ai](https://cloud.google.com/vertex-ai)) as the revision LLM. This revision LLM is prompted to rewrite the original sentence, specifically addressing the factual errors highlighted in the critique, while aiming to preserve relevant information and maintain stylistic coherence. The full Critic-and-Revise cycle—factuality classification by VNLI-Critique for all sentences, followed by critique-guided revision for those flagged as inaccurate—produces a revised caption with enhanced factual alignment to the image.

### 5.2 Correcting Synthetically Generated Captions

To demonstrate the downstream utility of our proposed Critic-and-Revise pipeline, we applied it to captions from two large-scale datasets known for detailed yet potentially unverified descriptions: PixelProse [32], featuring  16.9M synthetic caption pairs, and DetailCaps-4870 [9], a subset with 4,870 images each accompanied by three detailed synthetic captions. We conducted a human study to evaluate the pipeline’s effectiveness: after VNLI-Critique identified and critiqued inaccurate sentences, and the revision LLM corrected them, human evaluators assessed the factual correctness of both the original flagged sentences (for critic precision) and the revised sentences (for pipeline effectiveness).

The results, summarized in [Table 6](https://arxiv.org/html/2506.07631v1#S5.T6), highlight significant improvements. For DetailCaps-4870, while VNLI-Critique’s initial flagging showed a 15% false positive rate (original sentences deemed correct by humans), the pipeline successfully corrected a large portion of the genuinely inaccurate sentences, with human judges confirming 61% of the revised sentences as factually accurate. This represents a 46% gain in accuracy for the set of sentences initially considered incorrect by the critic. VNLI-Critique’s own re-evaluation classified 64% of these revised sentences as accurate, indicating strong self-consistency. Similar positive trends were observed for PixelProse, where human judges found 75% of revised sentences to be accurate (a 51% gain), demonstrating the pipeline’s capability to enhance factual accuracy in detailed image captions at scale. Qualitative examples illustrating the Critic-and-Revise process, including original incorrect sentences, critiques from VNLI-Critique, and the LLM-revised sentences, are provided in Appendix.

**Table 6: Critic-and-Revise Pipeline factuality: Human and VNLI-Critique judgments on original vs. revised claims. $\Delta$ = accuracy increase post-revision. The pipeline markedly improves claim accuracy (human-confirmed to 61% on DetailCaps, 75% on PixelProse for fixed claims, from low initial values). VNLI-Critique’s judgments align, showing high self-consistency.**
|  | DetailCaps-4870 | PixelProse |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Judge Type | Original | Fixed | $\Delta$ | Original | Fixed | $\Delta$ |
| Human Judge | 15% | 61% | +46% | 24% | 75% | +51% |
| VNLI-Critique as Judge | 0% | 64% | +64% | 0 | 61% | +61% |

## 6 Limitations and Future Work

The DOCCI-Critique benchmark, while richly annotated with 1,400 VLM-generated captions and over 10,000 sentence judgments, is constructed from a base set of 100 unique images. While these images provide diversity and the caption variations are extensive, expanding the number of base images could further enhance the benchmark’s statistical power and coverage. Nevertheless, our experiments demonstrate strong generalization capabilities. Specifically, VNLI-Critique, when trained for factuality classification on DOCCI-Critique, performs well on external, unseen claim verification datasets like M-HalDetect and CHOCOLATE ([Section 4.2](https://arxiv.org/html/2506.07631v1#S4.SS2)). Furthermore, our Critic-and-Revise pipeline, leveraging critiques from VNLI-Critique, effectively corrects captions on entirely different datasets such as DetailCaps-4870 and PixelProse ([Section 5.2](https://arxiv.org/html/2506.07631v1#S5.SS2)). This collective evidence of generalization across different tasks and datasets suggests the current DOCCI-Critique size is effective for developing robust and transferable evaluation models and correction methodologies.

Additionally, while VNLI-Critique achieves strong results in several settings, its performance is not perfect. Our pipeline evaluation ([Section 5](https://arxiv.org/html/2506.07631v1#S5)) indicates that its factuality classification can result in false positives and false negatives (e.g., a 15% false positive rate on DetailCaps-4870). The quality of its generated critiques, while generally high as shown by human evaluations in [Table 5](https://arxiv.org/html/2506.07631v1#S4.T5), can also exhibit variability. Future work could enhanceVNLI-Critique’s performance by further leveraging our rich annotations. For example, one could investigate methods to merge multiple rationales provided by different annotators into a single, more comprehensive and concise explanation, instead of solely using the longest rationale as the training target for critique generation. Furthermore, our annotation protocol captures whether a sentence’s factuality is dependent on the image content or relies on world knowledge (e.g., distinguishing “The cat is on the mat” which requires the image, from “A cat is a mammal” which does not). This currently unused label could enable a two-stage verification process: first classifying if image grounding is needed, and then applying either the visual fact-checker (VNLI-Critique) or a knowledge-based verifier accordingly, potentially improving overall accuracy and efficiency.

Regarding the Critic-and-Revise pipeline, its current design involves two distinct steps: critique generation by VNLI-Critique followed by revision using a separate LLM. While effective, this contrasts with a hypothetical end-to-end model that might directly output a corrected sentence. However, we argue that the two-step approach offers significant advantages in interpretability. Generating an explicit critique allows for a clear understanding of why a sentence was flagged and what specific error is being addressed. This insight into error types and sources is valuable for analyzing and improving the underlying captioning models, a benefit potentially lost in a direct, black-box correction approach. Therefore, while future work might explore direct revision models, the explanatory power of the intermediate critique remains a key strength of our pipeline.

Addressing these limitations and exploring the suggested avenues for leveraging the full extent of the dataset annotations offers exciting directions for future research in robust and interpretable evaluation of detailed image understanding.

## 7 Conclusion

This work tackled the critical challenge of evaluating and improving the factual accuracy of detailed, paragraph-length VLM-generated image captions. We introduced DOCCI-Critique, a novel benchmark featuring 1,400 VLM captions with over 10,216 sentence-level human annotations for factuality, including explanatory rationales for errors, providing a vital resource for fine-grained VLM assessment.
Building on this, we developed VNLI-Critique, a model proficient in automated factuality classification and critique generation. VNLI-Critique demonstrated strong generalization with state-of-the-art results on external datasets like M-HalDetect, and its use in the DOCCI-Critique showed high correlation with human judgments (0.98 Spearman).
Furthermore, we presented a novel Critic-and-Revise pipeline where VNLI-Critique’s critiques guide an LLM to automatically correct factual errors, significantly improving caption accuracy as confirmed by human evaluation.
Collectively, DOCCI-Critique, VNLI-Critique, and the Critic-and-Revise pipeline offer essential tools and methodologies for advancing VLMs towards generating more detailed, fluent, and factually reliable image descriptions. Future work, as outlined in [Section 6](https://arxiv.org/html/2506.07631v1#S6), will explore expanding the benchmark and further enhancing the pipeline’s capabilities.

## References

- Bai et al. [2025]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin.
Qwen2.5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Banerjee and Lavie [2005]
Satanjeev Banerjee and Alon Lavie.
METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.
In *Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization*, pages 65–72, Ann Arbor, Michigan, 2005. Association for Computational Linguistics.
- Chen et al. [2023]
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme Ruiz, Andreas Peter Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.
PaLI: A jointly-scaled multilingual language-image model.
In *The Eleventh International Conference on Learning Representations*, 2023.
- Chen et al. [2025]
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan.
Janus-pro: Unified multimodal understanding and generation with data and model scaling.
*arXiv preprint arXiv:2501.17811*, 2025.
- Chiang et al. [2023]
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.
- Chung et al. [2024]
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tai, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.
Scaling instruction-finetuned language models.
*J. Mach. Learn. Res.*, 25(1), 2024.
- Dai et al. [2023]
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.
Instructblip: towards general-purpose vision-language models with instruction tuning.
In *Proceedings of the 37th International Conference on Neural Information Processing Systems*, Red Hook, NY, USA, 2023. Curran Associates Inc.
- Deitke et al. [2024]
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, Yen-Sung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Jen Dumas, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross B. Girshick, Ali Farhadi, and Aniruddha Kembhavi.
Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.
*CoRR*, abs/2409.17146, 2024.
- Dong et al. [2024]
Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo.
Benchmarking and improving detail image caption.
*arXiv preprint arXiv:2405.19092*, 2024.
- Dosovitskiy et al. [2021]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at scale.
In *9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021*. OpenReview.net, 2021.
- Garg et al. [2024]
Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Michael Baldridge, and Radu Soricut.
ImageInWords: Unlocking hyper-detailed image descriptions.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 93–127, Miami, Florida, USA, 2024. Association for Computational Linguistics.
- Google Cloud [20xx]
Google Cloud.
Introduction to Cloud TPU.
[https://cloud.google.com/tpu/docs/intro-to-tpu](https://cloud.google.com/tpu/docs/intro-to-tpu), 20xx.
Accessed: 2024-07-04.
- Gordon et al. [2024]
Brian Gordon, Yonatan Bitton, Yonatan Shafir, Roopal Garg, Xi Chen, Dani Lischinski, Daniel Cohen-Or, and Idan Szpektor.
Mismatch quest: Visual and textual feedback for image-text misalignment.
In *Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LVII*, page 310–328, Berlin, Heidelberg, 2024. Springer-Verlag.
- Gunjal et al. [2024]
Anisha Gunjal, Jihan Yin, and Erhan Bas.
Detecting and preventing hallucinations in large vision language models.
In *Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence*. AAAI Press, 2024.
- Hessel et al. [2021]
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.
CLIPScore: A reference-free evaluation metric for image captioning.
In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 7514–7528, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.
- Hodosh et al. [2013]
Micah Hodosh, Peter Young, and Julia Hockenmaier.
Framing image description as a ranking task: data, models and evaluation metrics.
*J. Artif. Int. Res.*, 47(1):853–899, 2013.
- Hu et al. [2023]
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith.
TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering .
In *2023 IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 20349–20360, Los Alamitos, CA, USA, 2023. IEEE Computer Society.
- Huang et al. [2024]
Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji.
Do LVLMs understand charts? analyzing and correcting factual errors in chart captioning.
In *Findings of the Association for Computational Linguistics: ACL 2024*, pages 730–749, Bangkok, Thailand, 2024. Association for Computational Linguistics.
- Kingma and Ba [2015]
Diederik P. Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In *ICLR (Poster)*, 2015.
- Li et al. [2024]
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li.
Llava-onevision: Easy visual task transfer.
*arXiv preprint arXiv:2408.03326*, 2024.
- Lin et al. [2024]
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han.
Vila: On pre-training for visual language models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 26689–26699, 2024.
- Lin et al. [2014]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.
Microsoft coco: Common objects in context.
In *Computer Vision – ECCV 2014*, pages 740–755, Cham, 2014. Springer International Publishing.
- Lin et al. [2025]
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan.
Evaluating text-to-visual generation with image-to-text generation.
In *Computer Vision – ECCV 2024*, pages 366–384, Cham, 2025. Springer Nature Switzerland.
- Liu et al. [2024]
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 26296–26306, 2024.
- Onoe et al. [2024]
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, and Jason Baldridge.
Docci: Descriptions of connected and contrasting images.
In *Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LX*, page 291–309, Berlin, Heidelberg, 2024. Springer-Verlag.
- OpenAI et al. [2024]
OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu,
Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi
Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O’Connell, Ian O’Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang
Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal
Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam
Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov.
Gpt-4o system card, 2024.
- Papineni et al. [2002]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
Bleu: a method for automatic evaluation of machine translation.
In *Proceedings of the 40th Annual Meeting on Association for Computational Linguistics*, page 311–318, USA, 2002. Association for Computational Linguistics.
- Radford et al. [2021]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision.
In *Proceedings of the 38th International Conference on Machine Learning*, pages 8748–8763. PMLR, 2021.
- Rivière et al. [2024]
Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozinska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi,
Hanna Klimczak-Plucinska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjösund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, and Lilly McNealus.
Gemma 2: Improving open language models at a practical size.
*CoRR*, abs/2408.00118, 2024.
- Saouabe et al. [2023]
Abdelkrim Saouabe, Said Tkatek, Merouane Mazar, and Imad Mourtaji.
Evolution of image captioning models: An overview.
In *2023 10th International Conference on Wireless Networks and Mobile Communications (WINCOM)*, pages 1–5, 2023.
- Schuhmann et al. [2022]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.
Laion-5b: an open large-scale dataset for training next generation image-text models.
In *Proceedings of the 36th International Conference on Neural Information Processing Systems*, Red Hook, NY, USA, 2022. Curran Associates Inc.
- Singla et al. [2024]
Vasu Singla, Kaiyu Yue, Sukriti Paul, Reza Shirkavand, Mayuka Jayawardhana, Alireza Ganjdanesh, Heng Huang, Abhinav Bhatele, Gowthami Somepalli, and Tom Goldstein.
From pixels to prose: A large dataset of dense image captions, 2024.
- Stefanini et al. [2023]
Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara.
From Show to Tell: A Survey on Deep Learning-Based Image Captioning .
*IEEE Transactions on Pattern Analysis & Machine Intelligence*, 45(01):539–559, 2023.
- Steiner et al. [2024]
Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdulmohsin, Lucas Beyer, and Xiaohua Zhai.
Paligemma 2: A family of versatile vlms for transfer.
*arXiv preprint arXiv:2412.03555*, 2024.
- Team et al. [2024]
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcan Sercinoglu, Zach Gleicher, Juliette Love, Paul Voigtlaender, Rohan Jain, Gabriela Surita, Kareem Mohamed, Rory Blevins, Junwhan Ahn, Tao Zhu, Kornraphop Kawintiranon, Orhan Firat, Yiming Gu, Yujing Zhang, Matthew Rahtz, Manaal Faruqui, Natalie Clay, Justin Gilmer, JD Co-Reyes, Ivo Penchev, Rui Zhu, Nobuyuki Morioka, Kevin Hui, Krishna Haridasan, Victor Campos, Mahdis Mahdieh, Mandy Guo, Samer Hassan, Kevin Kilgour, Arpi Vezer, Heng-Tze Cheng, Raoul de Liedekerke, Siddharth Goyal, Paul Barham, DJ Strouse, Seb Noury, Jonas Adler, Mukund Sundararajan, Sharad Vikram, Dmitry Lepikhin, Michela Paganini, Xavier Garcia, Fan Yang, Dasha Valter, Maja Trebacz, Kiran Vodrahalli, Chulayuth
Asawaroengchai, Roman Ring, Norbert Kalb, Livio Baldini Soares, Siddhartha Brahma, David Steiner, Tianhe Yu, Fabian Mentzer, Antoine He, Lucas Gonzalez, Bibo Xu, Raphael Lopez Kaufman, Laurent El Shafey, Junhyuk Oh, Tom Hennigan, George van den Driessche, Seth Odoom, Mario Lucic, Becca Roelofs, Sid Lall, Amit Marathe, Betty Chan, Santiago Ontanon, Luheng He, Denis Teplyashin, Jonathan Lai, Phil Crone, Bogdan Damoc, Lewis Ho, Sebastian Riedel, Karel Lenc, Chih-Kuan Yeh, Aakanksha Chowdhery, Yang Xu, Mehran Kazemi, Ehsan Amid, Anastasia Petrushkina, Kevin Swersky, Ali Khodaei, Gowoon Chen, Chris Larkin, Mario Pinto, Geng Yan, Adria Puigdomenech Badia, Piyush Patil, Steven Hansen, Dave Orr, Sebastien M. R. Arnold, Jordan Grimstad, Andrew Dai, Sholto Douglas, Rishika Sinha, Vikas Yadav, Xi Chen, Elena Gribovskaya, Jacob Austin, Jeffrey Zhao, Kaushal Patel, Paul Komarek, Sophia Austin, Sebastian Borgeaud, Linda Friso, Abhimanyu Goyal, Ben Caine, Kris Cao, Da-Woon Chung, Matthew Lamm, Gabe Barth-Maron, Thais
Kagohara, Kate Olszewska, Mia Chen, Kaushik Shivakumar, Rishabh Agarwal, Harshal Godhia, Ravi Rajwar, Javier Snaider, Xerxes Dotiwalla, Yuan Liu, Aditya Barua, Victor Ungureanu, Yuan Zhang, Bat-Orgil Batsaikhan, Mateo Wirth, James Qin, Ivo Danihelka, Tulsee Doshi, Martin Chadwick, Jilin Chen, Sanil Jain, Quoc Le, Arjun Kar, Madhu Gurumurthy, Cheng Li, Ruoxin Sang, Fangyu Liu, Lampros Lamprou, Rich Munoz, Nathan Lintz, Harsh Mehta, Heidi Howard, Malcolm Reynolds, Lora Aroyo, Quan Wang, Lorenzo Blanco, Albin Cassirer, Jordan Griffith, Dipanjan Das, Stephan Lee, Jakub Sygnowski, Zach Fisher, James Besley, Richard Powell, Zafarali Ahmed, Dominik Paulus, David Reitter, Zalan Borsos, Rishabh Joshi, Aedan Pope, Steven Hand, Vittorio Selo, Vihan Jain, Nikhil Sethi, Megha Goel, Takaki Makino, Rhys May, Zhen Yang, Johan Schalkwyk, Christina Butterfield, Anja Hauth, Alex Goldin, Will Hawkins, Evan Senter, Sergey Brin, Oliver Woodman, Marvin Ritter, Eric Noland, Minh Giang, Vijay Bolina, Lisa Lee, Tim Blyth, Ian
Mackinnon, Machel Reid, Obaid Sarvana, David Silver, Alexander Chen, Lily Wang, Loren Maggiore, Oscar Chang, Nithya Attaluri, Gregory Thornton, Chung-Cheng Chiu, Oskar Bunyan, Nir Levine, Timothy Chung, Evgenii Eltyshev, Xiance Si, Timothy Lillicrap, Demetra Brady, Vaibhav Aggarwal, Boxi Wu, Yuanzhong Xu, Ross McIlroy, Kartikeya Badola, Paramjit Sandhu, Erica Moreira, Wojciech Stokowiec, Ross Hemsley, Dong Li, Alex Tudor, Pranav Shyam, Elahe Rahimtoroghi, Salem Haykal, Pablo Sprechmann, Xiang Zhou, Diana Mincu, Yujia Li, Ravi Addanki, Kalpesh Krishna, Xiao Wu, Alexandre Frechette, Matan Eyal, Allan Dafoe, Dave Lacey, Jay Whang, Thi Avrahami, Ye Zhang, Emanuel Taropa, Hanzhao Lin, Daniel Toyama, Eliza Rutherford, Motoki Sano, HyunJeong Choe, Alex Tomala, Chalence Safranek-Shrader, Nora Kassner, Mantas Pajarskas, Matt Harvey, Sean Sechrist, Meire Fortunato, Christina Lyu, Gamaleldin Elsayed, Chenkai Kuang, James Lottes, Eric Chu, Chao Jia, Chih-Wei Chen, Peter Humphreys, Kate Baumli, Connie Tao, Rajkumar
Samuel, Cicero Nogueira dos Santos, Anders Andreassen, Nemanja Rakićević, Dominik Grewe, Aviral Kumar, Stephanie Winkler, Jonathan Caton, Andrew Brock, Sid Dalmia, Hannah Sheahan, Iain Barr, Yingjie Miao, Paul Natsev, Jacob Devlin, Feryal Behbahani, Flavien Prost, Yanhua Sun, Artiom Myaskovsky, Thanumalayan Sankaranarayana Pillai, Dan Hurt, Angeliki Lazaridou, Xi Xiong, Ce Zheng, Fabio Pardo, Xiaowei Li, Dan Horgan, Joe Stanton, Moran Ambar, Fei Xia, Alejandro Lince, Mingqiu Wang, Basil Mustafa, Albert Webson, Hyo Lee, Rohan Anil, Martin Wicke, Timothy Dozat, Abhishek Sinha, Enrique Piqueras, Elahe Dabir, Shyam Upadhyay, Anudhyan Boral, Lisa Anne Hendricks, Corey Fry, Josip Djolonga, Yi Su, Jake Walker, Jane Labanowski, Ronny Huang, Vedant Misra, Jeremy Chen, RJ Skerry-Ryan, Avi Singh, Shruti Rijhwani, Dian Yu, Alex Castro-Ros, Beer Changpinyo, Romina Datta, Sumit Bagri, Arnar Mar Hrafnkelsson, Marcello Maggioni, Daniel Zheng, Yury Sulsky, Shaobo Hou, Tom Le Paine, Antoine Yang, Jason Riesa, Dominika
Rogozinska, Dror Marcus, Dalia El Badawy, Qiao Zhang, Luyu Wang, Helen Miller, Jeremy Greer, Lars Lowe Sjos, Azade Nova, Heiga Zen, Rahma Chaabouni, Mihaela Rosca, Jiepu Jiang, Charlie Chen, Ruibo Liu, Tara Sainath, Maxim Krikun, Alex Polozov, Jean-Baptiste Lespiau, Josh Newlan, Zeyncep Cankara, Soo Kwak, Yunhan Xu, Phil Chen, Andy Coenen, Clemens Meyer, Katerina Tsihlas, Ada Ma, Juraj Gottweis, Jinwei Xing, Chenjie Gu, Jin Miao, Christian Frank, Zeynep Cankara, Sanjay Ganapathy, Ishita Dasgupta, Steph Hughes-Fitt, Heng Chen, David Reid, Keran Rong, Hongmin Fan, Joost van Amersfoort, Vincent Zhuang, Aaron Cohen, Shixiang Shane Gu, Anhad Mohananey, Anastasija Ilic, Taylor Tobin, John Wieting, Anna Bortsova, Phoebe Thacker, Emma Wang, Emily Caveness, Justin Chiu, Eren Sezener, Alex Kaskasoli, Steven Baker, Katie Millican, Mohamed Elhawaty, Kostas Aisopos, Carl Lebsack, Nathan Byrd, Hanjun Dai, Wenhao Jia, Matthew Wiethoff, Elnaz Davoodi, Albert Weston, Lakshman Yagati, Arun Ahuja, Isabel Gao, Golan Pundak,
Susan Zhang, Michael Azzam, Khe Chai Sim, Sergi Caelles, James Keeling, Abhanshu Sharma, Andy Swing, YaGuang Li, Chenxi Liu, Carrie Grimes Bostock, Yamini Bansal, Zachary Nado, Ankesh Anand, Josh Lipschultz, Abhijit Karmarkar, Lev Proleev, Abe Ittycheriah, Soheil Hassas Yeganeh, George Polovets, Aleksandra Faust, Jiao Sun, Alban Rrustemi, Pen Li, Rakesh Shivanna, Jeremiah Liu, Chris Welty, Federico Lebron, Anirudh Baddepudi, Sebastian Krause, Emilio Parisotto, Radu Soricut, Zheng Xu, Dawn Bloxwich, Melvin Johnson, Behnam Neyshabur, Justin Mao-Jones, Renshen Wang, Vinay Ramasesh, Zaheer Abbas, Arthur Guez, Constant Segal, Duc Dung Nguyen, James Svensson, Le Hou, Sarah York, Kieran Milan, Sophie Bridgers, Wiktor Gworek, Marco Tagliasacchi, James Lee-Thorp, Michael Chang, Alexey Guseynov, Ale Jakse Hartman, Michael Kwong, Ruizhe Zhao, Sheleem Kashem, Elizabeth Cole, Antoine Miech, Richard Tanburn, Mary Phuong, Filip Pavetic, Sebastien Cevey, Ramona Comanescu, Richard Ives, Sherry Yang, Cosmo Du, Bo Li, Zizhao
Zhang, Mariko Iinuma, Clara Huiyi Hu, Aurko Roy, Shaan Bijwadia, Zhenkai Zhu, Danilo Martins, Rachel Saputro, Anita Gergely, Steven Zheng, Dawei Jia, Ioannis Antonoglou, Adam Sadovsky, Shane Gu, Yingying Bi, Alek Andreev, Sina Samangooei, Mina Khan, Tomas Kocisky, Angelos Filos, Chintu Kumar, Colton Bishop, Adams Yu, Sarah Hodkinson, Sid Mittal, Premal Shah, Alexandre Moufarek, Yong Cheng, Adam Bloniarz, Jaehoon Lee, Pedram Pejman, Paul Michel, Stephen Spencer, Vladimir Feinberg, Xuehan Xiong, Nikolay Savinov, Charlotte Smith, Siamak Shakeri, Dustin Tran, Mary Chesus, Bernd Bohnet, George Tucker, Tamara von Glehn, Carrie Muir, Yiran Mao, Hideto Kazawa, Ambrose Slone, Kedar Soparkar, Disha Shrivastava, James Cobon-Kerr, Michael Sharman, Jay Pavagadhi, Carlos Araya, Karolis Misiunas, Nimesh Ghelani, Michael Laskin, David Barker, Qiujia Li, Anton Briukhov, Neil Houlsby, Mia Glaese, Balaji Lakshminarayanan, Nathan Schucher, Yunhao Tang, Eli Collins, Hyeontaek Lim, Fangxiaoyu Feng, Adria Recasens, Guangda Lai,
Alberto Magni, Nicola De Cao, Aditya Siddhant, Zoe Ashwood, Jordi Orbay, Mostafa Dehghani, Jenny Brennan, Yifan He, Kelvin Xu, Yang Gao, Carl Saroufim, James Molloy, Xinyi Wu, Seb Arnold, Solomon Chang, Julian Schrittwieser, Elena Buchatskaya, Soroush Radpour, Martin Polacek, Skye Giordano, Ankur Bapna, Simon Tokumine, Vincent Hellendoorn, Thibault Sottiaux, Sarah Cogan, Aliaksei Severyn, Mohammad Saleh, Shantanu Thakoor, Laurent Shefey, Siyuan Qiao, Meenu Gaba, Shuo yiin Chang, Craig Swanson, Biao Zhang, Benjamin Lee, Paul Kishan Rubenstein, Gan Song, Tom Kwiatkowski, Anna Koop, Ajay Kannan, David Kao, Parker Schuh, Axel Stjerngren, Golnaz Ghiasi, Gena Gibson, Luke Vilnis, Ye Yuan, Felipe Tiengo Ferreira, Aishwarya Kamath, Ted Klimenko, Ken Franko, Kefan Xiao, Indro Bhattacharya, Miteyan Patel, Rui Wang, Alex Morris, Robin Strudel, Vivek Sharma, Peter Choy, Sayed Hadi Hashemi, Jessica Landon, Mara Finkelstein, Priya Jhakra, Justin Frye, Megan Barnes, Matthew Mauger, Dennis Daun, Khuslen Baatarsukh, Matthew
Tung, Wael Farhan, Henryk Michalewski, Fabio Viola, Felix de Chaumont Quitry, Charline Le Lan, Tom Hudson, Qingze Wang, Felix Fischer, Ivy Zheng, Elspeth White, Anca Dragan, Jean baptiste Alayrac, Eric Ni, Alexander Pritzel, Adam Iwanicki, Michael Isard, Anna Bulanova, Lukas Zilka, Ethan Dyer, Devendra Sachan, Srivatsan Srinivasan, Hannah Muckenhirn, Honglong Cai, Amol Mandhane, Mukarram Tariq, Jack W. Rae, Gary Wang, Kareem Ayoub, Nicholas FitzGerald, Yao Zhao, Woohyun Han, Chris Alberti, Dan Garrette, Kashyap Krishnakumar, Mai Gimenez, Anselm Levskaya, Daniel Sohn, Josip Matak, Inaki Iturrate, Michael B. Chang, Jackie Xiang, Yuan Cao, Nishant Ranka, Geoff Brown, Adrian Hutter, Vahab Mirrokni, Nanxin Chen, Kaisheng Yao, Zoltan Egyed, Francois Galilee, Tyler Liechty, Praveen Kallakuri, Evan Palmer, Sanjay Ghemawat, Jasmine Liu, David Tao, Chloe Thornton, Tim Green, Mimi Jasarevic, Sharon Lin, Victor Cotruta, Yi-Xuan Tan, Noah Fiedel, Hongkun Yu, Ed Chi, Alexander Neitz, Jens Heitkaemper, Anu Sinha, Denny
Zhou, Yi Sun, Charbel Kaed, Brice Hulse, Swaroop Mishra, Maria Georgaki, Sneha Kudugunta, Clement Farabet, Izhak Shafran, Daniel Vlasic, Anton Tsitsulin, Rajagopal Ananthanarayanan, Alen Carin, Guolong Su, Pei Sun, Shashank V, Gabriel Carvajal, Josef Broder, Iulia Comsa, Alena Repina, William Wong, Warren Weilun Chen, Peter Hawkins, Egor Filonov, Lucia Loher, Christoph Hirnschall, Weiyi Wang, Jingchen Ye, Andrea Burns, Hardie Cate, Diana Gage Wright, Federico Piccinini, Lei Zhang, Chu-Cheng Lin, Ionel Gog, Yana Kulizhskaya, Ashwin Sreevatsa, Shuang Song, Luis C. Cobo, Anand Iyer, Chetan Tekur, Guillermo Garrido, Zhuyun Xiao, Rupert Kemp, Huaixiu Steven Zheng, Hui Li, Ananth Agarwal, Christel Ngani, Kati Goshvadi, Rebeca Santamaria-Fernandez, Wojciech Fica, Xinyun Chen, Chris Gorgolewski, Sean Sun, Roopal Garg, Xinyu Ye, S. M. Ali Eslami, Nan Hua, Jon Simon, Pratik Joshi, Yelin Kim, Ian Tenney, Sahitya Potluri, Lam Nguyen Thiet, Quan Yuan, Florian Luisier, Alexandra Chronopoulou, Salvatore Scellato, Praveen
Srinivasan, Minmin Chen, Vinod Koverkathu, Valentin Dalibard, Yaming Xu, Brennan Saeta, Keith Anderson, Thibault Sellam, Nick Fernando, Fantine Huot, Junehyuk Jung, Mani Varadarajan, Michael Quinn, Amit Raul, Maigo Le, Ruslan Habalov, Jon Clark, Komal Jalan, Kalesha Bullard, Achintya Singhal, Thang Luong, Boyu Wang, Sujeevan Rajayogam, Julian Eisenschlos, Johnson Jia, Daniel Finchelstein, Alex Yakubovich, Daniel Balle, Michael Fink, Sameer Agarwal, Jing Li, Dj Dvijotham, Shalini Pal, Kai Kang, Jaclyn Konzelmann, Jennifer Beattie, Olivier Dousse, Diane Wu, Remi Crocker, Chen Elkind, Siddhartha Reddy Jonnalagadda, Jong Lee, Dan Holtmann-Rice, Krystal Kallarackal, Rosanne Liu, Denis Vnukov, Neera Vats, Luca Invernizzi, Mohsen Jafari, Huanjie Zhou, Lilly Taylor, Jennifer Prendki, Marcus Wu, Tom Eccles, Tianqi Liu, Kavya Kopparapu, Francoise Beaufays, Christof Angermueller, Andreea Marzoca, Shourya Sarcar, Hilal Dib, Jeff Stanway, Frank Perbet, Nejc Trdin, Rachel Sterneck, Andrey Khorlin, Dinghua Li, Xihui Wu,
Sonam Goenka, David Madras, Sasha Goldshtein, Willi Gierke, Tong Zhou, Yaxin Liu, Yannie Liang, Anais White, Yunjie Li, Shreya Singh, Sanaz Bahargam, Mark Epstein, Sujoy Basu, Li Lao, Adnan Ozturel, Carl Crous, Alex Zhai, Han Lu, Zora Tung, Neeraj Gaur, Alanna Walton, Lucas Dixon, Ming Zhang, Amir Globerson, Grant Uy, Andrew Bolt, Olivia Wiles, Milad Nasr, Ilia Shumailov, Marco Selvi, Francesco Piccinno, Ricardo Aguilar, Sara McCarthy, Misha Khalman, Mrinal Shukla, Vlado Galic, John Carpenter, Kevin Villela, Haibin Zhang, Harry Richardson, James Martens, Matko Bosnjak, Shreyas Rammohan Belle, Jeff Seibert, Mahmoud Alnahlawi, Brian McWilliams, Sankalp Singh, Annie Louis, Wen Ding, Dan Popovici, Lenin Simicich, Laura Knight, Pulkit Mehta, Nishesh Gupta, Chongyang Shi, Saaber Fatehi, Jovana Mitrovic, Alex Grills, Joseph Pagadora, Tsendsuren Munkhdalai, Dessie Petrova, Danielle Eisenbud, Zhishuai Zhang, Damion Yates, Bhavishya Mittal, Nilesh Tripuraneni, Yannis Assael, Thomas Brovelli, Prateek Jain, Mihajlo
Velimirovic, Canfer Akbulut, Jiaqi Mu, Wolfgang Macherey, Ravin Kumar, Jun Xu, Haroon Qureshi, Gheorghe Comanici, Jeremy Wiesner, Zhitao Gong, Anton Ruddock, Matthias Bauer, Nick Felt, Anirudh GP, Anurag Arnab, Dustin Zelle, Jonas Rothfuss, Bill Rosgen, Ashish Shenoy, Bryan Seybold, Xinjian Li, Jayaram Mudigonda, Goker Erdogan, Jiawei Xia, Jiri Simsa, Andrea Michi, Yi Yao, Christopher Yew, Steven Kan, Isaac Caswell, Carey Radebaugh, Andre Elisseeff, Pedro Valenzuela, Kay McKinney, Kim Paterson, Albert Cui, Eri Latorre-Chimoto, Solomon Kim, William Zeng, Ken Durden, Priya Ponnapalli, Tiberiu Sosea, Christopher A. Choquette-Choo, James Manyika, Brona Robenek, Harsha Vashisht, Sebastien Pereira, Hoi Lam, Marko Velic, Denese Owusu-Afriyie, Katherine Lee, Tolga Bolukbasi, Alicia Parrish, Shawn Lu, Jane Park, Balaji Venkatraman, Alice Talbert, Lambert Rosique, Yuchung Cheng, Andrei Sozanschi, Adam Paszke, Praveen Kumar, Jessica Austin, Lu Li, Khalid Salama, Bartek Perz, Wooyeol Kim, Nandita Dukkipati, Anthony
Baryshnikov, Christos Kaplanis, XiangHai Sheng, Yuri Chervonyi, Caglar Unlu, Diego de Las Casas, Harry Askham, Kathryn Tunyasuvunakool, Felix Gimeno, Siim Poder, Chester Kwak, Matt Miecnikowski, Vahab Mirrokni, Alek Dimitriev, Aaron Parisi, Dangyi Liu, Tomy Tsai, Toby Shevlane, Christina Kouridi, Drew Garmon, Adrian Goedeckemeyer, Adam R. Brown, Anitha Vijayakumar, Ali Elqursh, Sadegh Jazayeri, Jin Huang, Sara Mc Carthy, Jay Hoover, Lucy Kim, Sandeep Kumar, Wei Chen, Courtney Biles, Garrett Bingham, Evan Rosen, Lisa Wang, Qijun Tan, David Engel, Francesco Pongetti, Dario de Cesare, Dongseong Hwang, Lily Yu, Jennifer Pullman, Srini Narayanan, Kyle Levin, Siddharth Gopal, Megan Li, Asaf Aharoni, Trieu Trinh, Jessica Lo, Norman Casagrande, Roopali Vij, Loic Matthey, Bramandia Ramadhana, Austin Matthews, CJ Carey, Matthew Johnson, Kremena Goranova, Rohin Shah, Shereen Ashraf, Kingshuk Dasgupta, Rasmus Larsen, Yicheng Wang, Manish Reddy Vuyyuru, Chong Jiang, Joana Ijazi, Kazuki Osawa, Celine Smith, Ramya Sree
Boppana, Taylan Bilal, Yuma Koizumi, Ying Xu, Yasemin Altun, Nir Shabat, Ben Bariach, Alex Korchemniy, Kiam Choo, Olaf Ronneberger, Chimezie Iwuanyanwu, Shubin Zhao, David Soergel, Cho-Jui Hsieh, Irene Cai, Shariq Iqbal, Martin Sundermeyer, Zhe Chen, Elie Bursztein, Chaitanya Malaviya, Fadi Biadsy, Prakash Shroff, Inderjit Dhillon, Tejasi Latkar, Chris Dyer, Hannah Forbes, Massimo Nicosia, Vitaly Nikolaev, Somer Greene, Marin Georgiev, Pidong Wang, Nina Martin, Hanie Sedghi, John Zhang, Praseem Banzal, Doug Fritz, Vikram Rao, Xuezhi Wang, Jiageng Zhang, Viorica Patraucean, Dayou Du, Igor Mordatch, Ivan Jurin, Lewis Liu, Ayush Dubey, Abhi Mohan, Janek Nowakowski, Vlad-Doru Ion, Nan Wei, Reiko Tojo, Maria Abi Raad, Drew A. Hudson, Vaishakh Keshava, Shubham Agrawal, Kevin Ramirez, Zhichun Wu, Hoang Nguyen, Ji Liu, Madhavi Sewak, Bryce Petrini, DongHyun Choi, Ivan Philips, Ziyue Wang, Ioana Bica, Ankush Garg, Jarek Wilkiewicz, Priyanka Agrawal, Xiaowei Li, Danhao Guo, Emily Xue, Naseer Shaik, Andrew Leach,
Sadh MNM Khan, Julia Wiesinger, Sammy Jerome, Abhishek Chakladar, Alek Wenjiao Wang, Tina Ornduff, Folake Abu, Alireza Ghaffarkhah, Marcus Wainwright, Mario Cortes, Frederick Liu, Joshua Maynez, Andreas Terzis, Pouya Samangouei, Riham Mansour, Tomasz Kępa, François-Xavier Aubet, Anton Algymr, Dan Banica, Agoston Weisz, Andras Orban, Alexandre Senges, Ewa Andrejczuk, Mark Geller, Niccolo Dal Santo, Valentin Anklin, Majd Al Merey, Martin Baeuml, Trevor Strohman, Junwen Bai, Slav Petrov, Yonghui Wu, Demis Hassabis, Koray Kavukcuoglu, Jeff Dean, and Oriol Vinyals.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
- Touvron et al. [2023]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas
Scialom.
Llama 2: Open foundation and fine-tuned chat models, 2023.
- Tschannen et al. [2025]
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai.
Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features, 2025.
- Urbanek et al. [2024]
Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano.
A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 26700–26709, 2024.
- Vedantam et al. [2015]
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh.
Cider: Consensus-based image description evaluation.
In *CVPR*, pages 4566–4575. IEEE Computer Society, 2015.
- Wang et al. [2024a]
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin.
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.
*arXiv preprint arXiv:2409.12191*, 2024a.
- Wang et al. [2024b]
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al.
Emu3: Next-token prediction is all you need.
*arXiv preprint arXiv:2409.18869*, 2024b.
- Wiles et al. [2025]
Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kajic, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Pinelopi Papalampidi, Ira Ktena, Christopher Knutsen, Cyrus Rashtchian, Anant Nawalgaria, Jordi Pont-Tuset, and Aida Nematzadeh.
Revisiting text-to-image evaluation with gecko: on metrics, prompts, and human rating.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Yang et al. [2024]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan.
Qwen2 technical report, 2024.
- Yarom et al. [2023]
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor.
What you see is what you read? improving text-image alignment evaluation.
In *Advances in Neural Information Processing Systems*, pages 1601–1619. Curran Associates, Inc., 2023.
- Ye et al. [2024a]
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou.
mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024a.
- Ye et al. [2024b]
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang.
mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 13040–13051, 2024b.
- Young et al. [2014]
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.
*Transactions of the Association for Computational Linguistics*, 2:67–78, 2014.
- Zhai et al. [2023]
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.
Sigmoid loss for language image pre-training.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 11975–11986, 2023.
- Zhu et al. [2023]
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.
Minigpt-4: Enhancing vision-language understanding with advanced large language models.
*arXiv preprint arXiv:2304.10592*, 2023.

## Appendix A DOCCI-Critique AutoRater Leaderboards

This appendix presents the complete leaderboards detailing the performance of various Vision-Language Models (VLMs) as automated rankers (AutoRaters) on the DOCCI-Critique benchmark. These tables supplement the summary correlation metrics (Spearman’s $\rho$ and Kendall’s $\tau$) found in [Section 4.2](https://arxiv.org/html/2506.07631v1#S4.SS2), [Table 3](https://arxiv.org/html/2506.07631v1#S4.T3). Our goal was to assess how well automated methods, including our VNLI-Critique, rank caption-generating VLMs by factual accuracy against human-derived Ground Truth rankings.

Three leaderboards are provided, each for a distinct factuality criterion:

- 1.
Response-Level Correctness: Percentage of entirely factually accurate paragraphs (Table [7](https://arxiv.org/html/2506.07631v1#A1.T7)).
- 2.
Correct Sentences Overall: Total percentage of correct sentences across all descriptions (Table [9](https://arxiv.org/html/2506.07631v1#A1.T9)).
- 3.
Correct Sentences per Description: Average percentage of correct sentences per description (Table [8](https://arxiv.org/html/2506.07631v1#A1.T8)).

In each leaderboard (Tables [7](https://arxiv.org/html/2506.07631v1#A1.T7), [9](https://arxiv.org/html/2506.07631v1#A1.T9), and [8](https://arxiv.org/html/2506.07631v1#A1.T8)), rows list the 14 caption-generating VLMs from DOCCI-Critique (details in [Table 2](https://arxiv.org/html/2506.07631v1#S3.T2)). Columns denote automated ranking methods (e.g., ‘Ours (VNLI-Critique)’, ‘GPT-4o’). Cells show the rank assigned by the column’s method to the row’s VLM, with the superscript indicating the raw metric score. The final two rows report Spearman’s $\rho$ (with p-value superscript) and Kendall’s $\tau$ correlations against the Ground Truth for that criterion, offering a nuanced view of each AutoRater’s performance.

**Table 7: VLM AutoRater rankings for Response-Level Correctness on DOCCI-Critique. Each cell shows $Rank^{Metric-Score}$. Final two rows: Spearman’s $\rho^{p-value}$ and Kendall’s $\tau^{p-value}$ correlation against Human.**
| Captioner VLM<br>Ranking Method | Human | Ours | Gemini<br>2.0-Flash | GPT-4o | InstructBLIP | LLaVa-OV | Janus-Pro-7B | Qwen2.5-VL | mPLUG-Owl3-7B | Emu3-Chat |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MiniGPT-4 | 14${}^{$0.040$}$ | 14${}^{$0.060$}$ | 14${}^{$0.090$}$ | 14${}^{$0.120$}$ | 7${}^{$0.960$}$ | 14${}^{$0.420$}$ | 13${}^{$0.350$}$ | 4${}^{$0.0$}$ | 13${}^{$0.110$}$ | 3${}^{$0.930$}$ |
| MPlugOwl-2 | 13${}^{$0.110$}$ | 12${}^{$0.120$}$ | 13${}^{$0.280$}$ | 12${}^{$0.180$}$ | 2${}^{$0.980$}$ | 12${}^{$0.560$}$ | 10${}^{$0.560$}$ | 4${}^{$0.0$}$ | 12${}^{$0.170$}$ | 9${}^{$0.550$}$ |
| LLaVA | 11${}^{$0.190$}$ | 11${}^{$0.170$}$ | 9${}^{$0.400$}$ | 11${}^{$0.2551$}$ | 3${}^{$0.970$}$ | 10${}^{$0.710$}$ | 5${}^{$0.680$}$ | 4${}^{$0.0$}$ | 8${}^{$0.210$}$ | 12${}^{$0.480$}$ |
| PALI-5B | 12${}^{$0.112$}$ | 13${}^{$0.092$}$ | 12${}^{$0.3093$}$ | 13${}^{$0.1735$}$ | 9${}^{$0.9592$}$ | 11${}^{$0.5816$}$ | 14${}^{$0.1939$}$ | 4${}^{$0.0$}$ | 14${}^{$0.0714$}$ | 2${}^{$0.9898$}$ |
| VILA | 10${}^{$0.210$}$ | 10${}^{$0.200$}$ | 10${}^{$0.390$}$ | 9${}^{$0.330$}$ | 13${}^{$0.910$}$ | 9${}^{$0.850$}$ | 1${}^{$0.790$}$ | 4${}^{$0.0$}$ | 8${}^{$0.210$}$ | 4${}^{$0.780$}$ |
| InstructBLIP | 9${}^{$0.260$}$ | 9${}^{$0.210$}$ | 11${}^{$0.3776$}$ | 10${}^{$0.310$}$ | 9${}^{$0.9592$}$ | 13${}^{$0.540$}$ | 9${}^{$0.600$}$ | 2${}^{$0.010$}$ | 6${}^{$0.260$}$ | 10${}^{$0.540$}$ |
| Molmo-7B-D | 8${}^{$0.310$}$ | 8${}^{$0.270$}$ | 7${}^{$0.680$}$ | 8${}^{$0.580$}$ | 1${}^{$0.990$}$ | 5${}^{$0.930$}$ | 4${}^{$0.700$}$ | 4${}^{$0.0$}$ | 8${}^{$0.210$}$ | 4${}^{$0.780$}$ |
| LLaVA-OV-7B-Chat | 7${}^{$0.360$}$ | 6${}^{$0.350$}$ | 8${}^{$0.630$}$ | 5${}^{$0.640$}$ | 12${}^{$0.930$}$ | 8${}^{$0.900$}$ | 8${}^{$0.660$}$ | 4${}^{$0.0$}$ | 4${}^{$0.300$}$ | 8${}^{$0.580$}$ |
| Qwen2-VL-7B-Instruct | 5${}^{$0.410$}$ | 4${}^{$0.450$}$ | 5${}^{$0.750$}$ | 4${}^{$0.650$}$ | 3${}^{$0.970$}$ | 6${}^{$0.910$}$ | 2${}^{$0.780$}$ | 4${}^{$0.0$}$ | 5${}^{$0.290$}$ | 11${}^{$0.490$}$ |
| LLaVA-OV-7B | 5${}^{$0.410$}$ | 7${}^{$0.340$}$ | 6${}^{$0.720$}$ | 7${}^{$0.630$}$ | 14${}^{$0.900$}$ | 2${}^{$0.970$}$ | 6${}^{$0.670$}$ | 2${}^{$0.010$}$ | 2${}^{$0.410$}$ | 7${}^{$0.610$}$ |
| Gemini-1.5-Pro | 4${}^{$0.640$}$ | 5${}^{$0.430$}$ | 4${}^{$0.840$}$ | 3${}^{$0.670$}$ | 3${}^{$0.970$}$ | 6${}^{$0.910$}$ | 12${}^{$0.490$}$ | 4${}^{$0.0$}$ | 7${}^{$0.240$}$ | 14${}^{$0.310$}$ |
| Gemini-1.5-Flash | 3${}^{$0.680$}$ | 3${}^{$0.520$}$ | 3${}^{$0.880$}$ | 5${}^{$0.640$}$ | 11${}^{$0.940$}$ | 4${}^{$0.940$}$ | 11${}^{$0.510$}$ | 4${}^{$0.0$}$ | 11${}^{$0.180$}$ | 13${}^{$0.400$}$ |
| mPLUG-Owl3-7B | 2${}^{$0.710$}$ | 2${}^{$0.690$}$ | 2${}^{$0.9326$}$ | 2${}^{$0.770$}$ | 7${}^{$0.960$}$ | 1${}^{$0.980$}$ | 3${}^{$0.730$}$ | 1${}^{$0.020$}$ | 1${}^{$0.750$}$ | 1${}^{$1.0$}$ |
| GPT-4o[2024-08-06] | 1${}^{$0.830$}$ | 1${}^{$0.730$}$ | 1${}^{$0.940$}$ | 1${}^{$0.890$}$ | 3${}^{$0.970$}$ | 2${}^{$0.970$}$ | 6${}^{$0.670$}$ | 4${}^{$0.0$}$ | 3${}^{$0.370$}$ | 6${}^{$0.740$}$ |
| Spearman’s Rank $\rho$ | – | 0.98^5e-10 | 0.97^6e-9 | 0.92^3e-6 | -0.06^8e-1 | 0.88^2e-5 | 0.30^3e-1 | 0.29^3e-1 | 0.73^3e-3 | -0.2^5e-1 |
| Kendall Tau $\tau$ | – | 0.93^4e-6 | 0.89^1e-5 | 0.82^5e-5 | -0.05^8e-1 | 0.76^2e-4 | 0.21^3e-1 | 0.25^3e-1 | 0.27^5e-3 | -0.17^4e-1 |

**Table 8: VLM AutoRater rankings for Average Percentage of Correct Sentences on DOCCI-Critique. Each cell shows $Rank^{Metric-Score}$. Final two rows: Spearman’s $\rho^{p-value}$ and Kendall’s $\tau^{p-value}$ correlation against Human.**
| Captioner VLM<br>Ranking Method | Human | Ours | Gemini<br>2.0-Flash | GPT-4o | InstructBLIP | LLaVa-OV | Janus-Pro-7B | Qwen2.5-VL | mPLUG-Owl3-7B | Emu3-Chat |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MiniGPT-4 | 14${}^{$0.456$}$ | 14${}^{$0.476$}$ | 14${}^{$0.529$}$ | 14${}^{$0.4242$}$ | 8${}^{$0.9931$}$ | 14${}^{$0.8385$}$ | 13${}^{$0.772$}$ | 13${}^{$0.0493$}$ | 14${}^{$0.5109$}$ | 3${}^{$0.9866$}$ |
| MPlugOwl-2 | 13${}^{$0.527$}$ | 13${}^{$0.543$}$ | 13${}^{$0.6597$}$ | 13${}^{$0.5573$}$ | 7${}^{$0.9935$}$ | 12${}^{$0.8853$}$ | 12${}^{$0.8582$}$ | 11${}^{$0.0538$}$ | 13${}^{$0.5343$}$ | 12${}^{$0.8617$}$ |
| LLaVA | 12${}^{$0.600$}$ | 11${}^{$0.589$}$ | 11${}^{$0.7475$}$ | 12${}^{$0.5876$}$ | 9${}^{$0.9917$}$ | 11${}^{$0.913$}$ | 9${}^{$0.9018$}$ | 9${}^{$0.0817$}$ | 11${}^{$0.5798$}$ | 14${}^{$0.8325$}$ |
| InstructBLIP | 11${}^{$0.613$}$ | 12${}^{$0.581$}$ | 12${}^{$0.7054$}$ | 11${}^{$0.6225$}$ | 12${}^{$0.9804$}$ | 13${}^{$0.8619$}$ | 10${}^{$0.8902$}$ | 10${}^{$0.0544$}$ | 12${}^{$0.5638$}$ | 13${}^{$0.840$}$ |
| PALI-5B | 10${}^{$0.672$}$ | 10${}^{$0.675$}$ | 10${}^{$0.7948$}$ | 10${}^{$0.6711$}$ | 3${}^{$0.9966$}$ | 10${}^{$0.9133$}$ | 14${}^{$0.7338$}$ | 12${}^{$0.0518$}$ | 10${}^{$0.6068$}$ | 2${}^{$0.9992$}$ |
| VILA | 9${}^{$0.781$}$ | 8${}^{$0.784$}$ | 9${}^{$0.8604$}$ | 9${}^{$0.8062$}$ | 11${}^{$0.9896$}$ | 9${}^{$0.9832$}$ | 1${}^{$0.9697$}$ | 4${}^{$0.2229$}$ | 8${}^{$0.7644$}$ | 4${}^{$0.9619$}$ |
| mPLUG-Owl3-7B | 8${}^{$0.804$}$ | 6${}^{$0.799$}$ | 4${}^{$0.9781$}$ | 8${}^{$0.8675$}$ | 13${}^{$0.9767$}$ | 5${}^{$0.9921$}$ | 11${}^{$0.8655$}$ | 14${}^{$0.0475$}$ | 2${}^{$0.8474$}$ | 1${}^{$1.0$}$ |
| LLaVA-OV-7B | 7${}^{$0.818$}$ | 6${}^{$0.799$}$ | 7${}^{$0.9433$}$ | 6${}^{$0.9052$}$ | 14${}^{$0.9736$}$ | 1${}^{$0.9962$}$ | 5${}^{$0.9366$}$ | 6${}^{$0.1959$}$ | 7${}^{$0.8102$}$ | 8${}^{$0.9189$}$ |
| Molmo-7B-D | 6${}^{$0.827$}$ | 9${}^{$0.783$}$ | 6${}^{$0.9457$}$ | 7${}^{$0.9015$}$ | 1${}^{$0.9983$}$ | 7${}^{$0.9895$}$ | 4${}^{$0.9385$}$ | 8${}^{$0.0987$}$ | 9${}^{$0.725$}$ | 5${}^{$0.953$}$ |
| LLaVA-OV-7B-Chat | 5${}^{$0.857$}$ | 5${}^{$0.854$}$ | 8${}^{$0.9415$}$ | 4${}^{$0.9364$}$ | 10${}^{$0.9912$}$ | 8${}^{$0.9883$}$ | 3${}^{$0.944$}$ | 1${}^{$0.2779$}$ | 1${}^{$0.8482$}$ | 7${}^{$0.9305$}$ |
| Qwen2-VL-7B-Instruct | 4${}^{$0.876$}$ | 4${}^{$0.884$}$ | 5${}^{$0.9677$}$ | 5${}^{$0.934$}$ | 4${}^{$0.9962$}$ | 6${}^{$0.9915$}$ | 2${}^{$0.9681$}$ | 5${}^{$0.2227$}$ | 5${}^{$0.8275$}$ | 9${}^{$0.9078$}$ |
| Gemini-1.5-Pro | 3${}^{$0.951$}$ | 3${}^{$0.925$}$ | 3${}^{$0.9856$}$ | 2${}^{$0.9664$}$ | 2${}^{$0.9968$}$ | 4${}^{$0.9928$}$ | 8${}^{$0.9282$}$ | 3${}^{$0.251$}$ | 4${}^{$0.8341$}$ | 11${}^{$0.8929$}$ |
| Gemini-1.5-Flash | 2${}^{$0.961$}$ | 2${}^{$0.939$}$ | 2${}^{$0.9894$}$ | 3${}^{$0.9544$}$ | 5${}^{$0.9942$}$ | 3${}^{$0.9948$}$ | 7${}^{$0.9342$}$ | 2${}^{$0.271$}$ | 3${}^{$0.8381$}$ | 10${}^{$0.9017$}$ |
| GPT-4o[2024-08-06] | 1${}^{$0.971$}$ | 1${}^{$0.949$}$ | 1${}^{$0.9914$}$ | 1${}^{$0.9801$}$ | 6${}^{$0.9942$}$ | 2${}^{$0.995$}$ | 6${}^{$0.9345$}$ | 7${}^{$0.1664$}$ | 6${}^{$0.8233$}$ | 6${}^{$0.9477$}$ |
| Spearman’s Rank $\rho$ | – | 0.97^1e-8 | 0.96^9e-8 | 0.99^7e-11 | 0.35^2e-1 | 0.85^1e-4 | 0.58^3e-2 | 0.70^5e-3 | 0.80^6e-4 | 0.00^1e-0 |
| Kendall Tau $\tau$ | – | 0.91^7e-6 | 0.90^2e-7 | 0.93^1e-8 | 0.14^5e-1 | 0.74^7e-5 | 0.40^4e-2 | 0.50^1e-2 | 0.65^7e-4 | -0.05^8e-1 |

**Table 9: VLM AutoRater rankings for Percentage of Correct Sentences Overall on DOCCI-Critique. Each cell shows $Rank^{Metric-Score}$. Final two rows: Spearman’s $\rho^{p-value}$ and Kendall’s $\tau^{p-value}$ correlation against Human.**
| Captioner VLM<br>Ranking Method | Human | Ours | Gemini<br>2.0-Flash | GPT-4o | InstructBLIP | LLaVa-OV | Janus-Pro-7B | Qwen2.5-VL | mPLUG-Owl3-7B | Emu3-Chat |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MiniGPT-4 | 14${}^{$0.477$}$ | 14${}^{$0.493$}$ | 14${}^{$0.550$}$ | 14${}^{$0.444$}$ | 7${}^{$0.993$}$ | 14${}^{$0.825$}$ | 13${}^{$0.768$}$ | 11${}^{$0.055$}$ | 13${}^{$0.518$}$ | 3${}^{$0.988$}$ |
| MPlugOwl-2 | 13${}^{$0.516$}$ | 13${}^{$0.532$}$ | 13${}^{$0.642$}$ | 13${}^{$0.562$}$ | 7${}^{$0.993$}$ | 12${}^{$0.872$}$ | 11${}^{$0.847$}$ | 13${}^{$0.052$}$ | 14${}^{$0.514$}$ | 12${}^{$0.863$}$ |
| InstructBLIP | 12${}^{$0.574$}$ | 12${}^{$0.567$}$ | 12${}^{$0.677$}$ | 11${}^{$0.588$}$ | 11${}^{$0.990$}$ | 13${}^{$0.839$}$ | 10${}^{$0.869$}$ | 12${}^{$0.054$}$ | 12${}^{$0.537$}$ | 14${}^{$0.824$}$ |
| LLaVA | 11${}^{$0.594$}$ | 11${}^{$0.585$}$ | 10${}^{$0.741$}$ | 12${}^{$0.587$}$ | 7${}^{$0.993$}$ | 10${}^{$0.903$}$ | 9${}^{$0.894$}$ | 9${}^{$0.083$}$ | 11${}^{$0.583$}$ | 13${}^{$0.828$}$ |
| PALI-5B | 10${}^{$0.671$}$ | 10${}^{$0.663$}$ | 10${}^{$0.741$}$ | 10${}^{$0.620$}$ | 4${}^{$0.996$}$ | 11${}^{$0.877$}$ | 14${}^{$0.725$}$ | 10${}^{$0.067$}$ | 10${}^{$0.591$}$ | 2${}^{$0.999$}$ |
| VILA | 9${}^{$0.789$}$ | 8${}^{$0.790$}$ | 9${}^{$0.859$}$ | 9${}^{$0.805$}$ | 11${}^{$0.990$}$ | 9${}^{$0.983$}$ | 1${}^{$0.969$}$ | 4${}^{$0.224$}$ | 8${}^{$0.769$}$ | 4${}^{$0.962$}$ |
| LLaVA-OV-7B | 8${}^{$0.815$}$ | 7${}^{$0.812$}$ | 8${}^{$0.945$}$ | 6${}^{$0.906$}$ | 13${}^{$0.984$}$ | 1${}^{$0.995$}$ | 5${}^{$0.934$}$ | 6${}^{$0.213$}$ | 2${}^{$0.845$}$ | 9${}^{$0.906$}$ |
| mPLUG-Owl3-7B | 7${}^{$0.822$}$ | 6${}^{$0.822$}$ | 4${}^{$0.960$}$ | 8${}^{$0.840$}$ | 14${}^{$0.980$}$ | 8${}^{$0.985$}$ | 12${}^{$0.812$}$ | 14${}^{$0.051$}$ | 3${}^{$0.838$}$ | 1${}^{$1.0$}$ |
| Molmo-7B-D | 6${}^{$0.831$}$ | 9${}^{$0.786$}$ | 6${}^{$0.946$}$ | 7${}^{$0.903$}$ | 1${}^{$0.998$}$ | 6${}^{$0.989$}$ | 4${}^{$0.937$}$ | 8${}^{$0.098$}$ | 9${}^{$0.718$}$ | 5${}^{$0.955$}$ |
| Qwen2-VL-7B-Instruct | 5${}^{$0.867$}$ | 4${}^{$0.877$}$ | 4${}^{$0.960$}$ | 5${}^{$0.929$}$ | 2${}^{$0.997$}$ | 5${}^{$0.991$}$ | 2${}^{$0.966$}$ | 5${}^{$0.219$}$ | 6${}^{$0.833$}$ | 8${}^{$0.907$}$ |
| LLaVA-OV-7B-Chat | 4${}^{$0.869$}$ | 5${}^{$0.875$}$ | 6${}^{$0.946$}$ | 4${}^{$0.935$}$ | 10${}^{$0.991$}$ | 7${}^{$0.988$}$ | 3${}^{$0.958$}$ | 1${}^{$0.311$}$ | 1${}^{$0.868$}$ | 7${}^{$0.924$}$ |
| Gemini-1.5-Pro | 3${}^{$0.954$}$ | 3${}^{$0.925$}$ | 3${}^{$0.985$}$ | 2${}^{$0.968$}$ | 2${}^{$0.997$}$ | 4${}^{$0.992$}$ | 8${}^{$0.929$}$ | 3${}^{$0.250$}$ | 5${}^{$0.835$}$ | 11${}^{$0.891$}$ |
| Gemini-1.5-Flash | 2${}^{$0.962$}$ | 2${}^{$0.937$}$ | 1${}^{$0.990$}$ | 3${}^{$0.953$}$ | 5${}^{$0.995$}$ | 1${}^{$0.995$}$ | 5${}^{$0.934$}$ | 2${}^{$0.269$}$ | 4${}^{$0.836$}$ | 9${}^{$0.906$}$ |
| GPT-4o[2024-08-06] | 1${}^{$0.969$}$ | 1${}^{$0.950$}$ | 1${}^{$0.990$}$ | 1${}^{$0.979$}$ | 5${}^{$0.995$}$ | 1${}^{$0.995$}$ | 5${}^{$0.934$}$ | 7${}^{$0.161$}$ | 7${}^{$0.808$}$ | 6${}^{$0.947$}$ |
| Spearman’s Rank $\rho$ | – | 0.98^1e-9 | 0.98^2e-9 | 0.97^2e-9 | 0.37^2e-1 | 0.86^10e-5 | 0.52^6e-2 | 0.69^6e-3 | 0.74^2e-3 | 0.06^8e-1 |
| Kendall Tau $\tau$ | – | 0.91^5e-8 | 0.91^5e-8 | 0.91^5e-8 | 0.19^4e-1 | 0.76^4e-5 | 0.34^1e-1 | 0.52^10e-3 | 0.58^3e-3 | 0.01^1e+0 |

## Appendix B VNLI-Critique Model Development Details

This section provides further details on the architecture, fine-tuning process, and computational resources utilized for the development of our VNLI-Critique model, as introduced in [Section 4](https://arxiv.org/html/2506.07631v1#S4) of the main paper.

### B.1 Model Architecture

VNLI-Critique is developed by fine-tuning the PaliGemma 10B architecture [34]. This architecture integrates a Gemma2-9B Large Language Model (LLM) [29] as its textual backbone and a SigLIP model [48] as its visual encoder. For visual processing, input images are standardized to a resolution of $448px^{2}$ pixels. At this resolution, the SigLIP visual encoder processes each image into a sequence of 1024 visual tokens, which are subsequently fed into the LLM component for multimodal understanding and generation tasks.

### B.2 Fine-tuning Procedure

We performed full fine-tuning of the PaliGemma 10B model to develop VNLI-Critique. The fine-tuning process was conducted for 5 epochs. A batch size of 128 was used, with a dropout rate of 0.1 applied to aid regularization. No weight decay was utilized during training. The Adam optimizer [19] was employed with its default hyperparameters, and a constant learning rate of $1\times 10^{-6}$ was maintained throughout the fine-tuning process.

### B.3 Computational Resources

The training of the VNLI-Critique model was executed on Google Cloud TPUv5e [12] accelerators. Specifically, a configuration of 128 TPUv5e chips was utilized for the fine-tuning task. The total training time for the 5 epochs was approximately 1 hour and 30 minutes. Based on an estimated cost of $1.20 per chip-hour, the total computational cost for training VNLI-Critique was approximately $230.40.

## Appendix C Human Annotation Details

The creation of the DOCCI-Critique benchmark and the evaluation of our models’ outputs, including critique generation and the Critic-and-Revise pipeline, relied on comprehensive human annotations. We engaged third-party human annotators sourced through Prolific(^2^22[https://www.prolific.com/](https://www.prolific.com/)). Each data entry subject to human evaluation, whether for sentence-level factuality in DOCCI-Critique or for the quality assessment of generated critiques, was independently assessed by five different annotators. This multi-annotator approach helps ensure robustness and mitigate individual biases in the collected judgments. Annotators were compensated at a rate of $20 per hour for their work.

The following subsections provide an illustrative overview of the annotation interfaces designed for the two primary human evaluation tasks: assessing the factuality of VLM-generated description sentences (Section [C.1](https://arxiv.org/html/2506.07631v1#A3.SS1)) and evaluating the quality of generated critiques (Section [C.2](https://arxiv.org/html/2506.07631v1#A3.SS2)).

### C.1 Description Sentences Annotation Interface

For the task of annotating sentence-level factuality within VLM-generated paragraph descriptions (as detailed in Section 3 for the DOCCI-Critique benchmark), annotators were presented with an interface displaying the source image, the full paragraph context, and the specific sentence under evaluation. Figure [3](https://arxiv.org/html/2506.07631v1#A3.F3) illustrates a representative example of this annotation interface. Annotators were asked to judge whether the sentence accurately described the image content, providing labels such as ’Entailment’, ’Neutral’, or ’Contradiction’, and to supply textual rationales for any non-entailed judgments.

Figure: Figure 3: Example of the Description Sentences Annotation Interface. Annotators are shown the image, the full VLM-generated paragraph, and a highlighted sentence. They assess its factuality by selecting a label (here, ‘Contradiction’) and providing a textual explanation for any inaccuracies observed.
Refer to caption: x30.png

### C.2 Critique Annotation Interface

To evaluate the quality of critiques generated by VNLI-Critique and other baseline models (as described in Section 4.3), a different interface was employed. This interface presented human annotators with the original image, the factually incorrect sentence that was critiqued, and the critique generated by the model under evaluation. Figure [4](https://arxiv.org/html/2506.07631v1#A3.F4) shows an example of this interface. Annotators were tasked with judging whether the provided critique accurately and relevantly identified the factual error(s) present in the original sentence when compared against the visual evidence in the image.

Figure: Figure 4: Example of the Critique Annotation Interface. Annotators assess if the ’Predicted Explanation’ correctly identifies the error in the VLM’s highlighted sentence relative to the image.
Refer to caption: x31.png

## Appendix D Qualitative Examples

To further illustrate the core components and outputs of our work, this section provides additional qualitative examples, complementing the discussions and aggregated results presented in the main paper.

Table [10](https://arxiv.org/html/2506.07631v1#A4.T10) showcases another detailed entry from the DOCCI-Critique benchmark. This example highlights the fine-grained nature of our sentence-level annotations, including the multi-rater judgments on whether a sentence makes a claim about the image, its factual correctness against the visual evidence, and the diverse human-written rationales provided by annotators for any identified inaccuracies. Such examples underscore the richness of the benchmark for evaluating nuanced understanding and error analysis.

Furthermore, Table [11](https://arxiv.org/html/2506.07631v1#A4.T11) provides a step-by-step walkthrough of our Critic-and-Revise pipeline operating on an image description sourced from the PixelProse [32] dataset. The example demonstrates: (1) the original VLM-generated description containing factual errors, (2) the specific unfactual sentences detected by VNLI-Critique, (3) the corresponding critiques generated by VNLI-Critique, (4) the individual sentence revisions made by the LLM based on these critiques, and (5) the final, more factually accurate revised description. This illustrates the practical application of our pipeline in automatically correcting errors in detailed image captions.

**Table 10: Additional DOCCI-Critique benchmark annotation example (5 raters per assessment). Details sentence-level claims, factuality, and diverse human rationales for errors, showing varied perspectives.**
| Image |  |  |  |
| --- | --- | --- | --- |
| Description<br>Sentence | “… Looking closely, we can see eight flamingos lined up. …” | “… They are standing in a body of water, their reflection is seen in the water, and there are trees in the background. …” | “… Flamingos primarily eat brine shrimp, blue-green algae, small insects, mollusks, and crustaceans …” |
| Does the sentence include a claim about the image? (Answers from 5 raters) | \scalerel<br>*○, \scalerel*○, \scalerel*○, \scalerel*○, \scalerel*○ | \scalerel<br>*○, \scalerel*○, \scalerel*○, \scalerel*○, \scalerel*○ | \scalerel<br>*○, \scalerel*○, \scalerel*○, \scalerel*○, \scalerel*○ |
| Is the sentence factual? (Answers from 5 raters) | \scalerel<br>*○, \scalerel*○, \scalerel*○, \scalerel*○, \scalerel*○ | \scalerel<br>*○, \scalerel*○, \scalerel*○, \scalerel*○, \scalerel*○ | \scalerel<br>*○, \scalerel*○, \scalerel*○, \scalerel*○, \scalerel*○ |
| Rationales | •<br>The count of eight flamingos is incorrect; I can see at least ten.<br>•<br>There are 11 flamingos in the image.<br>•<br>Incorrect number of flamingos stated, there appear to be more.<br>•<br>I see eleven flamingos, not eight.<br>•<br>Eleven flamingos are lined up, not eight. | •<br>I don’t see any prominent trees in the background, mostly just distant, blurry foliage or land.<br>•<br>The background appears to be more of a distant shoreline or low vegetation, not distinct trees. | - |

**Table 11: Table 11: Step-by-step illustration of the Critic-and-Revise pipeline in action on a sample from the PixelProse dataset. The ‘Original Description’ contains several inaccuracies. ‘Detected Unfactual Sentences by VNLI-Critique’ highlights these errors (e.g., regarding hand position, light source, text location). ‘Predicted Critiques by VNLI-Critique’ provides explanations for these errors. The ‘Critic-and-Revise output’ shows the LLM’s corrected individual sentences, guided by the critiques. Finally, the ’Revised Description’ integrates these corrections into a more factually grounded paragraph.**
| Image |  |  |  |
| --- | --- | --- | --- |
| Original Description | A young man with short brown hair and dark brown eyes.<br>He is wearing a black jacket and a white shirt.<br>He has a serious expression on his face.<br>He is looking at the viewer with his left hand on his chin and the other holding his jacket.<br>There is a dark background with some light coming from the left side of the image.<br>There is text at the top of the image that says “The right to use my friends as a weapon, that is the sinful crown I shall adorn - Shu Ouma”.<br>The text is in a white font. The image is in an anime style. |  |  |
| Detected Unfactual Sentences by VNLI-Critique | He is looking at the viewer with his left hand on his chin and the other holding his jacket. | There is a dark background with some light coming from the left side of the image. | There is text at the top of the image that says “The right to use my friends as a weapon, that is the sinful crown I shall adorn - Shu Ouma”. |
| Predicted Critiques by VNLI-Critique | He is looking at the viewer but his hands are not visible | The light is coming from the right side of the image, not the left. | The text is at the bottom of the image and not the top. |
| Critic-and-Revise output | He is looking at the viewer. | There is a dark background with some light coming from the right side of the image. | There is text at the bottom of the image that says “The right to use my friends as a weapon, that is the sinful crown I shall adorn - Shu Ouma”. |
| Revised Description | A young man with short brown hair and dark brown eyes.<br>He is wearing a black jacket and a white shirt.<br>He has a serious expression on his face.<br>He is looking at the viewer.<br>There is a dark background with some light coming from the right side of the image.<br>There is text at the bottom of the image that says “The right to use my friends as a weapon, that is the sinful crown I shall adorn - Shu Ouma”.<br>The text is in a white font. The image is in an anime style. |  |  |

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 4
questions: I would appreciate the authors’ clarification on the points raised in the Weaknesses section.
In addition, I would like to ask the following specific questions and would appreciate detailed responses:

- In the sentence-level annotations, sentences judged as factually correct do not appear to include any additional textual human rationales. Would providing further critiques or enhancement suggestions for such sentences—aimed at improving them toward ground-truth human caption quality—offer benefits in terms of expressiveness or linguistic diversity?

- The authors fine-tuned PaliGemma-2 for developing the VNLI-Critique model. How would the performance differ if other detailed-captioning models (e.g., Qwen2.5-VL-7B, as reported in Table 4) were fine-tuned under the same framework? In other words, is PaliGemma-2 empirically the optimal architecture for this task, or is the proposed approach model-agnostic and transferable to other multimodal backbones?

- Could the authors elaborate on the prompt design used for the revision LLM in the Critic-and-Revise pipeline? Understanding the structure and rationale of this prompt would clarify how the critique outputs are transformed into factual sentence revisions.
soundness: 3
strengths: The paper is clearly written and logically structured, with a well-motivated problem statement and a thorough discussion of limitations in prior research. The proposed benchmark, model, and pipeline are novel and supported by strong empirical evidence.

- Benchmark Design (DOCCI-Critique):
The authors present DOCCI-Critique, a carefully curated benchmark consisting of 1,400 captions generated by diverse VLMs and 10,216 sentence-level human annotations. Unlike prior VLM factuality benchmarks that focus on short or isolated descriptions, DOCCI-Critique provides context-aware, sentence-level factuality annotations for paragraph-length image descriptions. A key differentiator is the inclusion of fully human-written explanatory rationales for factual errors, enabling in-depth analyses of VLM behavior and offering a grounded reference for alignment with human factuality judgments.

- Automated Evaluation Model (VNLI-Critique):
The proposed VNLI-Critique model performs both sentence-level factuality classification and explanatory critique generation, effectively identifying discrepancies and articulating their causes. Despite being a 10B-parameter open-source model, it achieves performance comparable to large-scale commercial models across multiple benchmarks. Moreover, the model demonstrates strong generalization beyond the in-domain DOCCI dataset, achieving competitive results on external and unseen datasets such as M-HalDetect and CHOCOLATE.

- Critic-and-Revise Pipeline:
Building upon VNLI-Critique, the authors design a Critic-and-Revise pipeline that integrates critique-driven sentence correction via a revision LLM. This framework establishes a new protocol for fine-grained, interpretable evaluation and correction of VLM-generated captions. Empirical results show that the pipeline substantially improves the factual accuracy of captions, validating its effectiveness as both an evaluation and correction methodology.
summary: Previous approaches to evaluating the quality of generated image captions have largely relied on model-based alignment metrics—such as CLIPScore—which primarily assess overall image–text similarity rather than fine-grained factual accuracy. As a result, these methods struggle to identify which specific sentences or details in a caption are factually incorrect.
This paper introduces a novel benchmark (DOCCI-Critique), a sentence-level factuality model (VNLI-Critique), and an integrated Critic-and-Revise pipeline. Together, these contributions enable explainable and fine-grained evaluation of detailed image captions and further foster improvements in Vision-Language Model (VLM) image understanding and factual accuracy.
weaknesses: While the paper is overall well-executed, several aspects could be strengthened to enhance completeness and reproducibility.

- Dataset Scale and Annotation Cost:
As acknowledged by the authors, the limited sample size of DOCCI-Critique constrains its statistical robustness. The benchmark contains 10,216 sentence-level annotations, each requiring multi-rater validation and textual rationales. Given that generating these critiques demands human intervention per caption (as illustrated in Table 1 and Appendix C), the data collection pipeline incurs substantial human labor costs. It would be valuable for the authors to discuss the feasibility of automating rationale extraction—for instance, leveraging reliable MLLMs trained on similar factuality explanation tasks—and to clarify why such an approach may not yet yield satisfactory results. A quantitative or qualitative comparison between human-written and MLLM-generated rationales could make this limitation more convincing.

- Performance Gap on CHOCOLATE Dataset:
The paper shows that VNLI-Critique achieves SOTA results on M-HalDetect but lower performance on CHOCOLATE. Since CHOCOLATE mainly consists of chart- and plot-based visual reasoning tasks, this result suggests that the proposed critique model—optimized for naturalistic, object-centric imagery—may have limited transferability to abstract or structured visual domains. The paper would benefit from a deeper analysis of this discrepancy, explaining which aspects of the critique mechanism fail to generalize.

- Documentation of External Datasets:
The paper briefly mentions external benchmarks such as M-HalDetect and CHOCOLATE, but provides limited details on their structure or usage. Including short summaries in the appendix—covering dataset scope and task formulation—would slightly improve clarity and reproducibility, though this is a minor issue overall.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: What specific contributions does this work make beyond the existing studies mentioned above?
soundness: 3
strengths: 1. This paper is clear and well-organized overall.
2. I'd like to thank the authors for their labor-intensive, human-involved benchmark construction.
summary: The paper proposes DOCCI-Critique, a benchmark dataset comprising 1400 VLM-generated captions for 100 images and over 10,216 sentence-level human annotations. The authors finetune a pre-trained 10B VLM on a training set generated using the same process as DOCCI-Critique. The experiments show that the finetuned model can achieve better results in measuring the precision of given image captions compared to other baseline models, such as Gemini-2.0-Flash and GPT-4o. They also show that the critic-and-revise pipeline, revisiting each sentence within an image caption and revising it using an LLM, can improve the precision of image captions by a large margin.
weaknesses: 1. Several prior studies have proposed caption revision models to enhance image caption quality [1,2]. However, the paper does not clearly articulate how its contributions go beyond these existing approaches.
2. The proposed critic-and-revise pipeline appears to be highly similar to the method introduced in a recent study [3]. A detailed comparison with this work is necessary to clarify the novelty of the proposed approach.

[1] Zhou et al., "ANALYZING AND MITIGATING OBJECT HALLUCINATION IN LARGE VISION-LANGUAGE MODELS", ICLR 2024  
[2] Lee et al., "VOLCANO: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision" NAACL 2024  
[3] Lee et al., "Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage" ICML 2025

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: - How does the critic-and-revise pipeline behave when sentence-level errors arise from global inconsistencies rather than local factual mistakes?
- Does the method risk over-editing or introducing new errors when the critic misclassifies a factual sentence as incorrect?
soundness: 3
strengths: - The work provides a richly annotated, sentence-level factuality dataset that fills a clear gap in evaluating long VLM-generated captions.
- The VNLI-Critique model offers a scalable, generalizable evaluator that performs well across multiple external benchmarks.
- The Critic-and-Revise framework delivers meaningful factuality improvements, demonstrating real downstream utility.
summary: This paper tackles the challenge of evaluating the fine-grained factual accuracy of long, paragraph-level captions produced by modern Large Vision-Language Models (VLMs), where existing benchmarks and metrics fail to detect subtle or sentence-level errors. The authors introduce DOCCI-Critique, a new benchmark containing 1,400 VLM-generated detailed captions paired with 10,216 sentence-level human annotations that include both factuality labels and rich error rationales. Building on this dataset, they develop VNLI-Critique, a model capable of sentence-level factuality classification and critique generation, showing strong generalization on external benchmarks like M-HalDetect and CHOCOLATE. Leveraging this capability, they further propose a Critic-and-Revise pipeline, where VNLI-Critique identifies and explains errors and a separate LLM revises them, significantly improving caption factuality, even on large synthetic datasets such as PixelProse and DetailCaps.
weaknesses: - Because both the critic model and the revise pipeline rely heavily on DOCCI-style annotations, there is a possibility that the system implicitly learns annotator-specific bias rather than true general fine-grained factuality.
- Sentence-level evaluation may miss cross-sentence logical dependencies or global coherence errors, potentially encouraging overly localized correction strategies that do not improve holistic caption quality.
- Lack of necessary case studies and error analysis to intuitively illustrate the success and failure modes of the proposed work.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: See above weakness
soundness: 3
strengths: - The primary strength of this work is the DOCCI-Critique benchmark, which collects detailed captions from 14 diverse VLMs and gathers textual rationales for all identified errors.
- VNLI-Critique shows strong model performance and demonstrates generalizability to other benchmarks.
- The paper is well-written, and the problem is clearly motivated.
summary: This paper addresses the challenge of evaluating factual accuracy in detailed, paragraph-length image captions generated by large vision-language models (VLMs). The authors introduce DOCCI-Critique, a benchmark dataset built on 100 images from the DOCCI dataset, comprising 1,400 captions from 14 VLMs, with over 10,216 sentence-level human annotations for factuality, including explanatory rationales for errors. They then develop VNLI-Critique, a fine-tuned PaliGemma-2 model (10B parameters) that performs sentence-level factuality classification and generates critiques explaining inaccuracies, incorporating paragraph context to resolve ambiguities. The paper highlights three applications: (1) generalization to external benchmarks like M-HalDetect and CHOCOLATE, where VNLI-Critique achieves strong performance (e.g., 0.76 Macro-F1 on M-HalDetect); (2) an AutoRater for ranking VLMs on DOCCI-Critique, showing high correlation with human judgments (e.g., 0.98 Spearman); and (3) a Critic-and-Revise pipeline where VNLI-Critique's critiques guide an LLM (e.g., GPT-4o) to correct erroneous sentences, yielding significant factuality improvements (e.g., 46% on DetailCaps-4870). Overall, the work aims to advance fine-grained evaluation and correction of VLM hallucinations in detailed captions.
weaknesses: - A key weakness is the novelty of the "Critic-and-Revise" pipeline. While the performance is strong, the idea of using a model to critique and then revise its own or another model's output is a well-established paradigm in recent LLM/VLM literature (often termed self-correction and self-improvement). The paper should compare or at least state the difference between such work [1][2][3].
- The proposed benchmark is relatively small with only 100 unique images. The size may not cover diverse scenarios.
- The proprietary models evaluated are not the state-of-the-art models (gpt-4o, gemini-2.0). It would be better to evaluate on the stronger models like gemini-2.5-pro or gpt-5 to better contextualize the results.
- The author can elaborate more on the proposed method. The paper states that VNLI-Critique was not trained on DOCCI-Critique, but on a separate, specialized dataset generated from 70+ PaliGemma-2 variants. However, the details of this training set are not as clearly presented in the paper as the details for DOCCI-Critique. 


[1] Wu et al., 2025. VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning.

[2] Yu et al., 2024. Attention prompting on image for large vision-language models.

[3] Prabhakaran et al., 2025. VADE: Visual Attention Guided Hallucination Detection and Elimination


Decision: Reject