Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage
Authors: Saehyung Lee
Seunghyun Yoon
Trung Bui
Jing Shi
Sungroh Yoon
Venue: ICML 2025 poster (2025)
Keywords: Image captioning
Caption evaluation
Multimodal large language models
Large language models
Primary area: applications->computer_vision
TL;DR: We propose a multiagent LLM-MLLM method to correct hallucinated captions and introduce new evaluation metrics that better align with human judgments in detailed image captioning.

Abstract: Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations. Our analysis reveals that existing hallucination detection methods struggle with detailed captions. We attribute this to the increasing reliance of MLLMs on their generated text, rather than the input image, as the sequence length grows. To address this issue, we propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions. Additionally, we introduce an evaluation framework and a benchmark dataset to facilitate the systematic analysis of detailed captions. Our experiments demonstrate that the proposed evaluation method aligns better with human judgments of factuality than existing metrics. Moreover, we show that current approaches for enhancing MLLM factuality often fail in hyper-detailed image captioning tasks. In contrast, our approach significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V. Finally, we highlight a limitation of VQA-centric benchmarking by demonstrating that an MLLM's performance on VQA benchmarks may not correlate with its ability to generate detailed image captions.

Numerous image captioning methods utilizing deep neural networks (DNNs) have been proposed (Vinyals et al., 2015; Xu et al., 2015). However, they are generally limited to Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage
Saehyung Lee 1*
Seunghyun Yoon 2 Trung Bui? Jing Shi? Sungroh Yoon 13
Abstract
generating short captions, which constrains their broader application in real-world scenarios. For instance, in cases such as assistance for visually impaired individuals, where it necessary to provide highly detailed descriptions the scene in front of the user, these methods may not be suitable.
Multimodal large language models (MLLMs) ex- cel generating highly detailed captions but often produce hallucinations. Our analysis reveals that existing hallucination detection methods struggle with detailed captions. We attribute this to the in- creasing reliance of MLLMs on their generated text, rather than the input image, as the sequence length grows. To address this issue, we propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions. Addition- ally, we introduce an evaluation framework and a benchmark dataset to facilitate the systematic analysis of detailed captions. Our experiments demonstrate that our proposed evaluation method better aligns with human judgments of factuality than existing metrics and that existing approaches to improve the MILM factuality may fall short in hyper-detailed image captioning tasks. In con- trast, our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V. Finally, we highlight a limitation of VQA-centric benchmarking by demonstrating that an MILM's performance on VQA benchmarks may not correlate with its abil- ity generate detailed image captions. Our code and data are available at https: //git thub. com/ haw.obbbeereeeearrcaaaaaaaa
Following the recent success of large language models (LLMs) (Brown et al., 2020), there have been attempts to use text and information from other modalities as input to LLMs. Notably, many studies have explored multimodal large language models (MILMs) that incorporate visual information (Li et al., 2023a; Dai et al., 2023; Liu et al., 2024b). These models have demonstrated significantly su- perior performance compared to traditional models in tasks such as visual question answering (VQA) and captioning (Liu et al., 2024a). In particular, MLLMs, leveraging the ad- vanced language capabilities of L.M are able to generate much longer and more detailed captions than conventional captioning models. However, these generated captions fre- quently contain inaccurate information, including descrip- tions of objects that are not present in the input image (Leng et al., 2024). Such hallucination problems hinder the practi- cal application of MLLMs in real-world settings.
Three major approaches have been recently proposed to im- prove the factuality of MLLMs: (i) Decoding- methods (Leng et al., 2024) reduce the probabilities of hallucination- related tokens during the model's decoding process without requiring additional training; (ii) Training-base methods (Liiu al., 2023a) further train the models on curated mul- timodal datasets to ensure they generate only accurate re- sponses; (iii) Corrector- methods (Zhou et al., 2024) employ a corrector model that detects and either removes or revises hallucinations present in the model's 's responses.
1. Introduction
In this paper, we propose Caption factuality enhaneing MultiAgent System (CapMAS), a multiagent approach to correct hyper-detailed image captions. Unlike existing corrector-based approaches that require training a correc- tor (Lee et al., 2024), CapMAS improves the factuality of detailed image captions by leveraging the collaboration be- tween an LLM and MLLM, without the need for additional training. Moreover, unlike methods that target specific types of hallucinations (Li al., 2023b: Zhou et al., 2024), our ap- proach does not pre-define the hallucination types, allowing
'Wrk done during internship at Adobe Research. 'Department of Electrical and Computer Engineering, Seoul National University Adobbe Research Interdisciplinary Program in Artificial Intelli- gence, Scoul National University. Correspondence Sungroh Yoon <sryoon @sn>.ac.rr>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage
to address a broader range of issues. The method proceeds as follows: (i) an LLM decomposes a given detailed caption into atomic propositions; (ii) an MLLM verifies the truthful- ness of each atomic proposition based on the image; and (iii) the L.M revises the caption accordingly. Our design is par- ticularly motivated by the observation that, as the length of a model's response increases, hallucinations generated later in the sequence become more difficult for existing methods (Wang et al., 2023; Zhou et al., 2024) to detect.
Dai al., 2023) align the frozen vision encoder and LLM using lightweight transformer (Vaswani, 2017) called Q- Former. The trainable input tokens of the Q-Former interact with the output tokens from the vision encoder through cross-attention, transforming them into input tokens for the LLM. The LLaVA models (Liu et al., 2024b;a) use a sim- ple MLP connector to align the vision encoder with the LLM. All output tokens from the vision encoder, passed through the MLP connector, used input the LLM. The vision encoder's parameters remain fixed during the training of the MIP connector and the LLM. Unlike exist- ing MLLMs, the InternVL models (Chen et al., 2024c;b) have demonstrated the effectiveness of increasing the size of both the vision encoder and the lision-language connector. They utilize a 6-billion parameter vision encoder and an 8-billion parameter vision-language connector. The connec- tor is obtained by fine-tuning the pre-trained multilingual LLaMA (Cui et al., 2023). Despite the many advancements in open-source MLLMs, closed-source MLLMs such as GPT-4V or GPT-4o' still outperform them significantly. As a result, these GPT models represent the upper bound perfor- mance benchmarks and are commonly used to evaluate MLLMs (Petryk et al., 2024). In our work, we demonstrate that captions generated by GPT-4V can be improved using our method, and we use GPT-4o to evaluate captions.
Evaluating the factuality of detailed captions is not straight- forward. Through experiments, we demonstrate that conven- tional caption evaluation metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004). METEOR (Banerjee & Lavie, 2005), and CIDEr (Vedantam et al., 2015), as well as recently proposed methods (Hessel et al., 2021; Petryk et al., 2024), fail to accurately assess the factuality of de- tailed captions. To address this issue, we propose GPT- based method for factuality evaluation and validate its effectiveness through experiments that include human evaluations. Even if a caption contains factual information, however, it may still be considered inadequate if it does not sufficiently capture the visual information. To measure the coverage of captions, we construct a detailed VQA dataset through collaboration between humans and AI agent (Achiam et al., 2023). If a caption fully encap- sulates the information of a given image, questions about the image should be answerable accurately using only the caption, without referencing the image itself.
MLLM hallucinations and mitigation strategies. MLLMs sometimes generate inaccurate responses. For example, they may incorrectly describe the characteristics of objects in an input image, misrepresent relationships between objects, or even describe objects that do not exist. To mitigate these hallucination problems, decoding- methods apply penalties to the probabilities of tokens that are likely to be hallucinations during the decoding process. For instance, VCD (Leng et al., 2024) induces hallucinations using corrupted images, while OPERA (Huang et al., 2024) leverages the correlation between high attention weights assigned to few summary tokens and hallucinations. Training- exploring training data that can suppress the generation of hallucinations. Liu et al. (2023a) demonstrated that hallucinations can be alleviated by incorporating negative samples descrippionsssss that explicitly state the absence of certain objects in a given image-into visual instruction tuning datasets. Corrector-based methods (Zhou et al., 2024; Lee et al., 2024) detect, remove, and revise hallucinations present in MLLM responses by using a corrector model. This model is obtained by supervised fine-tuning pre-trained MLLM. The corrector model then revises the initial response based on the given image.
Our experiments surprisingly reveal that methods designed to improve the factuality of MI.MMs, which have proven effective in tasks like VQA (Huang et al., 2024), may be ineffective for hyper-detailed image captioning tasks that require longer responses. In contrast, CapMAS significantly enhances the factuality of captions and can be applied in a plug-and-play manner to any captioning model; notably, this improvement extends to captions generated by the state-of- the-art closed model, GPT-4V (Achiam et al., 2023). Finally, we highlight an issue with the current VQA-centric bench- marking (Duan et al., 2024) by showing that an MILM's performance on VQA benchmarks may not correlate with its ability to generate detailed image captions.
methods focus on
2. Related Work
Multimodal large language models. LLMs that process inputs from multiple modalities, including text and other types of data, are referred to as multimodal LLMs (Yin et al., 2023a). Among these, LLMs that handle visual input have been the most actively researched, and the MILMMs discussed in this paper are focused on this cate- gory. Research on these models primarily explores methods for fusing the output of an independent vision encoder into the input of an LLM. The BLIP models (Li et al., 2023a;
'nttps://opena wttp:/.oooooooiieeeeeee co-g:////////////eoooooot--
2

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
"Parse the given caption and list all objects that object detector could identify in the inage."
"Describe the given inage in a very detailed manner,"
'The inage presents captivating scene of circular design etched onto e gray stone surface. The design is a harmonious blend of two distinct elenents: serpent and flower. The serpent,
"serpent", "flower", "gray stone"
"serpent", "flover", "gray stone"
<input image>
<LLaV--NeXT>>
<GPT-4>
<Humanso
Figure 1: The process generating data sample for evaluating hallucination detection methods in detailed image captioning tasks. Human annotators identify and label object hallucinations within the caption generated by LLaVA-NeXT (Liu et al., 2024a) for an image.
Caption evaluation methods. Since short image captions relatively easy to obtain reference captions for, we can use sauching-based caption evaluation methods (Hossain et al., 2019) to assess them. However, for long and detailed captions generated by MLLMs, the number of reference captions required for such evaluations becomes exceedingly large. Thus, it becomes impractical to evaluate detailed cap- tioning using traditional approaches. Hessel et al. (2021) proposed CLIPScore, reference-free evaluation method. CLIPScore measures the distance between image and its caption within the joint representation space of CLIP (Radford et al., 2021). Additionally, the authors introduced RefCLIPScore, which uses both the image and reference captions within that same representation space. Chan et al. (2023) addressed the limitations of matching- based methods by utilizing an LLM. The L.LM-based metric they proposed, CLAIR, assigns scores to captions based on reference cap- tions using LLM. Similarly, ALOHa (Petryk et al., 2024) detects object hallucinations by comparing generated cap- tions to reference captions using an LLM.
23
30-
03
80
(a) Confidence- Tokken Index
(b) Consistency-Toker Index
Figure 2: The hallucination scores of the Confidence and Consis- tency methods based on object positions within detailed captions. Object hallucinations near the end of the captions (192+) are unde- tectable by both methods.
follows: (i) We prompt an MILM with "Describe the given image in a very detailed manner." and collect the model's responses for a specified image set; (ii) For the convenience of our analysis, we use an LLM to identify objects that may be hallucinations; (iii) Human annotators then label each parsed object as either a hallucination or not, based on the corresponding image. We use LLeXA-NeXT (Liu et al., 2024a) and GPT-4 as the MILM and LLM, respectively. Figure illustrates the process of constructing the dataset. To build the dataset, we use subset of IIW-400 (Garg et al., 2024). We detect object hallucinations using two of the most widely adopted hallucination detection methods:
3. Method
In this paper, we propose multiagent- based caption correc- tion method. Corrector-based methods typically detect and remove hallucinations within model responses. Unlike exist- ing approaches, which require the corrector model training, our method employs collaboration between an MLLM and L.M. Moreover, in contrast to previous methods that are limited correcting specific types of hallucinations (Zhou
1. Confidence (Zhang et al., 2023; Zhou et al., 2024): This method detects hallucinations using the predicted proba- bility Pobj for the object token during LLa VA-NeTT gen- eration. For multi-tokes objects, the token probabilities are multiplied. The hallucination score Hobj log Pobj- increases with the likelihood of hallucination.
al., 2024), our approach is free from such constraints. We also propose a framework for evaluating the detailed image captioning capabilities of an MILM. Unlike existing meth- ods, our proposed evaluation approach allows for assessing image captioning models in terms of both factuality and coverage, evaluating each of these aspects separately.
2. Consistency (Wang et al., 2023; Zhao et al., 2024): This method assumes that hallucinations are sensitive to de- coding randomness. Using stochastic decoding, we have LLaVA-NeXT generate 40 captions per image and count the occurrence lokj of each object in the dataset of Figure 1. The hallucination score is Habj = - llog faj 40
3.1. Motivating Observations
Here, we examine the performance of existing hallucination detection methods on tasks that require generating long responses. To facilitate these analyses, we construct a dataset
Figure 2 presents the hallucination scores of each method by the position of objects appearing within detailed cap-
3

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
Table Performance comparison of hallucination detection meth- ods for the dataset of Figure 1.
Comparison with existing studies. Actually, the concept of decomposing text into smaller units and assessing the factuality of each has been introduced in prior studies (Min et al., 2023; Jing et al., 2024). Here, we summarize the key differences between our study and previous research:
Method
AUROCT
F
Confidence
57.5
95.1
Consistency
73.5
75.6
1. Unlike existing studies, which do not rigorously justify the need for decomposition, we empirically demonstrate the motivation behind our approach (Section 3.1).
Object Detector
61.5
95.7
Isolation 81.4
71.7
2. While previous studies focused solely on proposing eval- uation metrics, our research advances further by introduc- ing system that leverages the decomposition process to generate improved image captions (Section 3.2).
3. Existing evaluation metrics assess factuality using uni- modal data (either text or image). In contrast, our pro- posed evaluation metric utilizes multimodal data (both text and image) for factuality assessment (Section 3.3). 4. Previous studies focus solely on measuring the factual- ity of generated text. In contrast, our study proposes a method that assesses both the factuality and coverage of any given image caption (Section 3.3).
tions. The horizontal axis of the graphs represents bins of object token indices, with larger token indices indicating positions closer to the end of the caption. The vertical axis represents the mean and standard deviation of the hallucina- tion scores within each bin. Note that Figure 2a reflects the positions and hallucination scores during greedy decoding, while Figure 2b is derived from the average positions and hallucination scores across 40 stochastic decoding iterations. Figure 2 demonstrates that hallucinations generated after the 192nd token are undetectable by the Confidence and Con- sistency methods. Based on these results, we can infer that existing hallucination detection methods may be ineffective in detecting hallucinations in long detailed captions.
5. We demonstrate that our metric correlates more strongly with human evaluations than existing metries and is ro- bust against their critical limitations (Section 4.2).
3.2. Caption Factuality Enhancing MultiAgent System
Our hypothesis regarding these results is that as MLLM out- puts become longer, they become more strongly grounded in the text they generate rather than the given image. In fact, our hypothesis is supported by several recent stud- ies. For example, Liu et al. (2024c) demonstrated that as MLLM responses lengthen, the attention weights assigned to image tokens decrease, and Zhong et al. (2024) showed that MILM responses are significantly influenced by prior dialogue. Based on this hypothesis, we test method for determining whether each object is a hallucination by dis- connecting it from its context (Isolation). The Isolation method involves querying the LLaVA-NeXT model with parsed objects using the prompt template, "Is there a O in the photo?" along with the image. When the probabil- ity of the "Yes" token for the object query is PYosjobj- the hallucination score is defined as Hoj - log PYeslubj- We compare the object ballucination detection performance of the Isolation method with that of the Confidence method, the Consistency method, and a method based on an object detector (Object Detector) introduced in recent studies ( (Yin et al., 2023c; Ge et al., 2024). We measure their detection performance on the dataset of Figure 1 using Area Under the Receiver Operating Characteristic (AUROC) and False Positive Rate at 95% true positive rate (FPR95). Table demonstrates that the Isolation method outperforms the oth- ers. This suggests that breaking long caption into smaller units and examining each individually can help detect hallu- cinations in detailed captions.
To address various types of hallucinations comprehensively, we first decompose each detailed caption into atomic propo- sitions using L.M.. An atomic proposition is claim or statement that must either be true false. For example, the caption "A house has a red roof and chimney" is broken down into "A house has a red roof" and "A house has a chim- ney." We use an LLM to perform this process, but we allow flexibility in cases where the results do not strictly conform to the definition of an atomic proposition. We then inves- tigate the truth of each decomposed unit using an MILM. Each unit is converted into a True/False question and inde- pendently fed to the MLLM. The hallucination score H (u) for the unit u is defined as follows:
-log (min (p (T|z, Q(w)) - -p (F(z, Q(4), (I)
p (T) and p (F) represent the MLLM's token probabilities for the "Truc" and "False" tokens, respectively. x and e denote the input image and a very small constant near zero. Q(-) is a function that converts the input text into True/False question, which we implement by prepending "True or False?" the input. Each unit is included in either the True set T or the False set F, based on its hallucination score. To achieve this, we introduce a hyperparameter such that T = (u) < ]} and FF = {u) > T). Finally, the initial caption, along with the corresponding sets T and F, is provided to an LLM, which corrects the initial caption to ensure it contains only factual information.

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
"Decompose given caption into atomic propositions."
<image>
"The inage captures scene of a sidewalk, where main focus rectangular sign. The sign, painted in striking shade of green, stands against the gray concrete of the sidewalk. The sign ..."
("The image captures sidewalk.", ", "The main focus is rectangular sign.", "The sign painted in striking shade of green. "The sign stands out against the gray concrete of the sidewalk.", ...]
Prepend "Tuee or False?"
<Decomposer LLM>
Fact-cheeker MLLM>
"Based on given propositions and their corresponding True/False, correct the caption."
[e.11, 1. 1.41, 8.79, 8.23, ...] <Hallucination scores>
<initial caption
thresholding
[True, False, False, True, ...]
cCorrector LLM>
"The inage captures scene sidewalk. sign stands against the gray concrete the sidewalk. sign carries message that reads \"Please Walk Bikes\", a clear directive for cyclists
<Corrected caption
Figure 3: Overview of CapMAS. The decomposer LLM breaks an initial caption into atomic units. These units are converted into True/False questions and fed into the MLLM along with the image, where each unit is assigned a hallucination score according to Equation (1), Units are classified True False based on the threshold v, and the corrector LLM then revises the initial caption accordingly.
We name this pipeline, which improves the factuality of detailed image captions through the collaboration of pre- trained LLM and MLLM, Caption l'actuality enhancing MultiAgent System (CapMAS). CapMAS is training-free and can be applied in a plug-and-plas manner to any cap- tioning model. Unlike existing methods that can only ad- de-
Table 2: Meta-evaluation results across various caption evaluation methods. DOCCI and its synthetic hallucinatory captions are used for the meta-evaluation. The highest-rated caption for each method is highlighted in bold. The full table is in Appendix D.
Evaluation Metric
Caption
CIDEr
CLIP-S
RefCLIP-S CLAIR ALOHa Ours
dress predefined types of hallucinations, CapMAS tect and correct all hallucinations the atomic unit level. The pipeline of CapMAS is illustrated in Figure 3.
Clean Object
6.4
81.3
86.9
36.2
62.8
4.8
81.0
75.3
85.2
31.5
52.3
Attribution
6.2
80.9
75.2
80.0
34.3
60.9
Relation
6.7
81.4
75.6
83.5
36.9
51.9
3.3. Evaluation Methods
Traditional caption evaluation methods rely on word match- ing with reference captions, suitable for short captions gen- erated by conventional models. However, MLLMs produce longer and more detailed captions, making it impractical to obtain sufficient reference captions for accurate evaluation. Given the enriched content of these image captions, rather than simply evaluating them good bad, aim assess them systematically by considering two key perspectives:
Factuality. If human were to measure the factuality of text, would be natural to decompose the text into units that be classified true or false, and then calculate the proportion of true units (Maynez et al., 2020). We adopt this approach to measure the factuality of captions, utilizing the state-of-the-art model GPT-4o. In our framework, GPT- 4o decomposes each caption into atomic propositions and determines their truthfulness based on the corresponding image and reference caption. If the number of atomic propo- sitions judged true and false T and F, respectively, the factuality of the caption is defined as 7-F T
Factuality: The degree which the content of the caption is factual and free from hallucinations.
Coverage: The extent to which the caption captures the information contained in the image.
To validate this evaluation method, we use the DOCCI dataset (Onoe et al., 2024), which contains human-annotated detailed image captions. Specifically, for each image in a subset of the dataset, we prepare the following four types of
We propose evaluation methods for detailed image captions from these two perspectives.
5

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
1. what is the main focus of the photo?
A) landscape B) television and decorations c) A group people building
2. where was the photo likely taken?
A) In a park B) Inside a house c) At a beach D) In museum
3. what is the situation depicted in the photo?
A) A fa gathering B) A decorated living space C) A work neeting D) A sporting event
4. what is in the center of the photo?
A) painting B) television C) person D) A window
47. what is on the left side of the table in the photo?
A) A Lamp B) A blue decorative tree c) A vase D) A stack of books
48. what is the texture of the table in the foreground?
A) Smooth and shiny B) Raugh and rustic C) Soft and cushioned D) Metallic and cold
49. what is in the background of the photo, to the right side?
A) A kitchen B) A Christmas tree c) A bookshelf D) window with curtains
se. what type ornanents are the triangular decoration on the table?
A) Animal figurines B) Christmas baubles Miniature houses D) Candles
Figure 4: An example of our coverage evaluation data sample. The dataset consists of multiple-choice questions with four or fewer options. As demonstrated, the dataset includes questions with varying levels of granularity, ranging from broad to highly detailed. We have an LLM solve these problems using only the provided captions.
captions (details provided in Appendix G):
Attribution hallucinations but fails to capture Relation hallu- cinations due to its algorithmic limitations. CLAIR detects and reflects all three types of hallucinations in the scores. However, CLAIR does not focus solely factuality; in- stead, it allows the GPT model to directly score each cap- tion, applying the evaluation criteria implicitly defined by the GPT model. In contrast, our metric exclusively considers the factuality of the caption. While does not assign a per- feet score to the Clean captions due to GPT-4o's limitations in image understanding, it successfully assigns the highest score to Clean among the four caption sets.
1. Clean: The original caption (e.g., An indoor view cap- tures a cat on a wooden floor, attempting to catch a large pale peacock feather flying above it).
2. Object: A description of object likely present but not in the image is added to the Clean caption (e.g., An indoor view captures cat on a wooden floor, attempting to catch a large pale peacock feather flying above it. A small red ball is rolling near the cat).
Attribution: Some object attributions in the Clean caption modified to be inconsistent with the image (e.g.. An indoor view captures cat on a metal floor, attempting to catch small dark peacock feather flying above it). 4. Relation: Some object relationships in the Clean caption are altered to conflict with the image (e.g., An indoor view captures a cat on a wooden floor, attempting to catch a large pale peacock feuther flying below it).
Coverage. An image caption with only factual informa- tion is not high-quality if it focuses solely trivial aspects of the image. To assess the coverage of captioning models, we propose a QA-based metric and a benchmark dataset. Our coverage evaluation method is based on the assump- tion that if an image caption fully captures the information in the image, visual questions about that image should be answerable by referencing the caption alone.
We evaluate the four types of captions using various metrics (BLEU, ROUGE, METEOR, CIDEr, CLIP-S, RefCLIP-S, CLAIR, and ALOHaa), including our own, determine whether the hallucinations in the three modified types are reflected in the scores. For a fair comparison, all methods requiring GPT (CLAIR, ALOHa, and ours) use GPT-4o, and all methods requiring reference captions (BLEU, ROUGE, METEOR, CIDEr CLAIR, ALOHa, and ours) use a sepa- rate set (Garg et al., 2024) of
Our goal is evaluate hyper-detailed image captions. There- fore, the visual questions for evaluation must include va- riety of detailed and nuanced questions about the images. Given the limitations of existing VQA datasets in this re- gard (Yin et al., 2023b; Li et al., 2023b; Yue et al., 2024), we construct a new VQA dataset. However, creating a new VQA dataset that includes variety of detailed questions requires substantial labor. To reduce the associated costs, we follow the process outlined below to construct our dataset:
captions.
Table 2 shows that existing metrics are unreliable for evalu- ating the factuality of detailed image captions. Specifically, CLIP can only process up to 77 tokens and operates like oag-of-words model (Yuksekgonul et al., 2023). This pre- vents CLIP-based metrics from capturing the full content of detailed image captions, particularly missing Relation hallucinations. ALOHa effectively addresses Object and
1. Generating more than 50 questions per image in the IIW- 400 dataset using GPT-4o.
2. Deduplicating the questions for each image using Sentence-BEi (Reimers & Gurevych, 2019).
6

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
3. Human labelers refine or remove ambiguous, flawed, or common-knowledge questions.
Table 3: Comparison of correlations between human preferences and automated metrics in terms of factuality.
4. Human labelers annotate the correct answers to the re- maining and revised questions.
FAITHSCORE
FACTSCORE
Ours
Spearman's 62.5
67.9
70.2
Our coverage evaluation dataset contains total of 19,899 multiple-choice questions, with each image averaging 49.8 questions. We present an example of our dataset in Figure 4. While our benchmark dataset can also be used to assess the visual understanding capabilities of MLLMs, we use it to evaluate the coverage of captioning models by having an LLM answer the questions based on the generated captions.
tions, introduce stylistic biases tied to the specific style, tone, or phrasing of the references, unfairly favoring or penalizing captions based on these factors. In contrast, demonstrated in Appendix B, our metric is free from such biases.
4.3. Improving Captioning Model Factuality
4. Experimental Results and Discussion
Our proposed CapMAS exhibits loose factuality-coverage trade-off depending on the hyperparameter л. Specifically, as π decreases, the threshold for determining factual propo- sitions becomes stricter, leading to more propositions being identified for correction. Consequently, factuality increases while coverage decreases (an ablation study on π is provided in Appendix C). We first investigate whether CapMAS can enhance the factuality of various MLLMS while minimizing the reduction in coverage.
4.1. Experimental Setup
We adopt LLaVA-vI.5-7B, LLaVA-NeXT-7B, LLaVA- NeXT-13B, InternV-CChat Ch..t. and GPT-4V as the mod- els for both captioning and CapMAS's s fact-checking. We use LLaMA-3-8B (AI@ Meta, 2024) or GPT-4 as the decom- poser and corrector LLMs in CapMAS. Our experiments utilize the IIW-400 and DOCCI datasets, which contain images paired with highly detailed, hallucination-free cap- tions. These high-quality reference captions enable precise evaluation of the captioning models.
Table 4 demonstrates that CapMAS can significantly en- hance the factuality of all tested MLLMs while mini- mizing coverage loss. The substantial improvement in fac- tuality, compared to the relatively minor coverage loss in the captioning models, is also reflected in the increased CLAIR scores. Using a more advanced LLM in CapMAS does not necessarily result in greater performance gains, When applying CapMAS to the LLaVA and Intern mod- els, there is minimal difference between the results obtained with LLaMA-3-8B and those with GPT-4. This suggests that the LLM's role in CapMAS is relatively straightfor- ward. CapMAS can improve detailed image captioning even for the state-of-the-art MLLM, GPT-4V. It can sig- nificantly enhance factuality even when used with MLLMs far less capable than GPT-4V. However, in such cases, there is a considerable loss in coverage, as many visual elements recognized by GPT-4V are identified as hallucinations by CapMAS. With Intern VL-Cha V1.5, CapMAS maintains GPT-4V's ' coverage while improving factuality. We addition- ally provide a qualitative comparison in Figure 5 between LLaVA-NeXT-7B with and without the application of Cap- MAS (referencing the first two rows of Table 4).
We employ our proposed factuality and coverage metrics, along with CLAIR, all based on GPT-4o, to evaluate the generated captions. To ensure robust evaluation, we sum- marize the captions (Ge et al., 2024) generated from five different prompts using LLaMA-3-8B. The only CapMAS hyperparameter, ग,, is tuned on a validation set of five exam- ples sampled from the DCI dataset (Urbanek et al., 2024). The prompt templates are provided in Appendix G.
4.2. Our Metric's Correlation with Human Evaluation
We obtain human evaluation data to validate the reliability of our factuality metric and compare it with existing ones. Human labelers assess which caption- MittiiinnLLLVA---. LLaVA--v.5-7BB's or InstructBL.IP's--5 more factual for each image in sub- set of the DOCCI test set (refer Appendix A for further details). Using this dataset, we compare our metric with FAITHSCORE (Jing et al., 2024) and FACTSCORE (Min et al., 2023), both of which evaluate the factuality of de- composed units: FactScore uses only the reference caption, FaithScore only the image, and our metric combines both.
4.4. Comparison with Other Methods
Table shows that our metric exhibits a stronger correla- tion with human evaluation than existing metrics. However, this is not the sole reason to adopt our proposed metric for evaluating the factuality of detailed image captions. Met- rics that rely solely on unimodal information are inherently susceptible to undesirable biases. For instance, metrics like FACTSCORE, which depend exclusively on reference cap-
Various methods have been proposed to mitigate hallucina- tions in MLLMs, and they have primarily been validated on VQA and simple captioning benchmarks. We compare CapMAS with three recent decoding based methods (VCD, OPERA, and SPARC (Jung et al., 2025)), two corrector- based methods (LURE and Volcano), and one training-
7

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
Table 4: Effectiveness of our proposed method across various captioning models. In the CapMAS column, the LLM represents the decomposer and corrector, while the MILM represents the fact-checker. Avg. denotes the average of CLAIR, Factuality, and Coverage.
CapMAS
Metric
Captioner
LLM
MILM
CLAIR
Factuality
Coverage
Avg.
68.8
59.9
47.9
58.9
LLaVA-NeXT-7B
LLaMA-3-8B GPT-4
LLaVA-NeXT-7B
74.1
72.2
46.9
64.4
LLaVA-NeXT-7B
74.6
73.4
46.2
64.7
70.2
62.1
48.5
60.3
LLaVA-NeXT-13B
LLaMA-3-8B
LLaVA-NeXT-13B
75.5
77.9
45.8
66.4
GPT-4
LLaVA-NeXT-13B
73.4
79.3
45.1
65.9
74.9
65.5
48.2
62.9
Intern WitehnL--Cht-Vi.
LLaMA-3-8B
Intern VL-Chat---
78.2
75.9
47.3
67.1
GPT-4
VL-Chat-
77.8
75.7
47.3
66.9
82.4
77.1
53.5
71.0
LLaMA-3-8B
LLaVA-NeXT-7B
83.3
83.3
50.8
72.4
GPT-4V
LLaMA-3-8B
LLaVA-NeXT-13B
81.9
85.3
48.4
71.9
LLaMA-3-8B
IntermVL-Chat-i.55
84.6
82.1
53.5
73.4
Table 5: Performance comparison between our proposed method other methods regarding detailed image captioning, Base refers to the default image captioning v...5.....
impact in terms of factuality, but it proves to be consider- ably beneficial with respect to coverage. Volcano yields only slight improvements in LLaVA's captions. However, Cap- MAS substantially enhances the factuality of the captioning model compared to the other methods. These results suggest that methods proposed to enhance MLLM factuality should
Method
CLAIR Factuality Coverage Avg.
Base
62.1
52.8
34.3
49.7
VCD (Leng al., 2024)
59.7
44.6
39.3
47.9
be evaluated not only tasks requiring short responses, such as VQA, but also detailed image captioning tasks.
OPERA (Huang al., 2024)
59.1
53.0
34.1
48.7
LURE (Zhou et al., 2024)
57.2
51.9
27.6
45.6
Volcano (Lee et al., 2024)
63.9
53.7
37.7
51.7
LRV (Liu et al., 2023a)
39.7
29.1
37.8
35.5
4.5. Consistency Between MLLM Captioning and VQA Evaluation results
SPARC (Jung al., 2025)
64.7
50.2
44.9
53.3
CapMAS (ours)
66.3
63.4
33.1
54.3
Currently, MLLM evaluations conducted tasks that require only short responses, such as VQA tasks (Duan et al., 2024). However, to assess the potential of MLLMs in real- world applications, such as visual assistants, it is essential to evaluate their detailed image captioning abilities. The rank- ing of models used in our experiments, including LLaVA LLaVA-NeXT-133 Intern Chat V1.5, and GPT-4V, is consistent across both cap- tioning evaluation results and widely used benchmarks like MMMU (Yue et al.. 2024). However, for instance, some MLLMs may be optimized for VQA tasks that require only short responses, allowing them to rank highly on common VQA benchmarks, yet their limited image captioning abili- ties could restrict their practical use. To investigate this, we evaluate the detailed image captioning capabilities of vari- ous MLLMs and examine whether their rankings consis- tent with their rankings on widely used VQA benchmarks. We adopt InstructBLIP-7B (Dai et al., 2023), Idefics2-8B (Laurençon et al., 2024), and MiniCPM-V-2.6 (Yao et al., 2024) as additional MLLMs for the experiment.
based method (LRV) from the perspective of detailed image captioning. All methods, except for LRV and LURF, use LLaVA-v1.5-7B, while the LRV and LURE methods em- ploy the MiniGPT-4 model (Zhu et al., 2023) as provided by their respective authors. For reference, Visheer (VFC) (Ge al., 2024) is also pipeline composed of pre- trained models that revise initial captions, similar to our approach. However, the inability to reproduce VFC, as its authors have not provided the necessary resources for re- production, prevents direct comparison with our method. Nonetheless, we can infer that our method outperforms VFC because 1) VFC specifically targets object hallucinations, and 2) employs object detector (Liu al., 2023b) for hallucination detection (see Table 1).
v1.5-7B, LLaVA-NeXT-Ne
Table 5 shows that the decoding-basee based methods are ineffec- tive for detailed image captioning. Ironically, applying VCD significantly reduces the factuality of the LLaVA model while increasing coverage. SPARC does not show notable
8

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
Table Detailed image captioning and VQA performance of various MLLMs. OpenCompass (Duan al., 2024) includes MMBench v1.1 (Liu et al., 2023e), MMStar (Chen et al., 2024a), MMMU val (Yue et al., 2024). Vash (Lu et al., 2024), OCRBench (Liu et al., 2024d), A12D (Kembhavi et al., 2016), HallusionBenchh (Guan et al., 2024), and MMVet (Yu et al.. 2023). For POPE (Li et al., 2023b). we report the average F1 score across the three categories: adversarial, popular, and random. We report the sum of the perception and cognition scores for MME (Yin al., 2023b). The best results for each metric shown in bold.
Detailed Image Captioning
Visual Question Answering
Model
CLAIR
Factuality
Coverage
Avg.
OpenCompass
MME
POPE
Avg.
InstructBLIP-7B
57.2
44.4
30.3
43.9
31.1
1391.4
86.1
38.4
LLaVA-v1.5-7B
61.1
56.3
30.5
49.3
36.9
1808.4
86.1
44.6
LLaVA-NexT-7BB
63.8
58.5
42.2
54.8
44.7
1769.1
87.5
50.8
LaVA-NeXT-133
64.5
62.8
43.0
56.8
47.6
1745.6
87.8
53.1
Idefics2-8B
58.1
85.2
13.4
52.2
53.0
1847.6
86.2
57.6
Intern VL-ehat-- V1.5
72.4
67.6
46.0
62.0
61.7
2189.6
87.5
65.9
MiniCPM-V-2.6
73.1
68.9
43.6
61.9
65.2
2268.7
83.2
68.6
GPT-4V
82.4
78.6
52.6
71.2
63.5
2070.2
81.8
66.4
Table 6 presents the evaluation results of MLLMs' responses to the prompt "Describe the given image in very detailed manner" as well as the performance of these models on various VQA tasks. From these results, we observe that the performance of an MLLM on widely used benchmarks does not necessarily reflect its capabilities detailed image captioning. Specifically, Idefies2-8B ranks mid-tier among the tested models in VQA tasks but falls into the lowest- performing group in terms of detailed image captioning. Its high factuality but low coverage indicates that Idefics2-8B has been trained to provide short and concise answers; this conclusion remains unchanged even when using Idefics2-
collaboration of pre-trained MLLM and LLM. In addition, we introduce framework and benchmark dataset for evalu- ating the factuality and coverage of captioning models. Our experiments validate the proposed evaluation framework and demonstrate that CapMAS significantly improves the factuality of captioning models. We additionally present the following key observations:
Methods designed to improve MILM factuality, which have been validated primarily on VQA or short captioning tasks, may be ineffective for detailed image captioning and can even reduce the factuality of the backbone model. High performance on commonly used VQA-centric bench- marks does not necessarily indicate that the model will excel in hyper-detailed image captioning.
8B-Chatty (Laurençon et al., 2024). Despite being a rela- tively small model, MiniCPM-V-2. attracted attention by outperforming GPT-4V on benchmarks. However, our sults show that the model significantly underperforms GPT- 4V in detailed image captioning. Additionally, we find that the factuality of the captions cannot be reliably predicted from the accuracy of MLLMs on POPE (Li et al., 2023b), which was proposed evaluate object hallucinations.
These observations raise concerns about the current VQA- centric trend in MLLM evaluation. We encourage the com- munity to evaluate MLLMs and related algorithms not only on VQA tasks but also detailed image captioning tasks gain more comprehensive understanding of their potential.
Based on these experimental results, we raise concerns about the current MLLM evaluations that are centered around VQA tasks. We encourage the community to also evaluate MLLMs from the perspective of detailed image captioning in order to showcase their full potential.
Impact Statement
We provide additional results on fluency, cost, and test-time scaling in Appendix F.
This research contributes to developing more accurate and reliable image captioning systems, which are crucial for ac- cessibility technologies. The proposed multiagent approach mitigates the risks of misinformation and hallucinations in Al-generated content, enhancing the safety and trustworthi- ness of Al systems. However, image captioning models become more detailed, ethical concerns, particularly privacy- related, may emerge. Future research should address these challenges to ensure the responsible deployment of this technology.
5. Conclusion
Detailed image captioning tasks are closely linked to criti- cal applications, such as visual assistance for the impaired. Our research aims to assess and enhance the potential of MILMs in these real-world contexts. We propose CapMAS, method that improves detailed image captions through the

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
Acknowledgements
generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024c.
This work was supported by Institute of Information & com- munications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No.RS-2021- I1211343, RS-2022-11220992 Artificial Intelligence Grad- uate School Program (Seoul National University), No.RS- 2025-02263754, Cllman-CCentic Embodied AI Agents with Autonomous Decision-Making], the National Research Foundation of Korea (NRF) grant funded by the Korea gov- ernment (MSIT) (No. 2022R1 A3B 1371707,,,), a grant from Yang Young Foundation, and the BK21 FOUR program of the Education and the Research Program for Future ICT Pioncers, Scoul National University in 2025.
Cui, Y., Yang, Z., and Yao, X. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint WrXiv:230.008177, 2023.
Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li B., Fung, P., and Hoi, S. InstructBLIP: Towards general-purpose vision-language models with instruc- tion tuning. In Thirly-seventh Conference on Neural Information Processing Systems, 2023. URL https: / //openreview net forum? www.umidd-vooo wiw.ridwwoowP.......
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang. Y., Zhang, P., Wang. J., Lin, D., and Chen, K. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024. URL https:/ //cl//arra /atx/aaavvv//.////////ee//eaaav/ org/ abs /2407 11691.
References
Achiam, J., Adler, S., Agarwal. S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Garg, R., Burns, A., Ayan, B. K., Bitton, Y., Montgomery, Onoe, Y., Bunner, A., Krishna, R., Baldridge, J., and Soricut, R. Imageinwords: Unlocking hyper-detailed im- age descriptions. arXiv preprint arXiv:2405.0m 02793, 2024.
AI@Meta. Llama model card. 2024. URL https://githui httpgggttuu.oomooeeeeo poom//mtaa- 1allllaaaaai aalaaaaalaaaaa 11ama3/ blob/main Bttp/MMDDL_LCCAM md.
Ge, Y., Zeng, X., Huffman, J. S., Lin, T-Y., Liu, M.- Y., and Cui, Y. Visual fact checker: Enabling high- fidelity detailed caption generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14033-14042, 2024.
Banerjee, S. and Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Guan, T., Liu, F., Wu, X., Xian, R., Li, Z.,, Liu, X., Wang. X., Chen, L.., Huang, F., Yacoob, Y., et al. Hallusionbench: an advanced diagnostic suite for entangled language hal- lucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14375- 14385, 2024.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. arXiv preprint 10Xx:22005.1165. 2020.
Chan, D., Petryk, S., Gonzalez, J., Darrell, T., and Canny, Clair: Evaluating image captions with large language models. In Proceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing, pp. 13638 13646, 2023.
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, 2021.
Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z.,
Duan, H., Wang, J., Qiao, Y., Lin, D., al. Are we on the right way for evaluating large iision-languae models? 2024a.
Hossain, M. Z., Sohel, F., Shiratuddin, M. F., and Laga, H. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CsUR), 51()) 36, 2019.
arXiv preprint 2rXiv:2433.203:
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma. Z., et al. How far are we to gpt-4v? closing the gap to commercial mul- timodal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024b.
Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., and Yu, N. Opera: Alleviating hallu- cination in multi-modal large language models via over- trust penalty and retrospection-allocaae of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for
In Proceedings
10

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
Jing, L., Li, R., Chen, Y., and Du, X. Faith- Score: Fine- Fine-grained evaluations of hallucinations in large vision-language models.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024a. URL
In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Find- ings of the Association for Computational Linguis- tics: EMNLP 2024, pp. 5042-5063, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. /1188/3////22224.innnnnan. 290. URL https: /ac///claanttt aaclanntooooyyyy org/2024. Sea.inns-emml...9..
git..bitthuu...... rembbiio/br io/blog/
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems,
36. 2024b.
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al. Grounding dino: Marry- ing dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0544 2023b.
Jung, M., Lee, S., Kim, E.,, and Yoon, S. Visual attention never fades: Selective progressive attention recalibration for detailed image captioning in multimodal large lan- guage models. arXiv preprint (xXiv:2502.01419 2025.
Liu, S., Zheng, K., and Chen, W. Paying more attention to image: A training-free method for alleviating hallucina- tion in lylms. arXiv preprint arXiv:2407.21771 2024c.
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth a dozen images. In Computer SsiionEECVV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 235-251. Springer, 2016.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.0668)) 2023c.
Laurengon, H., Tronchon, L., Cord, M., and Sanh, V. What matters when building vision-language models? arXiv preprint arXiv:2405.00 02246, 2024.
Liu, Y., Li, Z., Huang, M., Yang. B., Yu, W., Li, C., Yin, X., lin Liu, C., Jin, L.,, and Bai, X. On the hidden mystery of ocr in large multimodal models, 2024d. URL https: / / //aaxiiv. wrw/ www.//a///////oo////////oo/.abee abs///////////// 2305 07895.
Lee, S., Park, S., Jo, Y., and Seo, M. Volcano: Mitigating multimodal hallucination through self-feedback guided revision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 391-404, 2024.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng. H., Chang, K.-W., Galley, M., and Gao, J. Math- vista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing, L.. Mitigating object hallucinations in large vision- language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13872-13882, 2024.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L.., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing 2023.
Li, Li, D., Savarese, S., and Hoi, S. 2 Bootstrapping language-image pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Systems,
Maynez, J., Narayan, S., Bohnet, B., and McDonald, R. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 1919-1919, 2020.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.R. Evaluating object hallucination in large vision- language models. arXiv preprint 205.::2305.10355 2023b.
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-., Koh, P., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Lan- guage Processing, pp. 1076-12100, Singapore, Decem- ber 2023. Association for Computational Linguistics. doi: 2..18633/1/2022 w0002..mmllpmmmmm 741. URL ht aclanthol hollnnhology.oo ogy worg/202.......... emn momll-maiin 741.
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74- 81, 2004.
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. Mitigating hallucination large mudti-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2023a.

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
Onoe, Y., Rane, S., Berger, Z., Bitton, Y., Cho, J., Garg, R., Ku, A., Parekh, Z.., Pont-Tuust,, J., Tanzer, G., al. Doeci: Descriptions of connected and contrasting images. arXiv preprint arXiv:2404. 19755, 2024.
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudi- nov, R., Zemel, R., and Bengio, Y. Show, attend and tell: Neural image caption generation with visual atten- tion. In International conference on machine learning, pp. 2048-2055 PMLR, 2015.
Papinnni, K., Roukos, S., Ward, T., and Zhu, W.-J Bleu: method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Yao, Y., Yu, T., Zhang, A., Wang, C., J., H., Cai, T., Li, H., Zhao, W., He, Z., et al. Miniepm-v: A gpt-4v level milm on your phone, arXiv preprint arXiv:2408. 01800, 2024.
Petryk, S., Chan, D., Kachinthaya, A., Zou, H., Canny, J., Gonzalez, J., and Darrell, T. Aloha: new measure for hallucination in captioning models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 2: Short Papers), pp. 342-357, 2024.
Yin, S., Fu, C.,, Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey on multimodal large language models. arXiv preprint 2.ti:::300..154549 2023a.
Yin, S., Fu, C.,, Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Yin, S., Fu, C., Zhao, S., Xu, T., Wang, H., Sui, D., Shen, Y., Li, K., Sun, X., and Chen, E. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310. 16045, 2023c.
Yu, W., Yang. Z., Li, L.., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang. L. Mm-vet: Evaluating large multi- modal models for integrated capabilities. arXiv preprint wwXiv:2308002490 2023.
Reimers, N. and Gurevych. Sentence-bert, Sentence em- beddings using siamese nett-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computational Linguistics, 11 2019. URL https: http:///aaxvv..ogg// abs/ 1908.98.. 10084.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M.. Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Urbanek, J., Bordes, F., Astolfi, P., Williamson, M., Sharma, V., and Romero- A. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26700- 26709, 2024.
Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, J. When and why vision-language models behave like bags-of words, and what to do about it? In The Eleventh International Conference on Learning Repre- sentations, 2023.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus- based image description evaluation. In Pro- ceedings of the IEEE conference on computer vision and paltern recognition, pp. 4566 2015.
Zhang, T., Qiu, L.,, Guo, Q. Deng, C., Zhang, Y., Zhang, Z., Zhou, C., Wang. X., and Fu, L. Enhancing uncertainty- based hallucination detection with stronger focus. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 915-932, 2023.
Vinyals, O., Toshev, A., Bengio, S.. and Erhan, D. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156-3164, 2015.
Zhao, Y., Yan, L., Sun, W., Xing, G., Meng, C., Wang, S., Cheng, Z., Ren, Z., and Yin, D. Knowing what llms do not know: A simple yet effective self-detectio method. In Proceedings of the 2024 Conference of the North Amer- ican Chapter of the Association for Computational Lin- guistics: Human Language Technologies ( (Volume 1: Long Papers), pp. 7044-7056, 2024.
Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Conference on Learning Representations, 2023.
12

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
Zhong, W., Feng, X., Zhao, L.,, Li, Q., Huang. L.,, Gu, Y., Ma, W., Xu, Y., and Qin, B. Investigating and mitigating the multimodal hallucination snowballing in large vision- language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11991-,2011, 2024.
Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C.,, Bansal, M., and Yao, H. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Conference on Learning Represen- tations, 2024.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
13

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
A. Human Evaluation Dataset Construction
The results Table are obtained through the following process:
Captions are generated for 100 DOCCI test images using LLaVA-v1.5-7B and InstructBLIP.
2. Human labelers evaluate the captions from LLaVA-v1.5-7B and InstructBLIP for each image in terms of factuality.
3. Caption pairs with similar factuality quality are excluded.
4. For the remaining pairs, the correlation between human decisions and those made by each automated metric is measured.
B. Undesirable Bias in FACTSCORE
Metrics that rely solely on unimodal information are inherently susceptible to undesirable biases. For instance, metries like FACTSCORE, which depend exclusively on reference captions, introduce stylistic bias tied to the specific style, tone, or phrasing of the references, unfairly favoring penalizing captions based these factors. To demonstrate this, we compare FACTSCORE with our factuality metric using human-labeled captions that are hallucination-frrr but stylistically different from DOCCI captions (HUMAN) (Garg et al., 2024). Table shows that, due to its stylistic bias, FACTSCORE assigns lower scores to these human-labeled captions, even though they are clearly superior to LLaVA-v1.557D1 and InstructBLIP captions in terms of factuality. In contrast, our factuality metric remains robust against such bias.
Table Comparison correlations between human preferences and automated metrics in terms of factuality.
Spearman's p
Task
FACTSCORE
Ours
LLaVA-v1.5-72 vs. InstructBLIP
70.2 67.9
HUMAN vs. LLaVA-vI.5-7B vs. InstructBLIP
18.3
61.4
C. Ablation Study
Table 8: Effectiveness of our proposed method across various captioning models function of æ. In the CapMAS column, the LLM represents the decomposer and corrector, while the MILMM represents the fact-checker.
CapMAS
Metric
Captioner
LLM
MLLM
π
CLAIR
Factuality
Coverage
68.8
59.9
47.9
LLaMA-3-8B
-EX--7
1.0
74.1
72.2
46.9
LLaVA-NeXT-7B
LLaMA-3-8B
LLaVA-NeXT-7B
0.5
73.6
76.9
43.7
LLaMA-3-8B
LLaVA-NeXT-7B
0.3
72.2
76.8
40.0
70.2
62.1
48.5
LLLMA-3-8B
-A.VA-NeXT-13R
1.0
75.5
77.9
45.8
LLaVA-NeXT-IBR
LIaMA-3-88
LLaVA-NeXT-13B
0.5
74.8
79.9
42.1
LLaMA-3-8B
LLaVA-NeXT-133
0.3
72.6
80.5
39.6
74.9
65.5
48.2
LLMMA-388R
InterVL--Cien V1.5
78.2
75.9
47.3
InternVL-Chat-Ve 11.5
LLaMA-3-8B
Intem Inter VL-Chat- Chat-V1.5
0.5
79.0
78.8
46.0
LLaMA-3-SB
MiemVVL-Chat-II.
0.3
77.7
81.7
42.5
Our proposed method features single hyperparameter, π, which serves as the threshold for classifying atomic propositions as hallucinations or non-hallucination. Table 8 presents the effects of CapMAS across various models as function of т. The results reveal loose trade-off between factuality and coverage depending on æ. Specifically, all tested settings, as a increases, factuality tends to decrease while coverage increases.
14

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
D. The complete version of Table 2
Table Meta-evaluation results across various caption evaluation methods. DOCCI and synthetic hallucinatory captions are used for the meta-evaluation. The highest-rated caption for cach method is highlighted in bold.
Evaluation Metric
Caption
BLEU
ROUGE
METEOR
CIDEr
CLIP-S
RefCLIP-S
CLAIR
ALOHa
Ours
Clean
4.2
22.0
13.7
6.4
81.3
75.5
86.9
36.2
62.8
Object
4.9
22.3
14.5
4.8
81.0
75.3
85.2
31.5
52.3
Attribution
4.1
21.8
13.6
6.2
80.9
75.2
80.0
34.3
60.9
Relation
4.1
21.8
13.7
6.7
81.4
75.6
83.5
36.9
51.9
E. Case
serene outdoor scene features majestic black owl statue standing an a woaden past frant of paln tree adorned with yellow flowers, against clear blue sky with fluffy white clouds and warn sun shining brightly. The statue, with its wings spread wide, appears to be in mid- fid-flight, while the paln tree provides vibrant contrast to the black owl. The wooden post provides solid base for the statue, and the clear blue sky adds depth to the image. The yellow flowers the palm tree add touch of color to the scene, while the sun's 's warn glow illuminates the
Nex-7--
scene, creating peaceful tranquil atnosphere.
inage depicts serene sculpture standing on wooden post in garden setting. The owl has large eyes and detailed pattern of feathers, predominantlly lush garden with pain trees and ather plants, which provide textured appearance. The sun shines brightly in creating dramatic contrast with the cool blue of the clear sky in the background. The overall composition the inage presents peaceful the garden, and the silhouettes of the trees contributing to picturesque and harmonious balance. The wooden provides solid base for the owl sculpture, while the clear blue adds depth to the inage, creating a besutiful representation of nature and wildlife
scene featuring stylized,
dimensional
of gray black. wooden post situated
<CapMAS>
left
of the image, casting
the
tranquil setting, with the sunlight,
Figure 5: An example of caption generated by CapMAS, with LLaVA-NeXT-7B as both the captioning and fact-checking model and LLaMA-3-8B both the decomposer and corrector LLM.
F. Additional Experiments
F.1. Fluency Analysis
We investigate the effect of our framework on the fluency the generated captions. We employ an LLM-based evaluation to measure the basic quality of language in the captions generated by CapMAS, Specifically, we utilize GPT-4o with the following prompt:
You are a language expert evaluating the fluency of image captions.
Fluency refers to how grammatically correct, natural, and well-formed the text sounds to a native English speaker. A fluent caption should be grammatically correct, free of awkward phrasing, and read smoothly.
Evaluate the fluency of the following caption and return your output **strictly in JSON format** with:
"reason": key reason for your scoring
"score": number between 0 (completely disfluent) and 100 (perfect fluency)
Caption: "(caption]"
In addition to evaluating the captions generated by CapMAS, we also assess human-written, detailed captions for the same set of images.
The results in Table 10 demonstrate that the captions generated by CapMAS achieve even higher fluency scores than the human- human-generaan generated captions. This can be attributed to the final stage of CapMAS, in which the corrector LLM helps preserve or even improve the fluency of the captions.
15

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
Table Fluency evaluation results. The results of CapMAS (captioner) demonstrate that the proposed framework can generate captions with higher fluency than the human written reference captions (Human).
Captions generated by
Fluency 1
Human
89.0
CapMAS (LLaVA-v1.5-7B))
93.4
CapMAS (LLaVA-NeX1-7B)
93.4
CapMAS (LLaVA-NeXT-I3B)
93.6
CapMAS VL-eat- V1.5)
94.1
Table Effect Self-Refine (SR) on image captioning. The results show that quality of captions degrades as iterations increase.
Method CLAIR
Factuality
Coverage
Base 82,4
77.1
53.5
+SR (xl)
79.9
72.3
50.4
+SR (x2)
79.1
70,6
50.1
+SR (x3) 78.5
69.8
49.7
F22 Effect of Self-Refine on Image Captioning
CapMAS can be interpreted as inference-time scaling strategy. However, allocating more compute at inference time does not necessarily lead to better results. To demonstrate this, we test the effectiveness of inference-ting scaling through Self-Refine (SR) (Madaan et al., 2023). Since validating and revising one's output requires advanced reasoning capabilities, we conducted the experiments using GPT-4V. For SR, we used the following prompt:
You are given an image and its corresponding caption. Your task is to:
1. Analyze the image and compare it with the caption.
2. Identify and correct any factual or descriptive errors in the caption based on the image.
3. Refine the caption for clarity, correctness, and completeness - even if the original caption is mostly accurate.
Show your reasoning and then provide a final refined caption.
Table 11 shows that having the model revise its own captions iteratively does not lead better results.
F.3. Efficiency of CapMAS
CapMAS is an approach that achieves better results by incurring additional cost at inference time. Although it involves multi-model pipeline, CapMAS can offer better perforeeformeece trade-off than Self-Refine (SR) methods for the following reasons:
Most of CapMAS's cost lies in the final step, where the corrector L.M generates a refined caption based on the initial caption and the True/False classification results of the propositions. The decomposition step involves shorter sequences, and the MLLM used for proposition classification can process them in parallel, generating only a single token (True or False) per proposition. SR processes long sequences that include the original caption, detailed feedback, and the refined caption. Considering the length and complexity of the feedback (Madaan et al., 2023), CapMAS and can be seen comparable in cost.
CapMAS's performance does not heavily depend on the capability of the LLM used. This suggests that the LLM-related cost in the pipeline could potentially be further reduced.
G. Prompt Templates
16

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
prompt "Describe the given image in very detailed manner."
prompt "Provide detailed description the specifled image."
prompt_ 3 = "Elaborate on the details of the image provided."
prompt_ "Offer an in-depth description of the given image." prompt_5 = "Thoroughly describe the features of the specified image."
Figure 6: The five prompt inputs used to generate captions in experiments.
system:
I want to verify if the given CAPTION is accurate. To assist with this verification, decompose the given CAPTION into atomic propositions. All parts of the caption must be broken down into propositions. The outputs should follow the following format: '1 proposition one\n2. proposition two \n3. proposition three'. For example, break down 'He is tall, thin, and pale' into '1. He tall. \n2. He is thin. \n3. is pale.'
user:
CAPTION: {caption)
Figure 7: The prompt input LLaMA-3-8B serving the decomposer.
system:
I want to create caption that includes only facts. Please help correct the given caption. The given caption contain things that are not true. Based on the given FACTS and NON-FACTS remove the non-factual elements from the caption. Place the revised caption between '###'.
user:
Caption: (caption)\IFACTS:: \nFACTS: haaaaa/ssttttoo---aallalannntton among the atomic propositions)\nnon
FA |nnhttalluiinsniin among the atomic propositions}
Figure 8: The prompt input for LLaMA-3-83 serving the corrector.
system:
This is hard problem. Carefully summarize in ONE detailed caption based on the following captions by different people describing the image. Be to describe everything, and avoid hallucination. Provide the detailed caption in the format *### (Detailed caption) #s#'.
user:
Caption 1: {caption 1st) An Caption 2: {caption 2nd) \n Caption 3: (caption 3rd)An Caption 4: {caption An Caption 5: (caption 5th}An
Figure 9: The prompt input for LLaMA-3-8B serving as the summerizer. We use the prompt employed in the work of (Ge al., 2024).
17

---

A Multiagent Approach and Dual Evaluation Metries for Factuality and Coverage
if hallucination "Object"
prompt_sys "I want to inject incorrect information into the caption of the given photo. Your role to modify about THREE words from the latter part the given caption that describe the attributes of the objects so that they do not match the photo." "Attribution"
elif hallucination
prompt_sys "I want to inject incorrect information into the caption of the given photo. Your role is to imagine object that isn't actually in the image but could plausibly be there, add very brief part about to caption so that they do not match photo." elif hallucination
"Relation":
want to inject incorrect information into the caption the given photo. Your role is to change the spatial relationships between the objects so that they do not match the photo. For example, change 'A person is standing the right of the car' to 'A person is standing to the left of the car.' Do not change anything other than the spatial relationships between the objects."
prompt_sys
system: (prompt_sys)
user (presented with the image): Caption: (caption)
Figure prompt input for GPT-4o used create meta-evale evaluation dataset Table
system:
"I want to use an object detector to check the correctness of an image caption obtained by an image caption model. you help to parse the given CAPTION and list all objects that could be detected with an object detection model in the image? Please only list the object name and ignore the description. Please use the name in the CAPTION it is. Please concatenate them together with \":\" as separation."
user:
CAPTION: {caption}
Figure The prompt input for GPT-4 used create dataset Figure We use the prompt employed in work (Ge al., 2024).
18


Reviews:
[{"review_id": "0CTRhaUh9Y", "reviewer": "Reviewer_quCw", "summary": "This paper proposes a multiagent approach that leverages LLM and MLLM to correct given captions and designs two metrics for evaluating generated caption. A dataset is collected for one of the metrics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition of our method’s potential to improve the factual accuracy of image captions and our novel approach leveraging MLLM–LLM collaboration without training a separate corrector. We also thank you for your suggestions, which have enriched our work.\n\n**We are pleased to share that, in support of open-source research, we have decided to release our carefully curated VQA dataset and evaluation codes. This dataset includes 1k images, each paired with approximately 36 question-answer sets. We kindly ask that you consider this a contribution to the open-source community.**\n\nIn the subsequent sections, we address each of your concerns as follows:\n\n>**Q1**. In Table 4, after the captions were corrected, the coverage score decreased. This may because: 1) the design of the metric is not suitable, 2) the method sacrifices coverage for factuality, indicating that the method has some drawbacks. Same phenomenon is observed in Table 5, please provide more explanation.\n\n**A1**. We would like to clarify a potential misunderstanding regarding the decrease in the coverage score caused by the application of CapMAS. First and foremost, CapMAS was proposed to enhance the factuality of detailed image captions. Ideally, identifying and correcting hallucinations within captions would improve both factuality and coverage. However, through empirical analysis, we observed that such correction attempts often lead to the generation of new hallucinations due to the limitations of current MLLMs.\nTherefore, CapMAS is designed to remove detected hallucinations rather than correct them. As CapMAS may remove not only hallucinations but also factual content, it inherently involves a trade-off between factuality and coverage, as discussed in Section 4.3. This trade-off is controlled by its hyperparameter, $\\pi$.\nConsequently, the decrease in coverage scores observed in Tables 4 and 5 arises not from a flaw in the proposed metrics, but rather from the design of CapMAS, which reflects the current limitations of MLLMs.\n\n>**Q2**. Since Factuality and Coverage cannot reflect the basic quality of language, how about including the traditional metrics like CIDEr and METEOR?\n\n**A2**. We employ an LLM-based evaluation to measure the basic quality of language in the captions generated by CapMAS. This approach is motivated by recent studies [1,2] indicating that conventional automatic caption evaluation methods are biased and not well-suited for assessing detailed captions. Specifically, we utilize GPT-4o with the following prompt:\n```\nYou are a language expert evaluating the fluency of image captions.\n\nFluency refers to how grammatically correct, natural, and well-formed the text sounds to a native English speaker. A fluent  caption should be grammatically correct, free of awkward phrasing, and read smoothly.\n\nEvaluate the fluency of the following caption and return your output **strictly in JSON format** with:\n- \"reason\": a key reason for your scoring\n- \"score\": a number between 0 (completely disfluent) and 100 (perfect fluency)\n\nCaption: \"{caption}\"\n```\nIn addition to evaluating the captions generated by CapMAS, we also assess human-written detailed captions for the same set of images.\n\n**Table A**\n\n|Captions generated by|Fluency &#8593;|\n|-|-|\n|Human|89.0|\n|CapMAS (LLaVA-v1.5-7B)|93.4|\n|CapMAS (LLaVA-NeXT-7B)|93.4|\n|CapMAS (LLaVA-NeXT-13B)|93.6|\n|CapMAS (InternVL-Chat-V1.5)|94.1|\n\nThe results in Table A demonstrate that the captions generated by CapMAS achieve even higher fluency scores than the human-generated captions. This can be attributed to the final stage of CapMAS, in which the corrector LLM helps preserve or even improve the fluency of the captions.\n\nWe sincerely thank the reviewer for suggesting this experiment, which enabled us to highlight a new strength of CapMAS in terms of fluency.\n\n>**Q3**. In Table 2, it appears that the CLAIR metric can reflect the introduced noises. It's not as problematic as the article suggests.\n\n**A3**. Yes, CLAIR is indeed capable of reflecting the introduced noise, which is precisely why we adopted it in our experiments. However, as shown in the prompt used below, the meaning of the score it produces is not clearly defined:\n```\nOn a precise scale from 0 to 100, how likely is it that the candidate caption is describing the same image as the reference  caption? (JSON format, with a key \"score\", value between 0 and 100, and a key \"reason\" with a string value.)\n```\nIn contrast, our proposed metrics have clearly defined interpretations and enable a more detailed analysis of MLLMs in terms of both factuality and coverage.\n\n[1] ImageInWords: Unlocking Hyper-Detailed Image Descriptions  \n[2] Benchmarking and Improving Detail Image Caption\n\n**We have additional experimental results that we were unable to include here. To help us share them, we kindly ask you to click “Rebuttal Comment” to allow us to leave a supplementary comment.**", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "- In Table 4, after the captions were corrected, the coverage score decreased. This may because: 1) the design of the metric is not suitable, 2) the method sacrifices coverage for factuality, indicating that the method has some drawbacks. Same phenomenon is observed in Table 5, please provide more explanation.\n- Since Factuality and Coverage cannot reflect the basic quality of language, how about including the traditional metrics like CIDEr and METEOR?", "theoretical_claims": "Not Applicable.", "experimental_designs_or_analyses": "In Table 2, it appears that the CLAIR metric can reflect the introduced noises. It's not as problematic as the article suggests.", "relation_to_broader_scientific_literature": "This paper is related to helping MLLMs generating highly detailed captions.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "- Strengths:\n  - The proposed method enhances the factual accuracy of captions.\n  - Existing approaches require the corrector model training, while the proposed method employs collaboration between an MLLM and LLM. \n- Weaknesses:\n  - Previous studies focus on measuring the factuality of generated text, while the coverage metric is newly proposed. However, the rationale for using this metric is questionable, as the two metrics appear to be contradictory based on the experimental results. Or if there is no problem with the metric, does this mean that the proposed method may not work as intended?", "other_comments_or_suggestions": "None.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XaoYAmJ7mn", "reviewer": "Reviewer_QHjW", "summary": "This paper focuses on generating long, detailed captions for images. A key idea in the paper is to decompose long captions into atomic claims using an LLM, and then verify every claim individually in the context of the image using a VLM. The paper motivates this by showing that this approach outperforms alternative ways of identifying hallucinations, like token confidence or consistency-based approaches. Based on this observation, the paper proposes a pipeline called CapMAS, which decomposes a long caption into atomic claims, verifies the correctness of each claim, and removes incorrect claims from the caption using an LLM. The paper also proposes two metrics to measure factuality and coverage of claims in captions by using GPT-4o to break down captions into individual claims and verifying them. The paper compares these metrics to prior metrics like ROGUE and BLEU, and shows that it outperforms them wrt finding the correct caption for an image. Using these metrics, they evaluate CapMAS and prior approaches for the task of long-caption generation. CapMAS performs quite well on a wide range of VLMs, outperforming prior approaches. Finally, the paper highlights that current models are typically trained to generate short, concise responses and struggle with longer responses.\n\n## Update after rebuttal\nI think this is a solid paper. Thus, I increased my score to Accept. I hope the added results and discussions are reflected in the final version.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We are pleased to share that, in support of open-source research, we have decided to release our carefully curated VQA dataset and evaluation codes. This dataset includes 1k images, each paired with approximately 36 question-answer sets.**\n\nWe sincerely thank you for your thorough review. We have made our best efforts to address your remaining concerns as follows:\n>**Q1**. They also proposed a new metric for measuring coverage, but did not compare it to any prior metrics for this purpose – are there any relevant alternatives?\n\n**A1**. There have been prior attempts to measure the coverage of image captions [1]. These methods typically involve extracting visual entities from reference captions and checking whether they appear in the caption being evaluated. However, such approaches have notable limitations:\n1. MLLMs can describe the same visual content in highly diverse ways. As a result, it is often difficult to determine with precision whether the visual content in the reference captions is present in the evaluated caption.\n2. These methods can only evaluate pre-defined types of visual content—typically objects, attributes, and spatial relations—since they rely on accurate extraction and comparison of such elements within captions. As a result, they cannot assess whether a caption captures conceptual aspects of an image, such as mood or semantic relationships between entities.\n\nIn contrast, our coverage metric avoids these limitations. Any information can be turned into a question and given to an LLM grounded in the caption.\n\n[1] Object Hallucination in Image Captioning\n>**Q2**. Which datasets do Tables 4 and 5 correspond to?\n\n**A2**. For Tables 4 and 5, we used IIW-400, based on findings [2] indicating that IIW-400 serves as a better reference caption set than DOCCI. However, to demonstrate the generalizability of CapMAS, we additionally tested its effectiveness on a subset of DOCCI (400 samples).\n\n<Table A>\n\n|Method|CLAIR|Factuality|Coverage|\n|-|-|-|-|\n|LLaVA-NeXT-7b|68.0|56.2|54.0|\n| +CapMAS|72.8|68.1|53.5|\n|LLaVA-NeXT-13b|70.1|58.2|55.6|\n| +CapMAS|74.5|74.6|52.3|\n\nTable A demonstrates that CapMAS is also effective on DOCCI.\n\n[2] ImageInWords: Unlocking Hyper-Detailed Image Descriptions\n>**Q3**. I believe this pipeline cannot improve the coverage in captions (and only improve factuality). How can we improve coverage?\n\n**A3**. Through inference-time scaling, CapMAS can improve the coverage of captions while maintaining a certain level of factuality. Due to space limitations, we regret that we cannot provide further details here and kindly ask you to refer to our response **A4** to reviewer **fLiQ** for more information.\n>**Q4**. This is a minor point, but for the sake of completeness, you could also evaluate your pipeline on GPT-4o, as it is more widely used than GPT-4v.\n\n**A4**. We used GPT-4o to evaluate the captions, and it is known that LLMs tend to favor their own outputs [3]. Therefore, to ensure a fair comparison, we did not use GPT-4o at any stage of the proposed captioning pipeline.\n\n[3] LLM Evaluators Recognize and Favor Their Own Generations\n>**Q5**. How is coverage computed? I couldn’t find a clear formula for this in the paper.\n\n**A5**. We apologize for the confusion. Let N be the total number of VQA samples and C the number correctly answered by GPT-4o using only the captions. Then, the coverage score is C/N.\n>**Q6**. For measuring factuality, your method uses the image and the reference caption. If the reference caption is already available, is the image really required?\n\n**A6**. Metrics based solely on reference captions are prone to stylistic bias, favoring or penalizing captions based on phrasing. To illustrate this, we compare a reference-based metric with our factuality metric using human-labeled captions (HUMAN) that are hallucination-free but stylistically different from the references. As shown in Table B, the reference-based metric correlates poorly with human judgment by assigning lower scores to HUMAN while our factuality metric remains robust.\n\nTable B: Meta-evaluation with model- and human-generated captions; see Sec. 4.2 and Appx. B for details.\n\n|Method|Correlation with Human Evaluation &#8593;|\n|-|-|\n|Reference-based|18.3|\n|Ours|61.4|\n>**Q7**. The coverage of these questions themselves would be bottlenecked by the vision capabilities of GPT-4o, and the questions might miss some details.\n\n**A7**. Building a detailed VQA dataset is costly, so we relied on GPT-4o, making our dataset quality depend on its capabilities. To address this, we carefully guided human annotators during the labeling process.\n\nWhile we considered using reference captions, we first aimed to show that our coverage metric works reliably without them, as reliance on references limits its applicability. Our results confirm this, and we are exploring ways to further improve the method by incorporating reference captions.\n\n**We have more results—please click “Rebuttal Comment” to allow us to add a comment.**", "claims_and_evidence": "The claims are mostly supported by empirical evidence:\n- The paper motivated the use of decomposing long captions into atomic claims by showing that it is better at detecting hallucinations as compared to simple baselines like token confidence or consistency-based approaches. This forms the basis of their pipeline, CapMAS.\n- The paper proposed a metric for measuring factuality and showed that it is better than prior metrics for this purpose. \n- They also proposed a new metric for measuring coverage, but did not compare it to any prior metrics for this purpose – *are there any relevant alternatives?*\n- They evaluated their pipeline CapMAS using these metrics and demonstrated that it consistently improved factuality for a range of MLLMs (like LLaVA and GPT-4o), supporting the efficacy of their approach.\n- They also demonstrated that models which are good at visual question answering might not necessarily be good at detailed image captioning as they might be biased toward shorter responses, reducing their coverage.", "methods_and_evaluation_criteria": "Yes. The paper focuses on removing hallucinations from detailed captions. They identify two key criteria for this setting: factuality and coverage. They propose metrics for these criteria and demonstrate their efficacy. They evaluate their approach on two datasets – DOCCI and IIW-400 – which contain images and corresponding detailed, factual captions. They test their method on a wide range of MLLMs to assess how general it is.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, the experiments in the paper are mostly sound. The factuality metric is compared to multiple existing metrics on 3 types of perturbed captions that contain hallucinations, and it is the most accurate at identifying captions across all of them. Further, the CapMAS pipeline is also compared against prior works on a variety of MLLMs and shows strong performance consistently.\n\n*One major question is: which datasets do Tables 4 and 5 correspond to? This wasn’t clear, and is an important detail.*", "relation_to_broader_scientific_literature": "Prior literature in vision-language understanding has focused less on highly detailed captions. While approaches have been proposed to reduce hallucinations in shorter responses, the paper shows that these approaches don’t work well for longer captions. They propose a new method to improve the factuality of detailed captions and show that it outperforms existing approaches.", "essential_references_not_discussed": "Mostly, the literature is well-covered.\nYou could additionally discuss and compare against [1] as a baseline.\n\n[1] Petryk et al. Simple Token-Level Confidence Improves Caption Correctness. WACV'24", "other_strengths_and_weaknesses": "Strengths:\n- The proposed pipeline performs well on the task of reducing hallucinations in detailed captions. It leads to significant improvement in performance for a range of MLLMs (small models like LLaVA to large models like GPT-4v). It also outperforms prior approaches.\n- The pipeline can be easily plugged into existing MLLMs without requiring any additional training.\n- The paper is mostly well-written and easy to follow.\n\nWeaknesses:\n- A few details were not clear. Which datasets do Tables 4 and 5 correspond to? This wasn’t clear, and is an important detail.\n- While this approach helps improve factuality, it doesn’t seem to improve the coverage of captions, which is also important for detailed captions. (I understand that one can tune the hyperparameter pi to change coverage but coverage is still upper-bounded by the original captions generated by the MLLM).\n- This is a minor point, but for the sake of completeness, you could also evaluate your pipeline on GPT-4o, as it is more widely used than GPT-4v.", "other_comments_or_suggestions": "You can consider giving your proposed metrics a specific name (for example, the title of Section 4.2 reads a bit weird to me). Also, clearly highlight that this metric uses GPT-4o’s vision capabilities as well (i.e., there is no other model serving as the VLM, only GPT-4o).", "questions_for_authors": "- How is coverage computed? I couldn’t find a clear formula for this in the paper.\n- I believe this pipeline cannot improve the coverage in captions (and only improve factuality). How can we improve coverage?\n- For measuring factuality, your method uses the image and the reference caption. If the reference caption is already available, is the image really required? Can’t you verify the claim against the reference caption? An ablation about this might be interesting, as removing the image can save significant amounts of compute/ API credits.\n- Similarly, for coverage, the questions are generated based on the image. The coverage of these questions themselves would be bottlenecked by the vision capabilities of GPT-4o, and the questions might miss some details. Could you try incorporating the captions of images in the question-generation pipeline?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cMEPucklBh", "reviewer": "Reviewer_bFJq", "summary": "This paper looks at preventing hallucination in long-form image captions, proposing a system \"CapMAS\" which decomposes generated captions into atomic statements, which are then generated/corrected using a VLM. The paper also introduces two metrics for image caption evaluation based on a similar pipeline: Factuality (which represents the portion of \"true\" atomic statements), and Coverage (which uses a dataset of human annotated examples to evaluate if a caption can be used to answer all questions in an images). The paper shows that the resulting factuality metric correlates better with human judgements than both FAITHSCORE and FACTSCOR. The paper then shows that CapMAS outperforms base models alone when applied across several base captioning models (on CLAIR, Factuality, and Coverage scores), and significantly improves performance compared to hallucination reduction methods.\n\n## update after rebuttal\n\nThanks to the authors for providing some clarifications to my comments - I particularly appreciate the additional discussion comparing to SR, and would love to see this included in the paper (A3). In general, while the paper is similar to some released methods, and I agree with reviewer QHjW that it is unlikely to have broad impact due to it's limited scope, it's a solid paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We are pleased to share that, in support of open-source research, we have decided to release our carefully curated VQA dataset and evaluation codes. This dataset includes 1k images, each paired with approximately 36 question-answer sets.**\n\nWe sincerely thank you for your thorough review. We have made our best efforts to address your remaining concerns as follows:\n>**Q1**. The claim on L346 (Sec 4.3) that CapMAS exhibits a factuality-coverage tradeoff is somewhat over-stated. \n\n**A1**. CapMAS is designed to remove detected hallucinations rather than correct them. This design choice is based on our observation that attempts at correction often lead to the introduction of new hallucinations. As CapMAS may remove not only hallucinations but also factual content, it inherently involves a trade-off between factuality and coverage, as discussed in Section 4.3.\n\nNevertheless, through inference-time scaling, CapMAS can improve the coverage of captions while maintaining a certain level of factuality. Due to space limitations, we regret that we cannot provide further details here and kindly ask you to refer to our response **A4** to reviewer **fLiQ** for more information.\n>**Q2**. I would like to see some experiments on a wider set of images.\n\n**A2**. Using benchmarks like COCO for evaluating detailed image captioning can lead to misleading conclusions:\n1. Short captions in COCO bias evaluation metrics to favor simpler captions over accurate, detailed ones.\n2. Reference-free metrics, intended to remove this bias, often introduce another bias stemming from the evaluation model itself, which can significantly affect the scores [1].\n\nWe believe reliable evaluation requires detailed human supervision. To address your concern, we conducted additional experiments on DOCCI. Due to space limitations, we kindly ask you to refer to our response **A2** to reviewer **QHjW** for details, which confirm CapMAS is effective on DOCCI as well.\n\n[1] LLM Evaluators Recognize and Favor Their Own Generations\n>**Q3**. Is it necessary to break the caption down into atomic points before re-captioning, or is just asking the external VLM to re-caption sufficient?\n\n**A3**. CapMAS can be understood as an inference-time scaling strategy. Motivated by the reviewer’s comment, we tested the effectiveness of inference-time scaling via Self-Refine (SR) [2]. Since validating and revising one's output requires advanced reasoning capabilities, we conducted the experiments using GPT-4V. For SR, we used the following prompt:\n```\nYou are given an image and its corresponding caption.\nYour task is to:\n1. Analyze the image and compare it with the caption.\n2. Identify and correct any factual or descriptive errors in the caption based on the image.\n3. Refine the caption for clarity, correctness, and completeness — even if the original caption is mostly accurate.\n\nShow your reasoning and then provide a final refined caption.\n```\n<Table A>\n\n|Method|CLAIR|Factuality|Coverage|\n|-|-|-|-|\n|Base|82.4|77.1|53.5|\n|+SR (x1)|79.9|72.3|50.4|\n|+SR (x2)|79.1|70.6|50.1|\n|+SR (x3)|78.5|69.8|49.7|\n\nTable A shows that having the model revise its own captions iteratively does not lead to better results.\n\n[2] SELF-REFINE: Iterative Refinement with Self-Feedback\n>**Q4**. There's not really any explanation or discussion of the efficiency of the model\n\n**A4**. CapMAS is an approach that achieves better results by incurring additional cost at inference time. Although it involves a multi-model pipeline, CapMAS can offer a better cost-performance trade-off than Self-Refine (SR) methods for the following reasons:\n1. Most of CapMAS’s cost lies in the final step, where the corrector LLM generates a refined caption based on the initial caption and the True/False classification results of the propositions. The decomposition step involves shorter sequences, and the MLLM used for proposition classification can process them in parallel, generating only a single token (True or False) per proposition. SR processes long sequences that include the original caption, detailed feedback, and the refined caption. Considering the length and complexity of the feedback [2], CapMAS and SR can be seen as comparable in cost.\n2. As shown in Table 4, CapMAS’s performance does not heavily depend on the capability of the LLM used. This suggests that the LLM-related cost in the pipeline could potentially be further reduced.\n\n>**Q5**. I couldn't find the dataset used for evaluation in Tables 4 and 5, is this on IIW-400 or the DOCCI dataset?\n\n**A5**. For Tables 4 and 5, we used IIW-400, based on findings [3] indicating that IIW-400 serves as a better reference caption set than DOCCI. As noted in A2, our additional experimental results confirm that CapMAS is effective on both IIW-400 and DOCCI.\n\n[3] ImageInWords: Unlocking Hyper-Detailed Image Descriptions\n\n**We have additional results and discussions that we were unable to include here. We kindly ask you to click “Rebuttal Comment” to allow us to add a comment.**", "claims_and_evidence": "The paper makes several well-supported claims:\n- CapMAS improves on factuality/clair/coverage averages in the DOCCI/ IIW-400 dataset\n- VQA benchmarks do not correlate with captioning performance\n- Existing hallucination detection methods fail on long captions (Though it's worth noting that CLAIR appears to be as effective in the meta-evaluation in Table 2, and ALOHa performs well in Table 2 for object-hallucination, which it is designed for).\n\nThe claim on L346 (Sec 4.3) that CapMAS exhibits a factuality-coverage tradeoff is somewhat over-stated. While the factuality and coverage scores do indeed have opposite trends in Table D, it doesn't seem to me that these are inherently conflicting values (and indeed, this is just an artifact of the fact that the LLM cannot achieve well-grounded performance).", "methods_and_evaluation_criteria": "The methods/evaluation criteria are fairly well-designed, though I would like to see some experiments on a wider set of images (particularly compared to IIW-400 which is quite small at only 400 images). Experimental comparisons on COCO (though less centered on detail captions), would be quite helpful in placing this against other captioning methods.", "theoretical_claims": "This paper makes no theoretical claims.", "experimental_designs_or_analyses": "I did not validate the experimental design, however no tables in the paper have any estimate of variance (for example, standard error around the mean), making it challenging to determine which experiments have significant effects. Also notable is that there is little ablation of the CapMAS components, for example, is it necessary to break the caption down into atomic points before re-captioning, or is just asking the external VLM to re-caption sufficient?", "relation_to_broader_scientific_literature": "This paper consists of two key components, the method CapMAS and the evaluation measures/dataset. The evaluation measures are, while useful, quite similar to related work. Coverage is quite similar in structure to the POPE metric, albeit with a different goal (rather than hallucination detection, image factuality). Factuality is quite similar to the ALOHa metric, but expands somewhat from objects to atomic statements (though the motivating experiments seem to use a version that is object-detector based, which aligns quite well with ALOHa's contribution). CapMAS itself is quite similar to methods employed by both LURE and Woodpecker (See below), and uses rewriting models to directly rewrite output captions (though here, the focus is on coverage/correctness compared to hallucination). \n\nWhile the approach is quite similar to existing methods, this appears to be the first approach which puts these all together, and the simplicity of the combination is quite compelling. The performance seems strong, and demonstrates that many of the ideas introduced in prior work can be combined to strong effect.", "essential_references_not_discussed": "The references are generally sufficient, however the paper might consider discussing a comparison to Woodpecker [1], which is a method similar to LURE for caption hallucination reduction which detects captions using an external model, and corrects the caption using information from this process. The system prompt in Figure 9 is very closely related to the system prompt in [2], and should probably be cited. The paper could consider citing related work in NLP on self-rationalization models which look at breaking sentences down into atomic claims [3].\n\n[1] Yin, Shukang, et al. \"Woodpecker: Hallucination correction for multimodal large language models.\" Science China Information Sciences 67.12 (2024): 220105.\n[2] Chan, David, et al. \"IC3: Image Captioning by Committee Consensus.\" The 2023 Conference on Empirical Methods in Natural Language Processing.\n[3] Wiegreffe, Sarah, and Ana Marasović. \"Teach me to explain: A review of datasets for explainable natural language processing.\" arXiv preprint arXiv:2102.12060 (2021).", "other_strengths_and_weaknesses": "Strengths:\n- The paper is quite strong in terms of performance on the discussed metrics, and clearly leads to overall improvements on a limited data subset.\n- The paper requires no additional training (is fully zero-shot)\n- Factuality seems to be quite good compared to both baseline measures\n\nWeaknesses:\n- As discussed above, the test datasets used here are fairly small and limited in scope, and it would be good to see evaluations on something a bit broader.\n- There's not really any explanation or discussion of the efficiency of the model, which now requires several LLMs (and probably significantly increases the cost of generating individual captions)", "other_comments_or_suggestions": "- It would be quite helpful to have some additional qualitative analysis, or error analysis, to look at potential directions for future research, or to evaluate weaknesses in the approach (for both CapMAS and for the evaluation metrics).", "questions_for_authors": "- I couldn't find the dataset used for evaluation in Tables 4 and 5, is this on  IIW-400 or the DOCCI dataset?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9IIyCBDmmm", "reviewer": "Reviewer_fLiQ", "summary": "This paper studies how to evaluate and tackle the hallucination phenomenon of MLLM. It first conduct a motivation experiments and conclude that existing hallucination detection methods struggles with long captions. Then it proposes a new multi agent approach which involves a LLM to decompose the original long detailed caption into atomic propositions, and another MLLM for fact checking. Additionally, a new evaluation metric and benchmark for factuality and coverage are proposed. Experiments on IIW-400 and DOCCI datasets with various common LLM and MLLM demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We are pleased to share that, in support of open-source research, we have decided to release our carefully curated VQA dataset and evaluation codes. This dataset includes 1k images, each paired with approximately 36 question-answer sets.**\n\nWe sincerely thank you for your thorough review. We have made our best efforts to address your remaining concerns as follows:\n\n>**Q1**. Do the authors have some experimental evidence to prove that after CapMAS correction, the new caption is on par with or better than the original caption besides the hallucination degree?\n\n**A1**. We sincerely thank the reviewer for suggesting this experiment, which allowed us to highlight a new strength of CapMAS in terms of fluency. Due to space limitations, we regret that we cannot provide a detailed response here, and kindly ask you to refer to our response **A2** to reviewer **quCw** for further details.\n\n>**Q2**. Thus, one wonders if a truly powerful MLLM with superb long-context understanding capability is employed, how much gain would CapMAS bring?\n\n**A2**. CapMAS is designed to improve the captioning performance of MLLMs, which currently have limited long-context capabilities and are prone to hallucinations. As such, its effect may be less pronounced for future MLLMs with powerful capabilities. However, it is important to highlight that even for GPT-4V, one of the current SOTA MLLMs, we were able to improve factuality by over 5% points using CapMAS—without sacrificing coverage—by leveraging relatively weaker open-source models. Given that today’s best MLLMs still fall short of a truly reliable MLLM, our work carries meaningful implications, particularly for visually impaired users.\n\n>**Q3**. How does the corrector LLM take as input the $\\pi$-thresholding results and correct the original caption?\n\n**A3**. It can be easily understood by referring to the corrector LLM's prompt template:\n```\nsystem:\nI want to create a caption that includes only facts. Please help me correct the given caption.\nThe given caption contains things that are not true. Based on the given FACTS and NON-FACTS,\nremove the non-factual elements from the caption.\n\nuser:\nCaption: {caption}\nFACTS:\n{propositions classified as non-hallucinatory}\nNON-FACTS:\n{propositions classified as hallucinatory}\n```\n>**Q4**. I wonder have the authors check if the hallucination scores would continue to improve if CapMAS is applied iteratively?\n\n**A4**. CapMAS is designed to remove detected hallucinations rather than correct them. This design choice is based on our observation that attempts at correction often lead to the introduction of new hallucinations. As a result, applying CapMAS iteratively to a single caption has an effect equivalent to applying it once with a lower threshold $\\pi$.\n\nInstead, we demonstrate that applying CapMAS individually to multiple captions for a single image and subsequently summarizing the results can lead to an overall improvement in the caption quality for that image.\n\nTable B: Results of LLaVA-Next on a subset of IIW400\n\n|Number of CapMAS Applications|CLAIR|Factuality|Coverage|\n|-|-|-|-|\n|1|59.8|83.0|24.3|\n|2|63.3|76.8|32.6|\n|3|67.4|80.2|37.7|\n|4|68.2|79.9|38.9|\n|5|70.6|80.1|41.4|\n\nTable B demonstrates that the overall quality of the resulting captions improves as the number of CapMAS applications increases.\n\n>**Q5**. I checked Appendix D and found that a larger threshold generally lead to better performances. I wonder, will it eventually plateau?\n\n**A5**. In CapMAS, as the value of $\\pi$ increases, the criterion for judging a proposition as true becomes more relaxed. This results in improved coverage but also causes more propositions to be classified as true, thereby introducing more hallucinations and lowering factuality. Consequently, in Appendix D, further increases in $\\pi$ lead to a significant drop in the factuality score, ultimately degrading the overall quality of the captions. We summarize the results of additional experiments in the following table:\n\nTable C: Ablation results on $\\pi$ using LLaVA-NeXT-7B\n\n|$\\pi$|CLAIR|Factuality|Coverage|\n|-|-|-|-|\n|3.0|71.1|63.0|47.8|\n|2.0|70.6|65.0|47.1|\n|1.0|74.1|72.2|46.9|\n|0.5|73.6|76.9|43.7|\n\n>**Q6**. Could this strategy be interpreted as some kind of inference-time scaling?\n\n**A6**. Yes, CapMAS can indeed be considered an inference-time scaling strategy, and the experimental results, including those in Table B, support this interpretation. Additionally, we compared CapMAS with Self-Refine, another inference-time scaling method. CapMAS achieves significantly better performance while requiring a comparable level of cost. Due to space limitations, we regret that we cannot provide further details here and kindly ask you to refer to our responses **A3 and A4** to reviewer **bFJq** for more information.\n\n**We have additional experimental results that we were unable to include here. To help us share them, we kindly ask you to click “Rebuttal Comment” to allow us to leave a supplementary comment.**", "claims_and_evidence": "Yes, the motivation justified by experiments in section 3.1 is straightforward and makes intuitive sense to me. Long-context understanding has always been a problem for LLM/MLLM, and it naturally comes to one's mind that hallucination detectors would lean more on closer text and less on far-way image when the generated caption is extremely long.", "methods_and_evaluation_criteria": "Yes. Breaking the detailed caption into atomic propositions naturally tackles the long-context understanding issue.\n\nI wonder, do the authors have some experimental evidences to prove that after CapMAS correction, the new caption is on par with or better than the original caption besides hallucination degree? A high-quality image caption should not hallucinates, and also truly and faithfully describe the whole image content in fluent human-style language.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experiments in Table 4 does show non-trivial improvement when the proposed CapMAS is applied. However, with a stronger captioner like GPT-4V, the gain becomes smaller. Thus, one wonders if a truly powerful MLLM with superb long-context understanding capability is employed, how much gain would CapMAS bring?\n\nSecondly, how does the corrector LLM takes as input the $\\pi$-thresholding results and correct the original caption? I wonder have the authors check if the hallucination scores would continue to improve if CapMAS is applied iteratively?", "relation_to_broader_scientific_literature": "Tackling LLM/MLLM hallucination is an important aspect of current foundational model researches.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This CapMAS approach requires two extra LLM and one MLLM. I wonder, what are the memory and computational cost of CapMAS when compared with naive baseline and other hallucination tackling methods? Could this strategy be interpreted as some kind of inference-time scaling? If so, how well does CapMAS perform when compared to simple inference-time scaling strategy of MLLM image captioning?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the challenge of hallucinations in hyper-detailed image captions generated by Multimodal Large Language Models (MLLMs). The authors posit that existing hallucination detection methods falter with long captions because MLLMs increasingly rely on their own generated text rather than the input image as sequences grow. To mitigate this, they propose CapMAS (Caption Multi-Agent System), a training-free, multi-agent approach where an LLM decomposes a given detailed caption into atomic propositions, and an MLLM verifies the factuality of each proposition against the image. Based on this verification (and a tunable threshold), a corrector LLM removes the non-factual propositions to produce a revised caption. Alongside the method, the paper introduces an evaluation framework comprising two new metrics, Factuality and Coverage, calculated using a similar decomposition-and-verification pipeline leveraging GPT-4o. They also curated and plan to release a VQA benchmark dataset to support this evaluation.

Reviewers largely agreed that the paper tackles an important and timely problem: reducing hallucinations in increasingly detailed image captions generated by MLLMs. They found the core CapMAS approach, which uses LLM-MLLM collaboration for decomposition, verification, and correction, to be logical and empirically effective at improving caption factuality across various base models, including strong ones like GPT-4V. Reviewers appreciated the training-free nature of the method and the introduction of new evaluation metrics (Factuality and Coverage) designed specifically for detailed captions, noting that the Factuality metric showed strong correlation with human judgments.

Reviewers identified several weaknesses and areas needing clarification. They questioned the precise contribution and necessity of the Coverage metric, observing that the CapMAS method often improves factuality at the expense of coverage due to its strategy of removing, rather than rewriting, hallucinated content – an inherent trade-off the authors acknowledged. Some reviewers also felt the evaluation datasets (IIW-400, DOCCI subset) were relatively small and suggested exploring broader benchmarks (for example DCI or PixelProse). Furthermore, reviewers noted similarities between CapMAS components and the proposed metrics to concepts in prior work (e.g., Woodpecker, ALOHa, POPE, Self-Refine), suggesting the novelty lies more in the effective combination and application to hyper-detailed captioning rather than entirely new techniques.

Overall, I recommend acceptance. But please take into account the reviewer comments seriously. Especially:
- better comparison with prior work
- inclusion of the rebuttal experiments into the main paper (datasets, calculations, fluency, qualitative analysis)
- better justification of the key insights and the important parts of the multi-step system
- more explanations of the cost and scalability of the CapMAS approach vs other options
Decision: Accept (poster)