Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning
Authors: Qinghao Ye
Xianhan Zeng
Fu Li
Chunyuan Li
Haoqi Fan
Venue: ICLR 2025 Poster (2025)
Keywords: Vision-Language Model
Detailed Image Captioning
Caption Metric
Alignment
Preference Optimization
Large Language Models
Primary area: datasets and benchmarks
TL;DR: We introduce DeCapBench and a novel evaluation metric DCScore for detailed image captioning evaluation. While with a fine-grained feedback collection method enabling the model to achieve superior performance across various tasks.

Abstract: Image captioning has long been a pivotal task in visual understanding, with recent advancements in vision-language models (VLMs) significantly enhancing the ability to generate detailed image captions. However, the evaluation of detailed image captioning remains underexplored due to outdated evaluation metrics and coarse annotations. In this paper, we introduce DeCapBench along with a novel metric, DCScore, specifically designed for detailed captioning tasks. DCScore evaluates hallucinations and fine-grained comprehensiveness by deconstructing responses into the smallest self-sufficient units, termed primitive information units, and assessing them individually. Our evaluation shows that DCScore aligns more closely with human judgment than other rule-based or model-based metrics. Concurrently, DeCapBench exhibits a high correlation with VLM arena results on descriptive tasks, surpassing existing benchmarks for vision-language models. Additionally, we present an automatic fine-grained feedback collection method, FeedQuill, for preference optimization based on our advanced metric, demonstrating robust generalization capabilities across auto-generated preference data. Extensive experiments on multiple VLMs demonstrate that our method not only significantly reduces hallucinations but also enhances performance across various benchmarks, achieving superior detail captioning performance while surpassing GPT-4o.

Published as a conference paper at ICLR 2025
PAINTING WITH WORDS: ELEVATING DETAILED IM- AGE CAPTIONING WITH BENCHMARK AND ALIGNMENT LEARNING
Qinghao Ye', Xianhan Zeng", Fu Li, Chunyuan Li, Haoqi Fan ByteDance Research
ABSTRACT
Image captioning has long been a pivotal task in visual understanding, with recent advancements in vision-language models (VLMs) significantly enhancing the ability generate detailed image captions. However, the evaluation of detailed image captioning remains underexplored due to outdated evaluation metrics and coarse annotations. In this paper, we introduce DECAPBENCH along with a novel metric, DCSCORF, specifically designed for detailed captioning tasks. DCSCORE evaluates hallucinations and fine- lne-grained comprehensiveness by deconstructing responses into the smallest self-sufficient units, termed primitive information units, and assessing them individually. Our evaluation shows that DCSCORE aligns more closely with human judgment than other rule-based or model- madel-based metrics. Concurrently, DECAPBENCH exhibits a high correlation with VLM arena results descriptive tasks, surpassing existing benchmarks for vision- language models. Additionally, we present an automatic fine-grained feedback collection method, FEEDQUILI for preference optimization based on our advanced metric, showing robust generalization capabilities across auto-generee sito-generated preference data. Extensive experiments on multiple VLMs demonstrate that our method not only significantly reduces hallucinations but also enhances performance across various benchmarks, achieving superior detail captioning performance while surpassing GPT--4o. We release the evaluation code and the model on Github',
INTRODUCTION
Vision-Language Models (VLMs) (Zhu et al., 2023; Liu et al., 2024b; Ye et al., 2023; Bai et al., 2023) have risen prominence by integrating the strengths of pre-trained large language models (LLMs) and vision models, leveraging large-scale multi-modal corpora (Liu et al., 2024b; Dai et al., 2023; Li et al., 2024a). These models have demonstrated remarkable capabilities across diverse array of tasks. To assess their visual understanding capability, numerous benchmarks have been developed, focusing on question- answering tasks, such as MMVet (Yu et al., 2023), MMStar (Chen et al., 2024a), and MMMU (Yue et al., 2024). However, these benchmarks often rely manually defined queries and questions, which may only cover a limited domain and lead to biased evaluations (Chen et al., 2024a). Additionally, Chen et al. (2024a) highlights that poorly constructed questions could make the models rely more on textual knowledge from their training data, thus neglecting actual visual input.
question-answering
In this context, the image captioning has been fundamental task to evaluate the visual perception capabilities of VLMs. Yet, traditional image captioning benchmarks suffer from two significant limitations: (1) The evaluation metrics ( (Vedantam et al., 2015; Papineni et al., 2002; Lin, 2004; Hessel et al., 2021) are unreliable and show low correlation with human judgment and model capability, and (2) The captions are typically short and lack informative visual details, missing fine-grained descriptions. In contrast, modern VLMs are capable of generating hyper-detailed image captions rich in fine- ane-grained visual information (OpenAL., 2024a; Liu et al., 2024b). These models can even extend and infer non-descriptive elements, which are often not covered by the conventional short ground -ruth captions, leading to unsatisfying detail caption evaluation results. Additionally, many of the existing image captioning datasets (Lin et al., 2014; Sidorov et al., 2020) focus on short captions

---

Published as a conference paper at ICLR 2025
and have become outdated, necessitating more rigorous evaluation framework for modern VLMs. To address these limitations, it is crucial to develop new benchmarks and evaluation metrics that align closely with human judgment and accurately reflect the advanced capabilities of modern VLMs.
In this paper, we aim to assess the capabilities of modern VLMs in producing detailed image captions. We introduce novel metric, DCSCORE, and comprehensive evaluation benchmark, DECAPBENCH, designed to address the challenges of hallucination and fine-grained comprehensiveness! in image cap- tioning. Our approach involves breaking down captions into the smallest self-sufficient units, termed primitive information units. This decomposition reduces ambiguity and enhances the transparency and interpretability of the evaluation process. By individually assessing these units, we can accurately measure both descriptive and non non-descriptive parts of captions with fine granularity. Additionally, decomposing captions allows us to evaluate their coverage with high-quality, hyper-detailed reference captions. Our experiments reveal that DCSCORE achieves the highest consistency with human expert evaluations, outperforming all existing rule-based and model- metrics. Furthermore, we present DEC DECAPBENCH as detailed captioning dataset that excels in measuring hallucination and fine-grained comprehensiveness It demonstrates superior correlation with the VLM description tasks compared to other benchmarks such as MMVet and MMStar.
In addition, we embrace the concept of breaking down responses into primitive information units and introduce FEEDQUILL, aine-granned feedback collection strategy for preference optimization. Specifically, we generate several candidate responses and decompose them into verifiable statements. Using open-source VLMs (Liu et al., 2024a; Chen et al., 2024b), then validate the correctness of these statements and calculate a preference score to measure precision. To avoid bias towards overly concise responses, we also factor in the number of primitive information units feedback signals. Leveraging proximal policy optimization (PPO) (Schulman et al., 2017), we optimize preferences using a reward model trained on the collected preference data. Extensive experiments demonstrate that FEEDQUILL consistently enhances performance across various VLM models on both comprehensive and task-specific benchmarks, significantly reducing hallucinations by 40.5% relative points in mmHal-V. V. Furthermore, our model not only outperforms GPT--40 in detailed image captioning but also exceeds GPT-4V in visual chatting, underscoring its potential and effectiveness.
The contribution of this work can be summarized as: (1) We present DCSCORE, novel metric for image detail caption evaluation with both hallucination and comprehensiveness, and it achieves the highest consistency with human experts among existing caption metrics. (2) We introduce new detailed caption benchmark DECAPBENCH for evaluating the captioning capability of modern VLMs, which has the highest correlation with human judgement on description task compared to other public benchmarks. (3) We propose a simple but effective fine-grained feedback collection method FREDQUILL by decomposing responses into primitive information units and verify them individually, which is scalable for automatically collecting preference data. (4) Extensive experimental results demonstrate the efficacy of FEEDQUILL, showing reduced hallucinations, superior performance in visual chat compared to GPT-4v, and better detailed image captioning capabilities than GPT-4o.
2 RELATED WORK
Image Captioning Evaluation Metrics Image captioning tasks are fundamental to visual-language understanding, as they assess a model's ability to comprehend and describe images accurately. Modern vision-language models (Ye et al., 2024; Chen et al., 2024b; Liu et al., 2024a; Bai et al., 2023) equipped with massive data pre-training, are capable of generating diverse and detailed image captions. Despite these advancements, evaluating captions accurately and comprehensively remains challenging. Traditional metrics, such as BLEU (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005), and CIDEr (Vedantam et al., 2015), leverage N-gram and lexical similarity with human- annotated captions but suffer from instability due to variability in phrasing. To address this issue, model-based metries like SPICE (Anderson et al., 2016) and CAPTURE (Dong et al., 2024) parse captions using scene graphs to match ground-truth captions. Additionally, CLIPScore (Hessel et al., 2021) and PACScore (Sarto et al., 2023) utilize pre-trained vision-language models like CLIP (Radford et al., 2021) to measure the similarity between images and captions, well as between generated and reference captions. Recently, researchers have leveraged the powerful zero-shot capabilities of large language models (LLMs) to prompt LLMs for assessing the alignment between model-ga -enerate and aaman--nnotated captions (Chan et al., 2023; Lee et al., 2024; Liu al., 2024b). Despite their potential, L.LM-based evaluation methods face challenges in maintaining
2

---

Published as a conference paper at ICLR 2025
objectivity and comprehensiveness, particularly in extending evaluation to aspects such knowledge and atmosphere. To alleviate these problems, we propose DCSCORE, a novel image caption metric that evaluates image captions by incorporating both hallucination and comprehensiveness thoroughly.
Learning from Feedback for VLMs Learning from feedback (Yu et al., 2024a; Sun et al., 2023; Zhou et al., 2024a;b) is a core technique in the post-training stage of vision language models (VLMs). This approach enhances model performance on various tasks, such as question answering (Yue et al., 2024; Liu et al., 2023; Chen et al., 2024a) and reducing hallucinations (Li et al., 2023b), through alignment learning techniques like PPO (Schulman et al., 2017), DPO (Rafailov al., 2024), and RLOO (Ahmadian et al., 2024). The quality of feedback is crucial for aligning models with human preferences. Early works, such LLa ALaVA-RLHD (Sun et al., 2023) and RLHF-V (Yu et al., 2024a), relied heavily on human--ntensinn in model responses. To alleviate the demand for intensive human labeling, various approaches (Li et al., 2023a; Zhao et al., 2023; Yu et al., 2024b) have been proposed to collect or construct feedback with preferences automatically. For instance, Bai et al. (2023) prompt GPT-4v (OpenAI., 2024b) to collect preference pairs and distill them into a pre-trained VLM. While this method offers ease and convenience, the preference judgment of GPT-4v is not manually verified, posing risks of bias and unreliability. Approaches like HA-DPO (Zhao et al., 2023), POVID (Zhou et al., 2024a), and STIC (Deng et al., 2024) perturb the image and text prompts or inject false statements into model responses to heuristically construct preference pairs. Other techniques, such as RLAIF-V (Yu et al., 2024b) and CSR (Zhou et al., 2024b), employ self-rewarding mechanisms to attain correctness scores or vision- language alignment scores for preferences. In contrast, we propose a fine-grained, verifiable feedback approach that links specific categories of undesired behavior (e.g., false or irrelevant responses) to detailed text spans (c.g., sentences or sub-sentences), which provides more generalizable and reliable automatic feedback for improving learning through feedback.
labeling to collect high-quality feedback and correct mistakes
Inage& Quary
Caption wir
(1) Decomposition
(3) Veriffcation
The me teak art wnita alting on pluatic ticde ras dine The poatiooed
Prcso droocbe
cale vidlo.
appears
sabl planti
Matching
Deeaaeicceee Cheehee
Hypen-Detalled Human Caption dik
alamp.
a dock
in the pfitt The wnih biss neve blss soree and tas mucid brown underziee, art an--itaped and has disdrct ytrr rigi araund
a fiedd
The or striig 2n1 tabls Tie plack.
stable.
ussders
Figure I: Overview of the proposed DCSCORE for evaluating detailed image captioning. (1) Given the image and prompt, model generated responses and human written responses are decomposed into sets of primitive information units. (2) We match the primitive information units of generated response P and those of human written response O. (3) Each primitive information unit in P is verified individually by VLM given the content of images.
DECAPBENCH IMAGE CAPTIONING TESTBED FOR MODERN VLMS
Recent open-source VLMs have been significantly improved, narrowing their performance gap compared with GPT-4V on various benchmarks. However, this progress does not always translate into better image captioning abilities. The issue lies in the fact that while current VLMs can generate detailed captions with many fine-grained elements, existing metrics rely on coarse-grained ground- truth captions that overlook these details. Furthermore, traditional automatic evaluation metrics show lower correlation with human evaluations, raising questions about their effectiveness. To address these limitations, we propose DECAPBENCH, a new image captioning evaluation benchmark, along with a novel metric DCSCORF, as illustrated in Figure 1, that better captures the descriptive capabilities of VLMs. Our metric ensures that model rankings align more closely with results from the VL.M arena, which is based on diverse, crowd-sourced user votes for image description tasks.
3.1 DCSCORE EVALUATION METRIC
Previous image caption evaluation metrics (Papineni et al., 2002; Vedantam et al., 2015; Banerjee & Lavie, 2005; Hessel et al., 2021; Anderson et al., 2016) are designed for short caption evaluation.
3

---

Published as a conference paper at ICLR 2025
When applied to detailed captioning, these metrics suffer from limitations such as low-quality and uninformative annotations, as well as biased captioning patterns, resulting in failures to adequately assess hallucinations and the comprchensiveness of captions generated by VLMs. To address this issue, we propose DCSCORF, a novel metric for detailed image captioning that accounts for both hallucinations and fine-grained comprehensiveness. DCSCORE evaluates the quality of image captions by generating and assessing primitive information units, which are the smallest self-sufficient units of information within caption. This method reduces ambiguity and enhances the transparency of the evaluation process. The evaluation process consists of three steps, described as following.
Step 1: Decomposition. The extraction of primitive information units involves splitting the model- generated caption into distinct components, which can be done either manually or by large language model (LLM). For the ground-truth caption, we use human experts to decompose it into a set of primitive information units, denoted as O {01, 02. OMM 1. where M is the total number of extracted units. On the other hand, we prompt the LLM to decompose the model-generater caption on centence-by-sentenee basis into a set p {pI. P2, pN), where N represents the number of units extracted from the model's description. Since image captions can include elements that are not directly descriptive of the image, which may influence the overall quality and style of the caption, it is essential to evaluate these non-descriptive elements as part of the VLMs' captioning capabilities. To differentiate between descriptive and non-descriptive units, we prompt LLMs to perform a binary classification for each unit p: E P during decomposition. Detailed instructions for extracting primitive information units can be found in the Appendix.
Step 2: Matching. High-quality the reference captions without omissions. To evaluate this, we prompt L.LMs to assess whether each primitive information unit ps P from the generated caption is mentioned or can be logically inferred from the reference caption oj E O. The matching process is formally computed as Q = P n O, where the overlap of primitive information units between the generated and reference captions.
captions should incorporate all key elements from
Step 3: Verification. To verify the correctness of the primitive information units pt in the generated captions P. we use modern VI.Ms. Specifically, we employ GPT-4o (OpenAI., 2024a) to assess the accuracy of each unit by referencing the corresponding image. GPT-4o is prompted to provide a simple "yes" or "no" answer regarding the correctness each unit, without requiring further explanation, following the approach used by Li et al. (2023b).
After obtaining the model-generann set P, the reference set O, and their overlap Q, we compute both precision score sp hon-haalucination) and a recall score s, (comprehensiveness) as follows:
Pirrue PI
[ + Perue \ Q |
8p
Perre \ Q
where Perue {pslp: E P,pp is correct) represents the set of correct units in the set P.
We assess the overall caption quality using the Fl score sy, which balances the precision score Sp and recall score 8,. Additionally, we evaluate the descriptive elements of the caption by computing the Fl score syy for only the descriptive units. The final assessment score F is computed as:
F F= (
3.2 DECAPBENCH: A DETAILED IMAGE CAPTIONING EVALUATION BENCHMARK
Dataset. We consider the recently released Imageln Words dataset (Garg et al., 2024), and leverage 400 high-quality, human-curated public image detailed captions from as the ground-truth captioning. Compared with ImageIn
traditional caption datasets such as COCO (Sidorov et al., 2020; Lin et al., 2014; Agrawal et al., 2019) often contains short, coarse coarse-grained captions, and lack detailed information, making them inadequate for measuring the correctness and comprehensiveness of the models' generated detailed captions. In contrast, ImageIn Words considers human-in-thhe loop framework produces hyper-detailed and hallucination-fie image descriptions, by combining human annotators and seeded machine generations. Consequently, we constructed DECAPBENCE
4

---

Published as a conference paper at ICLR 2025
Metric
PCC (p)
1 12
Kd 7
Sp
Rule-Based Evaluation
BLEU-4 (Papineni et al., 2002)
0.3439
62.78
0.2693
0.2931
ROUGE (Lin, 2004)
0.2509
156.05
0.1886
0.1893
METEOR (Banerjee & Lavie, 2005)
0.3593
111.95
0.2417
0.2536
CIDEr (Vedantam et al., 2015)
0,0522
3.3e7
0.0635
0.0601
Model-Baan Baaee- Evaluation
SPICE (Anderson et al., 2016)
0.2218
156.11
0.1731
0.1907
CLIP-Score (Hessel et al., 2021)
0.2183
26.04
0.1724
0.1480
PAC-Score (Sarto et al., 2023)
0.1525
20.93
0.1117
0.1260
CAPTURE (Dong et al., 2024)
0,3521
7.62
0.2801
0.3449
CLAIR (Chan et al., 2023)
0.3815
1.98
0.3847
0.4552
FLEUR (Lee et al., 2024)
0.4230
3.01
0.4246
0.5325
GPT4-Eval (Liu et al., 2024b)
0.3976
2.95
0.3447
0.3866
Faithscore (Jing et al., 2023)
0.1937
3.22
0.1626
0,1115
RLAIF-V (Yu al., 2024b)
0.3547
5.32
0.2774
0.2544
DCSCORE
0.6605
1.54
0.5328
0.6166
Table I: Correlation of image captioning evaluation metrics and human judgements. All p-values < 0.001. The bold number indicates the highest human consistency among caption metrics.
by applying the proposed DCSCORE evaluation metric to the ImageInWords images and their corresponding hyper-detailed image captions.
Human consistency of DCSCORE. To demonstrate consistency with human expert judgments, we randomly selected 500 captions generated by different models and employed X experienced annotators to rate each caption. We then computed the statistical metries to compare the proposed DCSCORE with human ratings, including the Pearson correlation coefficient (PCC) p. coefficient of determination R², Kendall's r (Kd 7) and Sample-wise т (Sp r). The correlation statistics, as presented Figure 2 (Left), highlight the significant improvements brought by our proposed metric, DCSCORE. Compared to the state-of-the-art DCSCORE enhances PCC p by 0.2375 and boosts Kendall r by 0.10 These advancements suggest that our metric achieves superior linear correlation and pairwise ranking accuracy with human judgments. Hence, DCSCORE holds great potential for optimizing detailed captions produced by VLMs.
High-quality and hyper-detailed image descriptions are crucial for evaluating model-generated captions, as they closely mirror the content the image. To investigate this, we assess the impact of varying quality of ground-trute truth descriptions our proposed DCSCORE. As shown in Figure 2 (Left), descriptions with finer granularity achieve higher consistency with human judgments compared COCO-style concise captions. Specifically, detailed captions annotated by either humans or GPT-4o (OpenAI., 2024a) demonstrate superior alignment with human evaluators, highlighting the importance of granularity in image description for more reliable and accurate evaluation.
Source of Captions
PCC (p) f
R²
Kd
Sp
COCO-Style
0.5468
14.10
0.4375
0.5093
Instruct-BLIP
0.6062
5.50
0.4745
0.5620
DuCoppanch
6.5
GPT-4o
0.6497
0.5194
0.5745
Human Annotated
0.6605
1.54
0,5328
0.6166
0.60
0.00
0.25
0.71
0:41
0.89
3.00
Figure 2: (Left) Comparison of four sources for ground-truth captions in terms of correlation between DCSCORE and human judgments. All p-values are less than 0.001. (Right) DECAPBENCH achieves the highest correlation with Arena Elo, with a Spear- man's 's correlation of 0.90 among different VLM benchmarks.
N80
0.79
0.84
0.91
0.52
0.84
0.72
0.68
0.58
Sobuskobochh
0.56
0.47
0.69
0.52
0.82 0.72
0000
0.35
0.26
0.45
0.53 0.63 0.81 0:43
0.67 0.64
0.33
00
0:55
MMBeec cv 0.59 0.62 0.80 0.33
0.63 0.60
6.36
1.85
1:00
0.84
CCllcch 0.47 0.44 0.64 0.25
0.55
0.58
1.48
0.69
0:34
1.00

---

Published as a conference paper at ICLR 2025
Human consistency of DECAPBENCH. To further study the consistency between the proposed DECAPBENCH and human judgement in the wild, we select the subset of image description from the VLM arena, and compute the ranking correlation. Note that VLM arena is a public VLM evaluation platform, where two model responses for the same task prompt are voted by humans to reflect their preferences. Specifically, we compute human preferences using Elo ratings, derived from over 1, 1,000 pairwise comparisons with around 800 images across 13 different VLMs on image captioning tasks.
In Figure 2 (Right), we visualize the Spearman correlation heatmap among various automatically evaluated multi-modal benchmarks (Chen et al., 2024a; Liu et al., 2023; Yue et al., 2024; Kembhavi al., 2016) and human-voter preference benchmarks (Lu et al., 2024). From the figure, we observe that DECAPPENC achieves the highest correlation with Arena Elo at 0.90, indicating a high level of alignment with human preferences and strong consistency in ranking. This high correlation demonstrates the effectiveness of DECAPBENCH in capturing the nuances of human judgment, making it a reliable benchmark for evaluating the image captioning capabilities of VLMs.
Compared with existing multimodal benchmark, the proposed DECAPBENCI is unique in its dedication to the task of detailed captioning, verified by the highest correlation with Arena captoin subset. Note that MM (Yu et al., 2023) evaluates the models' ability to solve complex vision- language tasks. MMMU (Yue et al., 2024) and Vista (Lu et al., 2023) assess subject knowledge and mathematical reasoning in visual contexts, respectively, while HallusionBench focuses understanding visually misleading figures. The MMBench-seriees (Liu et al., 2023) (e.g., MMBBnnh- EN, MMBench-CN, and CCBench) concentrates on fine-graings perception and reasoning tasks using multiple-choice questions. Additionally, MMStar (Chen et al., 2024a) corrects the misjudgments of actual multi-modal performance.
4
LEARNING FROM FINE-GRAINED FEEDBACK FINE-GRAINED FEEDBACK COLLECTION
4.1
The feedback collected for preference learning consists of comparison pairs, where each pair includes preferred response and less preferred response to the same input. The model learns from this preference data to distinguish differences among its own generated candidate responses. To gather these candidate responses, generate multiple outputs for given images and prompts using nucleus sampling (Holtzman et al., 2019), varying the random seed to ensure diversity. By learning to rank these candidate responses based on the preference data, the model becomes capable of assessing the quality of its outputs and deriving appropriate signals for preference optimization.
However, judging the quality of different responses is complex, even for experienced human annota- tors (Sun et al., 2023), due to the semantic intricacies involved. Previous methods (Zhou et al., 2024a; Zhao et al., 2023) attempted to address this by manually modifying responses and injecting noise to create negative samples. However, these approaches suffer from poor generalization because of implicit patterns in the data. In contrast, by adapting the concept of primitive information units and step-by-step verification (Lightman et al., 2023), we propose FEED QUill for feedback collection, which leverages modern VLMs to generate fine-grained feedback in the following three steps:
Decomposition.
We prompt an LLM to decompose the response into a set of N primitive information units {p. IN on seateecce-b-senteene basis, rewriting them into self-sufficient and verifiable statements.
Scoring. We several powerful VLMs (Chen al., 2024b; Liu al., 2024a) to ver- ify these rewritten statements using the prompt: (STATEMENT) Is the st statement correct? Please only answer 'yes' or 'no"" ", To increase confidence in our judgments, we ensemble the results from multiple open-source VLMs for verification.
Preference. After obtaining the verification results for each primitive information unit, we calculate the preference score Co as the fraction of correct units: Cp E where a higher score indicates fewer hallucinations in the response. Given the scores of each response, we construct preference dataset D (Ii,, of "9 ) by treating the response with the higher score as the preferred response yi and the one with the lower score as the non-preferred response V.-
1pp 1}.
As discussed in Zhu et al. (2023), responses with fewer hallucinations are often inherently less helpful. Specificallyy, models are more likely to hallucinate when producing longer responses compared to

---

Published as a conference paper at ICLR 2025
shorter ones. To address this issue, construct preference dataset D, using the number of primitive information units as the preference score G,. A response with a higher score c, - indicating more primitive information units is considered more preferable. This approach encourages the model to generate responses that are not only accurate but also rich in helpful and detailed information.
4.2 PREFERENCE OPTIMIZATION
Preference optimization (Ouyang et al., 2022; Rafailov et al., 2024) has shown promise in fine-tuning language models and aligning their behavior with desired outcomes. Specially, we train the reward model rg with the preference set 'D and D, respectively, with the pairwise comparison loss (Ouyang et at., 2022) as LRM
P(=..t.w~>** [log (arrr y²) r y "))], where (-)) is the sigmoid function and (.) is the output score the reward model. To mitigate biased preferences, such as unhelpful responses, we opt against using a single scalar reward to represent response quality. Instead, we leverage rewards derived from multiple reward models, each contributing to distinct behaviors like hallucination (Cp.) and richness (c,). To optimize these preferences, we utilize proximal policy optimization (PPO) (Schulman et al., 2017), a widely adopted reinforcement learning algorithm. To fully exploit the characteristics of preferences related to hallucination and comprchensiveness, select captioning as the optimization task. For additional details, please refer to the Appendix.
5
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Model. We conduct our experiments based on series of LLaVA models (Liu et al., 2024b) with different sizes and capabilities. We initialize both the policy model and reward model with same parameters as well as same size for validating the effectiveness of our proposed method. For the main results, we report the performance of our model FEEDQUILLL-TD trained on LLLVA--Oee L.AA-OOvviiion-7B. one of the most capable models in the < 10B size category.
Training Dataset for PPO. The PPO is performed with the detailed captioning task. To ensure the model learns robust generalization capabilities, diversity image distributions is crucial. Therefore, we randomly sample images from wide range of datasets, including MSCOCO (Lin et al., 2014), Openlmages (Kuznetsova et al., 2020), and ShareGPT4V (Chen et al., 2023). Additionally, to maintain diversity of instructions during training, prompt GPT-4o (OpenAL, 2024a) to generate variety of caption prompts, and provide in Appendix.
5.2 ABLATIONS
Preference Data for Reward Model. To assess the ability of various preference data to generalize, we trained multiple reward models using the same SFT model. For evaluation, we randomly sampled portions of the preference data that were held out. The findings, presented in Table 2, reveal that our model achieved the highest accuracy across diverse preference datasets. Notably, with the same scale of training data, our reward model outperformed the human-labeled dataset RLHF-V by 9.9% in accuracy. also surpassed the RLAIF-V dataset, which, despite having over 80k training samples, was outperformed by our model that utilized smaller data size. Additionally, we observed that increasing the amount of training data led to an improvement average accuracy from 71.3% to 75.2%. highlighting the scalability of our approach.
Preference Data for Preference Optimization. We delve into how varying types of preference data impact preference optimization. Using -LaVA--15577 as our baseline model, we trained it with variety of preference datasets. The performance of these models of downstream benchmarks in a zero-shot context. As showcased in Table 3, our approach not only excels in captioning performance but also substantially cuts down hallucinations, achieving notable 0.75 improvement on mmHal-V compared to the baseline.
then assessed through range
Data Size. We scale up the training set of the reward model, and investigate the correlation between downstream performance through preference optimization. We evaluate different checkpoints ranging
7

---

Published as a conference paper at ICLR 2025
Held-Out Dotaset
Train Data
HA-DPO
RLHF-V
POVID
CSR
RLAIF-
STIC
Average
HA-DPO (Zhao et nl., 2023)
93.5
81.1
23.7
53.5
51.0
42.0
57.5
RLHP V et al., 2024a)
82.0
94.7
30.7
44.2
48.7
67.8
61.4
POVID (Thou al,, 2024s)
32.5
30.6
99.5
39.4
52.5
59.5
55.7
CSR (Zhou et al., 2024b)
60.3
87.5
23.6
RLAIE-V et 2024b)
69.5
77.6
55.5
66.8
64.5
STIC (Deng 2024)
48.0
59.7
26.8
43.3
50.1
99.9
54.6
PEEDQUILL* PEEDQUILL
78.0
64.1
87.4
59.7
64.7
74.1
71.3
76.5
71.9
93.2
55.2
69.4
84,9
75.2
Table 2: Reward model zero-shot accuracy on the held-out validation set trained with different preference data LLa VA-1.5--B. * indicates that we only utilize 10k preference data to match the size of other training set.
Method
MMBasch
VizWaz
MMS
WildVision
LLaVA-W
DECAPBENCH
mmHal-V
CHAIKS 1
CHAIRI
LLaVA-L5
64.8
50,0
33.1
11.18
65.3
24.50
1.85
47.8
25.3
w/ HA-DPO
64.3
33.5
15.17
22.45
2.12
49.3
25.5
w/ POVID
64.7
47.9
35.4
13.25
71.5
23.54
1.90
31.8
5.4
w/ RLAIF-Y
62.7
50.9
34.7
15.65
76.0
28.21
2.59
8.5
4.3
PuraQQuIL
66.3
358
19.68
34.52
5.1
Table 3: The performance of different preference data on LLa across different benchmarks.
from 5,000 to 200,000 training samples, using models of sizes 7B and 13B. The results are illustrated in Figure As the size of the preference data increased, the performance of mmHal-V improves from 2.05 to 2.6. Similarly, MMStar, which focuses on image understanding, shows consistent increase from 34.7 7 35.8, yielding a 1.1 point lift. This demonstrates that as the size of preference data for the reward model grows, the model's performance consistently improves since the better reward model provides more accurate signals for preference optimization.
2
2
1230
300
Dara 5:00
Sara Size
Cata Site
Figure 3: Impact of the preference dataset size in terms of downstream performance.
Source of Responses. We explore the effect of the source of model responses on preference data, based on the hypothesis that improvements might arise from the model's ability generalize across varying sources. To test this hypothesis, we use LLaVA-1.5-13B as the base model and examine responses sampled either from the same model from other models such as LLaVA-115-7BB LLaVA- 1.6-7B, and LLaVA-1.66 1..VA-1.--13. 13B.. Furthermore, we assess the impact of combining responses from these different sources. The results of these experiments are summarized Table 4. We observe that integrating responses generated by the same model only leads to significant performance boost compared to the baseline model. Conversely, integrating responses from different models only leads to larger performance gains DECAPBENCH by providing diverse responses, while smaller gains on other benchmarks. When combining responses from both sources, the model achieves superior performance, surpassing the use of either source alone. Specifically, this combination results in improvement 13.0 points on LLaVA-W W and 13.23 points on DECAPBENCH compared to baseline.
Source of Rewards. Table provides comparative analysis of incorporating the preference score for the number of primitive information units (c,) alongside the preference score for the proportion of correct units (Cp). Each preference score is obtained separately from different reward models, summed to final reward in PPO training procedure. We specifically evaluate our method against three distinct variants: (1) the base model without any preference optimization (Base); (2) model optimized solely with the ep score (Only cp); and (3) a model optimized exclusively with the c, score (Only c,)). This comparison enables thorough examination of the impact of each optimization strategy on model performance. Notably, models trained with the score consistently enhance performance both VA- and DECAPBENCH. Conversely, models trained with the e, score
8

---

Published as a conference paper at ICLR 2025
Same Model
Other Models
Response
MMSSar
LLaVA-W
mafal-V
DECAPBENCH
Methed
LLaVA-W
LLAV- 5-78
DeCAFRENCH
ELaVA-W
LLVA-IS-TTR
DECAFBENCH
26.50
Base
65.3
24.50
72.8
25.55
37.5
75.1
2,74
26.32
c
18.0
21.5
2.53
34.84
Only o
46.2
10.03
56.9
15.11
38,3
37.73
76,0
34,52
78.3
37.73
Table 4: Comparison of performance by varying sources of preference data.
Table 5: Ablation of using different reward scores during preference optimization.
yield poorer results on both datasets due to the absence of precision constraint. Furthermore, when both by and Gr are incorporated, our method exhibits significant improvements, notably 10.7% increase on LLaVA- 1A--.B and 5.5% boost on
Comprehensive Benchmark
Visual Hallucinaion
Visual Chat and Capsioning
Method
MMBench
MMStar
VizWiz
SaiQ
mmHal-V
LLaVA-W
WildVision
DECAPBENCH
L1aVA-155-78
64.8
33.1
50.0
66.8
1.85
65.3
14.48
24.50
PERDQUILL
(4.)
35.8 c2m
55.2 (43))
620
2.60 wamm
76.0.mm wmm
(1000)
LLAAA-15-138
68.7
34.3
53.6
71.6
2.33
728
16.17
25.55
PERDQUILL
69.2
38.3
568
73.4 w3.
2,83
78.3 uan
18.15
37,73 612.0
LLaVA-L6-78
67.1
37.6
57.6
70.2
2,58
79.8
26.15
35.74
FEEDQUILL
Lan
38.6 k..
63.4 van
wan
2,93 weues
41.16 wason
(1609)
LLaVA-L6-13B
69.3
40.4
60.5
73.6
2.95
85.2
33.69
36.28
PEEDQUILL
69.9
41.3 uam
66.7 (a
73.5 wuu
3,76
87.1 1as
49.69
$3.26
LLaVA-Onevisier
80.8
61.7
60.0
96.0
2.94
90.7
$1,50
43.19
FEEDQUILL
62.4 (um
60.4 (un
95.9
3.10 m00
100.5 (uin)
55.65 (13.14)
Table 6: Performance of FEEDQUILL with various VLM models on downstream tasks.
Compatibility Analysis. To validate the applicability of FEEDQUILL across various VLMs, we conduct experiments on various models. The summarized results in Table reveal that FEEDQUILL is effective regardless of model size, consistently enhancing performance on downstream tasks such MMBench, mmHal-V, and DECAPBENCH. This underscores the robust generalization capability of our proposed FEEDQUILL. Notably, LLaVA-1 1.44--66--- 1.6 16611B trained with FEEDQUILL exhibits large improvement on mmHal-V, increasing from 2.95 3.76. Simultancously, it significantly boosts performance on WildVision and DECAPBENCH, with gains of +16.0% and + 16.98% respectively.
5.3
MAIN RESULTS
Model
AIDD CharQAA MMBench SEEDBench
MME
MMMU MMVer MMSiur SaIQA LLaVA-W WildVisson DECAPBENCH
Proprietory Model
94.7
90.8
78.5
+
68.3
75.4
60.2
50.5
102.9
50.00
52.37
Geminn-11.-Prr
94.4
87.2
73.9
-1-
62.2
64.0
35.45
GPT-VV
78.2
78.5*
79.8
49.9
14091517
56.8
57.1
75.7
75.7
98.0
80.01
48.52
GPT-40
76.2
+/-
76.2
83.5
196.1
89.41
Open Sowrce Model
Camhrian-34B
79.7
73.8
81.4
-|-
49.7
53.2
85.6
67.8
35.12
VILA-40B
82.4
75.8
1762
51.9
51.2
54.2
38.02
XComposer225..7
81.5
82.2
82.2
75.4
2229
42.9
51.7
59.9
78.1
Intem/LL-2-BB
83.8
83.3
81.7
76.0
2210
49.3
60.0
59.4
97.0
81.5
15.55
IntennVL-2-268
84.5
84.9
83.4
76.8
2260
48.3
65.4
60.4
97.5
99.6
49. 59.59
LavA--Onvsssnnn778
81.1
80.0
80.8
75.4
1580K118
45.8
57.5
61.7
96.0
90.7
$1.50
43.19
FEEDQUILL-7B
81.3
80.3
80.5
75.8
15151450
47.9
59.3
62.4
95.9
100.5
$9.60
55.65
Table 7: Main experimental results of our method and other open-sourced state-of-the-art VLMs. *GPT-4V reports 4-shot results ChartQA. All results presented in the 0-shot setting.
We evaluate FREDQUILL-7B across a variety of multi-modal large language model benchmarks, including A12D (Kembhavi et al., 2016), ChartQA (Masry et al., 2022), MMBench (Liu et al., 2023), SEEDBench (Li et al., 2024b), MME (Fu et al., 2023), MMMU (Yue et al., 2024), MMVet (Yu et al., 2023), MMStar (Chen et al., 2024a), ScienceQA (Lu et al., 2022), LLaVA-W Liu et al. (2024b), Wild (Lu et al., 2024), and DECAPBENCH. These datasets are specifically designed to measure various capabilities of VLMs, including document understanding, question answering, visual chatting, visual perception, and detailed image captioning. Table 7 presents a comparative analysis of FEEDQUILII-BB against state-of-the-art VLMs, encompassing both proprietary and open-source

---

Published as a conference paper at ICLR 2025
models including Claude-3.5 3.5dle--.---nneet (Anthropic., 2024), Gemini-1.5- (Team et al., 2023), GPT-4v (OpenAI. 2024b), GPT-4o (OpenAI., 2024a), Cambrian- 34B (Tong et al., 2024), VILA-40BB (Lin (Chen et al., 2024b).
et al., 2024), XComposer.2.5-7B (Zhang et al., 2024), and
FEEDQUILL-7B achieves state-of-the-art performance in detailed image captioning, surpassing GPT-4o by 2.21 points. Remarkably, it also outperforms GPT--v on LLaVA-W, -WW showing strong capability in visual chatting. Despite being trained solely on the captioning task, our model maintains its strong performance while achieving 1.8-point improvement on MM Vet and a 0.7-point increase on MMStar compared to preference optimization a feat that many aligned models, such as BHDS (Amirloo al., 2024), CSR (Zhou et al., 2024b), and RLAIF-V (Yu et al., 2024b), fail to accomplish.
Additionally, it retains most of its capabilities after
5.4
CASE STUDY
LLaVA-OneVission778
dirryy if room group people are engrossed their phores. The person closest a black whirt, hokting a phone vibrant red screen, which is prominenty displyyed in the foreground. The phane, slock device with a small screen, held securaly their hand. further away, another person seen holding phone black screen,
postionedd upper right corner of the image. This phone, sightly larger than the one in the foroground, is also held in hand background, thore's person wearing a black screen, which postiooed in the upper left comner of the image. The phone slighly larger than the one foreground,
shirt, their hand holding phone
in
The room they're in is dinnly it, creating an atmosphare of facus and concentration, The people are all dressed moment modem
dark dothing, adcing the mood the scone. technology and social Interaction intersect.
Explain the visual content of the image in great detall.
FEEDQUILL-78 (This work)
The image is pholograph taken from an overheed perapactive, focusing ig on the hands and uppar bodkes of two individuals. The main sutjects are the hands of the duals which holding two different types mobile devices. The keypad, The person on the right holding Blackberrr device. which also open, disstaying is screen and keybbard The parson the wine glass. The inctvidual is wearing a black and white patternd garment, which could a scart or piece dothing with distnctive design. facus,
the holding flip
which open, revealing its screen
also holding glass containing red liquid, which appears be wine. The gloss traneperent and stem, suggesting
The beckground of the image dark and sourow of light camning from the screens the mobila devicis, which luninale hands and the glass wine.
attentinn to hands and the etjects they are haiding The lighting the dim, with the main
There are visibre saxts logos on the mobile devioos, and the image does not provide any additional context about the location the event taking place. The style the photograph is candid informal, capturng moment time without any staging posing. compostion the individualss and their devices, as well as the contrast between the modem technology and the traditonnl act of hoiding glass of wine.
image emphastaes Interaction between
Wrong Elements (Nolucinatiors)
Correct Bements
Additional Correct Elements
Figure 4: Qualitative results of FEEDQUMLLL78 compared with LLaVA-Oneviision-78 (Li et al., 2024a) in terms of image captioning.
We provide qualitative results of LI LLaVA-OOevisioon-B and FREDQUILI-7B in Figure 4 to illustrate the effectiveness of our proposed method. In the example above, identifies the red wine in the glasses as vibrant screen. In contrast, our model correctly identifies it as a red liquid with fewer instances of hallucination. Additionally, while LLaVA- -aaV---nvvisinnn78 generically names both phone as "cell phone", FEEDQUILL-7B specifically identifies them Blackberry device and a flip phone, showcasing its strong fine-grained captioning capabilities. We refer readers to the Appendix for more qualitative results.
incorrectly
6 CONCLUSION
We have described a novel metric, DCSCORE, designed to evaluate both hallucination and compre- hensiveness, the two critical challenges in detailed image captioning. Empirical validations show that DCSCORE achieves the highest consistency with human judgments, underscoring its reliability, Additionally, we present new detailed caption benchmark. DECAPBENCH, specifically for assessing the captioning capabilities of modern VLMs. Our results demonstrate that the correlation of DE- CAPBENCH with human judgment surpasses that of any other public benchmark in description tasks. Furthermore, we propose an effective fine-grained feedback collection method, FEEDQUILL. which decomposes responses into primitive information units for individual verification and subsequently learns improved model through preference optimization. Comprehensive experiments reveal that FEEDQUiLL is applicable across various models, achieving superior image captioning performance while reducing hallucinations, and setting new tfate-of-thh-art We believe that both DECAPBENCE and FEEDQUILL will serve as invaluable foundations for advancements in detailed image captioning and preference optimization.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale, In Proceedings of the IEEE/CVF international conference computer vision, pp. 8948-8957, 2019.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Ahmet Üsttin, and Sara Hooker. Back basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.147400 2024.
Elmira Amirloo, Jean-Philipppe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang. Zhe Gan, al. Understanding alignment in multimodal llms: A comprehensive study. arXiv preprint Xii:22407000477, 2024.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Ams- terdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 382-398. Springer, 2016.
Anthropic. The claude 3 model family: Opus, sonnet, haiku., 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 120.:22686...6.,... 2023.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
David Chan, Suzanne Petryk, Joseph Gonzalez, Trevor Darrell, and John Canny. CLAIR: Evalu- ating image captions with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 16638-133444 Singapore, December 2023. Association for Computational Linguis- tics. doi: 10. 12118866/111 10t33//11222.eere www.:3.mmmmpmmm...... URL http:: /atess//llllaaa httpa/caaaaaclocooooooooooooccca wwwpchholly..oor/o/..e or0/2023.. emn.pp 1o-maaioe in.iinn.. 841.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12799, 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330. 2024a.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far multimodal models with open-source suites, arXiv preprint arXiv:2404. 16821, 2024b.
gpt-4v? closing the gap commercial
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- Alice Oh, Tristan Naumann, Amir Globerson, Kate
language models with instruction tuning. Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. Enhancing large vision language models with self-traine self-training on image comprehension. arXiv preprint (4t.::2005.19116 2024.
Hongyuan Dong. Jiawen Li, Bohong Wu. Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Bench- marking and improving detail image caption. arXiv preprint arXiv:2405. 19092, 2024.
11

---

Published as a conference paper at ICLR 2025
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR, 2tts/2330.13394, 2023. doi: 10.48550/ ARXIV 110000611339 13394. URL https::///....... http//di.oggg/////////oo/oo org/10.............o 48550..5555aaaa 203. 13394.
Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, and Radu Soricut. Imageinwords: Unlocking hyper-detailed image descriptions. arXiv preprint DrXiv:2405.027933 2024.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hamlin Zhao, Hanyu Lai, Hao Yu, Hongming Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang. Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang. Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang. Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm- 130b to glm-4 all tools, 2024.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 3608-3617, 2018.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLiPScore: A reference-free evaluation metric for image captioning. In FraninFe Moens, Xuanjing Huang, cen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.180658 18653/v1/2021. emnlp-main.595, URL http::tt/tttt
Lucia Specia, and Scott Wen-tau
org/2021 emn p-pppmmmm ma n. 595.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv: 2019.
2019.
Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du Faithscore: Evaluating hallucinations in large vision-language models. arXiv preprint arXiv:231.01177, 2023.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 235-251. Springer, 2016.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 128(7):1956-199 2020.
Yebin Lee, Imseong Park, and Myungjoo Kang. Fleur: An explainable reference- frerence--fre evaluation metric for image captioning using a large multimodal model. arXiv preprint arXiv:2406. 06004, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.00 03326, 2024a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13299-13308, 2024b.
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang. Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312. 10665, 2023a.
12

---

Published as a conference paper at ICLR 2025
Yifan Li. Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating ob- ject hallucination in large vision-language models. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 292-305, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18///vv 00553//12222..mm/l 122023.emmllmmain.2222. URL https::/aclaatte
woo.orrg2022 2023. emnlp-ma in...........1 20.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.200500 2023.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74 81, 2004.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (6689-26699) 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lec. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint 23xiv:2377.0688. 2023.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chur Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 55:2507-2522 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Rei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint vrXiv:2310.02255 2023.
Yujic Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang. Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating lsiounlanguuae models in the wild with human preferences. arXiv preprint arXiv:2406.11063 2024.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shatiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244 2022.
OpenAl. Hello gpt-4o., 2024a. http:pps//ppppaa..ccccc///eeeeea
OpenAl. Gpt-4v., 2024b. http://////pppaa..oooo////////a/
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Kishore Papineni, Salim Roukos, Todd Ward. and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
13

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 9748-8763. PMLR, 2021.
Rafael Rafailov, Archit Sharma, Erie Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara, Positive- augmented contrastive learning for image and video captioning evaluation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6914-6924, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 742-758. Springer, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings the IEEE/CVE conference computer vision and pattern recognition, pp. 8317-8326. 2019.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:23309. 14525, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- can-tBat Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 2. 1805, 2023.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-l: : A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406. 16860, 2024.
Ramakrishna Vedantam, C Lawrence Zitnick and Devi Parikh. Cider: Consensus-basse image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 01 800, 2024.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint 2it4:2334414778 2023.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13040-13051, 2024.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 13807-13816, 2024a.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint a405v:2405.1 17220, 2024b.
14

---

Published as a conference paper at ICLR 2025
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. 0et..22000.00090, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-95677 2024.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao. Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, al. muermlm-xcooppoee-25555 A versatile large vision language model supporting long-contextual input and output. arXiv preprint urXhv:2407.033200 2024.
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing Ivims through hallucination-awel direct preference optimization. arXiv preprint arXiv:2311. 16839, 2023.
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.114: 1411, 2024a.
Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. arXiv preprint ar Ttt1:2405.162222 2024b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DISCUSSION
A.1.1 RELATED WORKS
Faithscore
RLAIF-V
Ours
Descriptive/Non- Desscrptiveee
v/x
v/x
/ V
Response Evaluation Coverage
Full
Partial
Full
Hallucination
Comprehensivenesss
X
Decomposition Method For Evaluation
Rewrite
Question-Answee Pairs
Rewrite
For Preference Leariing
X
Human Correlation (PCC p)
0.1937
0,3547
0.6605
Human Correlation (Kd r)
0,1626
0.2274
0.5328
Human Correlation (Sp r)
0.1115
0,2544
0.6166
Table 8: The comparison among related works.
We have compared Faithscore (Jing al., 2023) and RLAIF-V (Yu et al., 2024b), two metrics built on a similar conceptual foundation, and the distinctions are detailed Table Below, we summarize these differences to highlight our main contributions:
Granularity: While Faithscore and RLAIF-V evaluate the descriptive aspects of responses, they neglect the non-descrip elements, which are crucial for caption quality. For example, incor- rect assertions about the image's ' context and inferences can significantly impair understanding. However, in the realm of detailed image captioning, comprehensiveness is equally critical, as shorter captions may indeed exhibit lower hallucination rates but often suffer from a lack of informative value. Our approach uniquely addresses this by simultaneously considering both descriptive and non-descriptive components.
Decomposition Method: Like Faithscore, our method decomposes responses sentence-by-
sentence, yet it also includes non-descriptive elements. RLAIF-V, the other hand, generates question-ansee answer pairs for verification, potentially omitting crucial details.
Faithscore rates the proportion of correct statements, while RLAIF-V counts incorrect statements, which may encourage the model to avoid making any assertions or state irrelevant but correct information. Conversely, our approach evaluates both the proportion of correct statements for hallucination and the number of valid statements for comprehensiveness.
Score Generation:
Application: Our method, designed for detailed image captioning, serves both evaluation and preference learning within a unified framework. Faithscore and RLAIF-V are limited to evaluating or optimizing hallucinations independently.
Human Consistency: Our approach demonstrates the highest correlation with human judgment across various aspects, shown in the table, validating its effectiveness for detailed image captioning.
In essence, method introduces granular, comprehensive, and human-aligned evaluation framework that surpasses existing methods for detailed image captioning.
A.2 ADDITIONAL EXPERIMENTS
We investigated the influence omission elements and non-descriptive elements DCSCORE on its alignment with human judgment in Table 9 and Table 10 respectively. The results show that including omission elements and non-descriptive elements during detailed image caption evaluation achieves a higher correlation with human judgment. This improvement occurs because non-descriptive elements, such as background details and inferred information, provide additional context that leads to a more comprehensive understanding of the image content. Consequently, by including these elements,
16

---

Published as a conference paper at ICLR 2025
Omission in GT PCC (p) f
1- Rt
Kdr
Sp S
Non-Deseripive
POC
1 R²
Kd r
Sp
0.6151
0.72
0.5111
0.5916
0.6213
2.77
0.5048
0.5985
0.6605
0.5328
0.6166
0.6605
0.5328
0.6166
Table 9: Correlation of DCSCORE and human judgement in terms of considering omission in ground-truth annotation.
Correlation of DCSCORE and hu- man judgement in terms of considering non- descriptive elements in the captions.
Table 10:
DCSCORE captures subtle nuances and implicit information critical for fully understanding the image, thus more closely aligning with human judgment.
A.2.1 CASE STUDY
LLaVA.OneVision-T8
her attention captivated plate of donuta before har, holds one hand, polsed for while two rest pláta, unlouched. The table, draped in pristine white tablecloth, also holcs a black oell phone, liinn idle, The backsrop is cheirful yellow well, adorned with a winddww and painting, woding to the warn amolance the The woman's anticlpation palpable propares
wibrant
sweet
FEEDQUILL-7B (This work)
The image is color photog aphh feeturinn waman soatled sable with bowi of donuts front of har. The women appears
mid-30s 40l,, with whart and waering a light blue t-ahirt. is haliing donut her right hand and appears be the process eating Her expression is focused on donut, The bowi in front of her contains several donuts, some of which are visible. The donuts vary color and plantic
is looking down at
Can you desoribe what is happering in thes piclure?
having colorful aprnncees lying background
appsars
The table wooden and has dark finish. On the tatlee the left of the bowl, there is black cell phone image near the top. There is also a colorfl, abstract painting or artwork hainging on the wall to the right of the wamen. setting
yellow
squarn-saaped
opening
Elements (Hallucinations)
be indoers, is bright, and the averall wit aatrosphere seems relened and informal. There are no visizla taxta or dislincive brands
environment home. lighting
Correct Elements
Additional Cornect Elements
LLaVA--OeVisioone OneVinion-
The image captures a vibrant city street, bathed glow daylight. Dominating soone and white bus, ts yellow poles standing tall and proud. The bus parked on the right side of the street, patienty was grrting
journey.
On the left side of the street, a red brick building stands, It's adomeed with mural, a splash of col not conversation, telling their story
landecape, building and bue
be
in this utben symphony.
and bustle. medy to tothe the street light when right falls, A traffii light hangs overheed, its calors unaveen the ready
background,
parked, adding the
lamp sentinel,
daylight
the flow traffi.
Descrbb the following image.
Above atl,, the sky is carves blue, dotied with birds in fight. They add a touch of life to the scone, their they above the ciyscape. The snopehot urban each element twlling its own this urben tepestry
through
FEEDQUILL-7E (This work)
The image a color photograph depicling an utben street scene. In the foreground, there is sidewalk with a curb and a street. The sidewalk is adjanent to building graffil appaars be vitrant and covers the entre visible portion of the wall. On sidewalk, there handbags. The indrviduals are disperzed stong the sidewalk, with some doseer to the camera and others further away. stop.
calorful graffini, featuring
shapes, patterns,
range including yellow, purple.
people standing walking.
dressed casual clothing. and
backpacks
there bus visibble on Its front, The bus is connected to overhead wires, suggesting is an electro or hybnd vehicle. There are also several cars parked along the street, and more motion. background,
predominantly white with blue yallow
has the number "2242"
are mone buildings, some of have banners hanging from them. banners are reclangulerr feeetre images, the datai's are not clear enough read. Thare are also several birds in fight, castured against the sky, which appears to be overcast. The cvenal atyle the image is cendid and capturee moment of eweryday city Mis The photograph his natuneistic oolor palntis is taken from perspective us that places the siower on the sidewali, looking down the streat.
Wrong Elements (Hallucinations)
Correct Elements
Additional Correct Elements
Figure 5: Qualitative results of FEEDQUILL-7B compared with LLaVA-Onevision-77 (Li et al., 2024a) in terms of image captioning.
17

---

Published as a conference paper at ICLR 2025
LLaVA ViVV-nnViss
In the heart of tusting jitthen men in white shatt and black hot engrossed hhs work the stands in front while brick
ourved top reaching towards the colling, The many space.
wormhh and
confort, conterpicee
his two bange pizse rest on studes of while pistss teeer wtrant color pooplig agains the dest tonee kehen One pirma retomeed mediay of vegetartees theer fresh taae rinting the fincors The other pioze pensecsely loppac with mest promising a haae by and and slying moed
The kichan isal is soymphony of while and oflte likes covaring the mells and countar providing worlopace The ontoppatin of delicouss meal po poble in the dir. This image captures not, uus moment, but an entire story of dedication passion, and the an cooling
unsetverin
propaetes sidde
pirmen
Whaf' teppweing tie soone?
FEEDQUILL-TB (Thin work)
each pizza tap. The pizzas appeas tedaed with ee toppings indlucing what looks lin torreto sauce, cheese, and possitty vegetndeen meets The cruet the pizzas golden brown, indiceting they fawe cookec
phatoggach depicling inside pizoriia forepound, fare slacka
Bahind the stacko of plates, thane is main waeiing a wit t-shit and a cap. Hu facing away from the camera and appearr to engaged activily ral related to pizza making possiily chodking
plzzas in the owin presarig serve fhem The man is standing in frost of laras, dome- share-hhappd brick oven, ihich commion foatura arch.
tracitinall pizzrias
rustic appearance
the sids the imaige. there is frameed poster on is Slad in white, and there are various kitchen itams and containers visible on the counterr and shalives suggesting an achiv and funciional kichen anéronment.
bed! "Nippolatore" and illustration person chel's
bui indd oven
The lighting phookogruhh brigtt, with natural light caming from the side, Rurniniing the and crealing iembieenoo The from of the on the suljucts.
tadkground Hurnd which diily opanationa plzzaria
the placaag cardld caplusa momeent
Wrong Elenenta (tstuchnatooaa
Correct
LLaVA OneVision-BB
In the hvart of cany liing room, a beige couch and a makdiing smrcheer forn comfoorbeil seeeing srea. The
couch schamed blus and white pallened thow pillow posilianed siaé ust the wall while the armdhai sala inviinaly foroground.
while
roonts fourth
biing floor
the
renchsirr schling e trauch of werrdn to the some octfee table sis indicaling rolaned altamoon taa time. The walls the room are painted in southing light blue odlor providing oolm teckutop for sorns. Tso pairings
the couch hosting dlate with
cake and cup cotfee, pertapo
tabdo the ready sott glow needed. ioomis futher iluniinttdd by natural light streaminng through a window with white curtains. The curtains are crown back, suggesting k kriight be daydimee outsiide.
Ihe
giging an ael toach the
The room cludes a sonco of tranuuility and comtort makingt it local place for relacation and lessure.
What about?
FEEDQUILL-7B (This work)
inageis phokogragh depiding wails are painted light gray tone. landscape scene with buildings and a body of water. The smaller painting is hung above a side table and depicts a figure in landscape setting.
liing The dlimly lit, primary light comning lamp sidde
are framed paindings hanging
larger painting positioned above the couch and features
In the fangrruund, Fiere a beige sada with person onin lecing from the carrens Thés parson appsers to eneggeee in acdiviy possiblyy usinng au dlvvici reading On the sota, there paterned thow pillww in trort of the soln, there as wooden coffee biee with blue try on which holds hew tome, inclding what appears
food.
To right sofli, there women witting on wooden cheir with dark finish, She wearing black top and darkk pants, and shsi lookingg lowends the camera Hir posture suggaata she in rolaood aalaodd poaition.
On the floor, there two dogs One dog is lying down near the woman, and the other is closer the camera, also lying down Both dogs appear calm and are looking ciferent directions.
the sockground, there
French
muttppl
allowing light enter the The closed,
reflects
lighting
Thhe style of the shotogroph is condid and appears to captree a casusi, everyday moment a home setting The composttio of image is balanced, with the fumiture and poople anarged in that crealis seroa of space comfort.
Correct Elenerta
Additional Coreet Clmments
Figure 6: Qualitative results of FEEDQUILL-7B compared with LLaVA-Onevision--78 (Li et al., 2024a) in terms of image captioning.
As instances in Figure 5 and Figure 6 indicates, FEEDQUILL-7B not only significantly reduces hallucinations, but also remarkably improves the granularity and richness of descriptions compared with L.I. V-n-----iiis (Li et al., 2024a), which is the initial model of FEEDQUILLI-7B. From these case we can see the preference score of precision (cp) and the preference of recall (c₂) jointly determine the direction of preference optimization in FEEDQUILL, leading the descriptions of the images more precise and more comprehensive. Additionally, we present qualitative results of FEEDQUILL-7B and GPT4o (OpenAI., 2024a) in Figure 7. In these cases GPT4o still introduce hallucinations while FEEDQUILL-7B describe them precisely. From these examples we can get an intuitive understanding of the superior image captioning performance FEEDQUILL-7B achieves.
A.2.2 THE PERFORMANCE OF VLMs ON DECAPBENCH
We present the performance of various current VLMs on DECAPBENCH in Table 11. As shown, the performance in detailed image captioning consistently improves with increase in model size. For
18

---

Published as a conference paper at ICLR 2025
&e GPT4o
imagee fiod mes stredd aoee on read voly bay unter skioends Domindinn the fonegrouna is bright yellour hydhord with blsw lop and masis, firily pla laad on canovile siloools Surrounding the brse the Iyydrunn patch of graas fewr scattered weeds,, indicating is usustly exposed the stements. tn beckgruund slightly booth
necogrinstle visitle through colored jackut engaged aliver sedaa parked on Ble road slightly angled and posiloneed just bakind the pirone
gloss glass. the right of the Further down the aidreneik, you can new more pecberriess going about theer theyy llvee
The ameed taal ia fired with sell modirm buildings Nirking dowerioon sewa. Sone of the build have bagi glisso winedosss, while others adkmned
conorele facados. The wooling appears light coer entins scare Trees with rnturredd leaves lin the streed, golcen and brown legg prosictig contived urton strutures
sity which
mutedd
gray
Ovalft and. the wines and polios suagess a vell-dlookopee neig hootood hastuuuuuue. The overall abmosph ware of the image is busy rutrrcuiigg no, capturing tycccal moment urban setting where people about their daily actvilios arridst the stoody city
FEEDQUILL-78 (Thi work)
What
image &s shotograph dopldting unbon strect scone. In the foreground, there a hydrant painted in bright you ow with baue top and a blue side cop. The hydramt is stuleed on concrete sidewalk and appeore to sightty ted the right Them is some vegebritio growing acound the been of the hydtant indluding trrell struace and grees.
happoning in diss image?
Behind the hydrant there public telephone booth
exterior blue
roads "Belf in
white lettors. The booth glane
glass panel on side, through which
parton bee
people welling on Further back theoe atreed with veliciee, intluding a black car parked theokide are modemn with reflectve glass windows and concrete facades, The city is overcost, suggesting a doulty day.
badiground they appoor going stout cisyy
The bulkings becllgmandd
Wrong Blemssts (Aatosnations)
Correct Elereets
imege
the everytay etements Hiee locus fioe tystrm mein sutteed. chrowing Meiiinggnn
composi the phologreph plecen hydhert
beckkroundd
prosice
GPT4o
dy Jann is player drossed whill
daring muld oublior
while
holding lamiss racket his right hand
proparing to hit a you temms ball that visbb the faregronnd to the let. His lof am extended
culoaeds tailenos and is moving Eowwed uxhikling in intana locasion on the bail
The tennis court true, with visibre white lines marking the playing area Behind the playor, there cark wall that Roalures
logos white, the sidee the lexd
We
roods, NNBW comCerter com"" the right dide bears the home "Vestom & Southom Firancial Group
Mastars" and NONNNI 2004."
Fin
in the bookground, several spectators are seated what appcars a stadium soting, watching the match The weother
and indicated bright the count the
spectaiors cosust aftre like sunglosses. imege
energetio and compatiiive ammopphheee typical profeesional
FEEDQUILL-78 work)
Ana the imageina comprehonste and detslld
The iraepe color phologragh feeduning a make tamiss player recion The player is cheiseed while atim including a heaedhond strit shorts, and we iettbtnnds He as holding tennis micket the right handd and kepwars be the prociess of hitting termia ball which wiuble in
telt of the playerr His body extandeed bailind him, suggestingg folloos-thruugh
addwways the motion.
The playor' a oxpression focused, he lodking intertly complesinnn The backgroand the image indlucles a blke woll with bed and logos. The bod indducios
His harls short, and light
"WacommCantee tar com. "Waskerm& S Souther Finar cial Group Muskera and CENCINNATII 2004 " rid opectiiiins
photo that phologragh wax takan gn tamiis muach The style image dymarkk ciolurin moment bati anion. focusi the player interctionwll the ball with the background dlomnnts providing commrt for the ovent. The colors in image are verrant, with the blue of the wall contracting with the white of player' outt and the yel ow of the tennts ball.
backgrannt gigeoiing
Wrong tlements (Motactration)
Conrect Elamerza
Adsitonnal Correct Elcesatts
Figure 7: Qualitative results of FEEDQUILL-7B compared with GPT4o (OpenAI,, 2024a) in terms of image captioning.
instance, notable improvements are observed in the Intern series (8/26/40B) (Chen et al., 2024b) and the LLaVA-series (7/13/34B) (Liu et al., 2024a).
A.3 IMPLEMENTATION
A.3.1 TRAINING DETAILS
Reward Model We initialize the reward model with the parameters of the SFT model and adopt the pairwise comparison loss (Ouyang et al., 2022) for training. The training is conducted for epoch, with learning rates set to 5 for the 7B model and 5e for the 13B model. The weight decay is set to O. The training size of the reward model is set to 200,000 pairs unless otherwise specified. During inference, the reward model produces scalar outputs provide the score for the responses.
19

---

Published as a conference paper at ICLR 2025
Model
Language Model DCSCORE JF
Qwen-VL-CChtt-7B (Bai et al., 202.)
Qwen-7B
19.16
mPLUG-OWl2 (Ye al., 2024)
LLaMA-2-7B
23.27
LLaVA-1.5-7B (Liu et al., 2024b)
Vicuna-v1.544
24.50
LLaVA-1.5-13B (Liu et al., 2024b)
Vicuna-v1.5-13B
25,55
XComposer2.557B (Zhang al., 2024)
IntemLLM2.5-7B
29.60
Cambrian-34B (Tong al., 2024)
Yi-34B
35.12
LLaVA-1.6-7B (Liu et al., 2024a)
Vicuma-v1.5-14
36.21
MiniCPM-Llama3--225--3 (Yao et al., 2024)
LLaMA-3-SB
36.36
LLaVA-1.6-13B (Liu et al., 2024a)
Vicuna-v1.5-13B
37.98
ViLA-40B (Lin et al., 2024)
Yi-34B
38.02
IntemVL-11.5 20B (Chen al., 2024b)
IntemLMM-2000
39.28
LLaVA-1.6-34B (Liu et al., 2024a)
Yi-34B
40.46
LLaVA-Onevision-78 (Li al., 2024a)
Qwen2-7B
43.49
Gemini-Pro-1.5 (Team et al., 2023)
46.34
IntemVL-22885 (Chen al., 2024b)
InterL.M2.5-75
47.39
GPT-4v (OpenAL., 2024b)
48.52
IntemVL-222000 (Chen et al., 2024b)
(2..52
49,59
GLM-4v-9B (GLM et al., 2024)
GLM-4-9B
49.85
Intern VntemWV-22-40B (Chen al., 2024b)
Yi-34B
51.17
Claude-3.5-Sonnet (Anthropic, 2024)
52.37
GPT-4o (OpenAI., 2024a)
53.44
FEEDQUILL-7B
Qwen2-7B
55.65
Table 11: The performance of various VLMMs on DECAPBENCH.
Our implementation of the PPO algorithm is variant of (Ouyang al., 2022). We adopt two reward models: Cp RM and RM. The Cp RM is trained with the preference for the proportion of correct units, which measures the precision hallucination rate of the description of the image. The o, RM is trained with the preference for the number of primitive information units, which measures the richness the description the image. We sum the two RM outputs final reward: Cp + Cr, The arper-parameter a, controls the trade-off off between accuracy and richness, we set to 0.5 our experiments. We set temperature to 1.0 and top-P to 0.7 when sampling trajectories for the diversity of responses. The PPO training data is entirely composed of captioning task data, containing 100k images. Other PPO hyper-parameter are presented in Table 12.
PPO
Tlype--paramete
Default Value
Optimizer
AdamW (e = 1e 8)
Learning Rate
(actor), 5e-6 (critic)
Scheduler
Linear
Batch Size
256
3 (KL Penalty Coefficient)
0.05
Y (discount factor)
1.0
X (TD trade-off factor)
0.95
Number of Mini-batches
(Policy Clipping Coefficient)
0.2
(Value Clipping Coefficient)
0.2
Table 12: PPO hyper-parametert
A.3.2 EVALUATION METRICS AND BENCHMARKS
MMBench (Liu et al., 2023) introduces a diversity of evaluation questions, and use circular evaluation protocol for multiple choices that leverage GPT to transform free-form answer into the choice.
20

---

Published as a conference paper at ICLR 2025
MMStar (Chen et al., 2024a) is vision-critical curated challenge samples designed to evaluate core capabilities and 18 detailed axes of VLMs. It is enhanced by strict human review to ensure visual dependency.
benchmark with 1,500 human-
TextVQA (Singh et al., 2019) measures the capability of VLMS for answering question about the text the natural images.
VizWiz (Gurari al., 2018) comes from natural visual question answering dataset for blinding people.
ScienceQA (Lu et al., 2022) consists of approximate IK multi- multiple choice questions with diverse set of science topics and annotations of their answers with corresponding lectures and explanations.
mmHal-V V (Amirloo al., 2024) is a visual hallucination evaluation benchmarks for VLMs, which consists object attribute, adversarial object, comparison, counting, spatial relation, envi- ronment, holistic description, and other types.
LLaVA-W (Liu et al., 2024b) aims evaluate the model's capability in visual chatting, which including memes, indoor and outdoor scenes, painting, sketches, etc. Each each image is associated with highly-dctaleed and manually-curated description and a proper selection of questions, and utilize GPT to score the model's response.
WildVision (Lu et al., 2024) simulates the arena and evaluate the model with various real- world questions, while benchmarking human preference.
CHAIRs and CHAIR, (Chan et al., 2023) a reconierec tool for evaluating the incidence of object hallucination in image captioning tasks which assess object hallucination at the instance-level and sentence-level respectively.
MME (Fu et al., 2023) is comprehensive benchmark for evaluating the capabilities of VLMs in multi-modal tasks. systematically assesses models across two primary dimensions: perception and cogmition, through 14 meticulously designed subtasks that challenge the models' interpretive and analytical skills.
SeedBench (Li al., 2024b) consists of 19K multiple choice questions with accurate human annotations, and spans 12 evaluation dimensions including the comprehension of both the image and video modality.
MMMU (Yue et al., 2024) includes 11.5K meticulously collected multi-modal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering.
A.3.3 PREFERENCE OPTIMIZATION
The following algorithm demonstrates how to leverage PPO (Schulman et al., 2017) to optimize the base model (SFT Model) with reward models trained with preference data D for Cp and preference data D, for Cr.
A.3.4 EVALUATION PROMPT FOR DCSCORE
To measure the quality of the generated captions, we present prompts for decomposition in Table 13, matching Table 14, and verification in Table 15. We utilize GPT-4o (OpenAI., 2024a) through the whole evaluation process.
A.3.5 TRAINING PROMPT FOR PPO
We prompt GPT-4o (OpenAI, 2024a) to generate a series of image captioning prompts for PPO training, listed in Table 16.
21

---

Published as a conference paper at ICLR 2025
Algorithm Preference Optimization with FEEDQUILL
Input initial policy model : initial value model Ver reward models Rø trained from cp or
Cyi PPO training prompts Dú; PPO hyperparameters, 7/, X, c, B.
policy model Pe t- Pe... value model Vi + Vorm
T do 3: Sample a batch from Dr
2: for step
4: Sample output sequence y" Po (x)) for each prompt x B
5: Compute rewards {rp, + rr]t=1
1s" from the reward model Rø, and RS, for each y".
6: Compute advantages {A,]r=]
==1 and value targets (ve*(s for each y" with Vi--
1/"
7:
for PPO iteration = ..... f do
8:
Update the policy model by maximizing the PPO clipped surrogate objective:
B8
Po st) As, clip(er, - E, E)A, Pews (a: st)
8 arg
min
t=1
9:
Update the value model by minimizing L2 objective:
[8|
V - min
(Vo(s)) - (s)))
V
1
end for 11: end for
Output Pe
You are linguistic expert in extracting primitive information units in the image caption. In specific, "primitive information units" refer to the smallest standalone pieces of information that collectively represent the entire meaning of the sentence without losing any detail, which typically describe various properties the visual elements in an image. The primitive information unit should be simple statement. The fact must represent the smallest piece of information that cannot be further broken down without loss of meaning. Abstract concepts or broad interpretations should be reduced basic, constituent observations possible. The primitive information unit should only contain ONE primary element.
When extracting primitive information units from image caption. it is useful to assign unique identifiers the primary objects entities being discussed. This will help in maintaining clarity and preventing confusion, especially when there are multiple similar objects or entities. the caption mentions two cats, you can assign unique identifiers such as "cat," and "catz" to distinguish them. Besides, for cach attribute, you should also assign the identifier to the object it belongs to. Mcanwhile, for spatial relationships, you assign the identifier to the object that the subject of the relationship in the primitive information unit.
example,
For each primitive information unit, you should also need to justify whether the primitive information unit directly describe the image or not.
** "IMPORTANT** Please extract ALL of the primitive information units the image caption. DO NOT omit any information!
The output should he list of dict [[ [["fact": [PRIMITIVE INFORMATION UNTT], "identifier": [UNIQUE ID), "relevance": 1/0), into ISON format. The "identifier" would be optional, the item fact has already been identified with ids. The "relevance" would be if the primitive information unit directly describe the content of the image. Otherwise it would be 0 if the primitive information unit is inference extension to the description and not directly describe to the content of image.
Caption:
(Caption Here)
Table 13: The prompt for decomposing the generated captions into set of primitive information units.
22

---

Published as a conference paper at ICLR 2025
You are now a iisual-linguiste expert in matching two set of primitive information units generated from two captions.
You will be received set predicted primitive information units across varicty categories and a set of oracle primitive information units (ground truth). The set primitive information units is represented as a list of dict [[" [["fact": [PRIMITIVE INFORMATION UNITJ, "identifier": [UNIQUE IDI). within JSON format. In addition, each primitive information unit in the oracle set would be accompanied with unique "id" identify the oracle primitive information unit.
To match primitive information units from a predicted set in terms of the given image with oracle set of primitive information units. Here is the step by step instruction:
1. Preliminary Review: Conduct initial review of hoth sets of primitive information units, considering all primitive information units. Understand the details and context presented within each primitive information unit.
2. Inferring Identifier Mappings: Closely examine both sets to deduce potential correlations and mappings based on the content of the primitive information units. Determine if there any unique identifiers descriptors that hint matching entities between the
For example, "cato" in the predicted set's primitive information units may be mapped to "catı" in the oracle set's primitive information units. Consider the attribute and spatial relation in both sets for possible mapping. Please note that there might be some attribute and spatial errors when mapping the objects. Try find the most similar mapping if cxists (not need exact matching). If no oracle primitive information unit matches, simply matched oracle id "None".
**MMPORTANT*: Please consider each primitive information unit the set individually, and MUST NOT omit any primitive information units from the predicted set.
should only output the matching results which will formatted as a dict as [[ [("fact": [PRIMITIVE INFORMATION UNIT), "identifier": [UNIQUE ID), "matched_ oracle id": [CORRE- SPONDING ORACLE IDI), ...1 in JSON format. The "identifier" would be optional, if the item in the fact has already been identified with ids as illustrated in the predicted primitive information units. For key named "matched oracle id", the value of "matched, oracle id" should be the corresponding "id" of the primitive information unit in the oracle set. the primitive information unit in the predicted set which cannot be matched with any oracle primitive information unit, set the value caatchedd_oralle_eoe to "None".
>>> Set of Primitive information units: {set units generated caption]
>> Oracle Set of Primitive information units: (set of units for human-written caption)
Matching Result:
Table 14: The prompt for verifying the correctness of each primitive information units by utilizing both image and human-writt caption.
23

---

Published as a conference paper at ICLR 2025
You are an extraordinary visual- -isuul-inguuisti expert in verifying the correctness a set of primitive information units given the image and the corresponding reference caption. The set of primitive information units extracted from paragraph of machine-gennrrttdd image caption of that image.
set of primitive information units represented as list dict ["fact": [PRIMITIVE INFORMATION UNIT), "identifier": [UNIQUE ID). ...] within JSON format. The identifier is unique and to identify the primary objects or entities being discussed. This will help in maintaining clarity and preventing confusion, especially when there multiple similar objects entities. For example, if the caption mentions two cats, would assign unique identifiers such "cat," and "cat2" to distinguish them. Besides, for each attribute, also assigned the identifier to the object it belongs to. Mcanwhile, for spatial relationships, it assigned the identifier to the object that is the subject of the relationship in the primitive information unit.
should first go through of the primitive information units, understand the details context presented within each primitive information unit. Then you need to verify the correctness of each individual primitive information units by asking yourself: Statement: "[PRIMITIVE INFORMATION UNIT]" Does the statement correct according image reference caption?
The output for predicted set primitive information units should be formatted as list ["fact": : [PRIMITIVE INFORMATION UNIT], "identifier": [UNIQUE ID), "verification": 1/0, ...] JSON format, where represents the fact is correct and 0 represents the fact is incorrect. Other keys in the dictionary the the input. The "identifier" would be optional, the item in the fact has already been identified with as illustrated the input.
as
>> Reference Caption: [reference caption}
Primitive Information Units:
/primitive information units]
Table 15:
The prompt for verifying the correctness of each primitive information units by utilizing suman-written
both image and human written caption.
24

---

Published as a conference paper at ICLR 2025
What do you see happening in this image?
Can you describe what is happening in this picture?
What events taking place in this image?
What do you observe this photo?
Can you explain the scene depicted in this image?
What is this photo about?
What is the subject of this picture?
Can you explain the theme of this image?
What the focus of this photo?
What is the central topic of this picture?
What is the main idea of this image?
What is the essence this photo?
What is the core subject of this picture?
What the primary focus of this image?
What is the overall theme of this photo?
What is the main topic depicted in this picture?
Can you elahorate on the elements of picture provided?
Can you give more details about the components of this image?
What the various elements this picture?
Can you describe the different parts of this photo?
What the individual components of this image?
Can you break down elements of this picture?
What are the distinct features of this photo?
Can you provide more information the elements in this image?
What are the specific parts of this picture?
Can you detail the elements present in this photo?
are the various aspects of this image?
Analyze the image in a comprchensive and detailed manner.
Provide thorough analysis of this picture.
Can you give an in-depth examination of this image?
What is your detailed analysis of this photo?
Can you down this image comprehensively?
What is your extensive analysis of this picture?
Table 16: Part of example prompts for preference optimization.
25


Reviews:
[{"review_id": "oQDjsFN3h0", "reviewer": "Reviewer_A3YP", "summary": "Image captioning is an important task that has recently been explored using VLMs to generate detailed captions. Traditional metrics or coarse annotations may not be ideal to evaluate the performance of detailed image captioning. \n\nThis paper proposes evaluation metric for detailed captioning tasks, considering both hallucination and comprehensiveness. A public benchmarks has been proposed. The paper also introduces FEEDQUILL, a scalable method for fine-grained feedback collection by decomposing and verifying responses. Experimental results seem reasonable.", "questions": "To generate benchmark, why only 400 images are selected? How these images are selected? 400 images seems very small subset. \n\nWhy LLaVA models are used as base model? It will be good if more popularly used models can be investigated to demonstrate the effectiveness of the proposed fine-grained feedback collection.\n\nThe decomposition in section 4.1 seems similar as that in 3.1.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "To address the problem of evaluating the performance of detailed image captioning, this paper proposes a new evaluation metric to take both hallucination and comprehensiveness into consideration. It also constructed an evaluation benchmark using the proposed evaluation metric to the ImageInWords images and their corresponding hyper-detailed image captions. The experimental results seem reasonable.", "weaknesses": "To generate benchmark for detailed image captioning, 400 images from ImageInWords dataset are used to generate benchmark with the proposed evaluation metric. Only 400 images seems a very small subset. The uniqueness of the proposed benchmarks needs to be further clarified. \n\nThe performance of the proposed method does not always achieve the best results. More explanations and justifications are expected.\n\nThe organisation of the paper can be further improved. It would be good to have a self-contained version rather than leave some important content in appendix. \n\nSome notations are not clearly defined. For example, in 4.1, the definition of the fraction of correct units seems not easy to be understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GxHlu6uWOO", "reviewer": "Reviewer_Mger", "summary": "This paper proposes a DeCapBench together with a DCScore and FeedQuill preference optimization method to evaluate and improve the ability of detailed image captioning. More specifically, DCScore is implemented in an F1 score style to asses the hallucination and comprehensiveness of the output captions. The introduced detailed captioning benchmark DeCapBench is further conducted to evaluate the captioning capability of VLMs. In addition, this paper also proposes a fine-grained feedback collection method to formulate the reward function for model alignment. The experimental results demonstrate extensive comparisons and results against multiple closed- and open-sourced approaches.", "questions": "Please refer to the Weaknesses. The following is a minor question.\n\n1. This paper mainly considers LLaVA to be the VLM. Are other commonly used multimodal LLMs, such as VILA or InternVL-2, also applicable to this proposed RL alignment framework?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Evaluating detailed image captioning remains a challenging task since current mainstream captioning datasets (e.g., COCO) only contain relatively coarse-grained and short captions. Since most of the metrics for image captioning rely on ground-truth captions, designing a metric to properly assess the quality and hallucination degree of the output captions is crucial, especially in the multimodal LLM era.\n2. The proposed fine-grained feedback as the reward function and the PPO-based alignment framework is reasonable and technically sound.\n3. The quantitative comparisons and experimental analysis are comprehensive, and the performance of the proposed method is promising.\n4. The overall paper is well-structured and easy to follow.", "weaknesses": "1. As we know, the multimodal LLMs themself has inevitable hallucination issues. Does integrating VLMs into the verification step guarantee that the evaluation results are trustworthy? Or the evaluation results may still be affected by potential hallucinations or input prompts.\n2. While this paper provides results like Table 1 to show the proposed DCScore better aligns with human judgments, it remains unclear whether this behavior can be generalized to other datasets or tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lzVFUGVfNk", "reviewer": "Reviewer_DgKk", "summary": "This paper introduces a new metric DCScore and a new benchmark DECAPBENCH to evaluate the detailed image captioning capabilities of  VLMs. DCScore is designed to measure both the hallucination and comprehensiveness of generated captions. To calculate DCScore, ground-truth and generated captions are first broken down into primitive information units. Then, the primitive information units from the generated captions are compared with those from the ground-truth captions. In addition, GPT-4o is utilized to judge whether each primitive information unit from the generated captions corresponds to the image. Based on these results, a precision score and a recall score are derived, representing non-hallucination and comprehensiveness, respectively. Empirical study shows that DCScore is more aligned with human judgments than previous image captioning evaluation metrics. By combining the proposed DScore and 400 high-quality and detailed image-caption pairs, the benchmark DECAPBENCH is established.\nThe paper also proposes FEEDQUILL, an automatic fine-grained feedback collection method to collect preference data for model training. This method breaks down each response into primitive information units, ensembles multiple VLMs to score these units, and constructs preference data from these scores. Experiments demonstrate that models trained using FEEDQUILL outperform those trained with other preference data.", "questions": "About DCScore  \nStep 1 of the evaluation process is unclear to me.\n1. Who are the “human experts”? What’s the definition of “experts” in this paper?\n2. Why are the decomposers for generated captions and ground-truth captions different (LLM vs. human experts)? Can LLM be used for both?  \n\nStep 3  \n3. Is the goal of this step to compensate for the missing details in the ground-truth captions?\nWhat’s the difference between $P_{true}$ and $Q$?\n\nAbout DECAPBENCH  \n4. How are the 400 high-quality, human-curated public detailed captions chosen? Is there any criterion for this selection?\n\nAbout FEEDQUILL  \n5. In the related work section, the paper mentioned that using GPT-4v to collect preference data could pose risks of bias and unreliability as the preference judgment of GPT-4v is not manually verified. As FEEDQUILL also leverages multiple VLMs to collect preference pairs, aren't the collected data also likely to be influenced by these models' bias and unreliability?\n\nAbout experiments  \n6. Do other non-LLaVA VLMs, e.g. InternVL-2-8B, trained with the FEEDQUILL-collected preference data also show superior results on downstream tasks?  \n7. How many FEEDQUILL preference data are used for training in the last row of Table 2?\n\nMinor comments  \n8. Line 319: \"..., responses with fewer hallucinations are often inherently less helpful.\" Is this sentence correct?  \n9. Typo in line 334: \"In To fully exploit the characteristics ...\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A novel metric for evaluating detailed image captions and an automatic feedback collection method are proposed. The proposed metric aligns more closely with human judgments and can measure hallucination and comprehensiveness. The proposed feedback collection method is able to construct better preference data without human annotators than previous preference data collection methods.\n2. The experiments show that the proposed FEEDQUILL generalizes much better than other preference data when LLaVA models are used. In addition, the model trained with FEEDQUILL has better performance on various downstream benchmarks, demonstrating its effectiveness in enhancing models' image captioning capabilities.\n3. The paper is well-written and organized.", "weaknesses": "1. The explanation of the DCScore evaluation process is not entirely clear. Please see the questions below.\n2. When evaluating the effectiveness of FEEDQUILL with various VLMs, only LLaVA-family models are utilized. Why aren't any non-LLaVA-family models (e.g. InternVL-2-8B) used in Table 6?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y0yX8NYZ3Y", "reviewer": "Reviewer_swUh", "summary": "This paper propose:\n1. DCScore: a metric to evaluate both hallucination and comprehensiveness\n\n\n2. DeCapBench: an image captioning benchmark (contains only testing dataset) for hallucination evaluation\n\n\n3. FeedQuill: a method to mitigate hallucination in vision-language models (VLMs), which consists of the following steps:\n  - (1) Collect the responses from VLMs. \n  - (2) Employ LLM to decompose the responses into primitive information units. \n  - (3) Use an off-the-shelf VLMs to verify the correctness of each information units. \n  - (4) Label data as positive or negative based on the verification scores.\n  - (5) Train a reward model with preference dataset constructed in (4).\n  - (6) Fine-tune the target VLM to generate less hallucinated and more enriched captions through PPO.\n\nFinally, several vl benchmarks are achieved as SOTA performance.", "questions": "Please answer my questions in “weaknesses” section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Mitigate the hallucination issues in VLM is crucial, especially in the detailed image captioning task.\n- The proposed metric DCScore sounds reasonable, is provided with a comprehensive comparison with previous metrics (e.g., Faithscore and RLAIF-V), and is demonstrated high consistency with human evaluation.\n- The conducted experiments and related ablation studies are extensive.", "weaknesses": "[Metric - DCScore]\n1. Why are non-descriptive captions included as a part of this metric? \n2. This metric appears to rely on paid API (i.e., GPT-4o) for its evaluation process. It would be advantageous if the metric could also be adapted to work with open-source VLMs as alternatives to GPT-4o.\n\n[Benchmark - DeCapBench]\n1. The testing dataset in DeCapBench consists of only 400 samples. How does this compare to other visual data hallucination quality sets, such as HallusionBench [1]\n\n[Method - FeedQuill]\n1. In table3, there is a lacked experiment to compare the FeedQuill with the simplest cross-entropy loss (i.e., image caption loss) using the same PPO-finetuned dataset. A comparison of FeedQuill with cross-entropy loss on hallucination-measured datasets, such as mmHal-V and DeCapBench, would be valuable.\n2. In addition to MSCOCO, OpenImages, and ShareGPT4V, what other datasets are included in fine-tuning the VLM with PPO?\n3. Is the preference score $c_r$ a scalar value? If so, why is it necessary to train an additional reward model $R_{\\phi_r}$ to generate the $r_{r_t}$ in Algo 1? Could the $c_r$ be directly used as a part of reward?\n\n\n[1] HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (CVPR'24)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M0BghkOfRO", "reviewer": "Reviewer_t2Sb", "summary": "This work introduces a specialized metric (**DCScore**) and a benchmark (**DeCapBench**) for detailed image description evaluation. The core idea is to break down the reference and generated caption into the \"smallest self-sufficient units\", and then quantify the precision and recall of information units conveyed by the generated caption. The authors demonstrate that the new metric and benchmark achieve the best consistency with human evaluations. In addition, based on a similar concept, the authors propose a method (**FeedQuill**) for automatically constructing preference data for RLHF. Extensive experiments validate that the collected preference data can train a strong image captioning model.", "questions": "1. The main experiments employ LLaVA-Onevision-7B. Why was this setting not maintained consistently in other experiments? For instance, the ablation study “Preference Data for Reward Model” used LLaVA-1.5-7B, and the “Source of Response” experiments used LLaVA-1.5-13B.\n2. In Appendix A.1.1, the authors claim that DCScore accounts for non-descriptive elements, unlike other metrics. Could the authors further explain why it is important to consider **non-descriptive** elements in **image captioning** tasks, which aim to generate descriptions for images?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper introduces a new metric and benchmark for evaluating the quality of detailed image captions. Correlation analysis with human evaluations indicates that these new assessments are effective and superior.\n2. This paper presents an efficient method for collecting preference data and demonstrates that such data can be used to build more effective image captioning models.\n3. The experiments are comprehensive. The authors’ claims are well-supported by substantial experimental evidence, and they have conducted detailed ablation studies, providing valuable best practices for the community.\n4. The paper is well-written with clear figures and tables, effectively conveying information.", "weaknesses": "1. Section 3.2 does not introduce the instructions provided to human annotators for scoring image captions. Disclosing the task instruction for annotators is crucial; if the basis for scoring largely aligns with the design of an automated metric, the metric will likely benefit in correlation assessments.\n2. DCScore relies on the hyper-detailed human captions in the ImageInWords dataset. However, as \"a picture is worth a thousand words,\" reference descriptions might not fully reflect all the semantics of an image, while the model may describe image details that are correct but not mentioned in the reference caption.\n3. The proposed metric conceptually resembles prior works like FaithScore and RLAIF-V (I am delighted to see this discussed in Appendix). The divide-and-conquer approach and evaluation using LLMs is not novel. Collecting preference data for model optimization is also a consensus in the research community. While I see no other obvious flaws, **I am not fully convinced of the overall contribution**. I look forward to being further convinced by the authors and other reviewers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)