Calibrated Self-Rewarding Vision Language Models
Authors: Yiyang Zhou
Zhiyuan Fan
Dongjie Cheng
Sihan Yang
Zhaorun Chen
Chenhang Cui
Xiyao Wang
Yun Li
Linjun Zhang
Huaxiu Yao
Venue: NeurIPS 2024 poster (2024)
Keywords: Calibrated self-rewarding
large Vision-language models
Modality alignment
Primary area: deep_learning_architectures
TL;DR: Improving modality alignment in large vision-language models with calibrated self-rewarding

Abstract: Large Vision-Language Models (LVLMs) have made substantial progress by integrating pre-trained large language models (LLMs) and vision models through instruction tuning. Despite these advancements, LVLMs often exhibit the hallucination phenomenon, where generated text responses appear linguistically plausible but contradict the input image, indicating a misalignment between image and text pairs. This misalignment arises because the model tends to prioritize textual information over visual input, even when both the language model and visual representations are of high quality. Existing methods leverage additional models or human annotations to curate preference data and enhance modality alignment through preference optimization. These approaches are resource-intensive and may not effectively reflect the target LVLM's preferences, making the curated preferences easily distinguishable. Our work addresses these challenges by proposing the Calibrated Self-Rewarding (CSR) approach, which enables the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. In the reward modeling, we employ a step-wise strategy and incorporate visual constraints into the self-rewarding process to place greater emphasis on visual input. Empirical results demonstrate that CSR significantly enhances performance and reduces hallucinations across twelve benchmarks and tasks, achieving substantial improvements over existing methods by 7.62\%. Our empirical results are further supported by rigorous theoretical analysis, under mild assumptions, verifying the effectiveness of introducing visual constraints into the self-rewarding paradigm. Additionally, CSR shows compatibility with different vision-language models and the ability to incrementally improve performance through iterative fine-tuning.

Calibrated Self-Rewarding Vision Language Models
Yiyang Zhou', Zhiyuan Fan"", Dongjie Cheng Sihan Sihan Yang', Zhaorun Chen? Chenhang Cui", Xiyao Wang", Yun Li', Linjun Zhang", Huaxiu Yao UNC- Chapel Hill, *University of Chicago, UUniverstyy of Maryland Rutgers University, HKUST, PolyU, 'NTU, NUS y yiyangai www.eees..uuuceceeeeeee unc. edu, huaxiu@cs. unc. edu edu
Abstract
Large Vision-Language Models (LVLMs) have made substantial progress by in- tegrating pre- pre-trained large language models (LLMs) and vision models through instruction tuning. Despite these advancements, LVLMs often exhibit the hallu- cination phenomenon, where generated text responses appear linguistically plau- sible but contradict the input image, indicating misalignment between image and text pairs. This misalignment arises because the model tends to prioritize tex- tual information over visual input, even when both the language model and visual representations are of high quality. Existing methods leverage additional models or human annotations to curate preference data and enhance modality alignment through preference optimization. These approaches are resource-intensiv and may not effectively reflect the target LVLM's preferences, making the curated pref- erences easily distinguishable. Our work addresses these challenges by proposing the Calibrated SewarRne (CSR) approach, which enables the model to self- improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. In the reward mod- eling. employ step-wise strategy and incorporate visual constraints into the self-rewarding process to place greater emphasis on visual input. Empirical re- sults demonstrate that CSR significantly enhances performance and reduces hal- lucinations across ten benchmarks and tasks, achieving substantial improvements over existing methods by 7.62%. Our empirical results further supported by rigorous theoretical analysis, under mild assumptions, verifying the effectiveness of introducing visual constraints into the relf-rewad paradigm. Additionally, CSR shows compatibility with different vision-language models and the ability to incrementally improve performance through iterative fine-tuning. Our data and code are available at https://gtthub.co Yiyang/Zhu//CSR)
Introduction
Large Vision- Models (LVLMss) [1--I] have achieved significant success by incorporating pre-trained large language models (LLMs) and vision models through instruction tuning. However, these LVLMs suffer from the hallucination phenomenon [5]], which generates text responses that are linguistically plausible but contradict the visual information the accompanying image. For instance, the description generated by LVLMs may include visual elements that are not depicted in the image. This issue can also occur when the LLM is highly factual and the visual backbone is capable of producing sufficiently high-quality representations. As indicated in Cui et al. [B]] Guan et al. [Z]. the potential reason for this lies in the misaligument problem between image and text modalities in LVLMs, which causes the model to prioritize the text knowledge present in the training language data while ignoring the actual visual input information.
"quual contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Several works have been proposed to enhance modality alignment capability in LVLMs through pref- erence fine-tuning techniques, such as reinforcement learning from human feedback (RLHF) [8] and direct preference optimization (DPO) [H, [СO. However, these methods often either introduce addi- tional models, such as GPT-4, or depend on human annotation to generate preference data. This data generation process is not only resource-intensive but, more critically, fails to capture the inher- ent preferences of the target LVLM. Consequently, the target LVLM may easily discern preferences from such curated data, making them less effective (detailed analysis provided in Appendix A.4.). Recently, self-rewa approaches have emerged, utilizing a single LLM for both response genera- tion and preference modeling, showing promising results in LLM alignment [ I2]. Unlike LLMs, LVLMs face modality misalignment issues in both response generation and preference modeling stages, potentially resulting in self- geffeeeeedtedd preferences overlooking visual input information. Di- rectly applying these self-rewarding approaches to LVLMs is not capable of addressing the modality alignment problem and redirecting LVLM's s attention towards emphasizing input image information.
To tackle these challenges, our work introduces the Calibrated Self- Rewa-Reng (CSR) approach, aimed at calibrating the self-rewarding paradigm by incorporating visual constraints into the pref- erence modeling process. Specifically, we train the target LVLM using an iterative preference optimization framework that continuously generates preferences and optimizes the target LVLM over multiple iterations. Starting with a seed model, each iteration employs sentence-level beam search [T3 |El to produce fine- ane-grained candidate responses for each image and text prompt. During the beam search, for each generated sentence, we first utilize the language decoder to establish an initial reward (i.e., sentence-level cumulative probabilities). Subsequently, reward by incorporating an image-response relevance score, resulting in the calibrated reward score. These calibrated reward scores are utilized to guide the generation of the next batch of candidate sentences. Finally, responses with the highest and lowest cumulative calibrated reward scores are identified as preferred and dispreferred responses, respectively, for preference fine-tuning in the subsequent iteration.
calibrate this initial
The primary contribution of this paper is CSR, novel calibrated rewardigg paradigm for im- proving modality alignment in LVLMs. Theoret- ically, with mild assumptions, we show that in- troducing visual constraints in the self-rewarding paradigm can improve performance. Empiri- cally, when compared with other competitive ap- proaches (see Figure M for some representative methods), the results demonstrate that CSR is ca- pable of improving performance on comprehen- sive LVLM evaluation benchmarks, VQA tasks, and reducing hallucination, achieving up to a 7.62% improvement on average. Additionally, we demonstrate CSR is capable of continuously improving performance over iterations, compati- ble with different large vision-languaae backbone models, and redirecting the attention of LVLMs toward the visual modality to achieve stronger modality alignment.
LLAVA
66.5
#4.6
NHE5/4
32/4
SUN)
66.3
+22
100-CHAIR
12.0
COA
100-CHAIT,
POrt
LLAN-
ref-rewarding 78
CSR 78 (Ours)
Figure 1: Benchmark performance comparison.
2 Preliminaries
In this section, we will provide a brief overview of LVLM and preference optimization.
Large Vision Language Models. LVL.Ms extend LLMs to multimodal scenario, which progres- sively predict the probability distribution of the next token for each input prompt. Given <image To. text r,> pair as input prompt x, LVLM outputs a text response y.
Preference Optimization. Preference optimization has shown promise in fine-tuning language mod- els and aligning their behavior with desired outcomes. Given an input prompt & language model with policy can produce a conditional distribution Toly æ) with y the output text response. The preference data is defined D (tx()) ye
(6) (²)) ) where v and o denote the preferred

---

Calibrated Self Rewarding
image
Search space
Option
Describe the image and then tell me there a ball net in the picture?
Sentence group
High
Sentl
High
Sentence group
Option 1-5
"Predict the end"
Sentence
Sen12 :
Option 2-1
Souree Data
Sentence
Low
Sent5
Option
Sentence
Stronger VLLM
Direct Preference Optimization
Dynamic Preference Data
Choose the three with the highest scores the two with the loweal
"epcion ij" refirs the optica for the nesi samammo following i-th senienes.
Consider the response with the highest prefernd and the with the dispectiered
Figure 2: The CSR framework operates iterative process of preference data generation and learn- ing. During preference data generation, CSR utilizes a sentence-level beam search approach to con- struct responses sentence by sentence, assigning reward each sentence. This reward, initially generated by the model itself, is then calibrated using image-relevance information. Preferences are determined based the cumulative reward for each response. each iteration, CSR generates new preference data and performs preference learning based on this data, continuously enhancing the model's 's performance.
and dispreferred responses for the input prompt x (0) Preference optimization leverage the prefer- ence data optimize language models. Taking DPO [1S]] as a representative example,, it formulates the probability of obtaining each preference pair as p(u s) (r(2,, Mw) r(z, v1)), where (-))
the sigmoid function. DPO optimizes the language models with the following classification loss:
Te (Yw (r) Teer(Ww [z))
Ther (3) )
To ( (2r)
( Teet)
-E - (a log a log
a log
(1)
where (y(=x) represents the reference policy, i.e., the language model after performing supervised fine-tuning.
Calibrated Self- Self-Rewarding Vision Language Models
To address this challenge, we propose Calibrated Self- Rewarding (CSR), novel approach aimed at improving modality alignment in LVLMs by integrating visual constraints into the self-rewarding paradigm. As illustrated in Figure M CSR trains the target LVLM by alternately performing two stages: candidate response generation and preference curation and fine-tuning. In the candidate response generation stage, we employ sentence-level beam search for each input prompt to produce fine- finennrained candidate responses. During this process, the language decoder determines the initial reward for each generated sentence, which is then calibrated by incorporating an response relevance score. This calibrated reward score guides the generation of subsequent sentences and finally generate the entire response. Moving on to the preference curation and fine-tuning stage, we use the responses with the highest and lowest cumulative calibrated rewards to construct the preferred and dispreferred responses, and utilize the constructed preference pairs for fine-tuning. In the remaining of this section, we will provide detailed explanations of CSR.

---

3.1 Step-Level Reward Modeling and Calibration
Before delving into how to generate candidate response and construct preference data, in this section, we first discuss how to formulate the reward within CSR. The ideal reward in the LVLM fulfillss two specific criteria:
Vision-Constrained Reward: This aspect aims integrate image-relevance information into the reward definition of LVLMs. By doing so, we address the limitation of LVLM in overlooking image input data when generating preferences.
Step- Peep-Wise Reward: Instead of assigning single reward for the entire response, we opt for step- wise approach. This involves assigning rewards at each step of response generation. Compared to single reward, this finer-grained reward offers more detailed guidance and is more robust.
To fulfill these criteria, we propose a step-wise calibrated reward modeling strategy. Inspired by Process-Supervises Reward Models [TT6], assign reward score, R(s), to each generated sentence s during the sentence-lewe level beam search. This score is a combination of two components: the self- generated cnatuuctioo-followinn score, (s), and the image-response relevance score, (s).
Specifically, the self-generated instruction -aaruction-followinng score, (s)s), calculated using the language decoder of the LVLM. It represents the sentence- cumulative probability of generating sentence formulated as:
Rs(s) II P(r. 2,,1, To-11) t=1
where No is the number of tokens in sentence s and To represents token o in sentence A higher self-generated instruction-folloiin score indicates stronger capability of the generated response to follow instructions.
While the self-generated instruction-followee score partially reflects the LVLM's preference, it still suffers from modality misaligmment, potentially overlooking visual input information. To address this, we introduce image-response relevance score, Rr(s), to calibrate the reward score R (s). This score depicts the relevance between the generated sentence and input image t.p We leverage CLIP-score [17]] for this calculation, where the vision encoder in the CLIP model aligns with the vision encoder in the target LVLM. The image-response relevance score Ry (s) is defined as:
R max(100 (r)) 0),
where the (xx,) and (s) represent the visual CLIP embedding and textual CLIP embedding, respectively. Finally, the calibrated reward score R(s) for the generated sentence is defined
R(s) = A (s(s) + (1 (1- (s).
where À is a hyperparameter used to balance the language instruction woolowwfwloogggg and resgeeresponse relevance scores. By combining both scores, we aim to redirect the attention of LVLM towards the input visual information, thus enhancing its modality alignment ability.
3.2 Iterative Fine- Tine-Tuning
After establishing the reward framework in CSR, we next discuss our iterative fine-tuning process. Within this framework, we iteratively perform two essential steps, namely candidate response gen- eration and preference data curation and optimization. These steps are elaborated upon as follows:
3.2.1 Step-Level Candidate Response Generation
In candidate response generation, our objective is to generate responses to build preference data. To accomplish this, we employ sentence-level beam search strategy. Initially, we concurrently sample multiple candidate sentences, utilizing the "end of sub-sentence" marker (e.g., "." in English) as the delimiter. Subsequently, for each sentence s, we compute its reward score R(s) using Eqn. (e)) From these scores, we then select the top-k and bottom-k sentences with the highest and lowest reward scores, respectively, to proceed to the subsequent round of sentence-level beam search. This iterative process continues until reaching the "end of response," conventionally represented as (eos).
Once all sentences for response y {81, ...,.... &N,] } are generated, we calculate the cumulative reward score for the response as the sum of the reward scores for each sentence within it. This is

---

Algorithm Calibrated Self- Self-Rewarding
Require: Dataset: D = ),= Reference model: Number of iterations: T ,,....7'6 1
1: for ==
,
2:
for each x E D do
3: while not reach the end of response do
4:
Generate bunch of candidate sentences from last-round sentences
5:
for each candidate sentence do
6:
Compute the self-generated instruction-followinn score (s) by Eqn. (C) Calculate the image representation Fi (xx) and sentence representation Fr
7:
8:
Compute the image-response relevance score (s) by Eqn. (E)
9:
Compute the calibrated reward score R(s) ) by Eqn. (E!)
10:
Select top-k and bottom-k sentences with the highest and lowest reward scores Select the preferred response yo,, and dispreferred response ytt
11:
12:
Update arg ming (TTe; Teet), Tiet TT..
Ny R(s)), where Ng is the number of sentences in response y. The detailed defined R(y)= Σ i=1 algorithm for candidate response generation is outlined in Algorithm III
3.2.2 Preference Curation and Optimization
After generating candidate responses with their reward scores, next step is to curate preference dataset. Here, for each input prompt, we select the responses with the highest and lowest cumulative calibrated reward scores the preferred and dispreferred responses, respectively, preference dataset for fine-tuning. For each iteration I, we denote the constructed preference data as: D. = {(xx(), V.w.t: Vi,4 ) After obtaining the preference data, we fine-tune the target LVLM using DPO. At iteration
construct the
the last iteration fine-tuned model no,-, as the reference model. Following Eqn (I), the loss at iteration of CSR is defined as:
log Te (VW,,e) a log
To (M, no. (V1, (x))
)]
L. - (a. (.,,)... log
(S)
no, (Var, (1))
The training process of CSR is detailed in Algorithm M.
Experiment
In this section, empirically investigate CSR in addressing the modality misalignment problem of LVLMs, focusing on the following questions: (1) Can CSR help improve the performance of models on both comprehensive benchmarks and hallucination benchmarks? (2) Can CSR iteratively improve multimodal alignment progressively in LVLMs and lead to more factual LVLMs? (3) Is CSR compatible with different open-sourced LVLMs? (4) How does CSR change attention weights and preference pairs to align image and text modalities?
4.1 Experimental Setups
Implementation Details. We utilize LLaVA-1.5 7B and 13B [1] as the backbone models. During the preference learning process, we adapt LoRA fine-tuning [18]. The images and prompts used
to construct the preference data are randomly sampled from the detailed description and complex reasoning subclasses of the LLaVA150k dataset, totaling approximately 13,000 samples [[ is worth noting that each iteration uses the same prompt and image the previous round. Overall, the iterative training is conducted over three iterations, completed on one A100 80GB GPU. It takes roughly 3.5 and 5 hours for fine-tuning VLa 7B and LLaVA-1.5 13B, respectively. For more detailed information on training hyperparameters and training data, please refer to Appendix A.
Evaluation Benchmarks. We conducted evaluations on three types of benchmarks: comprehensive benchmarks, general VQA and hallucination benchmarks. Specifically, this includes: (1) Compre- hensive benchmarks (MME [20]] SEEDbench [2T]], LLaVA W [19], MMbench [22]], MM-Vet (23]);;
(2) General VQA (ScienceQA (SQA) [24]], VisWiz. [25]], GQA [26]); (3) Hallucination benchmark More detailed description are discussed in Appendix A.
(POPE [7]]
CHAIR [[]])
5

---

Table 1: The performance of CSR LLaVA-1.5 across all benchmarks is presented. Most baseline results, except those for self-rewarding. are sourced from Zhou et al. [
Comprehensive Benchmark
General VQA
Hallucination Benchmark
Mothod
MME"
MMF
SEED
LLaVA W MMB
MM-Vet
SQA'
VisWiz
GQA
POPE
CHAIRs
CHAIR;
LLaVA 1,5.70
1510.7
348.2
58.6
63.4
64.3
30,5
66.8
50.0
62.0
85,90
48.8
14.9
Vifeedback
1432.7
321.8
59.3
64.0
13.2
Human-Prefer
1490.6
335.0
58.1
63.7
63.4
31.1
65.8
51.7
61.3
81.50
38.7
11.3
POVID
1452.8
325.3
60.2
68.7
64.9
31.8
68.8
51.6
61.7
86.90
35.2
8.3
RLHF-V
1489.2
349.4
60.1
63.6
30.9
67.1
54.2
62.1
86,20
29.7
Self-rewarding
1505.6
362.5
60.0
61.2
64.5
31.4
69.6
53.9
61.7
86.88
24.0
6.7
CSR (Ours)
1524.2
367,9
65.4
70.7
1.. 1531.3
295.4
61.6
70.7
67.7
35.4
71.6
53.6
63.3
85,90
48.3
14.1
Self-rewarding
1529.0
300.
62.8
64.5
35.3
74.3
63.2
86.58
37.0
CSR (Ours)
1530.6
303,9
62.9
74.7
68.8
37.8
75.1
56.8
63.7
87.30
28.0
7,3
Baselines. We will first compare CSR with the self-rewarding approach described by Yuan et al. [29]. Here, directly apply self-rewarding to LVLM, using the prompts and experimental settings outlined in Yuan et al. [29]] (see detailed settings in Appendix A and Table () We also com- pared CSR with several data-driven preference learning methods, including Silkie (VIfeedback) [2], LLaVA-RLHF (Human-presd premeeeneeeeeecee [8]. POVID [TTO], and RLHF-V [30]. Furthermore, we com- pared the performance of the optimized LLaVA-1.5 via CSR with other state-of- of-t----tthe-tttt open-source LVLMs, including InstructBLIP [S]I], Qwen- Qwen-VL-Chai [62]] mPLUG-Ows
[[5]] BLIP-2 [64]], and IDEFICS [35]], after the final rounds of training (CSR with iteration 3). Additionally, evaluate the effectiveness of CSR on other LVLMs, we applied CSR to a recent LVLM called Vila [36]] For
more information these baselines, please refer to Appendix A.ll
4.2 Results
CSR Continuously Improves Model Perfor- mance over Iterations. In Figure D. we report
72.0
the average performance of LLaVA-1.5 7B and
71.0
70.0
13B models concerning the number of training it- crations on comprehensive benchmarks, general VQA tasks, and hallucination benchmarks. To fa- cilitate score calculation, we first calculated an average score
60.0
70.00
09.0
67.0
12-4 th-theel
68.0
LLoVA Basclne
Base
Hor
Iter
Bteratoo
100-point scale by adjusting the original values: MMEF was divided by 16, and MMEC divided by 4, corresponding the number of categories in MME. Additionally, since a lower CHAIR value indicates better performance, we standardized all metrics to follow higher is better approach by transforming the CHAIRs and CHAIR; metrics into 100 CHAIRs and 100 CHAIR, We then calculated the average score by averaging these standardized values, which were used to compute the average percentage increase. In the experiment, the 7B model achieved an improvement of approximately 7.62% across all benchmarks through online iterative updates, while the 13B model saw an improvement of approximately 5.25%. According to the full results in Table & and Table of Appendix A.S. the improvement is particularly significant on the LLaVA W and CHAIR benchmarks, with improvements of 8.9% and 49.50% respectively. The results indicate that CSR is capable incrementally improving model performance over iterations, demonstrating its effectiveness in self-improving the quality of generated preference data and leading to stronger modality alignment. The degree improvement gradually becomes smaller, which not surprising, indicating that the model is gradually converging.
of CSR different it- erations over all benchmarks (see Table B and Table in Appendix A.3 for full results).
Figure 3: Average
Ablation study of vision-text reward score.
Table 2:
CSR Outperforms Competitive Preference Fine- Tine-Tuning Base- lines. Compared to preference data curation approaches (e.g., POVID, RHLF-V) that generate preference data from either ad- ditional models or human annotations, the superiority of CSR in- dicates that adapting a self-rewarding paradigm better captures the inherent preferences of the target LVLMs, achieving stronger modality alignment. Furthermore, CSR outperforms existing self- rewarding methods, with an average performance improvement of
Method
7B
13B
Base
66.61
68.08
Only R
68.46
68.12
Only R
67.49
69.23
CSR (Ours)
72.39
71.95

---

2.43%, demonstrating its effectiveness in calibrating the reward model by incorporating image- response relevance scores. This mitigates the potential issue of overlooking visual input information when estimating self-generated preferences.
In addition, we compare the performance of L.L after three rounds of online CSR with other state-of-the-an open-soureed VLLMs and report the results in Table EI of Appendix A Although different open-sourced VLLMs utilize various image and text encoders, CSR still outperforms other open-sourced VLLMs in 9 out of 10 benchmarks, further corroborating the effectiveness of CSR in improving modality alignment.
4.3 Analysis
Ablation Study. To validate the effectivvness of using the image-response relevance score (R₃) to complement the self-generated instruction following score (Rr), we specifically compare CSR with three variants: (1) without applying CSR LLaVA 1.5 (Base); (2) using CSR with only the self-generated instruction following score (Only Rr); and (3) using CSR with only the image- response relevance score (Only Ry). The results are reported in Table D. We first observe that CSR improves performance by jointly considering both the self-generated instruction following and image-response relevance scores. This verifies its effectiveness enhancing modality alignment by calibrating the language-driven self-rewarding paradigm with visual constraints. Additionally, we further conduct the analysis on the change of in Equ (e) and found that incorporating external vi- sual scores to calibrate the models rewarding process effectively enhances performance (see detailed results in Appendix (A.5)
Compatibility Analysis. To validate CSR for its applicability to other LVLMs, we deployed CSR Vila 7B and conducted three rounds of online iterations. We conducted experiments on all evaluation benchmarks and tasks, and the results are shown in Figure g. Similar to the findings in Figure B. Vila demonstrates similar phenomenon during the online it- erations of CSR, where it can self-correct preferences, lead-
Vila 78
werage Score
72.0
71.0
ing to gradual improvements in all benchmarks. For Vila, the
overall performance improved by 3.37% after three rounds of CSR iterations, with particularly notable increases of 8.48% on VisWiz and 14.0% on MM-Vet. The compatibility analy- sis further corroborates the generalizability and effectiveness of CSR in enhancing the performance of LVLMs.
Base
iter 1
iter 2
iter 3
Figure Average scores of CSR in Vila 7B at different iterations over all benchmarks (see Table & in Appendix A. for full results).
How Does CSR Change the Image-Response Relevance Over Iterations? To investigate how CSR gradually improve
the performance over iterations, we analyzed the change of self- self--enertted preference data with the LLaVA-1.5 7B model. In Figure E. we illustrated the distribution of image-response relevance scores of three iterations over 500 examples from VLaV [T9]. We first observe that both the chosen (preferred) and rejected (dispreferred) responses achieve higher image-response relevance scores af- ter the model undergoes CSR online iterations. This indicates that, following CSR, the responses generated by LVLMs are more closely aligned with the image information. Secondly, it can be observed that after multiple rounds of online iterations with CSR, the average mnage-response rel- evance scores for the rejected and chosen responses become closer to each other. This makes the self self-generated preference data during CSR iterations more challenging to distinguish, while further strengthening the learning process.
How Does CSR Improve Modality Alignment? To fur- ther understand how CSR affects modality alignment, in
Base Rejected
6.2
Dasn Chasnn
CSR: (Durs) Chossn
Figure E. we present the changes in image and text atten- tion maps for three models: the original LLaVA-1.5 1.5 7B model, the self-rewarding approach, and CSR. These at- tention maps illustrate the distribution of attention scores over image and text tokens. We observe that applying CSR strengthens the model's attention to certain visual tokens. Simultaneously, the change of attention values of the text tokens indicates that CSR is capable of alleviating
030
0.02
0.00
10
15
Image Saggp---aaaeaaaeeo Faleaaca-Scorm
20
2
50
Figure 5: Image relevance scores before and after employing CSR.
7

---

Prompt: clements picture?
Vinaal Tokees
Tokens
+50 HRT ESIA
Self- Reward
LLaVA-15
Gemerate Responses
Visuslize Attemtions
CSR (Ours)
white driving down a city strect, the midile of the stree, and its large sine and promitent destination sign make it eaeily noticuable to pedostrians and other vehicles. additiion the bus, the also includes driving behind the indicating that both vehicles sharing roadway
image features
destination cleadyy displayod on "Aneeis." bus positioned
Self RelfRR
An articulated city bus. which is blue and white, driving down street in large city. la front the bus track, and the bus positioned behind Several people can be seen walking along or standing near both the bus und truck. They are likely pedestrans going about duily activities or waiting for public tansportation, buses, plays crucial communing and going about their daily rourines.
the bustling atmosghere busy daily city dwellers.
with
highlights public typical day large metropolitan area with people
trasportation
1LVA--15
image fentures blue significatt porticn
large huilding background, street, with car positioned down the streat on the right side There are seveni people in the soene, with one person standing nou the left edge of the image, another person closer to the and third penson right side of street. They appear pedestriass going about their daily activiiies, Additionally traffic visible the background, indicasing the bus driving urbun
driving image. cas
Excus occapying bus, another
traffic regulatios.
Figure 6: Comparison of attention maps After optimizing the model with CSR, the attention scores allocated visual tokens increase, indicating that CSR effectively redirects the model's attention toward the input visual information during the response generation process.
the issue of over-reliance on context mentioned in Huang et al. 5 Additionally, compared with the self-rewarding approach, CSR shows more effective distribution of attention between image and text tokens. These findings indicate that with CSR, LVLMs can better align different modali- ties through calibrated self-rewarding strategy, focusing more on the visual modality rather than over-relying contextual text.
4.4
Case Study
In this section, we use LLaVA-1.5 13B as an example to illustrate changes in the models own re- sponses during CSR iterations and the preference data sampled in the CSR learning process, with hallucinations and errors highlighted in red. The results are shown in Figures M and A respectively. As shown Figures with each iteration of CSR, hallucinations in the models responses notice- ably decrease. This indicates that CSR effectively refines the models preferences through iterative preference learning, making the models responses more accurate. In Figure B of the Appendix, we present sampled preference data pair from the model during the CSR learning process. It can be seen that through CSR, the model not only gradually refines its own preferences but also obtains high-quality preference data pairs without human annotation.
Theoretical Explanation
In this section, present theoretical framework to explain the empirical phenomenon that incor- porating an image-response relevance score can calibrate the self-rewarding procedure, ultimately improving generation accuracy.
As we consider an LVLM, to facilitate the analysis, we decompose the input prompt into x (Iv, 22) E dd x Rd, representing the image and text prompts respectively. Although text data

---

ISAM DOL
REAS
Describe this Image
Deseribe detull.
1)
t20 CSR (Deeattion 31
The image Soturs red mack rnterne to as Pelpero Tham Co. has trveling on the stret with screnu pasergans on boad. The pavergars, indlddin siddls und chilkhen, appear to be erjeying tiimealivss es they ride along the oour The has is postroned the cester the scens. tacins saberzziaa portize of inaae. are for pessengers, enjying sirvophars
boldina child been They surddnng diee the artesd, peesiily ftading obseninn up chee. The woraa and the clidd agpear be the cecte of ccttrtic in soene.
insracciaa
like
so they are fieuadd in inc improsese aseral The waree ie weating a his ahirt and shart, which acceet the Serily-or enviooment. The origre and sieee-pp insrrction betwoee the
bahind
visille stree, some Googroend others
a
enparing
mocker diil,, the asizal saganb menerraalle and alectioona esperiane
@ CSR (focatiin 3)
dis CSR (lteration 2)
the saage faaaree warnn ausding next two childnes, who are aakding onto
fhis perticalo dhikdinn appar be ot iranneting with boows hornal
The image shosesses old and bus with Petyere Trurs Co. wittes it, diiving along
they sight forthe childion
country moze The hos in lnaded with paoempon, inclading lead sino people stehte thoouhh
te windows, shting and liiig in a relaoed reaznee As the bas travds down the street
The waron, chlivees and Yok all store the same space, and it seems as ('thee having
appeare in he dluerinn and orogalii tmp for the pauurgan, with thvdy not color the
mummmaale and clacaiona experkmne agather the eoe, The waren scom to be cuusuin: with lids is ectiorng the edtiii and staring trarveing infernation about the Yoi, while cojie usiqu encoute
boo ciding touch of passeniity to the rnd setling.
dis (lteration I)
The image fnstures larg, dld fishiooed und Ias, posiitt molley parked and wailing on
ab CSR (lecatios 1)
cit strae. The tax packed with passengers who can be son through the windows, with The mmage depins femily wath young cxte and baby steting a patting soo, where they aee
lost aise people visitie oeboad. It appeas the the bus other picking up or Enoping off
weding feeding a large funy brwn askeal posdlly yau. caas loiding is
peovangers as moves thouuhh the cit.
enpagee
kage
parked in a buildin, surounded otser vehides noaty. There sponod anordd exponsing has haating appansncc
with
joyful
ively amncopiore.
rellecing lygansc
A hareh can som furthr beck in the scree sugaorine the onr has seainng anca fir vistore
to ras and cajoy the viow The grounce the lergy anlind and the heppy people samvesting
crean warm and fieedly catinnes the pening coe.
LLaVA-1.5118
db LLLV3-15--18
pescibly harope. The hes a fall of pasenger waiing on the dive ana of the vehids, white
The inage shows rod men thi parkodd on the side of sneet with mrss the baokgroood
the image, nwo litle girls are being coddied by thair mother on a street They semr to be stjeiing fasr lane asather onaiks bedkgioand, asevd of people can be sons giving ieprosmion
oelookers wash the scese Boon nearly. tuul, cleves people cas seen other walling oz the has eanding by the enezt.
oleers benckes unieettss scatered. Adltionally a aceee The procece the clock aggesta that the Incation could be on aos with dieps or catte A handhng notioaale belonging one of the adults in sceae
therr
In adition red barn
two cans visibee is some. Oue ou behial and lu the found
clock viable
the right iide ereet. louted dose un paoceggers ledkeging
There
Figure 7: Two cases selected from the CSR- esR-genertted datasets.
typically comprises discrete tokens, we follow the CLIP theory literature [38 [I] in modeling them as continuous-value random vectors in this section to elucidate the rationale behind our proposed method. More specifically, we assume the following data generative model for Iy and v4:
Fr. = U121 E1, and xy U2 + E2.
where U, E Or and U2 E 04 xr are two orthonormal matrixces, representing decoders that transform the latent (low-dimensional) signals 21, IR" images and text respectively. We assume the covariance matrices of ø1, 22 are identity matrices. Ei E RRd and & E IR4 are noise vectors, and assume they follow sub-gaussian distributions with well-conditioned covariance matrices and sub-gaussian norms upper bounded by a universal constant. We consider the infinite data setting. This is a widely used simplification avoid the influence of sample randomness [II]- A3 According to [[8]. with an abundance of image-text pairs, the learned visual CLIP embedding Fi (Ix) and textual CLIP embedding Fr (z:) converge to UT xy and U. 2, respectively. To simplify our analysis without loss of generality, we consider single score for each response y and define the image-response relevance score R₁ (g) (U, Ze. U2 v).
We assume the ground truth Vtrwth Vi* x. + Viz,, + Ey with weights Vi E Rd and V e Rd. In CSR, we assume the conditional distribution at iteration t, Te, (y x)) with O, (Vi 1/2). follows a Gaussian distribution Te, (y x) x exp(-lly
(Vix, + V22,) (bxx:)1///// /2//"), where Vi2 E Rd,xd, and V/2 Rdd, are the weights matrices for the image and text imputs respectively, and > 0 is the standard deviation. As the likelihood is monotonically decreasing with respect to Ily 11y-60 (Vix, + V2 we consider the self-generated cnstruction-following score Rr(y) - lly (V + V2 )l1 Then the calibrated reward score becomes R(y) = X (v( + (1 - A) (v), for some E [0, 1]. In theoretical analysis, we consider a simpler version of CSR, where we assume yau arg max, R(y) (whose distribution is denoted by ph, (v/ x)), and y1 is the text output generated by no, (y/ x:). As R(y) depends on X, we denote the solution 0.+1 by 0 In the
special case where = 1, this corresponds to the setting where we do not use the image-response relevance score at all.
9

---

To evaluate the quality of the text output y, we consider regression problem where there is an outcome 2 associated with the ground -ruth text output ytruth 2 S Mtrwth with 5 E R We evaluate the quality of by considering the loss function I(y) = min seR* El(2 g) We then have the following theorem.
Theorem 5.1. Suppose that Tे. (y x) lies in the LLM space { x) θ). | T || I-1l and [13** Vi 3*1l < 113 1/2 11 then there exists X < 1, such that
Eme 1a (v [[y)] < Brg, 1c>(ol))))))) [Ly)]
Our theoretical analysis implies that as long as V |-1| < VT the model tends to prioritize textual information over visual input. By incorporating the image- response relevance score (corresponding to À < 1), CSR is able to increase the attention on image signals in generating y. As result, the solution produced by CSR will be better than the method without using the image-response relevance score (corresponding to x=
3+1|, VT which happens when
6 Related Work
Large Langal-L Model Hallucination. Recently, the rapid development of visual-language
alignment methods [19 PED [o and LLMs [500-44] has significantly accelerated the progress of LVLMs, which extend LLMS with visual modalities and demonstrate impressive visual understand- ing by unifying the encoding of visual and text tokens [3A], 55-577. However, LVLMs still face the problem of hallucination [58, 59], where generated text descriptions contradict the visual modal- ity information. Various approaches have been proposed to address hallucination in LVLMs, in- cluding enhancing dataset quality for fine-tuning [600, B. 611, 9I, manipulating the decoding pro- cess [37] 62 666], and leveraging external closed-source models to facilitate post-hoc mitigation of hallucination [58, 677770 Though these approaches alleviate hallucination to some extent, they do not focus directly on improving modality alignment.
Preference and Modality Alignment. In large models, alignment is necessary to ensure their be- havior aligns with human preferences [ZT], IS. 72]. In LVLMs, alignment manifests modality misalignment, where the generated textual responses are supposed to follow the input visual in- formation. Recently, preference optimization has been used to address the modality misalignment problem. These optimizations involve preference data curated by human annotators [8, 600, 30] and additional models (e.g.. GPT-4) [X IOJ. While these methods improve the ability of LVLMs to align modalities, their reliance on human annotation or additional models is resource-intensive and may introduce additional biases. Furthermore, these models cannot fully capture the inherent preferences of LVLMMs making the curated preference data less effective. Instead, CSR leverages a calibrated self-rewarding strategy, aiming to stimulate the LVLMs' self-correction and enhancement capabili- ties, thereby further improving modality alignment.
Self- Seff-Improvement in Large Language Models. Self-improvement emerges powerful paradigm for LLMs to enhance themselves without significant external intervention. For exam- ple, self-rewarding and online alignment propose method that selects consistent answers generated thereby improving its reasoning ability. Similarly, Chen et al. [022] utilizes self-play to enhance the model's performance by distinguishing its self -eferaeddd responses from those in annotated training data. Unlike prior methods that primarily target LLMs,, CSR addresses the modality misalignment issue LVLMS during the preference modeling process by introducing visual constraints, making it particularly well-suited for LVLMs.
by the model to fine-tune itself [77 [Z!]
7 Conclusion
In this paper, we investigate the challenge of enhancing modality alignment in LVLMs by intro- ducing a calibrated self-rewarding approach, which integrates visual constraints into the preference modeling process of the self-rewarding paradigm. Empirically, CSR enhances the alignment be- tween image and text modalities, significantly reducing hallucination and improving performance on various LVLM evaluation benchmarks. These empirical results are further supported by rigorous theoretical findings. Additionally, CSR is capable of continuously enhancing LVLM capabilities over iterations, leading to better utilization of visual information.
10

---

Acknowledgement
We thank the Center for AI Safety for supporting our computing needs. This research was supported by Cisco Faculty Research Award.
References
[1] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024.
[2] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general purpose vision- language models with instruction tuning, 2023.
[3] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[4] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vi vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.. 12966, 2023.
[5] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint vrXiv:1809.02/555 2018.
[6] Chenhang Cui, Yiyang Zhou, Xinyu Yang. Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:2311.03253 2023.
[7] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: An advanced diagnostic suite for entangled language hallucination visual illusion large vision-language models. arXiv preprint arXiv:23100 14566, 2023.
[8] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yar Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
[9] Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.10665 2023.
[10] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Align- ing modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.1 1411, 2024.
[11] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024.
[12] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.0133 2024.
[13] Alex Graves. Sequence transduction with recurrent neural networks. arXiv: 2012.
arXiv preprint
[14] Ilya Sutskever, Oriol Vinyals. and Quoc V Lc. Sequence to sequence learning with neural networks. Advances neural information processing systems, 27. 2014.
[15] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In seveth Conference on Neural Information Processing Systems, 2023. URL https:
//arxiv. wwg.abs.////oeooooooo...oee.eert
abs/ 2305 18290.

---

[16] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
[17] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. CoRR, abs/2104.08718, 2021. URL
https://arxiv. org/ hbtp//ab/.2000000 08718.
[18] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[19] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint 2044:2234.088485, 2023.
[20] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
[21] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprehension, 2023.
[22] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024.
[23] Weihao Yu. Zhengyuan Yang. Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabil- ities, 2023.
[24] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Pro- cessing Systems (NeurIPS), 2022.
[25] Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people, 2018.
[26] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual rea- soning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-67000 2019.
[27] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision vasion-language models. arXiv preprint arXiv:2305. 10355, 2023.
[28) Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning, 2019.
[29] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024.
[30] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. arXiv preprint 2.1v:23112.08445 2023.
[31] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards purposs vision- language models with instruction tuning, 2023.
[32] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vasion-language model for understanding. lo- calization, text reading, and beyond, 2023.
12

---

[33] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration, 2023. 2023.
[34]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2 Bootstrapping language- image pre-training with frozen image encoders and large language models, 2023.
[35] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset interleaved image-text documents, 2023.
[36] Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, arXiv preprint arXiv: 2023.
[37] Qidong Huang. Xiaoyi Dong, Pan Zhang, Bin Wang. Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and citospection-allocatio arXiv preprint arXiv:2311.17911 2023.
[38] Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji. James Zou, and Linjun Zhang Understanding multimodal contrastive learning and incorporating unpaired data. In Interna- tional Conference on Artificial Intelligence and Statistics, pages 4348-4380. PMLR, 2023.
[39] Zixiang Chen, Yihe Deng, Yuanzhi Li, and Quanquan Gu. Understanding transferable repre- sentation learning and zero-shot transfer in clip. arXiv preprint arXiv:2310.0092) 2023.
[40] Yuhang Liu, Zhen Zhang, Dong Gong, Biwei Huang. Mingming Gong, Anton den Hengel, Kun Zhang, and Javen Qinfeng Shi. Revealing multimodal contrastive representation learning through latent partial causal models. arXiv preprint arXiv:2402.0622: 2024.
[41] Michael P Kim, Amirata Ghorbani, and James Zou. Multiaceuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 247-254, 2019.
[42] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two- layers neural networks in high dimension. 2021.
[43] Haotian Ye, James Zou, and Linjun Zhang. Freeze then train: Towards provable representa- tion learning under spurious correlations and feature noise. In International Conference on Artificial Intelligence and Statistics, pages 8968-8990. PMLR, 2023.
[44] Jean-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Miko- laj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. 2022.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[46] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024.
[47] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9561 2024.
[48] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023.
13

---

[49] Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li, al. Mmie: Massive multimodal interleaved com- prehension benchmark for large vision-language models. arXiv preprint 2024.
[50] Wei Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez. lon Stoica, and Eric P. Xing. Vi- cuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality, March 2023, URL
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 139711 2023.
[52] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[53] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
[54] 01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang, Tao Yu. Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01 01.ai, 2024.
[55] Wenliang Dai. Junnan Li. Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[56] Deyao Zhu, Jun Chen, Xiaogian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models, 2023.
[57] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023. URL https:
/ //www.ades ai/ wwwbblolgguuuue-
[58] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mo- hit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision- language models. arXiv preprini 1rX0v:2310.00754. 2023.
[59] Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, and Yu- Yu-Gang Jiang. Eventhallu- sion: Diagnosing event hallucinations in video llms. arXiv preprint arXiv:2409. 2697977799,6697, 2024.
[60] Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models, 2024.
[61] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.1456 14565, 2023.
[62] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large lisiounlanguage models through visual con- trastive decoding. arXiv preprint arXiv:2311. 2.:::2311.1000 16922, 2023.
[63] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual in- struction data, 2024.
14

---

[64] Zongbo Han, Zechen Bai, Haiyang Mei, Qianli Xu, Changqing Zhang, and Mike Zheng Shou. Skip: A simple method to reduce hallucination in large laiguale models. arXiv preprint arXiv2240..01345, 2024.
[65] Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint 2033::2403.00425, 2024.
[66] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Li Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. ArXiv, abs/2311..19222 2023. URL https://api. semanticacholarr org/ CorpusLD:26b466823
[67] Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li. Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint 1IXi:22310116055 2023.
[68] Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable multimodal rag for factuality medical vision language models. arXiv preprint 2xXiv:2477.0513/ 2024.
[69] Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. Mmed-rag: Versatile multimodal rag system for medical vision language models. arXiv preprint arXiv:2410. 13085, 2024.
[70] Hao Shao. Shengju Qian, Han Xiao, Guanglu Song. Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of-thoup reasoning in multi-modal language models. arXiv preprint arXiv:2403. 16999, 2024.
[71] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020.
[72] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of human preferences in dialog. 2020. URL abtps://opeareviev..no wwt.orrrrreeetrrr
Id=r.
[73] Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self- ilf-oier 2022.
[74] Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, and Huaxiu Yao. Cream: Consistency regularized self-rewarding language models. arXiv preprint arXiv:2410.1221 12735, 2024.
[75] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Zitnick, and Piotr Dollár, Microsoft coco: Com- mon objects in context, 2015.
15

---

A Additional Results
A.1 Experimental Setup
A.1.1 Hyperparameter Settings
Sentence-Level Beam Search. We configure our pa- rameters follows to ensure both diversity and qual- ity in the sampled data. The num beamsparameter, set to 5, determines the capacity of input at each search layer. Additionally, set to ensures that each beam search returns token-level search results. The cos cos_ token id is set to the token for a period, effectively controlling the sentence-by be-setence generation process. The max max_length parameter, set to 1024, prevents trunca- tion errors and infinite repetitions by controlling the maximum length, while max_ __ww sew_toeess, set to 74, limits the maximum length of newly generated con- tent to avoid exceeding the CLIP encoding limit.
60
LVLM Pretemed
LUM Dispreforied
CP Disprefereed
also
0 0.2
0.3
0.4
0.5
0.6
0.7
0.0
0.9
1.0
Sampling Probability Score
Figure 8: Distribution of preferred responses and dispreferred responses based on the sampling probability scores generated by LVLMs' language models.
To further enhance data diversity, we utilize group beam search by setting the sum_beam_gree
parameter to This approach, when matched with token-level search, significantly boosts the diversity of each data point. The diversity pennllyyeelltyy parameter, set to value of 3.0, effectively controls the diversity and quality of the sampled data among different beam groups.
Calibrated Rewarding. We set the clip score weight to 0.9 and the language score weight to 0.1 when calculating the scores, giving greater emphasis to visual calibration.
A.2 Evaluation Metrics and Benchmarks
MME [20]] is comprehensive benchmark for assessing the capabilities of LVLMs in multimodal tasks. systematically evaluates models across two primary dimensions: perception and cog- nition, through 14 meticulously designed subtasks that challenge the models' interpretative and analytical skills.
SEED-Bench [21]] is designed to evaluate the generative comprehension capabilities of LVLMs. It features an extensive dataset of 19K multiple-choice questions with precise human annotations, covering 12 distinct evaluation dimensions that assess both spatial and temporal understanding across image and video modalities.
LLaVA W | is comprehensive benchmark for evaluating visual reasoning models. It comprises 24 diverse images with a total of 60 questions, covering a range of scenarios from indoor and outdoor settings to abstract art.
MMBench 2 introduces a dual-pronged approach: a meticulously curated dataset that signif- icantly expands the scope and diversity of evaluation questions, and pioneering CircularEval strategy that leverages ChatGPT to transform free-form predictions into structured choices.
MM-Vet [23]] is an evaluation benchmark tailored for assessing the multifaceted competencies of LVLMs. It systematically structures complex multimodal tasks into 16 distinct integrations derived from combination of core rision-language capabilities, providing a granular analysis of model performance across diverse question types and answer styles.
ScienceQA [2a|] is multimodal benchmark designed evaluate and diagnose the multi-hop rea- soning ability and interpretability of AI systems within the domain of science. It offers an expan- sive dataset of approximately 21k multiple-choice questions across a broad spectrum of scientific topics, complemented by detailed answer annotations, associated lectures, and explanations.
VizWiz [25] is a dataset the field visual question answering (VQA), derived from a natural- istic setting with over 31,000 visual questions. It is distinguished by its goal-oriented approach, featuring images captured by blind individuals and accompanied by their spoken queries, along with crowdsoureed answers.
16

---

is a dataset engineered for advanced real-world visual reasoning, utilizing scene graph- based structures to generate 22 million diverse. comanticlly-programmine questions. It incorpo- rates a novel evaluation metries suite focused on consistency, grounding, and plausibility, estab- lishing rigorous standard for assessing in vision-language tasks.
GQA
[
POPE [27]] an assessment methodology designed to scrutinize object hallucination in LVL.Ms It reformulates the evaluation into a binary classification task, prompting LVLMs with straight- forward Yes-or-No queries to identify hallucinated objects. POPE offers a stable and adaptable approach, utilizing various object sampling strategies to reveal model tendencies towards halluci- nation.
CHAIR [28] is a widely-recognized tool for evaluating the incidence of object hallucination in image captioning tasks, which has two variants: CHAIR, and CHAIRs. which assess object hallu- cination the instance and sentence levels, respectively. Formulated as:
CHAIR, |hhalluiiaatdd objects]|
CHAIRg = |cpptions with hallucinated objects}|
[all mentioned objects}||
{all captions]|
Specifically, we randomly sampled 500 images from the COCO validation set and evaluated object hallucination using the CHAIR metric.
A.3 Overview of the Baselines
LLaVA-1.5 [T]] is an improvement based on the original LLaVA [ model demonstrating excep- tional performance and data efficiency through visual instruction tuning. It enhanced with CLIP- ViT-L-336px visual backbone and MLP projection. By incorporating academic-taskkooee task-oriented VQA data and simple response formatting prompts. LLaVA-1.5 achieves the state-of state-of-the-ar -hrttrrtt re- sults at that time with remarkably modest dataset of just 1.2 million public images.
InstructBLIP [+T] leverages instruction tuning on pretrained BLIP-2 models, integrating an instruction-aware Query Transformer enhance feature extraction for diverse vision-language tasks. It achieved state-of-the-art zero-shot performance at the time across 13 datasets and ex- celled in fine-tuned downstream tasks, such as ScienceQA, showcasing its advantage con- temporancous multimodal models.
Dwen-VL--CHe [52] is built upon the Qwen-LM [ with a specialized visual receptor and input- output interface. It is trained through 3-stage process and enhanced with multilingual multi-
modal corpus, enabling advanced grounding and text-reading capabilities.
mPLUG-Ow12 [ES] employs a modular network design with a language decoder interface for unified modality management. It integrates shared modules for cross-modal collaboration and modality-adaptive components for feature retention, enhancing generalization in both text-only and multimodal tasks.
BLIP-2 [64] vision- lssiuull pre-training framework that efficiently leverages off-the-sheli frozen image encoders and LLMs. Employing a two-stage pre- treiirrining strategy with lightweight Querying Transformer, BLIP-2 bridges the modality gap between vision and language. enabling zero-shot image-to-tex- generation that adheres to natural language instructions while maintaining high compute-efficient
IDEFICS [3>}] is an open-access visual language model that expands upon the Flamingo [+] architecture, offering both base and instructed variants with 9 billion and 80 billion parameter sizes. It is developed using solely publicly available data and models.
POVID [MC]] is novel training paradigm aligns the preferences of VLLMs through external pref- erence data from GPT4 and the inherent hallucination patterns within the model triggered by noisy images.
RLHF-V [50] collected fine- -ne-grrined paragraph-Ieve corrections from humans on hallucinations and performing dense direct preference optimization on the human feedback.
Silkie [9]] constructed a VLFeedback dataset using VLLMs annotation. Specifically, the responses were generated by 12 LVLMs models conditioned on multimodal instructions extracted from dif- ferent datasets. The entire dataset was evaluated using GPT-4V to assess the generated outputs in terms of helpfulness, visual faithfulness, and ethical considerations. this paper, the WT.eeebbcck dataset was utilized to perform one round of DPO on ILLVA-1.5
17

---

LLaVA-RLHP [8] proposes novel alignment algorithm called Factually Augmented RLHF, which enhances the reward model by incorporating additional factual information such as im- age captions and ground-truth multi-choice options. In this paper, the annotated preference data used to conduct one round of preference learning on LLaVA1.5.
Selardew [M introduces a method for self-fcedback learning in LLMs and serves as base- line for our approach, referred to as CSR. Specifically, for each input image and prompt, two outputs are sampled from LLaVA-1.5. The model provided with the prompt mentioned Ta- ble 3 and is tasked with determining which output is better. Finally, LLaVA-1.5 is fine-tuned using the collected preference data, with the entire setup and the images and prompts used for inference matching those of CSR.
A.4 Do Different Sources of Preference Data Have Different Impacts?
The sources of preference data generally fall into two main categories; external preference data and celf-generated data. External preference data typically represent preferences obtained from human annotations GPT-4. Although externll preference data generally have higher quality compared to celf-generated data, are they really more effective? We conducted an analysis using 500 samples obtained from the original VA-1.5 7B model. Following the same pipcline CSR, selected samples with the highest and lowest rewards as preferred (chosen) and dispreferred (rejected) re- sponses. We further employed the GPT-4 API to transform preferred responses into dispreferred ones, with specific prompts referenced in Table !L
In Figure & we present the distribution based on both the sampling probabilities score generated by the target LVLM, which describes the probability the LVLM generating this response. Clearly, compared to the model's own generated dispreferred responses, the dispreferred responses modified by GPT-4V are not as easily confusable for the model. This result partially supports the idea that dispreferred responses generated by external models are more easily distinguishable by the target LVLM, making them less effective.
Table 3: Prompt for self-reward: utilizing the model itself judge to determine whether the corresponding response is chosen response or a reject response.
Now you act as a judge, helping me determine which of the two texts provide closer to the given image and has fewer errors.
Response
(response 1}
Response 2:
(response 2)
Please strictly follow the following format requirements when outputting, and don't have any other unnecessary words.
Output Format:
response or response 2.
A.5 Additional Experiments
In this subsection, we provide a comparison of CSR with other state-of-the-art models, a perfor- mance comparison of different CSR iterations, comparison of hallucinations. in different CSR iterations, validation experiments of CSR on other models, ablation study on À in Eqn (E)) and the relationship between reward score and average performance score. Experiments strongly demon- strate the effectiveness of CSR.
For the ablation study À in Eqn (H)), our training settings are consistent with Table II with three rounds of iteration. The experimental results Table show that as the value of À increases, the models performance various benchmarks improves. This suggests that calibrating the models
18

---

Table Prompt for GPT-4 API: transform the provided response into negative ones based on the provided image.
Transform the provided response into negative ones based on the provided image. **
Response:
[chosen response from another LVLM or ground truth)
Requirements:
(1) Revise the response while maintaining its original format and order as much possible.
(2) Based on the provided image, primarily add, replace, modify entities in the imput response to make them related to the image but incorrect. Adjust their attributes and logical relationships accordingly.
(3) The modifications in (2) must align with the image information, making the revised result difficult to discern.
M W- WWw.AA/wwwwww............
Please strictly follow the following format requirements when outputting, and don't have any other unnecessary words.
Output Format: negative response
Table 5: Comparison of LLaVA-1.5 with CSR and other open-sourced state-of-the-art LVLMs.
Comprehensive Benchmark
General VQA
Method
MME
MME
SEED
LLaVA W
MMB MM-Vas Vet
SQA!
VisWiz GQA
BLIP-2
1293.8
290.0
46.4
38.1
22.4
61.0
19.6
41.0
InstructBLIP
1212.8
291.8
53.4
60.9
36.0
26.2
60.5
34.5
49.2
IDEFICS
1177.3
45.0
45.0
48.2
30.0
35.5
38.4
Qwen-VL-Chat
1487.6
360.7
58.2
67.7
60.6
47.3
68.2
38.9
57.5
mPI.U--w122
1450.2
313.2
57.8
59.9
64.5
36.2
68.7
54.5
CSR ittr-3
1524.2
367.9
71.1
65.4
33.9
70.7
54.1
CSR iter-3 13B 1530.6
303.9
62.9
74.7
68.8
37.8
75.1
56.8
63.7
Table 6: The performance of CSR online iteration with LLa as the backbone on comprehen- sive benchmarks and general VQA.
Comprehensive Benchmark
General VQA
Method
MME
MMEC
SEED LLaVA
MMB MM- Vet SQA'
VisWiz GQA
ALLVA-1.55-B
1510.7
348.2
58.6
63.4
30.5
66.8
50.0
62.0
CSR iter-1
1500.6
367.5
60.4
69.7
64.7
32.2
70.3
54.0
62.1
+ CSR iter-2
1519.0
368.9
60.3
70.4
65.2
33.7
70.1
54.0
62.3
CSR iter-3
1524.2
367.9
60.3
71.1
65.4
33.9
70.7
54.1
62.3
CSR iter-4
1524.6
368.8
60.4
71.0
65.3
33,9
70.4
54.0
62.2
CSR iter-5
1520.1
367.2
60.5
71.3
65.4
70.8
54.2
62.4
LLLVA--.5-153
1531.3
295.4
61.6
70.7
67.7
35.4
71.6
53.6
63.3
+ CSR iter-1
1533.1
303.6
63.0
74.4
68.4
37.4
74.8
56.8
63.2
CSR iter-2
1530.4
301.1
63.0
74.3
68.5
37.2
75.0
56.0
63.2
CSR iter-3
1530.6
303.9
62.9
74.7
68.8
37.8
75.1
56.8
63.7
CSR iter-4
1530.4
301.4
63.0
74.2
68.3
37.3
75.2
56.6
63.4
CSR iter-5
1531.1
302.2
62.8
74.0
68.2
37.4
74.8
56.7
63.7
rewarding process using the visual score can enhance the preference learning process, thereby boost- ing performance.
19

---

Table 7: The performance of CSR online iteration with LLaVA-1.5 as the backbone hallucination benchmarks.
Hallucination Benchmark
Method
POPEacc
РOРEn
CHAIRs CHAIR: Avg Length
LLaVA-I.5-7BB
85.90
84.29
48.8
14.9
89.03
CSR iter-1
86.94
85.80
7.2
80.59
CSR iter-2
86.82
85.62
23.0
6.1
82.62
CSR iter-3
87.01
85,93
21.0
6.0
83.29
CSR iter-4
87.05
85.95
19.0
5.9
81.34
CSR iter-5
87.16
85.98
18.3
5.4
82.07
LLaVA-1.5-133
85.90
84.87
48.3
14.1
89.73
CSR iter-1
87.28
86.29
9.0
98.85
CSR iter-2
87.33
86.36
36.0
7.8
105.0
CSR iter-3
87.30
86.31
28.0
7.3
107.8
CSR iter-4
87.20
86.58
27.4
7.4
112.3
CSR iter-5
87.18
86.51
28.0
7.3
102.4
Table The performance of CSR online iteration with Vila 7B as the backbone.
Comprehensive Benchmark
General VQA
Hallucination Benehmark
Method
MME
MME
SEED
LLaVA W MMB
MM-Vet
SQA'
VisWiz
GQA
POPE
CHAIRs CHAIR;
Vila
1533.0
316.4
61.1
69.7
68.9
34.9
68.2
57.8
62.3
85,50
31.0
8.8
CSR
321.9
63.2
71.9
CSR iter-
1536.0
322.6
63.4
74.2
69.1
39.7
72.3
62.6
62.1
87.30
28.2
8.0
1542.2
321.5
63.4
69.3
72.2
62.7
62.4
Table 9: Performance comparison of CSR on LLaVA-1.5 7B with different À values on various benchmarks.
Method
MME
MMF
SEED
I.a.VA W
MMB
MM.
SQA!
VieWiz
GQA
POPE CHAIRs CHAIR;
(A 0.1)
1508.6
369.3
60.0
66.7
64.9
31.6
70.0
54.0
62.0
86.90
40.8
10.2
0.5)
32.4
0.9)
1524.2
367.9
60.3
71.1
65.4
33,9
70.7
54.1
62.3
87.01
21.0
6.0
Table 10: Reward score and average performance score across multiple iterations of CSR on LLaVA- 1.5 7B.
Iteration
Iter-1
Iter-2
Iter-3
Iter-4
Iter-5
Chosen reward
0.4885
0.5040
0,5052
0.5055
0.5066
Rejected reward
0.4551
0.4788
0.4789
0.4794
0.4799
Avg performance score
66.61
71.02
71.74
72.09
72.24
B Proofs
Proof of Theorem 5. Let us first denote the distribution of yo by Tū,(1) x). As take yor arg max, R(y). this distribution a point mass. As result. the global minimizer to (E) will then converge to Tē. (y z).
In the following, we analyze how тह (y x) is shaped.
By the CSR procedure, we have
Yo arg max (UT F.. UT) - Alyy Vi + V20x212=
U₁U We + Via, + V2s
X
We that CSR up-weights the signal of the image input.
20

---

Model Inputs
Chosen by CSR
The image fissaros wooden fance with sign attahed to ii, waming visiiee to destroy dhiir sell phones before entering Numerous old cel. phones are uttached the lenze in various positions, creating inique znd eye-cunching display. Seme the call phones plooo clour the sign, while others scallened throghott the fancr. stoog message
diverity technelogy the devices, Overill serves remiinee the histariaal evolution cell phones impact they The wooden lence and the sign create a vistally interesting contrett with the cell phones, emphasizing the message the sign and drowing atiention to the display.
diflarsst
sizes,
importince discomoeting
present
also
our
Rejected CSR
A wooden fince adomeed with mineroee cell phones hanging on There phones are of various sizes and styles, creasing unique display, total, Shere are cell phones visble on the feace.
The cell phones are aranneed in way that they cover the entire fence, with some phones hanging higher than others. that these plance lave hoan cuillated and
There mis of old and now wzll phanes in this collastion. displayed
and nostalgi ethiit..
Prompt:
Adkitivenlly. there
the ride fanas people observing the phone display. right side. Bods individuals seen providing a place for people sit and enjoy this unique sight.
Desembee the follewing image.
alo seen the background,
Figure 9: A case including both self-generated preferred and dispreferred responses.
Then
L(y) = min E[(2 - y) RER4
min B[(B* Ytruth - y)
SER4:
min BI(B" (Vi + [Vzx+)) - =} + Var Var(e))113M SER4:
We have
E((B* E[(BTT(Vi'z (Vitz, + +[)x))) - 3T y) = (Vi + v1/x())
BT (D~ABiU + Vi)a + V2 )
As we assume V
Vill and due to the smoothness over parameters. Without loss of generality, we prove the claim for the case where [ || = 0. that is V/=0..
Vi
114 B
VTT
In this case, want show that there exists À (0, 1), such that
E[(3T (V v(z=,) 3T ( 1 U.UT)a Varre) ) )²]
min 6ER4
< min E (Vi* + v(zz,) - 3 (V2x:)) BEX
Due to the independence between Iv and Ft, the right-hand sides is lower bounded by 3 V/Coor(z;) 3*.
The left-hand side, on the other hand, can be upper bounded by the value when we take Bo such that L,A Bo = U,U,T Vi 3r, which equals to 3 T (i - U,U, )ou(()))))))))))) V 8r.
As we assume ||3- Vi+ 8*1 > 1|8 V/2 1* this is a dominating term when the left-hand side is evaluated Su.
In addition, we assume Cor(E1) well-conditioned, implying Cov(x;) is well-conditioned, and therefore
3r <
8r.
We complete the proof.
C Limitations
Due to limitations computing resources, we conducted only three iterations of CSR. Additionally, experiments were confined to 7B and 13B models. This restriction prevents us from determiing
21

---

if our method adheres to scaling law. We hope to continue iterative training in the future and to train larger models, given access more computing resources, explore the upper limits of our method.
D Broader Impacts
Our approach requires no additional human annotations and significantly enhances model perfor- mance using the model itself. Technically, our method may inspire more researchers to explore how multimodal models can learn from themselves. From a socictal impact perspective, our method significantly reduces hallucinations in LVLMs, major factor affecting the application of Al in real- world scenarios. Our approach promotes more responsible use of LVLMs. However, it is important note that while method significantly reduces hallucinations, they still occur. Therefore, it is crucial to employ various measures to ensure safety and stability when applying this approach in real-world scenarios.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we have thoroughly detailed the background, motivation, scope, main experimental results, and contributions of our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section C we discussed that due to a lack of computing resources, our method was not trained for more iterations and was not tested on larger models.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification: In Section B and Appendix I. we provide a complete theoretical analysis and proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear the main paper or supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple-
mented by formal proof' provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In our paper, we provide comprehensive details about the backbone, dataset, algorithm, training hyperparameters, and hardware platform used ensure experiment re- producibility, Additionally, we will open-source all training code and logs.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
24

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
The data we used is publicly accessible, and we will release the code upon
acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ce
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: for more details.
/nips. wtt.cc/ppcccgcuuiieeeee httpl////g..ddd/ddddeaaiiii.sees bobdeesubmsssoooooliiioi
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed experimental setups and specifics in Section and Appendix [. of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We adopt standard evaluation benchmarks and metries, which are accompa- nied by statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the enror bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experiments are sufficiently discussed to be run by others.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The conducted research conforms in every respect with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have explained the impact of our work from both technical and societal perspectives in Section D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e-g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not involve any high risks or potential misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions to access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All the mentioned previous work are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: The code will be made public upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection. cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "lMSUdS1ICL", "reviewer": "Reviewer_GvTb", "summary": "This paper proposes a new method for preference alignment with LVLMs. Specifically, the reward is computed using its own LLM (text only) and an external CLIP model. The optimization is done with DPO. This process can be iterated for several times.", "questions": "Q1. How did the performance improve over each preference data generation stage?\n\nQ2. Scaling study as in Gao et al. [1]?\n\n\n[1] Gao et al., \"Scaling Laws for Reward Model Overoptimization\", ICML 2023", "limitations": "Limitations and broader impact are discussed in the appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. Preference optimization in LVLM is under explored compared to those pure language ones. This submission proposed an effective approach.\n\nS2. Both theoretical and empirical contribution.", "weaknesses": "I don't find significant weaknesses. However, I do think the experiments can be improved. Therefore, I give a conservative score of borderline accept. I may consider increasing the rating if more thorough studies are conducted (see next section).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback to help us improve our paper. We detail our response below and please kindly let us know if our response addresses your concerns.\n\n> **Q1**: How did the performance improve over each preference data generation stage?\n\n**A1**: In the first round of training, our model (referred to as model-iter0) generated a batch of preference data based on the approach discussed in Section 3.2.1. Then, by employing preference optimization, we optimized model-iter0 and obtained model-iter1. Compared to model-iter0, model-iter1 exhibits better performance and generates responses that are better aligned with the image input.\nIn the second round of training, similarly, we used model-iter1 to generate a batch of preference data. Due to the improved performance, this batch of data is of higher quality compared to the previous round’s preference data and is more challenging to distinguish between preferred and dispreferred data. Consequently, using this new preference data to perform preference optimization on model-iter1 yields a stronger model, i.e., model-iter2.\n\nIn this way, we can perform more rounds of iteration until convergence (see empirical results in Figure 3 in the paper).\n\nIn summary, because the quality of preference data can improve over different rounds and it becomes increasingly difficult for the model to distinguish preferences, performing preference optimization each round will enhance the model’s performance.\n\n---\n\n> **Q2**: .Scaling study-Gao et al., \"Scaling Laws for Reward Model Overoptimization\", ICML 2023\n\n**A2**: Since CSR utilizes the model itself to establish an initial reward and then calibrates this initial reward by incorporating an image-response relevance score computed by the CLIP model, we analyzed the effect of the reward model's (CLIP model) size. We conducted experiments with different sizes of CLIP models to calibrate the initial reward. Due to rebuttal time constraints, all experiments were conducted with only one round of CSR training on LLaVA-1.5-7B. The results are reported in Table R1. The experimental results show that using larger and more powerful CLIP models as the reward model reduces the model's hallucinations and improves comprehensive benchmark performance. The results meet with our expectations, as stronger models can better align image and text responses.\n\n\n**Table R1:** Performance with respect to different sizes of CLIP models\n| CLIP models (from small to large)       | MME$^P$   | MME$^C$  | SEED  | LLaVA$^W$ | MMB   | MM-Vet | SQA$^I$  | VisWiz | GQA   | POPE  | CHAIR$_S$ | CHAIR$_I$ |\n|---------------|--------|-------|-------|--------|-------|--------|-------|--------|-------|-------|--------|--------|\n| clip-vit-base-patch16   | 1504.1 | 350.3 | 60.3  | 65.3   | 64.9  | 31.6   | 70.1  | 53.6   | 62.1  | 85.98 | 31.7   | 8.4    |\n| clip-vit-l-336px     | 1500.6 | 367.5 | 60.4  | 69.7   | 64.7  | 32.2   | 70.3  | 54.0   | 62.1  | 86.94 | 26.6   | 7.2    |\n| clip-vit-g-14   | 1511.3 | 367.1 | 60.6  | 70.6   | 65.3  | 33.0   | 70.3  | 54.3   | 62.3  | 87.02 | 26.2   | 7.1    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9tgkjN3cWp", "reviewer": "Reviewer_oVPD", "summary": "The paper proposes a new approach to addressing the hallucination problem in Large Vision-Language Models (LVLMs). This phenomenon occurs when generated text responses appear linguistically plausible but contradict the visual input, indicating a misalignment between image and text pairs. The proposed solution, Calibrated Self-Rewarding (CSR), allows the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. This method incorporates visual constraints into the self-rewarding process, emphasizing visual input. Empirical results show performance improvements and reduced hallucinations across various benchmarks and tasks.", "questions": "How is the improvement of approximately 7.62% computed? Please provide a detailed explanation of the calculation method used to arrive at this figure.\n\nWhat is the upper bound in the performance gain of CSR? A case study showing its potential on one or several benchmarks with LLaVA-1.5 13B would be helpful, given the increasing trend shown in Figure 3.\n\nFigure 2 is unclear. What do the green and red circles represent? What does \"Option\" mean? Is it possible to visualize the generated preference data in Figure 2 or elsewhere? Additionally, Figure 1 appears confusing regarding the shapes and score positions. Can these be clarified and improved for better understanding?", "limitations": "The authors acknowledge some limitations of their work, such as conducting three iterations of CSR. While this approach shows promise, its technical novelty compared to existing methods is somewhat limited. \n\nAdditionally, the performance improvements are not uniformly substantial across all benchmarks. The paper could benefit from a more detailed discussion of these limitations and potential strategies for addressing them in future work. \n\nFurthermore, the potential negative societal impact of the work should be considered, particularly in terms of the ethical implications of improving LVLMs that might be used in sensitive applications. Providing constructive suggestions for mitigating any negative impacts would be valuable.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses the critical issue of hallucination in LVLMs by introducing a new Calibrated Self-Rewarding (CSR) approach. This method is new in its use of visual constraints within the self-rewarding paradigm to enhance image-text alignment.\n\nThe empirical evaluation is thorough, encompassing twelve benchmarks and tasks. The results show improvements over existing methods, demonstrating the effectiveness of CSR.\n\nThe paper is well-organized, providing a clear explanation of the problem, proposed solution, and empirical results. The inclusion of theoretical analysis further strengthens the clarity and credibility of the work.\n\nThe proposed CSR method has implications for improving the reliability and accuracy of LVLMs, which is crucial for their application in various real-world scenarios.", "weaknesses": "**Technical Novelty:** The primary distinction of the proposed method from previous approaches is the introduction of a new reward score based on vision-language similarity and step-wise evaluation. While this is a meaningful contribution, the paper could benefit from a more detailed discussion on how this approach fundamentally differs from and improves upon existing methods.\n\n**Performance Improvements:** Although CSR shows substantial improvements on average, the performance gains in some benchmarks, such as LLaVA-1.5-13B, are not very substantial. It would be beneficial to explore the reasons behind these limited improvements and suggest potential avenues for further enhancement.\n\n**Figures:** Figures 1 and 2 lack clarity. The visual representation of preference data and the meaning of green and red circles in Figure 2, as well as the shapes and score positions in Figure 1, need to be better explained and presented more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and providing valuable feedback. We address your concerns point by point below and would appreciate knowing if our responses address them.\n\n> **Q1**: Technical Novelty: The primary distinction of the proposed method … differs from and improves upon existing methods.\n\n**A1**: \n- Compared with self-rewarding in LLM, the fundamental difference in LVLM is that using the target LVLM as both the generator and judge (as is done in LLM self-rewarding) can amplify the misalignment issues between text and image modalities. This happens because the judge itself also tends to be biased toward the contextual information, often ignoring the input image. By incorporating visual constraints, we can effectively mitigate this issue, as it encourages better alignment between text and image modalities within the judge. We believe this is the fundamental difference between our approach and LLM self-rewarding.\n- Compared with other modality alignment enhancement approaches, CSR does not rely on additional VLMs or human annotators to generate preference data, which allows it to better capture and correct the inherent preferences of the target LVLM.\n\nWe believe our contributions are significant and fundamentally different from existing methods.\n\n---\n\n> **Q2**: It would be beneficial to explore the reasons behind these limited improvements … further enhancement.\n\n**A2**: Based on Table 1, CSR performs significantly better on hallucination benchmarks that evaluate image captioning quality compared to closed-ended VQA benchmarks. This is because preference optimization may be better suited for enhancing the performance of open-ended questions. In preference optimization, the preferred and dispreferred responses in the preference data are possibly partially correct, with the preferred response simply being better than the dispreferred one. Optimizing the model with this preference data can strengthen its ability to distinguish between both responses and accurately capture more details in long-form, open-ended image captioning tasks. On the other hand, some closed-ended VQA tasks use a fixed set of response options (e.g., yes/no, multiple-choice), which rely more on the visual model's perception capability and may not be as easily improved by preference optimization compared to long-form image captioning. Similar characteristics are exhibited across POVID [1], RLHF-V [2], Human-Prefer [3], and Vlfeedback [4].\n\n---\n\n> **Q3**: How is the improvement of approximately 7.62% computed? \n\n**A3**: As mentioned in Line 204, the improvements refer to the average percentage increase across all benchmarks when comparing CSR (7B) and LLaVA-1.5 7B. The method we used to calculate the average percentage increase is as follows:\n\nFirst, to calculate an average score on a 100-point scale, we adjusted the original values: MME$^P$ was divided by 16, and MME$^C$ was divided by 4, according to \\# categories in MME. Additionally, since a lower CHAIR value indicates better performance, we standardized all metrics to follow a \"higher is better\" approach by transforming the CHAIR$_S$ and CHAIR$_I$ metrics into 100 - CHAIR$_S$ and 100 - CHAIR$_I$. Then, we calculated the average score by averaging the standardized values, which were used to compute the average percentage increase.\n\nRegarding the specific value, we apologize for a computational error in calculating the average percentage increase using the data from Table 1. The correct value is 8.23%.\n\n---\n\n> **Q4**: What is the upper bound in the performance gain of CSR? A case study would be helpful.\n\n**A4**: We have included two case studies selected from the CSR-generated datasets in Figure R1 in the supplementary PDF. As the CSR iteration increases, we observe performance gains, the alleviation of challenging hallucinations (such as counting issues), and enhanced fine-grained perception. Combined with the qualitative improvements of CSR in various benchmarks, these additional case studies further support the effectiveness of CSR.\n\n---\n\n> **Q5**: Figure 2 is unclear. \n\n**A5**: Since CSR uses step-level reward modeling, we initially generate a few sentences (five for illustration). Each of these sentences is then used to sample five possible next sentences, resulting in a total of 25 sentences. To improve efficiency, only the top 3 sentences (indicated by green dots in the figure) and the bottom 2 sentences (indicated by red dots) are retained. The retained five sentences are then used to generate the third step sentence, and this process is repeated until the response is complete. We have refined Figure 2 in the supplementary PDF (see Figure R2).\n\n---\n\n> **Q6**: visualize the generated preference data\n\n**A6**: We visualized a case of preference data generated by the model itself in the CSR process in the supplementary PDF (see Figure R3).\n\n---\n\n> **Q7**: Additionally, Figure 1 appears confusing. \n\n**A7**: Figure 1’s radar chart shows the scores of self-rewarding, LLaVA-1.5-7B, and CSR-7B across all benchmarks. A larger area in the radar chart indicates higher scores. To clarify it, we polish Figure 1 in the supplementary PDF (see Figure R4).\n\n---\n\n> **Q8**: The potential negative societal impact of the work should be considered\n\n**A8**:  CSR depends on the quality of self-generated preference data. Although it achieves overall higher performance, there is still a possibility that the data can contain errors, which may misguide the learning process in some situations and ultimately lead to erroneous decisions in safety-critical applications like healthcare.\n\n---\n\n**References**\n\n[1] Aligning modalities in vision large language models via preference fine-tuning.\n\n[2] Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. \n\n[3] Aligning large multimodal models with factually augmented rlhf.\n\n[4] Silkie: Preference distillation for large visual language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0soXWXySJ7", "reviewer": "Reviewer_myph", "summary": "The paper addresses the challenge of hallucination in Large Vision-Language Models (LVLMs), where generated text responses appear plausible but contradict the input image. This misalignment occurs because the models prioritize textual information over visual input, even with high-quality representations. Existing methods to address this issue involve resource-intensive preference optimization through additional models or human annotations, which may not align well with the LVLM’s preferences.\n\nTo overcome these challenges, the authors propose the Calibrated Self-Rewarding (CSR) approach. CSR allows the model to self-improve by generating candidate responses, evaluating rewards for each, and curating preference data for fine-tuning. The process emphasizes visual input through a step-wise strategy and incorporates visual constraints into the self-rewarding mechanism.", "questions": "See weaknesses above", "limitations": "The authors addressed the limitations of the work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed method reduces the resources required to align a VLM model for less hallucination compared to previous methods. \n\n* The paper is clearly written.\n\n* The authors show theoretical explanation to validate the proposed method.", "weaknesses": "* In the limitation section, the authors mention that they could only run three iterations due to computation issues. However, in Section 4.1, the authors say they used one A100 80GB GPU, which takes roughly 3.5 and 5 hours to fine-tune LLaVA-1.5 7B and LLaVA-1.5 13B, respectively. It seems like such an experiment could be done with more iterations to see how the score trend continues across more iterations. This could supplement the claim of increased performance concerning iterations made in Section 4.2. \n\n* The authors use the CLIP similarity score to align the image and text response. However, there is a concern that CLIP may primarily focus on prominent objects in the foreground, potentially overlooking smaller details in the background. If this is the case, using CLIP for reward calculation might inadvertently cause the Vision-Language Model (VLM) to miss fine details in the image, even though it would reduce hallucinations overall.\n\n* Although Table 2 shows an analysis of the effect of each reward term ($R_I$ and $R_T$), it would be beneficial to see how varying $\\lambda$ terms affect the score. \n\n* The benefit of the proposed method, apart from the resulting benchmark score, is that it takes much less resources to align the model. Can a rough estimate of how different the computing resource would be, including the time to gather the preference data for previous methods? \n\n* (minor) To this reviewer, the overall framework figure (Figure 2) seems hard to understand even after understanding the proposed method through the texts in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. Below are our responses to your questions. Please let us know if they address your concerns.\n\n> **Q1**: In the limitation section, the authors mention that they could only run three iterations due to computation issues. However, in Section 4.1, … supplement the claim of increased performance concerning iterations made in Section 4.2. \n\n**A1**: Using LLaVA-7B as an example, we performed two additional rounds of iterative training on top of the three rounds of CSR training detailed in Table 7. The results are presented in Table R1. According to the results, we observe that more iterations can still improve model performance, especially in mitigating hallucination. However, the overall improvement tends to slow down with additional iterations (see Figure R5 in Supplementary PDF), indicating performance convergence.\n\n**Table R1**: Results of more rounds of evaluation \n| Method        | MME$^P$   | MME$^C$  | SEED  | LLaVA$^W$ | MMB   | MM-Vet | SQA$^I$  | VisWiz | GQA   | POPE  | CHAIR$_S$ | CHAIR$_I$ |\n|---------------|--------|-------|-------|--------|-------|--------|-------|--------|-------|-------|--------|--------|\n| CSR iter-3  | 1524.2 | 367.9 | 60.3  | 71.1   | 65.4  | 33.9   | 70.7  | 54.1   | 62.3  | 87.01 | 21.0   | 6.0    |\n| CSR iter-4  | 1524.6 | 368.8 | 60.4  | 71.0   | 65.3  | 33.9   | 70.4  | 54.0   | 62.2  | 87.05 | 19.0   | 5.9    |\n| CSR iter-5  | 1520.1 | 367.2 | 60.5  | 71.3   | 65.4  | 33.8   | 70.8  | 54.2   | 62.4  | 87.16 | 18.3   | 5.4    |\n\n---\n\n> **Q2**: Concern about the CLIP model in capturing details.\n\n**A2**: First, we would like to clarify that our proposed CSR framework is general enough to incorporate different vision-centric models used to calculate image-response relevance scores, even though CLIP is used in this paper. To further demonstrate the compatibility of CSR, we conducted an experiment using GPT-4o to calculate the image-response relevance score, which offers stronger image perception capabilities. We used the first iteration of CSR training on LLaVA-1.5-7B as an example, and the results are shown in Table R2. The results indicate that using a model with stronger image perception capabilities to calibrate the initial reward can provide improvements compared to using CLIP, although CLIP is sufficiently strong and lightweight. We plan to include the performance of using more vision-centric models to calibrate the initial reward in a future version.\n\n**Table R2**: Comparison of different models used to calculate image-response relevance scores\n \n| Method       | MME$^P$   | MME$^C$  | SEED  | LLaVA$^W$ | MMB   | MM-Vet | SQA$^I$  | VisWiz | GQA   | POPE  | CHAIR$_S$ | CHAIR$_I$ |\n|---------------|--------|-------|-------|--------|-------|--------|-------|--------|-------|-------|--------|--------|\n| CSR iter-1    | 1500.6 | 367.5 | 60.4  | 69.7   | 64.7  | 32.2   | 70.3  | 54.0   | 62.1  | 86.94 | 26.6   | 7.2    |\n| CSR iter-1 (GPT-4o)    | 1509.4 | 366.2 | 60.4  | 70.2   | 65.1  | 31.8   | 70.4  | 54.0   | 62.2  | 87.11 | 24.2   | 6.6    |\n\n---\n\n> **Q3**: how varying $\\lambda$ terms affect the score.\n\n**A3**: We evaluate the effectiveness of varying $\\lambda$ on the performance. The overall training settings are the same as in Table 1, with three rounds of iteration. The results are shown in Table R3 and indicate that placing greater emphasis (i.e., $\\lambda=0.9$) on visual calibration performs best, further strengthening our contribution to reward calibration.\n\n**Table R3**:  Performance w.r.t. $\\lambda$\n| Method           | MME$^P$   | MME$^C$  | SEED  | LLaVA$^W$ | MMB   | MM-Vet | SQA$^I$  | VisWiz | GQA   | POPE  | CHAIR$_S$ | CHAIR$_I$ |\n|-------------------|-------|------|------|--------|------|--------|------|--------|------|------|--------|--------|\n| CSR-7B  ($\\lambda$=0.1)      | 1508.6| **369.3**| 60.0 | 66.7   | 64.9 | 31.6   | 70.0 | 54.0   | 62.0 | 86.90| 40.8   | 10.2   |\n| CSR-7B  ($\\lambda$=0.5)  | 1515.4| 364.5| 60.1 | 68.2   | 64.9 | 32.4   | 69.7 | 54.0   | 62.1 | 86.90| 28.2   | 6.7    |\n| CSR-7B  ($\\lambda$=0.9)      | **1524.2**| 367.9 | **60.3** | **71.1**   | **65.4** | **33.9**   | **70.7** | **54.1**   | **62.3** | **87.01** | **21.0**   | **6.0**  |\n\n---\n\n> **Q4**: Can a rough estimate of how different the computing resource would be, including the time to gather the preference data for previous methods?\n\n**A4**: In CSR, the data is constructed by the target LVLMs themselves and calibrated using the CLIP model, which eliminates the need for any human effort or costly closed-source models (e.g., GPT-4). In terms of the time required to gather the data, using LLaVA-1.5-13B as an example, it takes around 12 hours to obtain 13,000 pairs of high-quality preference data on an A6000ada GPU node. In contrast, manual data correction represented by RLHF-V [1] significantly involves human effort, as well as time and monetary costs. For POVID [2], constructing one dataset (which may not be optimal) requires approximately $300-400 in GPT-4 API costs.\n\n---\n\n> **Q5**:  (minor) The overall framework figure (Figure 2) seems hard to understand.\n\n**A5**: We have polished Figure 2 and put it in the supplementary PDF (see Figure R2). Here is an explanation of this figure: Since CSR uses step-level reward modeling, we initially generate a few sentences (five for illustration). Each of these sentences is then used to sample five possible next sentences, resulting in a total of 25 sentences. To improve efficiency, only the top 3 sentences (indicated by green dots in the figure) and the bottom 2 sentences (indicated by red dots) are retained. The retained five sentences are then used to generate the third step sentence, and this process is repeated until the response is complete. \n\n---\n\n**References**\n\n[1] Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.\n\n[2] Aligning modalities in vision large language models via preference fine-tuning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jHAeNe6Nzr", "reviewer": "Reviewer_bHbK", "summary": "The paper generally follows self-rewarding language models and applies the idea to vision-language models. The method first ask a VLM to self-generate candidates, based on which they score the candidates with the VLM itself and CLIPScore, and then perform DPO on the generated candidates. Experiments on LLaVA demonstrate improvements over baselines such as self-rewarding.", "questions": "Please refer to the weakness section.", "limitations": "the authors adequately addressed the limitations", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper investigates the application self-rewarding proposed in NLP to the vision-language domain and proposes VL-specific changes.\n2. The baselines are fairly strong and they perform evaluations on multiple standard benchmarks, making their results trustworthy.\n3. The paper is well-written.", "weaknesses": "1. The methodological contribution is not sufficient as it basically follows self-rewarding language models [11] and applies it in another domain. The proposed techniques such as using CLIPScore to calibrate the self-generated scores are incremental changes to me.\n2. The empirical results do not seem significant in many of the datasets. In Table 1, on popular benchmarks such as GQA and SEED, the performance improvements are marginal (one exception may be LLaVA-W, but this dataset is small and relies on GPT evaluations). Results on VQAv2 and MMStar should also be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have answered your questions below, and we would appreciate it if you could let us know whether our responses address your concerns.\n\n> **Q1**: The methodological contribution is not sufficient as it basically follows self-rewarding language models [11] and applies it in another domain. The proposed techniques such as using CLIPScore to calibrate the self-generated scores are incremental changes to me.\n\n**A1**: We would like to highlight our contribution in comparison to LLM self-rewarding. Compared with self-rewarding in LLM, the fundamental difference in LVLM is that using the target LVLM as both the generator and judge (as is done in LLM self-rewarding) can amplify the misalignment issues between text and image modalities. This happens because the judge itself also tends to be biased toward the contextual information, often ignoring the input image. By incorporating visual constraints, we can effectively mitigate this issue, as it encourages better alignment between text and image modalities within the judge. We believe this is the fundamental difference between our approach and LLM self-rewarding. We believe these contributions are significant.\n\n---\n\n> **Q2**: The empirical results do not seem significant in many of the datasets. In Table 1, on popular benchmarks such as GQA and SEED, the performance improvements are marginal (one exception may be LLaVA-W, but this dataset is small and relies on GPT evaluations). Results on VQAv2 and MMStar should also be added. \n\n**A2**: We would like to emphasize that the primary goal of CSR is to enhance the alignment between image and text modalities and minimize hallucination. CSR has demonstrated substantial improvements over other methods across the majority of the datasets evaluated. These improvements are significant; we achieved average improvements of 8.23% and 2.59% compared to the original 7B model and the strongest baseline (i.e., self-rewarding), respectively.\n\nTo further evaluate the effectiveness of CSR, we evaluated LLaVA-1.5 7B and 13B of CSR for VQAv2 and MMStar, following the setup in Table 6. The results are shown in Table R1, including the original LLaVA, the strongest baseline (self-rewarding), and CSR. The performance gain of CSR on these additional benchmarks further demonstrates the effectiveness of CSR.\n\n**Table R1**: Performance on VQAv2 and MMStar\n| Model              | VQAv2   | MMStar (Avg performance) | \n|---------------------|--------------|--------------|\n| LLaVA-1.5-7B           | 78.5         | 30.3 |\n| +Self-rewarding  | 78.5          | 31.5 |\n| **+CSR**  | **78.7**          | **34.0**  |\n|---------------------|--------------|--------------|\n| LLaVA-1.5-13B           | 80.0         | 32.8 |\n| +Self-rewarding  | 80.2          | 34.7  |\n| **+CSR**   | **80.5**          | **36.6**  |\n\n---\n\n**References**\n\n[1] Zhou Y, Cui C, Rafailov R, et al. Aligning modalities in vision large language models via preference fine-tuning[J]. arXiv preprint arXiv:2402.11411, 2024.\n\n[2] Yu T, Yao Y, Zhang H, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13807-13816.\n\n[3] Sun Z, Shen S, Cao S, et al. Aligning large multimodal models with factually augmented rlhf[J]. arXiv preprint arXiv:2309.14525, 2023.\n\n[4] Li L, Xie Z, Li M, et al. Silkie: Preference distillation for large visual language models[J]. arXiv preprint arXiv:2312.10665, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FZeCoSqFEN", "reviewer": "Reviewer_4vrA", "summary": "This paper addresses an important and tough issue in LVLM – hallucination, which is usually caused by the misalignment of the image and text modalities. A new method CSR is proposed, by extending the language self-awarding approach to multimodality, considering both instruction-following (text) and  image-text alignment (multimodality) awarding.\n\nThe experiments are very comprehensive and solid, demonstrating the effectiveness of CSR in a wide range of tasks, especially for hallucination specific tasks.", "questions": "It would be more insightful if we can distinguish 1) comprehensive, 2) general VQA and 3) hallucination scores (e.g. avg score per type of benchmarks) in the main ablation, i.e., Table 2. I would expect “Only RI” to increase 3) but hurt 1) and 2); however, this is not verifiable given the current average scores.\n\nThe terminologies “sentence group” and “option” in Figure 2 are confusing (especially before readding 3.2). More legend or explanation (or re-structuring the figure) might be needed for better readability.\n\nWhat is the consideration to use the same examples in different iterations? Using random examples may lead to even better results (to converge later)?", "limitations": "N/A", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The issue this paper addresses is important and still an open question for LVLM training. The method proposed by this paper is simple yet effective, thus may be very insightful for the community.\n\nThe evaluation is comprehensive and solid, including 3 types of benchmarks: comprehensive benchmarks, general VQA and hallucination benchmarks, with 10 tasks in total. CSR is also compared with self-rewarding and several data-driven preference learning methods to demonstrate its effectiveness. Also different model sizes and model architectures are verified. The addtional Theoretical Explanation and attention map sections make the paper even more convincing.\n\nThe proposed method is easy to implement and extend to other modalities, so may have larger impact for other directions in the community.\n\nThis paper is very well written and readable.", "weaknesses": "The ablation of the weight of instruction-following score and image-text alignment score is missing, which is important to help understand how important of each score for CSR. \n\nIt might worth inverstigating the impact of different image-text alignment scoring approaches. At least comparing different image-text models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We've addressed your questions below and would appreciate it if you could let us know whether our responses meet your expectations.\n\n> **Q1**: The ablation of the weight of instruction-following score and image-text alignment score is missing.\n\n**A1**: We evaluate the effectiveness of varying $\\lambda$ on the performance. The results are shown in Table R1 and indicate that placing greater emphasis on visual calibration significantly improves performance, further strengthening our contribution to reward calibration.\n\n**Table R1**: Performance vs $\\lambda$\n| Category            | VQA  | Hallucination | Comp_bench |\n|-------------------|-------|------|------|\n| CSR-7B  ($\\lambda$=0.9)     | 62.4|86.7| 69.7 |\n| CSR-7B  ($\\lambda$=0.5)     | 61.9|84.0| 68.6 |\n| CSR-7B  ($\\lambda$=0.1)     | 62.0|78.6| 68.3 |\n\n---\n\n> **Q2**: Inverstigating the impact of different image-text alignment scoring approaches.\n\n**A2**: To further demonstrate the compatibility of CSR, we conducted an experiment using GPT-4o to calculate the image-response relevance score, which offers stronger image perception capabilities. We used the first iteration of CSR training on LLaVA-1.5-7B as an example, and the results are shown in Table R2. The results indicate that using a model with stronger image perception capabilities to calibrate the initial reward can provide improvements. We plan to include the performance of using more vision-centric models to calibrate the initial reward in the next version.\n\n**Table R2**: Comparison between different models that are used to calculate image-response relevance score \n| Method       | MME$^P$   | MME$^C$  | SEED  | LLaVA$^W$ | MMB   | MM-Vet | SQA$^I$  | VisWiz | GQA   | POPE  | CHAIR$_S$ | CHAIR$_I$ |\n|---------------|--------|-------|-------|--------|-------|--------|-------|--------|-------|-------|--------|--------|\n| CSR iter-1    | 1500.6 | 367.5 | 60.4  | 69.7   | 64.7  | 32.2   | 70.3  | 54.0   | 62.1  | 86.94 | 26.6   | 7.2    |\n| CSR iter-1 (GPT-4o)    | 1509.4 | 366.2 | 60.4  | 70.2   | 65.1  | 31.8   | 70.4  | 54.0   | 62.2  | 87.11 | 24.2   | 6.6    |\n\n---\n\n> **Q3**:  Distinguish 1) comprehensive, 2) general VQA and 3) hallucination scores in the main ablation.\n\n**A3**: We have split the ablation results and reported them in Table R3. The results indicate that relying solely on the image-response relevance score increases the hallucination score and slightly negatively impacts, or at best, only marginally improves, the VQA and comprehensive benchmarks (likely affecting only the VQA performance in these benchmarks).\n\n**Table R3**: Fine-grained results of ablation study\n| Category            | VQA  | Hallucination | Comp_bench |\n|-------------------|-------|------|------|\n| LLaVA-1.5-7B     | 59.6| 74.1|66.4 |\n| Only $R_T$     | 60.0|75.0| 69.4 |\n| Only $R_I$     | 59.1 | 76.7 | 67.1 |\n| CSR (Ours) 7B     |62.4| 86.7 | 69.7 |\n|-------------------|-------|------|------|\n| LLaVA-1.5-13B  | 62.8| 74.5| 67.5 |\n| Only $R_T$   |63.4| 74.7| 67.2 |\n| Only $R_I$      |62.7 | 77.0 | 68.6 |\n| CSR (Ours) 13B    |65.2| 84.0| 69.3 |\n\n---\n\n> **Q4**: The terminologies “sentence group” and “option” in Figure 2 are confusing.\n\n**A4**: We have polished Figure 2 and put it in the supplementary PDF (see Figure R2). Here is an explanation of this figure: Since CSR uses step-level reward modeling, we initially generate a few sentences (five for illustration). Each of these sentences is then used to sample five possible next sentences, resulting in a total of 25 sentences, which is named **“Option”**. To improve efficiency, only the top 3 sentences (indicated by green dots in the figure) and the bottom 2 sentences (indicated by red dots) are retained, which is named as **“Sentence Group”**. The retained five sentences are then used to generate the third step sentence, and this process is repeated until the response is complete. \n\n---\n\n> **Q5**: What is the consideration to use the same examples in different iterations? \n\n**A5**: In CSR, we use the same samples for iterative learning for two main reasons:\n\n- A single round of preference learning may not fully resolve all issues. Multiple iterations allow for the incremental correction of the model's errors on these samples, leading to further improvements in model performance.\n\n- Using the same samples helps us investigate whether the model's responses to these samples improve across different iterations. This consistency is useful for analytical experiments (e.g. Figure 5).\n\nAdditionally, we conducted an experiment during the second iteration of CSR with LLaVA-1.5-7B, where we selected 13,000 images that were entirely distinct from those used in the first iteration. The results of this experiment are reported in Table R4. It can be observed that using different samples across iterations performs similarly to using the same samples. We plan to conduct further investigations on this topic in the next version.\n\n**Table R4**: Results w/ random image batch over different iterations\n| CLIP model        | MME$^P$   | MME$^C$  | SEED  | LLaVA$^W$ | MMB   | MM-Vet | SQA$^I$  | VisWiz | GQA   | POPE  | CHAIR$_S$ | CHAIR$_I$ |\n|-------------------|-------|------|------|--------|------|--------|------|--------|------|------|--------|--------|\n| CSR iter-1  | 1500.6| 367.5| **60.4** | 69.7   | 64.5 | 32.2   | 70.3 | 54.0   | 62.1 | **86.94** | 26.6   | 7.2    |\n| CSR iter-2     | **1519.0** | 368.9 | 60.3 | 70.4   | **65.2** | **33.7**   | 70.1 | 54.0   | **62.3** | 86.82 | 23.0   | 6.1    |\n| CSR iter-2 (**other data**)     | 1513.4 | **369.1** | 60.2 | **70.7**   | 64.9 | 33.6   | **70.5** | **54.1**   | 62.2 | 86.90 | **21.3**   | **5.8**    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their constructive feedback. Below is a summary of the information covered in the attached PDF:

- **Figure R1**: Two cases selected from the CSR-generated datasets (Reviewer oVPD).

- **Figure R2**: Polished Figure 2 of the main paper (Reviewers oVPD, myph, 4vrA).

- **Figure R3**: A case including both self-generated preferred and dispreferred responses (Reviewer oVPD).

- **Figure R4**: Polished Figure 1 of the main paper (Reviewer oVPD).

- **Figure R5**: The average score of CSR across different iterations, with additional iterations included.

- **Figure R6 & Table A1**: We apologize for a computational error in calculating the average score of our method and LLaVA-1.5 using the data from Table 1 in the main paper. This mistake slightly affected specific values in Figure 3 and Table 2, but the improvement ratio and the overall conclusion completely remain unchanged. We have corrected this issue and updated Figure 3 in Figure R6 and Table 2 in Table A1, respectively.

For all reviewer reference, the way we **calculate the average score** is: 

First, to facilitate the calculation of an average score on a 100-point scale, we adjusted the benchmarks as follows: we divided the MME$^P$ by 16 and the MME$^C$ by 4, according to the number of categories in MME. Additionally, since the CHAIR metric is better when it's lower, we unified all metrics to follow a "higher is better" approach by transforming the CHAIR$_S$ and CHAIR$_I$ metrics into 100 - CHAIR$_S$ and 100 - CHAIR$_I$, respectively. After these adjustments, we calculated the average score by summing the standardized values and dividing by the total number of datasets. We will clarify this calculation process in the revised version.

Meta-review: This paper proposes a new method called Calibrated Self-Rewarding (CSR) to address hallucination issues in large vision-language models. It allows the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. The reward is computed using its own LLM (text only) and an external CLIP model.  Overall, all reviewers agree the approach is novel and the empirical evaluation is thorough, encompassing twelve benchmarks and tasks. However, there are some weaknesses including technical novelty is somewhat limited, and performance gains are modest on some benchmarks.
Overall, while not groundbreaking, this paper makes a solid contribution to an important problem in multimodal AI. The AC recommend weak accept.
Decision: Accept (poster)