Calibrated Self-Rewarding Vision Language Models

Authors: yiyang  zhou
zhiyuan  fan
dongjie  cheng
sihan  yang
zhaorun  chen
chenhang  cui
xiyao  wang
yun  li
linjun  zhang
huaxiu  yao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 07:40:08

Abstract: Large Vision-Language Models (LVLMs) have made substantial progress by integrating pre-trained large language models (LLMs) and vision models through instruction tuning. Despite these advancements, LVLMs often exhibit the hallucination phenomenon, where generated text responses appear linguistically plausible but contradict the input image, indicating a misalignment between image and text pairs. This misalignment arises because the model tends to prioritize textual information over visual input, even when both the language model and visual representations are of high quality. Existing methods leverage additional models or human annotations to curate preference data and enhance modality alignment through preference optimization. These approaches are resource-intensive and may not effectively reflect the target LVLM's preferences, making the curated preferences easily distinguishable. Our work addresses these challenges by proposing the Calibrated Self-Rewarding (CSR) approach, which enables the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. In the reward modeling, we employ a step-wise strategy and incorporate visual constraints into the self-rewarding process to place greater emphasis on visual input. Empirical results demonstrate that CSR significantly enhances performance and reduces hallucinations across twelve benchmarks and tasks, achieving substantial improvements over existing methods by 7.62\%. Our empirical results are further supported by rigorous theoretical analysis, under mild assumptions, verifying the effectiveness of introducing visual constraints into the self-rewarding paradigm. Additionally, CSR shows compatibility with different vision-language models and the ability to incrementally improve performance through iterative fine-tuning.

Main paper: 
 introduction: 
 Large Vision-Language Models (LVLMs) [1] [2] [3] [4] have achieved significant success by incorporating pre-trained large language models (LLMs) and vision models through instruction tuning. However, these LVLMs suffer from the hallucination phenomenon [5] , which generates text responses that are linguistically plausible but contradict the visual information in the accompanying image. For instance, the description generated by LVLMs may include visual elements that are not depicted in the image. This issue can also occur when the LLM is highly factual and the visual backbone is capable of producing sufficiently high-quality representations. As indicated in Cui et al. [6] , Guan et al. [7] , the potential reason for this lies in the misalignment problem between image and text modalities in LVLMs, which causes the model to prioritize the text knowledge present in the training language data while ignoring the actual visual input information.Several works have been proposed to enhance modality alignment capability in LVLMs through preference fine-tuning techniques, such as reinforcement learning from human feedback (RLHF) [8] and direct preference optimization (DPO) [9, 10] . However, these methods often either introduce additional models, such as GPT-4, or depend on human annotation to generate preference data. This data generation process is not only resource-intensive but, more critically, fails to capture the inherent preferences of the target LVLM. Consequently, the target LVLM may easily discern preferences from such curated data, making them less effective (detailed analysis provided in Appendix A.4). Recently, self-rewarding approaches have emerged, utilizing a single LLM for both response generation and preference modeling, showing promising results in LLM alignment [11, 12] . Unlike LLMs, LVLMs face modality misalignment issues in both response generation and preference modeling stages, potentially resulting in self-generated preferences overlooking visual input information. Directly applying these self-rewarding approaches to LVLMs is not capable of addressing the modality alignment problem and redirecting LVLM's attention towards emphasizing input image information.To tackle these challenges, our work introduces the Calibrated Self-Rewarding (CSR) approach, aimed at calibrating the self-rewarding paradigm by incorporating visual constraints into the preference modeling process. Specifically, we train the target LVLM using an iterative preference optimization framework that continuously generates preferences and optimizes the target LVLM over multiple iterations. Starting with a seed model, each iteration employs sentence-level beam search [13, 14] to produce fine-grained candidate responses for each image and text prompt. During the beam search, for each generated sentence, we first utilize the language decoder to establish an initial reward (i.e., sentence-level cumulative probabilities). Subsequently, we calibrate this initial reward by incorporating an image-response relevance score, resulting in the calibrated reward score. These calibrated reward scores are utilized to guide the generation of the next batch of candidate sentences. Finally, responses with the highest and lowest cumulative calibrated reward scores are identified as preferred and dispreferred responses, respectively, for preference fine-tuning in the subsequent iteration. The primary contribution of this paper is CSR, a novel calibrated self-rewarding paradigm for improving modality alignment in LVLMs. Theoretically, with mild assumptions, we show that introducing visual constraints in the self-rewarding paradigm can improve performance. Empirically, when compared with other competitive approaches (see Figure 1 for some representative methods), the results demonstrate that CSR is capable of improving performance on comprehensive LVLM evaluation benchmarks, VQA tasks, and reducing hallucination, achieving up to a 7.62% improvement on average. Additionally, we demonstrate CSR is capable of continuously improving performance over iterations, compatible with different large vision-language backbone models, and redirecting the attention of LVLMs toward the visual modality to achieve stronger modality alignment. 
preliminaries: 
 In this section, we will provide a brief overview of LVLM and preference optimization.Large Vision Language Models. LVLMs extend LLMs to multimodal scenario, which progressively predict the probability distribution of the next token for each input prompt. Given an <image x v , text x t > pair as input prompt x, LVLM outputs a text response y.Preference Optimization. Preference optimization has shown promise in fine-tuning language models and aligning their behavior with desired outcomes. Given an input prompt x , a language model with policy π θ can produce a conditional distribution π θ (y | x) with y as the output text response. The preference data is defined as D = {(x (i) , y (i) w , y(i) l )} N i=1, where y (i) w and y (i) l denote the preferred Figure 2 : The CSR framework operates an iterative process of preference data generation and learning. During preference data generation, CSR utilizes a sentence-level beam search approach to construct responses sentence by sentence, assigning a reward to each sentence. This reward, initially generated by the model itself, is then calibrated using image-relevance information. Preferences are determined based on the cumulative reward for each response. In each iteration, CSR generates new preference data and performs preference learning based on this data, continuously enhancing the model's performance.and dispreferred responses for the input prompt x (i) . Preference optimization leverage the preference data to optimize language models. Taking DPO [15] as a representative example" it formulates the probability of obtaining each preference pair as p(yw ≻ y l ) = σ(r(x, yw) -r(x, y l )), where σ(•) is the sigmoid function. DPO optimizes the language models with the following classification loss:EQUATIONwhere π ref (y|x) represents the reference policy, i.e., the language model after performing supervised fine-tuning. 
calibrated self-rewarding vision language models: 
 To address this challenge, we propose Calibrated Self-Rewarding (CSR), a novel approach aimed at improving modality alignment in LVLMs by integrating visual constraints into the self-rewarding paradigm. As illustrated in Figure 2 , CSR trains the target LVLM by alternately performing two stages: candidate response generation and preference curation and fine-tuning. In the candidate response generation stage, we employ sentence-level beam search for each input prompt to produce fine-grained candidate responses. During this process, the language decoder determines the initial reward for each generated sentence, which is then calibrated by incorporating an image-response relevance score. This calibrated reward score guides the generation of subsequent sentences and finally generate the entire response. Moving on to the preference curation and fine-tuning stage, we use the responses with the highest and lowest cumulative calibrated rewards to construct the preferred and dispreferred responses, and utilize the constructed preference pairs for fine-tuning. In the remaining of this section, we will provide detailed explanations of CSR.Before delving into how to generate candidate response and construct preference data, in this section, we first discuss how to formulate the reward within CSR. The ideal reward in the LVLM fulfills two specific criteria:• Vision-Constrained Reward: This aspect aims to integrate image-relevance information into the reward definition of LVLMs. By doing so, we address the limitation of LVLM in overlooking image input data when generating preferences. • Step-Wise Reward: Instead of assigning a single reward for the entire response, we opt for a stepwise approach. This involves assigning rewards at each step of response generation. Compared to a single reward, this finer-grained reward offers more detailed guidance and is more robust.To fulfill these criteria, we propose a step-wise calibrated reward modeling strategy. Inspired by Process-Supervised Reward Models [16] , we assign a reward score, R(s), to each generated sentence s during the sentence-level beam search. This score is a combination of two components: the selfgenerated instruction-following score, R T (s), and the image-response relevance score, R I (s).Specifically, the self-generated instruction-following score, R T (s), is calculated using the language decoder of the LVLM. It represents the sentence-level cumulative probability of generating sentence s, formulated as:EQUATIONwhere N o is the number of tokens in sentence s and r o represents token o in sentence s. A higher self-generated instruction-following score indicates a stronger capability of the generated response to follow instructions.While the self-generated instruction-following score partially reflects the LVLM's preference, it still suffers from modality misalignment, potentially overlooking visual input information. To address this, we introduce an image-response relevance score, R I (s), to calibrate the reward score R T (s). This score depicts the relevance between the generated sentence s and input image x v . We leverage CLIP-score [17] for this calculation, where the vision encoder in the CLIP model aligns with the vision encoder in the target LVLM. The image-response relevance score R I (s) is defined as:EQUATIONwhere the F I (x v ) and F T (s) represent the visual CLIP embedding and textual CLIP embedding, respectively. Finally, the calibrated reward score R(s) for the generated sentence s is defined as:EQUATIONwhere λ is a hyperparameter used to balance the language instruction-following and image-response relevance scores. By combining both scores, we aim to redirect the attention of LVLM towards the input visual information, thus enhancing its modality alignment ability.After establishing the reward framework in CSR, we next discuss our iterative fine-tuning process. Within this framework, we iteratively perform two essential steps, namely candidate response generation and preference data curation and optimization. These steps are elaborated upon as follows:In candidate response generation, our objective is to generate responses to build preference data. To accomplish this, we employ a sentence-level beam search strategy. Initially, we concurrently sample multiple candidate sentences, utilizing the "end of sub-sentence" marker (e.g., "." in English) as the delimiter. Subsequently, for each sentence s, we compute its reward score R(s) using Eqn. (4) .From these scores, we then select the top-k and bottom-k sentences with the highest and lowest reward scores, respectively, to proceed to the subsequent round of sentence-level beam search. This iterative process continues until reaching the "end of response," conventionally represented as ⟨eos⟩.Once all sentences for a response y = {s for each x ∈ D do 3:while not reach the end of response do 4:Generate a bunch of candidate sentences from last-round sentences 5:for each candidate sentence s do Select the preferred response y w,t and dispreferred response y l,t 12:Update π θ ← arg min θ L t (π θ ; π ref ), π ref ← π θ .defined as: R(y) = ∑ Ny i=1 R(s i ), where N y is the number of sentences in response y. The detailed algorithm for candidate response generation is outlined in Algorithm 1.After generating candidate responses with their reward scores, our next step is to curate preference dataset. Here, for each input prompt, we select the responses with the highest and lowest cumulative calibrated reward scores as the preferred and dispreferred responses, respectively, to construct the preference dataset for fine-tuning. For each iteration t, we denote the constructed preference data as:Dt = {(x (i) , y (i) w,t , y (i) l,t )} N i=1 .After obtaining the preference data, we fine-tune the target LVLM using DPO. At iteration t, we use the last iteration fine-tuned model π θt-1 as the reference model. Following Eqn (1) , the loss at iteration t of CSR is defined as:EQUATION)The training process of CSR is detailed in Algorithm 1. 
experiment: 
 In this section, we empirically investigate CSR in addressing the modality misalignment problem of LVLMs, focusing on the following questions: (1) Can CSR help improve the performance of models on both comprehensive benchmarks and hallucination benchmarks? (2) Can CSR iteratively improve multimodal alignment progressively in LVLMs and lead to more factual LVLMs? (3) Is CSR compatible with different open-sourced LVLMs? (4) How does CSR change attention weights and preference pairs to align image and text modalities?Implementation Details. We utilize LLaVA-1.5 7B and 13B [1] as the backbone models. During the preference learning process, we adapt LoRA fine-tuning [18] . The images and prompts used to construct the preference data are randomly sampled from the detailed description and complex reasoning subclasses of the LLaVA150k dataset, totaling approximately 13,000 samples [19] . It is worth noting that each iteration uses the same prompt and image as the previous round. Overall, the iterative training is conducted over three iterations, completed on one A100 80GB GPU. It takes roughly 3.5 and 5 hours for fine-tuning LLaVA-1.5 7B and LLaVA-1.5 13B, respectively. For more detailed information on training hyperparameters and training data, please refer to Appendix A.1.Evaluation Benchmarks. We conducted evaluations on three types of benchmarks: comprehensive benchmarks, general VQA and hallucination benchmarks. Specifically, this includes: (1) Comprehensive benchmarks (MME [20] , SEEDbench [21] , LLaVA W [19] , MMbench [22] , MM-Vet [23] );(2) General VQA (ScienceQA (SQA) [24] , VisWiz [25] , GQA [26] ); (3) Hallucination benchmark (POPE [27] , CHAIR [28] ). More detailed description are discussed in Appendix A.1. Baselines. We will first compare CSR with the self-rewarding approach described by Yuan et al. [29] . Here, we directly apply self-rewarding to LVLM, using the prompts and experimental settings outlined in Yuan et al. [29] (see detailed settings in Appendix A.1 and Table 3 ). We also compared CSR with several data-driven preference learning methods, including Silkie (Vlfeedback) [9] , LLaVA-RLHF (Human-preference) [8] , POVID [10] , and RLHF-V [30] . Furthermore, we compared the performance of the optimized LLaVA-1.5 via CSR with other state-of-the-art open-source LVLMs, including InstructBLIP [31] , Qwen-VL-Chat [32] , mPLUG-Owl2 [33] , BLIP-2 [34] , and IDEFICS [35] , after the final rounds of training (CSR with iteration = 3). Additionally, to evaluate the effectiveness of CSR on other LVLMs, we applied CSR to a recent LVLM called Vila [36] . For more information on these baselines, please refer to Appendix A.1. 6 and Table 7 in Appendix A.5 for full results).CSR Continuously Improves Model Performance over Iterations. In Figure 3 , we report the average performance of LLaVA-1.5 7B and 13B models concerning the number of training iterations on comprehensive benchmarks, general VQA tasks, and hallucination benchmarks. To facilitate score calculation, we first calculated an average score on a 100-point scale by adjusting the original values: MME P was divided by 16, and MME C was divided by 4, corresponding to the number of categories in MME. Additionally, since a lower CHAIR value indicates better performance, we standardized all metrics to follow a higher is better approach by transforming the CHAIR S and CHAIR I metrics into 100 -CHAIR S and 100 -CHAIR I . We then calculated the average score by averaging these standardized values, which were used to compute the average percentage increase. In the experiment, the 7B model achieved an improvement of approximately 7.62% across all benchmarks through online iterative updates, while the 13B model saw an improvement of approximately 5.25%. According to the full results in Table 6 and Table 7 of Appendix A.5, the improvement is particularly significant on the LLaVA W and CHAIR benchmarks, with improvements of 8.9% and 49.50%, respectively. The results indicate that CSR is capable of incrementally improving model performance over iterations, demonstrating its effectiveness in self-improving the quality of generated preference data and leading to stronger modality alignment. The degree of improvement gradually becomes smaller, which is not surprising, indicating that the model is gradually converging. CSR Outperforms Competitive Preference Fine-Tuning Baselines. Compared to preference data curation approaches (e.g., POVID, RHLF-V) that generate preference data from either additional models or human annotations, the superiority of CSR indicates that adapting a self-rewarding paradigm better captures the inherent preferences of the target LVLMs, achieving stronger modality alignment. Furthermore, CSR outperforms existing selfrewarding methods, with an average performance improvement of 2.43%, demonstrating its effectiveness in calibrating the reward model by incorporating imageresponse relevance scores. This mitigates the potential issue of overlooking visual input information when estimating self-generated preferences.In addition, we compare the performance of LLaVA-1.5 after three rounds of online CSR with other state-of-the-art open-sourced VLLMs and report the results in Table 5 of Ablation Study. To validate the effectiveness of using the image-response relevance score (R I ) to complement the self-generated instruction following score (R T ), we specifically compare CSR with three variants: (1) without applying CSR on LLaVA 1.5 (Base); (2) using CSR with only the self-generated instruction following score (Only R T ); and (3) using CSR with only the imageresponse relevance score (Only R I ). The results are reported in Table 2 . We first observe that CSR improves performance by jointly considering both the self-generated instruction following and image-response relevance scores. This verifies its effectiveness in enhancing modality alignment by calibrating the language-driven self-rewarding paradigm with visual constraints. Additionally, we further conduct the analysis on the change of λ in Eqn (4) and found that incorporating external visual scores to calibrate the models rewarding process effectively enhances performance (see detailed results in Appendix A.5.) Figure 4 : Average scores of CSR in Vila 7B at different iterations over all benchmarks (see Table 8 in Appendix A.5 for full results).Compatibility Analysis. To validate CSR for its applicability to other LVLMs, we deployed CSR on Vila 7B and conducted three rounds of online iterations. We conducted experiments on all ten evaluation benchmarks and tasks, and the results are shown in Figure 4 . Similar to the findings in Figure 3 , Vila demonstrates a similar phenomenon during the online iterations of CSR, where it can self-correct preferences, leading to gradual improvements in all benchmarks. For Vila, the overall performance improved by 3.37% after three rounds of CSR iterations, with particularly notable increases of 8.48% on VisWiz and 14.0% on MM-Vet. The compatibility analysis further corroborates the generalizability and effectiveness of CSR in enhancing the performance of LVLMs.Over Iterations? To investigate how CSR gradually improve the performance over iterations, we analyzed the change of self-generated preference data with the LLaVA-1.5 7B model. In Figure 5 , we illustrated the distribution of image-response relevance scores of three iterations over 500 examples from LLaVA-150k [19] . We first observe that both the chosen (preferred) and rejected (dispreferred) responses achieve higher image-response relevance scores after the model undergoes CSR online iterations. This indicates that, following CSR, the responses generated by LVLMs are more closely aligned with the image information. Secondly, it can be observed that after multiple rounds of online iterations with CSR, the average image-response relevance scores for the rejected and chosen responses become closer to each other. This makes the self-generated preference data during CSR iterations more challenging to distinguish, while further strengthening the learning process. How Does CSR Improve Modality Alignment? To further understand how CSR affects modality alignment, in Figure 6 , we present the changes in image and text attention maps for three models: the original LLaVA-1.5 7B model, the self-rewarding approach, and CSR. These attention maps illustrate the distribution of attention scores over image and text tokens. We observe that applying CSR strengthens the model's attention to certain visual tokens. Simultaneously, the change of attention values of the text tokens indicates that CSR is capable of alleviating Prompt: What are the key elements in this picture?The image features a blue and white bus driving down a city street, with the bus's destination clearly displayed on the front: "Artesia." The bus is positioned in the middle of the street, and its large size and prominent destination sign make it easily noticeable to pedestrians and other vehicles. In addition to the bus, the image also includes a car driving behind the bus, indicating that both vehicles are sharing the roadway in the city.An articulated city bus, which is blue and white, is driving down a street in a large city. In front of the bus is a truck, and the bus is positioned behind it. Several people can be seen walking along or standing near both the bus and truck. They are likely pedestrians going about daily activities or waiting for public transportation. The scene captures the bustling atmosphere of a busy urban environment with various modes transportation in use. It highlights how public transportation, such as buses, plays a crucial role in the daily lives of city dwellers. Overall, it is a typical day in a large metropolitan area with people commuting and going about their daily routines.The image features a blue and white bus driving down a city street, with a large building in the background. The bus is the main focus of the scene, occupying a significant portion of the image. A few cars can be seen on the street, with one car positioned behind and to the left of the bus, and another car located further down the street on the right side.There are several people in the scene, with one person standing near the left edge of the image, another person closer to the center, and a third person on the right side of the street. They appear to be pedestrians going about their daily activities. Additionally, a traffic light is visible in the background, indicating that the bus is driving in an urban environment with traffic regulations.Text Tokens CSR (Ours)LLaVA-1.5Visualize Attentions the issue of over-reliance on context mentioned in Huang et al. [37] . Additionally, compared with the self-rewarding approach, CSR shows a more effective distribution of attention between image and text tokens. These findings indicate that with CSR, LVLMs can better align different modalities through a calibrated self-rewarding strategy, focusing more on the visual modality rather than over-relying on contextual text.In this section, we use LLaVA-1.5 13B as an example to illustrate changes in the models own responses during CSR iterations and the preference data sampled in the CSR learning process, with hallucinations and errors highlighted in red. The results are shown in Figures 7 and 9 , respectively. As shown in Figures 7, with each iteration of CSR, hallucinations in the models responses noticeably decrease. This indicates that CSR effectively refines the models preferences through iterative preference learning, making the models responses more accurate. In Figure 9 of the Appendix, we present a sampled preference data pair from the model during the CSR learning process. It can be seen that through CSR, the model not only gradually refines its own preferences but also obtains high-quality preference data pairs without human annotation.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: In our paper, we provide comprehensive details about the backbone, dataset, algorithm, training hyperparameters, and hardware platform used to ensure experiment reproducibility. Additionally, we will open-source all training code and logs. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We have explained the impact of our work from both technical and societal perspectives in Section D. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
theoretical explanation: 
 In this section, we present a theoretical framework to explain the empirical phenomenon that incorporating an image-response relevance score can calibrate the self-rewarding procedure, ultimately improving generation accuracy.As we consider an LVLM, to facilitate the analysis, we decompose the input prompt into x = (x v , x t ) ∈ R dv × R dt , representing the image and text prompts respectively. Although text data typically comprises discrete tokens, we follow the CLIP theory literature [38] [39] [40] in modeling them as continuous-value random vectors in this section to elucidate the rationale behind our proposed method. More specifically, we assume the following data generative model for x v and x t :x v = U 1 z 1 + ξ 1 , and x t = U 2 z 2 + ξ 2 ,where U 1 ∈ O dv×r and U 2 ∈ O dt×r are two orthonormal matrixces, representing decoders that transform the latent (low-dimensional) signals z 1 , z 2 ∈ R r to images and text respectively. We assume the covariance matrices of z 1 , z 2 are identity matrices. ξ 1 ∈ R dv and ξ 2 ∈ R dt are noise vectors, and we assume they follow sub-gaussian distributions with well-conditioned covariance matrices and sub-gaussian norms upper bounded by a universal constant. We consider the infinite data setting. This is a widely used simplification to avoid the influence of sample randomness [41] [42] [43] . According to [38] , with an abundance of image-text pairs, the learned visual CLIP embedding F I (x v ) and textual CLIP embedding F T (x t ) converge to U ⊤ 1 x v and U ⊤ 2 x t respectively. To simplify our analysis without loss of generality, we consider a single score for each response y and define the image-response relevance score R I (y) = ⟨U ⊤ 1 x v , U ⊤ 2 y⟩. We assume the ground truthy truth = V * 1 x v + V * 2 x t + ϵ y with weights V * 1 ∈ R dv×dv and V * 2 ∈ R dv×dt .In CSR, we assume the conditional distribution at iteration t, π θt (y | x) withθ t = (V 1 , V 2 ), follows a Gaussian distribution π θt (y | x) ∝ exp(-∥y -(V 1 x v + V 2 x t )∥ 2 /σ 2 ), where V 1 ∈ R dv×dv and V 2 ∈ R dv×dtare the weights matrices for the image and text inputs respectively, and σ > 0 is the standard deviation. As the likelihood is monotonically decreasing with respect to ∥y -(V 1 x v + V 2 x t )∥ 2 , we consider the self-generated instruction-following score R T (y) = -∥y - (V 1 x v + V 2 x t )∥ 2 . Then the calibrated reward score becomes R(y) = λ • R I (y) + (1 -λ) • R T (y),for some λ ∈ [0, 1]. In theoretical analysis, we consider a simpler version of CSR, where we assume y w = arg max y R(y) (whose distribution is denoted by p * θt (y | x)), and y l is the text output generated by π θt (y | x). As R(y) depends on λ, we denote the solution θ t+1 by θ t+1 (λ). In the special case where λ = 1, this corresponds to the setting where we do not use the image-response relevance score at all.To evaluate the quality of the text output y, we consider a regression problem where there is an outcome z associated with the ground-truth text output y truth : z = β * ⊤ y truth with β * ∈ R dt . We evaluate the quality of y by considering the loss function L(y) = min β∈R d t E[(z -β ⊤ y) 2 ]. We then have the following theorem.Theorem 5.1. Suppose that π * θt (y | x) lies in the LLM space {π θ (y | x) : θ ∈ Θ}, ∥β * ⊤ V * ⊤ 1 β * ∥ ≫ ∥β * ⊤ V * ⊤ 2 β * ∥ and ∥β * ⊤ V ⊤ 1 β * ∥ ≪ ∥β * ⊤ V ⊤ 2 β * ∥, then there exists λ < 1, such that E π θ t+1 (λ) (y|x) [L(y)] < E π θ t+1 (1) (y|x) [L(y)].Our theoretical analysis implies that as long as∥β * ⊤ V ⊤ 1 β * ∥ ≪ ∥β * ⊤ V ⊤ 2 β * ∥,which happens when the model tends to prioritize textual information over visual input. By incorporating the imageresponse relevance score (corresponding to λ < 1), CSR is able to increase the attention on image signals in generating y. As a result, the solution produced by CSR will be better than the method without using the image-response relevance score (corresponding to λ = 1). 
related work: 
 Large Visual-Language Model Hallucination. Recently, the rapid development of visual-language alignment methods [19, [44] [45] [46] [47] [48] [49] and LLMs [50] [51] [52] [53] [54] has significantly accelerated the progress of LVLMs, which extend LLMs with visual modalities and demonstrate impressive visual understanding by unifying the encoding of visual and text tokens [34, [55] [56] [57] . However, LVLMs still face the problem of hallucination [58, 59] , where generated text descriptions contradict the visual modality information. Various approaches have been proposed to address hallucination in LVLMs, including enhancing dataset quality for fine-tuning [60, 8, 61, 9] , manipulating the decoding process [37, [62] [63] [64] [65] [66] , and leveraging external closed-source models to facilitate post-hoc mitigation of hallucination [58, [67] [68] [69] [70] . Though these approaches alleviate hallucination to some extent, they do not focus directly on improving modality alignment.Preference and Modality Alignment. In large models, alignment is necessary to ensure their behavior aligns with human preferences [71, 15, 72] . In LVLMs, alignment manifests as modality misalignment, where the generated textual responses are supposed to follow the input visual information. Recently, preference optimization has been used to address the modality misalignment problem. These optimizations involve preference data curated by human annotators [8, 60, 30] and additional models (e.g., GPT-4) [9, 10] . While these methods improve the ability of LVLMs to align modalities, their reliance on human annotation or additional models is resource-intensive and may introduce additional biases. Furthermore, these models cannot fully capture the inherent preferences of LVLMs, making the curated preference data less effective. Instead, CSR leverages a calibrated self-rewarding strategy, aiming to stimulate the LVLMs' self-correction and enhancement capabilities, thereby further improving modality alignment.Self-Improvement in Large Language Models. Self-improvement emerges as a powerful paradigm for LLMs to enhance themselves without significant external intervention. For example, self-rewarding and online alignment propose a method that selects consistent answers generated by the model to fine-tune itself [73, 74] , thereby improving its reasoning ability. Similarly, Chen et al. [12] utilizes self-play to enhance the model's performance by distinguishing its self-generated responses from those in human-annotated training data. Unlike prior methods that primarily target LLMs, CSR addresses the modality misalignment issue in LVLMs during the preference modeling process by introducing visual constraints, making it particularly well-suited for LVLMs. 
conclusion: 
 In this paper, we investigate the challenge of enhancing modality alignment in LVLMs by introducing a calibrated self-rewarding approach, which integrates visual constraints into the preference modeling process of the self-rewarding paradigm. Empirically, CSR enhances the alignment between image and text modalities, significantly reducing hallucination and improving performance on various LVLM evaluation benchmarks. These empirical results are further supported by rigorous theoretical findings. Additionally, CSR is capable of continuously enhancing LVLM capabilities over iterations, leading to better utilization of visual information. Sentence-Level Beam Search. We configure our parameters as follows to ensure both diversity and quality in the sampled data. The num_beamsparameter, set to 5, determines the capacity of input at each search layer. Additionally, num_token_beams, also set to 5, ensures that each beam search returns 5 token-level search results. The eos_token_id is set to the token for a period, effectively controlling the sentence-by-sentence generation process. The max_length parameter, set to 1024, prevents truncation errors and infinite repetitions by controlling the maximum length, while max_new_tokens, set to 74, limits the maximum length of newly generated content to avoid exceeding the CLIP encoding limit. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not involve any high risks or potential misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All the mentioned previous work are properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To further enhance data diversity, we utilize group beam search by setting the num_beam_group parameter to 5. This approach, when matched with token-level search, significantly boosts the diversity of each data point. The diversity_penalty parameter, set to a value of 3.0, effectively controls the diversity and quality of the sampled data among different beam groups. Calibrated Rewarding. We set the clip score weight to 0.9 and the language score weight to 0.1 when calculating the scores, giving greater emphasis to visual calibration.• MME [20] is a comprehensive benchmark for assessing the capabilities of LVLMs in multimodal tasks. It systematically evaluates models across two primary dimensions: perception and cognition, through 14 meticulously designed subtasks that challenge the models' interpretative and analytical skills.• SEED-Bench [21] is designed to evaluate the generative comprehension capabilities of LVLMs.It features an extensive dataset of 19K multiple-choice questions with precise human annotations, covering 12 distinct evaluation dimensions that assess both spatial and temporal understanding across image and video modalities.• LLaVA W [19] is a comprehensive benchmark for evaluating visual reasoning models. It comprises 24 diverse images with a total of 60 questions, covering a range of scenarios from indoor and outdoor settings to abstract art.• MMBench [22] introduces a dual-pronged approach: a meticulously curated dataset that significantly expands the scope and diversity of evaluation questions, and a pioneering CircularEval strategy that leverages ChatGPT to transform free-form predictions into structured choices.• MM-Vet [23] is an evaluation benchmark tailored for assessing the multifaceted competencies of LVLMs. It systematically structures complex multimodal tasks into 16 distinct integrations derived from a combination of 6 core vision-language capabilities, providing a granular analysis of model performance across diverse question types and answer styles.• ScienceQA [24] is a multimodal benchmark designed to evaluate and diagnose the multi-hop reasoning ability and interpretability of AI systems within the domain of science. It offers an expansive dataset of approximately 21k multiple-choice questions across a broad spectrum of scientific topics, complemented by detailed answer annotations, associated lectures, and explanations.• VizWiz [25] is a dataset in the field of visual question answering (VQA), derived from a naturalistic setting with over 31,000 visual questions. It is distinguished by its goal-oriented approach, featuring images captured by blind individuals and accompanied by their spoken queries, along with crowdsourced answers.• GQA [26] is a dataset engineered for advanced real-world visual reasoning, utilizing scene graphbased structures to generate 22 million diverse, semantically-programmed questions. It incorporates a novel evaluation metrics suite focused on consistency, grounding, and plausibility, establishing a rigorous standard for assessing in vision-language tasks.• POPE [27] is an assessment methodology designed to scrutinize object hallucination in LVLMs.It reformulates the evaluation into a binary classification task, prompting LVLMs with straightforward Yes-or-No queries to identify hallucinated objects. POPE offers a stable and adaptable approach, utilizing various object sampling strategies to reveal model tendencies towards hallucination.• CHAIR [28] is a widely-recognized tool for evaluating the incidence of object hallucination in image captioning tasks, which has two variants: CHAIR I and CHAIR S , which assess object hallucination at the instance and sentence levels, respectively. Formulated as:CHAIR I = |{hallucinated objects}| |{all mentioned objects}| CHAIR S = |{captions with hallucinated objects}| |{all captions}|Specifically, we randomly sampled 500 images from the COCO [75] validation set and evaluated object hallucination using the CHAIR metric.• LLaVA-1.5 [1] is an improvement based on the original LLaVA [19] model demonstrating exceptional performance and data efficiency through visual instruction tuning. It enhanced with a CLIP-ViT-L-336px visual backbone and MLP projection. By incorporating academic-task-oriented VQA data and simple response formatting prompts, LLaVA-1.5 achieves the state-of-the-art results at that time with a remarkably modest dataset of just 1.2 million public images.• InstructBLIP [31] leverages instruction tuning on pretrained BLIP-2 models, integrating an instruction-aware Query Transformer to enhance feature extraction for diverse vision-language tasks. It achieved state-of-the-art zero-shot performance at the time across 13 datasets and excelled in fine-tuned downstream tasks, such as ScienceQA, showcasing its advantage over contemporaneous multimodal models.• Qwen-VL-Chat [32] is built upon the Qwen-LM [4] with a specialized visual receptor and inputoutput interface. It is trained through a 3-stage process and enhanced with a multilingual multimodal corpus, enabling advanced grounding and text-reading capabilities.• mPLUG-Owl2 [33] employs a modular network design with a language decoder interface for unified modality management. It integrates shared modules for cross-modal collaboration and modality-adaptive components for feature retention, enhancing generalization in both text-only and multimodal tasks.• BLIP-2 [34] is a vision-language pre-training framework that efficiently leverages off-the-shelf frozen image encoders and LLMs. Employing a two-stage pre-training strategy with a lightweight Querying Transformer, BLIP-2 bridges the modality gap between vision and language, enabling zero-shot image-to-text generation that adheres to natural language instructions while maintaining high compute-efficiency.• IDEFICS [35] is an open-access visual language model that expands upon the Flamingo [44] architecture, offering both base and instructed variants with 9 billion and 80 billion parameter sizes. It is developed using solely publicly available data and models.• POVID [10] is a novel training paradigm aligns the preferences of VLLMs through external preference data from GPT4 and the inherent hallucination patterns within the model triggered by noisy images.• RLHF-V [30] collected fine-grained paragraph-level corrections from humans on hallucinations and performing dense direct preference optimization on the human feedback.• Silkie [9] constructed a VLFeedback dataset using VLLMs annotation. Specifically, the responses were generated by 12 LVLMs models conditioned on multimodal instructions extracted from different datasets. The entire dataset was evaluated using GPT-4V to assess the generated outputs in terms of helpfulness, visual faithfulness, and ethical considerations. In this paper, the VLFeedback dataset was utilized to perform one round of DPO on LLaVA-1.5.• LLaVA-RLHF [8] proposes a novel alignment algorithm called Factually Augmented RLHF, which enhances the reward model by incorporating additional factual information such as image captions and ground-truth multi-choice options. In this paper, the annotated preference data is used to conduct one round of preference learning on LLaVA1.5.• Self-rewarding [29] introduces a method for self-feedback learning in LLMs and serves as a baseline for our approach, referred to as CSR. Specifically, for each input image and prompt, two outputs are sampled from LLaVA-1.5. The model is provided with the prompt mentioned in Table 3 and is tasked with determining which output is better. Finally, LLaVA-1.5 is fine-tuned using the collected preference data, with the entire setup and the images and prompts used for inference matching those of CSR.The sources of preference data generally fall into two main categories: external preference data and self-generated data. External preference data typically represent preferences obtained from human annotations or GPT-4. Although external preference data generally have higher quality compared to self-generated data, are they really more effective? We conducted an analysis using 500 samples obtained from the original LLaVA-1.5 7B model. Following the same pipeline as CSR, we selected samples with the highest and lowest rewards as preferred (chosen) and dispreferred (rejected) responses. We further employed the GPT-4 API to transform preferred responses into dispreferred ones, with specific prompts referenced in Table 4 .In Figure 8 , we present the distribution based on both the sampling probabilities score generated by the target LVLM, which describes the probability of the LVLM generating this response. Clearly, compared to the model's own generated dispreferred responses, the dispreferred responses modified by GPT-4V are not as easily confusable for the model. This result partially supports the idea that dispreferred responses generated by external models are more easily distinguishable by the target LVLM, making them less effective.Table 3 : Prompt for self-reward: utilizing the model itself as a judge to determine whether the corresponding response is a chosen response or a reject response. Output Format: response 1 or response 2.In this subsection, we provide a comparison of CSR with other state-of-the-art models, a performance comparison of different CSR iterations, a comparison of hallucinations in different CSR iterations, validation experiments of CSR on other models, ablation study on λ in Eqn (4), and the relationship between reward score and average performance score. Experiments strongly demonstrate the effectiveness of CSR.For the ablation study on λ in Eqn (4), our training settings are consistent with Table 1 , with three rounds of iteration. The experimental results in Table 9 show that as the value of λ increases, the models performance on various benchmarks improves. This suggests that calibrating the models (1) Revise the response while maintaining its original format and order as much as possible.(2) Based on the provided image, primarily add, replace, or modify entities in the input response to make them related to the image but incorrect. Adjust their attributes and logical relationships accordingly.( rewarding process using the visual score can enhance the preference learning process, thereby boosting performance. Proof of Theorem 5.1. Let us first denote the distribution of y w by π * θt (y | x). As we take y w = arg max y R(y), this distribution is a point mass. As a result, the global minimizer to (5) will then converge to π * θt (y | x). In the following, we analyze how π * θt (y | x) is shaped. By the CSR procedure, we havey w = arg max y (1 -λ))⟨U ⊤ 1 x v , U ⊤ 1 y⟩ -λ∥y -V 1 x v + V 2 x t ∥ 2 = 1 -λ λ U 1 U ⊤ 1 x v + V 1 x v + V 2 x t .We can see that CSR up-weights the signal of the image input. Then L(y) = minβ∈R d t E[(z -β ⊤ y) 2 ] = min β∈R d t E[(β * ⊤ y truth -β ⊤ y) 2 ]= minβ∈R d t E[(β * ⊤ (V * 1 x v + V * 2 x t )) -β ⊤ y) 2 ] + V ar(ϵ y )∥β * ∥ 2We haveE[(β * ⊤ (V * 1 x v + V * 2 x t )) -β ⊤ y) 2 ] = E[(β * ⊤ (V * 1 x v + V * 2 x t )) -β ⊤ ( ( 1 -λ λ U 1 U ⊤ 1 + V 1 )x v + V 2 x t ) ) 2 ]As we assume ∥V1∥∥β * ⊤ V * 1 ∥ ≪ ∥V2∥ ∥β * ⊤ V *2 ∥ and due to the smoothness over parameters. Without loss of generality, we prove the claim for the case where∥V 1 ∥ = 0, that is V 1 =0.In this case, we want to show that there exists λ ∈ (0, 1), such that minβ∈R d t E[(β * ⊤ (V * 1 x v + V * 2 x t )) -β ⊤ ( ( 1 -λ λ U 1 U ⊤ 1 )x v + V 2 x t ) ) 2 ]< minβ∈R d t E[(β * ⊤ (V * 1 x v + V * 2 x t )) -β ⊤ (V 2 x t )) 2 ]Due to the independence between x v and x t , the right-hand sides is lower bounded byβ * ⊤ V * 1 Cov(x t )V * ⊤ 1 β * .The left-hand side, on the other hand, can be upper bounded by the value when we take β 0 such that1-λ λ U 1 U ⊤ 1 β 0 = U 1 U ⊤ 1 V * ⊤ 1 β * , which equals to β * ⊤ V * 1 (I -U 1 U ⊤ 1 )Cov(x t )(I -U 1 U ⊤ 1 )V * ⊤ 1 β * . As we assume ∥β * ⊤ V * ⊤ 1 β * ∥ ≫ ∥β * ⊤ V * ⊤ 2 β * ∥,this is a dominating term when the left-hand side is evaluated at β 0 .In addition, we assume Cov(ξ 1 ) is well-conditioned, implying Cov(x t ) is well-conditioned, and thereforeβ * ⊤ V * 1 (I -U 1 U ⊤ 1 )Cov(x t )(I -U 1 U ⊤ 1 )V * ⊤ 1 β * < β * ⊤ V * 1 Cov(x t )V * ⊤ 1 β * .We complete the proof.Due to limitations in computing resources, we conducted only three iterations of CSR. Additionally, our experiments were confined to 7B and 13B models. This restriction prevents us from determiing if our method adheres to a scaling law. We hope to continue iterative training in the future and to train larger models, given access to more computing resources, to explore the upper limits of our method.Our approach requires no additional human annotations and significantly enhances model performance using the model itself. Technically, our method may inspire more researchers to explore how multimodal models can learn from themselves. From a societal impact perspective, our method significantly reduces hallucinations in LVLMs, a major factor affecting the application of AI in realworld scenarios. Our approach promotes more responsible use of LVLMs. However, it is important to note that while our method significantly reduces hallucinations, they still occur. Therefore, it is crucial to employ various measures to ensure safety and stability when applying this approach in real-world scenarios.Answer: [Yes] Justification: In Section 5 and Appendix B, we provide a complete theoretical analysis and proof. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 


Decision: This paper proposes a new method called Calibrated Self-Rewarding (CSR) to address hallucination issues in large vision-language models. It allows the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. The reward is computed using its own LLM (text only) and an external CLIP model.  Overall, all reviewers agree the approach is novel and the empirical evaluation is thorough, encompassing twelve benchmarks and tasks. However, there are some weaknesses including technical novelty is somewhat limited, and performance gains are modest on some benchmarks.
Overall, while not groundbreaking, this paper makes a solid contribution to an important problem in multimodal AI. The AC recommend weak accept.

Reviews: review: paper_summary: paper_summary: This paper proposes a new method for preference alignment with LVLMs. Specifically, the reward is computed using its own LLM (text only) and an external CLIP model. The optimization is done with DPO. This process can be iterated for several times.
strength_weakness: strength_weakness: I don't find significant weaknesses. However, I do think the experiments can be improved. Therefore, I give a conservative score of borderline accept. I may consider increasing the rating if more thorough studies are conducted (see next section).
questions: questions: Q1. How did the performance improve over each preference data generation stage?

Q2. Scaling study as in Gao et al. [1]?


[1] Gao et al., "Scaling Laws for Reward Model Overoptimization", ICML 2023
limitations: limitations: Limitations and broader impact are discussed in the appendix.
paper_summary: paper_summary: The paper proposes a new approach to addressing the hallucination problem in Large Vision-Language Models (LVLMs). This phenomenon occurs when generated text responses appear linguistically plausible but contradict the visual input, indicating a misalignment between image and text pairs. The proposed solution, Calibrated Self-Rewarding (CSR), allows the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. This method incorporates visual constraints into the self-rewarding process, emphasizing visual input. Empirical results show performance improvements and reduced hallucinations across various benchmarks and tasks.
strength_weakness: strength_weakness: **Technical Novelty:** The primary distinction of the proposed method from previous approaches is the introduction of a new reward score based on vision-language similarity and step-wise evaluation. While this is a meaningful contribution, the paper could benefit from a more detailed discussion on how this approach fundamentally differs from and improves upon existing methods.

**Performance Improvements:** Although CSR shows substantial improvements on average, the performance gains in some benchmarks, such as LLaVA-1.5-13B, are not very substantial. It would be beneficial to explore the reasons behind these limited improvements and suggest potential avenues for further enhancement.

**Figures:** Figures 1 and 2 lack clarity. The visual representation of preference data and the meaning of green and red circles in Figure 2, as well as the shapes and score positions in Figure 1, need to be better explained and presented more clearly.
questions: questions: How is the improvement of approximately 7.62% computed? Please provide a detailed explanation of the calculation method used to arrive at this figure.

What is the upper bound in the performance gain of CSR? A case study showing its potential on one or several benchmarks with LLaVA-1.5 13B would be helpful, given the increasing trend shown in Figure 3.

Figure 2 is unclear. What do the green and red circles represent? What does "Option" mean? Is it possible to visualize the generated preference data in Figure 2 or elsewhere? Additionally, Figure 1 appears confusing regarding the shapes and score positions. Can these be clarified and improved for better understanding?
limitations: limitations: The authors acknowledge some limitations of their work, such as conducting three iterations of CSR. While this approach shows promise, its technical novelty compared to existing methods is somewhat limited. 

Additionally, the performance improvements are not uniformly substantial across all benchmarks. The paper could benefit from a more detailed discussion of these limitations and potential strategies for addressing them in future work. 

Furthermore, the potential negative societal impact of the work should be considered, particularly in terms of the ethical implications of improving LVLMs that might be used in sensitive applications. Providing constructive suggestions for mitigating any negative impacts would be valuable.
paper_summary: paper_summary: The paper addresses the challenge of hallucination in Large Vision-Language Models (LVLMs), where generated text responses appear plausible but contradict the input image. This misalignment occurs because the models prioritize textual information over visual input, even with high-quality representations. Existing methods to address this issue involve resource-intensive preference optimization through additional models or human annotations, which may not align well with the LVLM’s preferences.

To overcome these challenges, the authors propose the Calibrated Self-Rewarding (CSR) approach. CSR allows the model to self-improve by generating candidate responses, evaluating rewards for each, and curating preference data for fine-tuning. The process emphasizes visual input through a step-wise strategy and incorporates visual constraints into the self-rewarding mechanism.
strength_weakness: strength_weakness: * In the limitation section, the authors mention that they could only run three iterations due to computation issues. However, in Section 4.1, the authors say they used one A100 80GB GPU, which takes roughly 3.5 and 5 hours to fine-tune LLaVA-1.5 7B and LLaVA-1.5 13B, respectively. It seems like such an experiment could be done with more iterations to see how the score trend continues across more iterations. This could supplement the claim of increased performance concerning iterations made in Section 4.2. 

* The authors use the CLIP similarity score to align the image and text response. However, there is a concern that CLIP may primarily focus on prominent objects in the foreground, potentially overlooking smaller details in the background. If this is the case, using CLIP for reward calculation might inadvertently cause the Vision-Language Model (VLM) to miss fine details in the image, even though it would reduce hallucinations overall.

* Although Table 2 shows an analysis of the effect of each reward term ($R_I$ and $R_T$), it would be beneficial to see how varying $\lambda$ terms affect the score. 

* The benefit of the proposed method, apart from the resulting benchmark score, is that it takes much less resources to align the model. Can a rough estimate of how different the computing resource would be, including the time to gather the preference data for previous methods? 

* (minor) To this reviewer, the overall framework figure (Figure 2) seems hard to understand even after understanding the proposed method through the texts in the paper.
questions: questions: See weaknesses above
limitations: limitations: The authors addressed the limitations of the work.
paper_summary: paper_summary: The paper generally follows self-rewarding language models and applies the idea to vision-language models. The method first ask a VLM to self-generate candidates, based on which they score the candidates with the VLM itself and CLIPScore, and then perform DPO on the generated candidates. Experiments on LLaVA demonstrate improvements over baselines such as self-rewarding.
strength_weakness: strength_weakness: 1. The methodological contribution is not sufficient as it basically follows self-rewarding language models [11] and applies it in another domain. The proposed techniques such as using CLIPScore to calibrate the self-generated scores are incremental changes to me.
2. The empirical results do not seem significant in many of the datasets. In Table 1, on popular benchmarks such as GQA and SEED, the performance improvements are marginal (one exception may be LLaVA-W, but this dataset is small and relies on GPT evaluations). Results on VQAv2 and MMStar should also be added.
questions: questions: Please refer to the weakness section.
limitations: limitations: the authors adequately addressed the limitations
paper_summary: paper_summary: This paper addresses an important and tough issue in LVLM – hallucination, which is usually caused by the misalignment of the image and text modalities. A new method CSR is proposed, by extending the language self-awarding approach to multimodality, considering both instruction-following (text) and  image-text alignment (multimodality) awarding.

The experiments are very comprehensive and solid, demonstrating the effectiveness of CSR in a wide range of tasks, especially for hallucination specific tasks.
strength_weakness: strength_weakness: The ablation of the weight of instruction-following score and image-text alignment score is missing, which is important to help understand how important of each score for CSR. 

It might worth inverstigating the impact of different image-text alignment scoring approaches. At least comparing different image-text models.
questions: questions: It would be more insightful if we can distinguish 1) comprehensive, 2) general VQA and 3) hallucination scores (e.g. avg score per type of benchmarks) in the main ablation, i.e., Table 2. I would expect “Only RI” to increase 3) but hurt 1) and 2); however, this is not verifiable given the current average scores.

The terminologies “sentence group” and “option” in Figure 2 are confusing (especially before readding 3.2). More legend or explanation (or re-structuring the figure) might be needed for better readability.

What is the consideration to use the same examples in different iterations? Using random examples may lead to even better results (to converge later)?
limitations: limitations: N/A

Comments: title: Response to Reviewer GvTb
Response to Reviewer bHbK
Thank you for the response!
Response to Follow-Up Questions from Reviewer GvTb (2/2)
Response to Follow-Up Questions from Reviewer GvTb (1/2)
Please discuss with authors
Please discuss with authors
Response to Reviewer myph
Response to Reviewer 4vrA
comment: Thanks a lot for raising your score and your constructive comments!
Thanks for the additional results! I'm further increasing my ratings and confidence.
Dear Reviewer bHbK,

Thank you for your response and for increasing your score. We're delighted that our answers addressed your questions. We'll continue exploring ways to further calibrate reward scores in the future.
Thank you for your response!

I've read the rebuttal and the other reviews. 

My concern about the effectiveness of the method is mostly addressed (though the improvements on VQAv2 seem rather marginal.)

Regarding the technical contribution, I agree that using an LVLM as a judge may be biased toward the contextual information. Although interpolating the CLIPScore may not be a perfect solution, the empirical improvements suggest it is a simple and effective approach, even though less principled to me.

Considering these factors and the other positive reviews, I will raise my score to borderline accept.
> **Q3**: Following Q2&A2, I believe readers will also be interested in how reward model sizes affect the over-optimization

**A3**: Due to the time constraints of the discussion period, we conducted experiments on LLaVA-1.5-7B using the following reward models (CLIP models): 7 iterations with clip-vit-l-336px (the model used in the paper, which is a relatively small model); and 3 iterations with clip-vit-g-14 (relatively large model). The results are reported in Tables R4 and R5. We observe that while the stronger reward model provides better feedback earlier, it also causes the target LVLM to converge more quickly.

**Table R4**: Performance of **clip-vit-l-336px (small model)** w.r.t. CSR iterations

| Method        | MME$^P$   | MME$^C$  | SEED  | LLaVA$^W$ | MMB   | MM-Vet | SQA$^I$  | VisWiz | GQA   | POPE  | CHAIR$_S$ | CHAIR$_I$ |**Avg score**|
|---------------|--------|-------|-------|--------|-------|--------|-------|--------|-------|-------|--------|--------|--------|
| CSR iter-1  | 1500.6| 367.5| 60.4| 69.7| 64.7| 32.2| 70.3| 54.0| 62.1  | 86.94 | 26.6   | 7.2    |71.02|
| CSR iter-2  | 1519.0| 368.9 |60.3| 70.4| 65.2| 33.7 |70.1 |54.0 |62.3 | 86.82 | 23.0   | 6.1    |71.74|
| CSR iter-3  | 1524.2 | 367.9 | 60.3  | 71.1   | 65.4  | 33.9   | 70.7  | 54.1   | 62.3  | 87.01 | 21.0   | 6.0    |72.09|
| CSR iter-4  | 1524.6 | 368.8 | 60.4  | 71.0   | 65.3  | 33.9   | 70.4  | 54.0   | 62.2  | 87.05 | 19.0   | 5.9    |72.24|
| CSR iter-5  | 1520.1 | 367.2 | 60.5  | 71.3   | 65.4  | 33.8   | 70.8  | 54.2   | 62.4  | 87.16 | 18.3   | 5.4    |72.39|
| CSR iter-6  | 1521.3 | 365.4 | 60.4  | 70.9   | 65.3  | 34.0   | 70.7  | 54.0   | 62.4  | 87.15 | 17.9   | 5.2    |72.35|
| CSR iter-7  | 1520.8 | 360.3 | 60.2  | 71.1   | 65.2  | 33.7   | 70.3  | 54.1   | 62.3  | 87.12 | 18.8   | 5.6    |72.06|

**Table R5**: Performance of **clip-vit-l-336px  (large model)** w.r.t. CSR iterations

| Method        | MME$^P$   | MME$^C$  | SEED  | LLaVA$^W$ | MMB   | MM-Vet | SQA$^I$  | VisWiz | GQA   | POPE  | CHAIR$_S$ | CHAIR$_I$ |**Avg score**|
|---------------|--------|-------|-------|--------|-------|--------|-------|--------|-------|-------|--------|--------|--------|
| CSR iter-1   | 1511.3 | 367.1 | 60.6  | 70.6   | 65.3  | 33.0   | 70.3  | 54.3   | 62.3  | 87.02 | 26.2   | 7.1    | 72.06|
| CSR iter-2  | 1519.6 | 365.9 | 60.5  | 71.9   | 65.4  | 34.0   | 70.7  | 54.1   | 62.2  | 87.21 | 20.4   | 7.0    |72.09|
| CSR iter-3  | 1512.1| 367.2 |60.4| 70.8| 65.2| 33.2 |70.2 |54.0 |62.4 | 87.13 | 23.9   | 7.4    |71.53|
Thanks a lot for increasing your score, we are happy that our response addresses your concerns. For the follow up questions, we detail our response below and please kindly let us know if our response addresses your concerns. 


> **Q1**: Following A1, is it possible to add iter-4 results (even it's negative)? Since for the 13B model it seems the performance not yet converges.

**A1**: We added additional iteration results for the LLaVA-7B and 13B models, which are presented in Tables R1 and R2, respectively. Due to the time constraints of the discussion period, we conducted four additional iterations for the 7B model (iter-4 through iter-7) and two additional iterations for the 13B model (iter-4 and iter-5). As shown in the "Avg score" column in both Table R1 and Table R2, the overall improvement tends to slow down or even fluctuate with each additional iteration, indicating that the performance is converging.

**Table R1**: Results of additional iteration evaluation of 7B model
| Method        | MME$^P$   | MME$^C$  | SEED  | LLaVA$^W$ | MMB   | MM-Vet | SQA$^I$  | VisWiz | GQA   | POPE  | CHAIR$_S$ | CHAIR$_I$ | **Avg score** |
|---------------|--------|-------|-------|--------|-------|--------|-------|--------|-------|-------|--------|--------|--------|
| CSR iter-3  | 1524.2 | 367.9 | 60.3  | 71.1   | 65.4  | 33.9   | 70.7  | 54.1   | 62.3  | 87.01 | 21.0   | 6.0    |72.09|
| CSR iter-4  | 1524.6 | 368.8 | 60.4  | 71.0   | 65.3  | 33.9   | 70.4  | 54.0   | 62.2  | 87.05 | 19.0   | 5.9    |72.24|
| CSR iter-5  | 1520.1 | 367.2 | 60.5  | 71.3   | 65.4  | 33.8   | 70.8  | 54.2   | 62.4  | 87.16 | 18.3   | 5.4    |72.39|
| CSR iter-6  | 1521.3 | 365.4 | 60.4  | 70.9   | 65.3  | 34.0   | 70.7  | 54.0   | 62.4  | 87.15 | 17.9   | 5.2    |72.35|
| CSR iter-7  | 1520.8 | 360.3 | 60.2  | 71.1   | 65.2  | 33.7   | 70.3  | 54.1   | 62.3  | 87.12 | 18.8   | 5.6    |72.06|

**Table R2**: Results of additional iteration evaluation of 13B model
| Method        | MME$^P$   | MME$^C$  | SEED  | LLaVA$^W$ | MMB   | MM-Vet | SQA$^I$  | VisWiz | GQA   | POPE  | CHAIR$_S$ | CHAIR$_I$ |**Avg score**|
|---------------|--------|-------|-------|--------|-------|--------|-------|--------|-------|-------|--------|--------|--------|
| CSR iter-3  | 1530.6 |303.9| 62.9| 74.7| 68.8| 37.8| 75.1| 56.8| 63.7 | 87.30 | 28.0   | 7.3    |71.95|
| CSR iter-4  | 1530.4 | 301.4 | 63.0  | 74.2   | 68.3  | 37.3   | 75.2  | 56.6   | 63.4  | 87.20 | 27.4   | 7.4    |71.78|
| CSR iter-5  | 1531.1 | 302.2 | 62.8  | 74.0   | 68.2  | 37.4   | 74.8  | 56.7   | 63.7  | 87.18 | 27.2   | 7.6    |71.77|


---

> **Q2**: do you observe "model collapse" such that the reward keep increasing yet the benchmark score drops

**A2**: We analyzed the relationship between the reward score and the final average performance of LLaVA-7B, with the results presented in Table R3. In the later stages of training, it can be observed that a slight increase in the reward score coincided with a slight decrease in model performance. One possible reason is that the model's performance had mostly converged, yet it continued to overfit the training data in pursuit of a higher reward score. Another explanation could be the inherent limitations of the target LVLM and the reward-calibrating model, making it increasingly challenging to accurately assign rewards to certain difficult data points. Early in training, these mis-rewarded data points may not have had a significant impact because the majority of the data was useful. However, as the model's capabilities improved, the mis-rewarded data began to affect overall performance. We believe that introducing more diverse vision-centric tools for reward calibration, along with a more thoughtful data selection mechanism, could potentially alleviate this issue. We plan to systematically explore these methods further in the future.

**Table R3**: Results of rewards w.r.t. average score
|     | iter-1 | iter-2 | iter-3 |  iter-4| iter-5| iter-6| iter-7|
|-------------|--------|--------|--------|--------|--------|--------|--------|
| Chosen reward          | 0.4885 | 0.5040 | 0.5052 | 0.5055| 0.5066|0.5078|0.5079|
| Rejected reward          | 0.4551 | 0.4788 | 0.4789 |0.4794|0.4799|0.4805|0.4812|
| Avg performance score         |66.61 | 71.02 | 71.74 | 72.09 | 72.24 | 72.39 | 72.35 | 72.06 |
Dear Reviewer bHbK:

Please respond to author rebuttal and discuss with authors. 

Thanks,

Your AC
Dear Reviewer oVPD:

Please respond to author rebuttal and discuss with authors. 

Thanks,

Your AC
Dear Reviewer myph,

Thank you for your response and for raising your score. We’re very pleased that our answers addressed your questions.
Dear Reviewer 4vrA,

Thank you for your response. We are pleased to hear that our answers have addressed all of your questions.
Thanks for the updates and the detailed explanation, which have addressed all my questions. I am happy to maintain the previous rating.
Thanks for the detailed response. They have resolved my issues and I have raised the score accordingly.
Thanks for the response, I appreciate the additional results. My main concerns are addressed so I will raise my rating to 6. (Sorry I overlooked Fig. 3 in my initial review.)

Following A1, is it possible to add iter-4 results (even it's negative)? Since for the 13B model it seems the performance not yet converges. Moreover, do you observe "model collapse" such that the reward keep increasing yet the benchmark score drops? If so, I think plotting the curve along the training progress could improve this submission. Following Q2&A2, I believe readers will also be interested in how reward model sizes affect the over-optimization. I'm open to raising the score again if the authors post new interesting results during the discussion period.