Title: Calibrated Self-Rewarding Vision Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Calibrated self-rewarding

large Vision-language models

Modality alignment
Abstract: Large Vision-Language Models (LVLMs) have made substantial progress by integrating pre-trained large language models (LLMs) and vision models through instruction tuning. Despite these advancements, LVLMs often exhibit the hallucination phenomenon, where generated text responses appear linguistically plausible but contradict the input image, indicating a misalignment between image and text pairs. This misalignment arises because the model tends to prioritize textual information over visual input, even when both the language model and visual representations are of high quality. Existing methods leverage additional models or human annotations to curate preference data and enhance modality alignment through preference optimization. These approaches are resource-intensive and may not effectively reflect the target LVLM's preferences, making the curated preferences easily distinguishable. Our work addresses these challenges by proposing the Calibrated Self-Rewarding (CSR) approach, which enables the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. In the reward modeling, we employ a step-wise strategy and incorporate visual constraints into the self-rewarding process to place greater emphasis on visual input. Empirical results demonstrate that CSR significantly enhances performance and reduces hallucinations across twelve benchmarks and tasks, achieving substantial improvements over existing methods by 7.62\%. Our empirical results are further supported by rigorous theoretical analysis, under mild assumptions, verifying the effectiveness of introducing visual constraints into the self-rewarding paradigm. Additionally, CSR shows compatibility with different vision-language models and the ability to incrementally improve performance through iterative fine-tuning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses an important and tough issue in LVLM – hallucination, which is usually caused by the misalignment of the image and text modalities. A new method CSR is proposed, by extending the language self-awarding approach to multimodality, considering both instruction-following (text) and  image-text alignment (multimodality) awarding.\n\nThe experiments are very comprehensive and solid, demonstrating the effectiveness of CSR in a wide range of tasks, especially for hallucination specific tasks.\n\nSTRENGTHS:\nThe issue this paper addresses is important and still an open question for LVLM training. The method proposed by this paper is simple yet effective, thus may be very insightful for the community.\n\nThe evaluation is comprehensive and solid, including 3 types of benchmarks: comprehensive benchmarks, general VQA and hallucination benchmarks, with 10 tasks in total. CSR is also compared with self-rewarding and several data-driven preference learning methods to demonstrate its effectiveness. Also different model sizes and model architectures are verified. The addtional Theoretical Explanation and attention map sections make the paper even more convincing.\n\nThe proposed method is easy to implement and extend to other modalities, so may have larger impact for other directions in the community.\n\nThis paper is very well written and readable.\n\nWEAKNESSES:\nThe ablation of the weight of instruction-following score and image-text alignment score is missing, which is important to help understand how important of each score for CSR. \n\nIt might worth inverstigating the impact of different image-text alignment scoring approaches. At least comparing different image-text models.\n\nQUESTIONS:\nIt would be more insightful if we can distinguish 1) comprehensive, 2) general VQA and 3) hallucination scores (e.g. avg score per type of benchmarks) in the main ablation, i.e., Table 2. I would expect “Only RI” to increase 3) but hurt 1) and 2); however, this is not verifiable given the current average scores.\n\nThe terminologies “sentence group” and “option” in Figure 2 are confusing (especially before readding 3.2). More legend or explanation (or re-structuring the figure) might be needed for better readability.\n\nWhat is the consideration to use the same examples in different iterations? Using random examples may lead to even better results (to converge later)?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

comments: SUMMARY:
The paper generally follows self-rewarding language models and applies the idea to vision-language models. The method first ask a VLM to self-generate candidates, based on which they score the candidates with the VLM itself and CLIPScore, and then perform DPO on the generated candidates. Experiments on LLaVA demonstrate improvements over baselines such as self-rewarding.

STRENGTHS:
1. The paper investigates the application self-rewarding proposed in NLP to the vision-language domain and proposes VL-specific changes.
2. The baselines are fairly strong and they perform evaluations on multiple standard benchmarks, making their results trustworthy.
3. The paper is well-written.

WEAKNESSES:
1. The methodological contribution is not sufficient as it basically follows self-rewarding language models [11] and applies it in another domain. The proposed techniques such as using CLIPScore to calibrate the self-generated scores are incremental changes to me.
2. The empirical results do not seem significant in many of the datasets. In Table 1, on popular benchmarks such as GQA and SEED, the performance improvements are marginal (one exception may be LLaVA-W, but this dataset is small and relies on GPT evaluations). Results on VQAv2 and MMStar should also be added.

QUESTIONS:
Please refer to the weakness section.

LIMITATIONS:
the authors adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of hallucination in Large Vision-Language Models (LVLMs), where generated text responses appear plausible but contradict the input image. This misalignment occurs because the models prioritize textual information over visual input, even with high-quality representations. Existing methods to address this issue involve resource-intensive preference optimization through additional models or human annotations, which may not align well with the LVLM’s preferences.\n\nTo overcome these challenges, the authors propose the Calibrated Self-Rewarding (CSR) approach. CSR allows the model to self-improve by generating candidate responses, evaluating rewards for each, and curating preference data for fine-tuning. The process emphasizes visual input through a step-wise strategy and incorporates visual constraints into the self-rewarding mechanism.\n\nSTRENGTHS:\n* The proposed method reduces the resources required to align a VLM model for less hallucination compared to previous methods. \n\n* The paper is clearly written.\n\n* The authors show theoretical explanation to validate the proposed method.\n\nWEAKNESSES:\n* In the limitation section, the authors mention that they could only run three iterations due to computation issues. However, in Section 4.1, the authors say they used one A100 80GB GPU, which takes roughly 3.5 and 5 hours to fine-tune LLaVA-1.5 7B and LLaVA-1.5 13B, respectively. It seems like such an experiment could be done with more iterations to see how the score trend continues across more iterations. This could supplement the claim of increased performance concerning iterations made in Section 4.2. \n\n* The authors use the CLIP similarity score to align the image and text response. However, there is a concern that CLIP may primarily focus on prominent objects in the foreground, potentially overlooking smaller details in the background. If this is the case, using CLIP for reward calculation might inadvertently cause the Vision-Language Model (VLM) to miss fine details in the image, even though it would reduce hallucinations overall.\n\n* Although Table 2 shows an analysis of the effect of each reward term ($R_I$ and $R_T$), it would be beneficial to see how varying $\\lambda$ terms affect the score. \n\n* The benefit of the proposed method, apart from the resulting benchmark score, is that it takes much less resources to align the model. Can a rough estimate of how different the computing resource would be, including the time to gather the preference data for previous methods? \n\n* (minor) To this reviewer, the overall framework figure (Figure 2) seems hard to understand even after understanding the proposed method through the texts in the paper.\n\nQUESTIONS:\nSee weaknesses above\n\nLIMITATIONS:\nThe authors addressed the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new approach to addressing the hallucination problem in Large Vision-Language Models (LVLMs). This phenomenon occurs when generated text responses appear linguistically plausible but contradict the visual input, indicating a misalignment between image and text pairs. The proposed solution, Calibrated Self-Rewarding (CSR), allows the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. This method incorporates visual constraints into the self-rewarding process, emphasizing visual input. Empirical results show performance improvements and reduced hallucinations across various benchmarks and tasks.\n\nSTRENGTHS:\nThe paper addresses the critical issue of hallucination in LVLMs by introducing a new Calibrated Self-Rewarding (CSR) approach. This method is new in its use of visual constraints within the self-rewarding paradigm to enhance image-text alignment.\n\nThe empirical evaluation is thorough, encompassing twelve benchmarks and tasks. The results show improvements over existing methods, demonstrating the effectiveness of CSR.\n\nThe paper is well-organized, providing a clear explanation of the problem, proposed solution, and empirical results. The inclusion of theoretical analysis further strengthens the clarity and credibility of the work.\n\nThe proposed CSR method has implications for improving the reliability and accuracy of LVLMs, which is crucial for their application in various real-world scenarios.\n\nWEAKNESSES:\n**Technical Novelty:** The primary distinction of the proposed method from previous approaches is the introduction of a new reward score based on vision-language similarity and step-wise evaluation. While this is a meaningful contribution, the paper could benefit from a more detailed discussion on how this approach fundamentally differs from and improves upon existing methods.\n\n**Performance Improvements:** Although CSR shows substantial improvements on average, the performance gains in some benchmarks, such as LLaVA-1.5-13B, are not very substantial. It would be beneficial to explore the reasons behind these limited improvements and suggest potential avenues for further enhancement.\n\n**Figures:** Figures 1 and 2 lack clarity. The visual representation of preference data and the meaning of green and red circles in Figure 2, as well as the shapes and score positions in Figure 1, need to be better explained and presented more clearly.\n\nQUESTIONS:\nHow is the improvement of approximately 7.62% computed? Please provide a detailed explanation of the calculation method used to arrive at this figure.\n\nWhat is the upper bound in the performance gain of CSR? A case study showing its potential on one or several benchmarks with LLaVA-1.5 13B would be helpful, given the increasing trend shown in Figure 3.\n\nFigure 2 is unclear. What do the green and red circles represent? What does \"Option\" mean? Is it possible to visualize the generated preference data in Figure 2 or elsewhere? Additionally, Figure 1 appears confusing regarding the shapes and score positions. Can these be clarified and improved for better understanding?\n\nLIMITATIONS:\nThe authors acknowledge some limitations of their work, such as conducting three iterations of CSR. While this approach shows promise, its technical novelty compared to existing methods is somewhat limited. \n\nAdditionally, the performance improvements are not uniformly substantial across all benchmarks. The paper could benefit from a more detailed discussion of these limitations and potential strategies for addressing them in future work. \n\nFurthermore, the potential negative societal impact of the work should be considered, particularly in terms of the ethical implications of improving LVLMs that might be used in sensitive applications. Providing constructive suggestions for mitigating any negative impacts would be valuable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper proposes a new method for preference alignment with LVLMs. Specifically, the reward is computed using its own LLM (text only) and an external CLIP model. The optimization is done with DPO. This process can be iterated for several times.

STRENGTHS:
S1. Preference optimization in LVLM is under explored compared to those pure language ones. This submission proposed an effective approach.

S2. Both theoretical and empirical contribution.

WEAKNESSES:
I don't find significant weaknesses. However, I do think the experiments can be improved. Therefore, I give a conservative score of borderline accept. I may consider increasing the rating if more thorough studies are conducted (see next section).

QUESTIONS:
Q1. How did the performance improve over each preference data generation stage?

Q2. Scaling study as in Gao et al. [1]?


[1] Gao et al., "Scaling Laws for Reward Model Overoptimization", ICML 2023

LIMITATIONS:
Limitations and broader impact are discussed in the appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16