Have the VLMs Lost Confidence? A Study of Sycophancy in VLMs

Authors: Shuo Li, Tao Ji, Xiaoran Fan, Linsheng Lu, Leyi Yang, Yuming Yang, Zhiheng Xi, Rui Zheng, Yuran Wang, xh.zhao, Tao Gui, Qi Zhang, Xuanjing Huang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Multi-modal Model, Visual-Language Model, Sycophancy, Hallucination

Abstract:
In the study of LLMs, sycophancy represents a prevalent hallucination that poses significant challenges to these models. Specifically, LLMs often fail to adhere to original correct responses, instead blindly agreeing with users' opinions, even when those opinions are incorrect or malicious. However, research on sycophancy in visual language models (VLMs) has been scarce. In this work, we extend the exploration of sycophancy from LLMs to VLMs, introducing the MM-SY benchmark to evaluate this phenomenon. We present evaluation results from multiple representative models, addressing the gap in sycophancy research for VLMs. To mitigate sycophancy, we propose a synthetic dataset for training and employ methods based on prompts, supervised fine-tuning, and DPO. Our experiments demonstrate that these methods effectively alleviate sycophancy in VLMs. Additionally, we probe VLMs to assess the semantic impact of sycophancy and analyze the attention distribution of visual tokens. Our findings indicate that the ability to prevent sycophancy is predominantly observed in higher layers of the model. The lack of attention to image knowledge in these higher layers may contribute to sycophancy, and enhancing image attention at high layers proves beneficial in mitigating this issue.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730453324480

thread_type:
official_review

<thread_object>
creator: reviewer_mdmb

<content>
summary:
This paper focuses on the study of sycophancy, a prevalent hallucination issue in Vision-Language Models (VLM). Firstly, a benchmark named MM-SY is introduced to evaluate the severity of sycophancy in VLMs. Subsequently, three methods—prompt guidance, Supervised Fine-Tuning (SFT), and Direct Policy Optimization (DPO)—are explored to mitigate sycophancy. Finally, the author analyzes the impact of attention weights on the sycophancy problem through experiments and proposes a simple, training-free method to alleviate this issue.

strengths:
1. This paper is well written, clearly articulating the progressively detailed research approach to the sycophancy issue in VLMs.

2. Experiments are comprehensive, thoroughly testing multiple VLM models, various tasks, and different user preferences, and analyzing the relationship between sycophancy and various dimensions.

3. By studying the attention weights at different layers, this work reveals the model's performance in mitigating the sycophancy problem.

weaknesses:
1. This paper mentions the contradiction between sycophancy and stubbornness issues, so for the VLM model, the real problem that needs to be addressed is to reduce sycophancy while maintaining the acceptance of correct opinions. However, methods such as prompt guidance, DPO, and amplify attention seem to reduce sycophancy but at the same time increase stubbornness to an equal extent. This does not truly solve the problem. It is merely shifting the imbalance from one side of the seesaw to the other. Only the SFT method shows a lower increase in stubbornness compared to the alleviation of flattery, but the paper does not provide a thorough analysis of this point.

2. This paper identifies the impact of amplifying high layer attention on the sycophancy problem but does not propose effective solutions based on this finding to truly address both sycophancy and stubbornness issues.

questions:
As in Weaknesses, why does SFT perform better than other methods? Does high layer attention help in truly addressing the issues of flattery and stubbornness?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687834021

thread_type:
official_review

<thread_object>
creator: reviewer_eesv

<content>
summary:
The paper "Have the Vision-Language Models Lost Confidence? A Study of Sycophancy in VLMs" introduces the concept of sycophancy in vision-language models (VLMs), where models blindly agree with user inputs despite contradictory visual evidence. The authors present the MM-SY benchmark, the first evaluation benchmark for sycophancy in VLMs across ten visual understanding tasks. They find that VLMs exhibit significant sycophantic behavior, influenced by factors like task type, user tone, and model size. To address this, the paper explores three mitigation methods: prompt-based, supervised fine-tuning, and direct preference optimization, showing progressive improvements in reducing sycophancy. However, these methods also make VLMs more resistant to corrections. The authors propose a training-free approach by amplifying high-layer vision attention, which effectively mitigates sycophancy without compromising the model's receptiveness to corrections.

strengths:
1. The paper offers a thorough analysis of the factors influencing sycophancy in VLMs, providing valuable insights into model behavior across different conditions.

2. The exploration of three distinct mitigation methods, each with varying degrees of success, contributes to the understanding of how to manage sycophantic behavior in VLMs.

3. The proposal of a simple, training-free method to reduce sycophancy by amplifying high-layer vision attention is innovative and has practical implications for model development.

weaknesses:
1. The mitigation methods were only validated on a single VLM (LLaVA-1.5-7B), which limits the generalizability of the findings. It's unclear how these methods would perform across different VLM architectures.

2. The paper mentions that due to time and computational resource constraints, the analysis was limited. This suggests that the findings may not be exhaustive and could benefit from further exploration with additional resources.

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702672342

thread_type:
official_review

<thread_object>
creator: reviewer_cv64

<content>
summary:
This paper investigates the sycophancy problem in VLMs, which is also a common hallucination issue in LLMs. The authors first design an evaluation benchmark along with 10 visual question answering (VQA) tasks to assess the sycophancy problem in popular VLMs. They then propose three methods from the perspective of prompt engineering, supervised fine-tuning, and direct preference optimization to mitigate this issue.

strengths:
+ This is the first paper to investigate the hallucination problem in multi-modality language models. To address this issue, the authors construct a new evaluation benchmark that includes 10 different visual question answering (VQA) tasks.

+ Based on the designed benchmark, the authors investigate this problem on various popular VLMs and provide comprehensive experimental results.

+ Besides revealing the sycophancy phenomenon, the authors also provide three different kinds of solution to alleviate this hallucination problem.

weaknesses:
+ It seems that the definition of sycophancy rate is missing. Could the authors present it in Section 2? This is important for the readers to understand Table 1 and Figure 2.

+ In addition to revealing the sycophancy phenomenon, it would be beneficial to analyze why the current model tends to exhibit sycophancy. For example, is this comes from the training data or the network architecture?"

questions:
+ From Table 1, it seems the sycophancy rate is not correlated to the designed types of tones. Could the authors provide the analysis for this?
+ In addition to the sycophancy rate, it would be beneficial to also present the baseline accuracy. It is interesting to see if the model's sycophancy rate related to its performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732619391729

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comment

comment:
We would like to thank the reviewers for their efforts in assessing our work and the valuable feedback. We have accordingly made significant improvements following the suggestions and comments. For convenience, *we have marked in the revised PDF in blue* the added sections and the added results in tables. Light blue prefixes indicate the Reviewer ID and question ID for easy reference (it will not appear in the next version).

> R1 corresponds to Reviewer CV64.
>
> R2 corresponds to Reviewer eeSv.
>
> R3 corresponds to Reviewer MDMB.

Below we summarise our changes:  
- **Formal definition of sycophancy:** Following Reviewer CV64's request, we have added this in Appendix A.2.  
- **Discussion on potential causes of sycophancy:** Following Reviewer CV64's request, we have provided a discussion in Appendix A.5, covering aspects such as whether the LLM underwent RLHF training, image resolution, the use of image-text interleaved data, and multimodal SFT training.  
- **Relationship between sycophancy and tone:** Following Reviewer CV64's request, we have emphasized this in Section 2.2 RQ2.  
- **Relationship between sycophancy and baseline performance:** Following the requests from Reviewers CV64 and eeSv, we have added analyses in Appendix A.6, including:  
  1. A comparison of sycophancy and baseline performance across multiple VLMs.  
  2. An analysis of baseline performance for the same VLM (LLaVA-1.5) under different sycophancy mitigation methods.  
- **Sycophancy mitigation experiments for InternVL-1.5-26B:** Following Reviewer eeSv's request, we have included InternVL-1.5-26B in the main results in Table 2.  
- **New setup for correction experiments:** Following Reviewer MDMB's request, we have expanded the correction experiments to include a new setup where prompts do not contain answers (only indicating that the first-round answer is incorrect). This effectively differentiates the correction ability derived from *sycophancy* and the *helpfulness* of the VLM.  
- **New correction experiments for the Amplified Image Attention method across three VLMs (Cor w/o Answer):** Following Reviewer MDMB's request, we have extended the correction experiments for the proposed Amplified Image Attention method. The results demonstrate that this method can mitigate sycophancy and slightly enhance correction ability—or, more accurately, the "helpful ability" of the VLM—in a training-free manner, validating our hypothesis that "high-level attention lacks focus on image tokens."

We hope that our revisions have strengthened our contributions and would like to thank the reviewers for their valuable suggestions. We look forward to productive rebuttal discussions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates the hallucination problem, specifically sycophancy, in multi-modality language models (VLMs) by constructing a new benchmark for evaluating 10 different VQA tasks. The paper provides comprehensive experiments across popular VLMs, offering valuable insights into sycophantic behavior and the factors influencing it. It proposes three mitigation methods, with varying success, and introduces a novel, training-free approach to reduce sycophancy by amplifying high-layer vision attention, demonstrating both theoretical and practical contributions.

All reviewers recognize the contributions of the paper, with feedback generally leaning towards acceptance. The AC thoroughly reviewed the paper and rebuttal, agreeing with the consensus recommendation for acceptance due to the consistency of the feedback.