Title: More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reasoning model

Multimodality

Hallucination
Abstract: Test-time compute has empowered multimodal large language models to generate extended reasoning chains, yielding strong performance on tasks such as multimodal math reasoning. However, we observe that this improved reasoning ability often comes with increased hallucination: as generations become longer, models tend to drift away from image-grounded content and rely more on language priors. Attention analysis reveals that longer reasoning chains reduce focus on visual inputs, contributing to hallucination. To systematically study this phenomenon, we introduce RH-AUC, a metric that quantifies how a model's perception accuracy changes with reasoning length, enabling evaluation of whether the model preserves visual grounding while reasoning. We also release RH-Bench, a diagnostic benchmark covering diverse multimodal tasks, designed to jointly assess the balance of reasoning ability and hallucination. We find that (i) larger models generally exhibit a better balance between reasoning and perception; (ii) reasoning and perception balance depends more on the types and domains of the training data than its volume. Our findings highlight the need for evaluation frameworks that account for both reasoning quality and perceptual reliability.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the trade-off between reasoning and visual grounding in multimodal large language models (MLLMs), showing that while extended reasoning chains enhance task performance, they often amplify visual hallucinations due to reduced attention to image inputs and increased reliance on language priors. To quantify this trade-off, the authors introduce RH-AUC, a novel metric that captures the balance between reasoning accuracy and hallucination across varying reasoning lengths. They also release RH-Bench, a diagnostic benchmark comprising 1,000 reasoning and perception tasks to evaluate model performance holistically. Using attention analysis, the study finds that hallucination increases with longer reasoning chains, but the relationship is non-monotonic and task-dependent. Moreover, models trained with reinforcement learning (RL-only) outperform those using supervised fine-tuning plus RL (SFT+RL) in maintaining this balance, and the quality and domain of training data are more influential than data volume in achieving robust visual grounding.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths \n\n1- Strong and Actionable Conclusions: The paper draws clear conclusions from its analyses, showing that longer reasoning can lead to more hallucination and that the best reasoning length depends on the task. These findings offer useful guidance for designing and fine-tuning multimodal reasoning models.\n\n2- Strong Benchmark and Metric: The authors introduce RH-Bench and RH-AUC, which together offer a solid and practical way to measure how well models balance reasoning with staying grounded in the image.\n\n3- Clear and Easy to Follow: The paper is well-organized and clearly written, with helpful figures and examples that make even complex ideas, like reasoning length control, easy to understand.\n\n### Weaknesses \n\n1- Narrow Choice of Non-Reasoning Baselines: The paper primarily uses Qwen2.5-VL as the non-reasoning baseline across all comparisons, limiting the diversity of grounding-focused models in the study. This makes it unclear whether the observed hallucination trends generalize across other non-reasoning MLLMs such as LLaVA, or MiniGPT-4, which may have different visual grounding behaviors and could serve as stronger or more varied baselines.\n\nQUESTIONS:\n1- Could the authors elaborate on the specific aspects of the R1-OneVision training paradigm that may limit its ability to balance reasoning and perception, despite its access to extensive visual perception data? For example, are there architectural constraints, insufficient multi-modal supervision signals, or imbalanced task objectives contributing to this outcome?\n\n2- Given that RH-Bench includes diverse tasks beyond mathematics, could the authors clarify whether the observed RH-AUC gains for math-trained models (e.g., MM-Eureka, MM-R1) reflect general improvements in perception-reasoning balance, or are primarily driven by performance on math-like subsets of the benchmark? Would domain-wise breakdowns of RH-AUC help clarify this?\n\nLIMITATIONS:\nThe paper makes a strong and valuable contribution by introducing a benchmark and analysis framework for evaluating perception-reasoning balance in multi-modal models. The overall framing is thoughtful and well-executed. However, some limitations could be more clearly acknowledged. For instance, the choice of non-reasoning baseline is somewhat narrow, which may affect the generalizability of certain claims about grounding behavior. Additionally, a few interpretive points—such as how model training paradigms affect performance or how domain-specific data impacts general reasoning—would benefit from further clarification. These are not critical weaknesses, but addressing them would improve the transparency and depth of the work. Overall, the paper is solid and deserves acceptance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the hallucination observed in recent reasoning VLMs. The authors begin with a preliminary analysis showing that compared to the original VLMs, reasoning VLMs with reasoning chains tend to produce more perception-related hallucinations. Through attention-based analysis, they identify that reasoning models tend to reduce visual attention over time, which leads to the rise in hallucination. Further investigation reveals a non-monotonic relationship between reasoning length and  reasoning-perception performance.\n\nTo quantify the trade-off between reasoning and hallucination in VLMs, the authors propose: (1) RH-AUC, a metric that measures the perception accuracy as reasoning length varies; and (2) RH-Bench, a diagnostic benchmark designed to evaluate both reasoning and hallucination across diverse multimodal tasks. Experimental results show that larger models generally achieve a better balance between reasoning and hallucination, and that the types and domains of training data significantly influence the performance.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengthes\n- This paper addresses a timely and important problem concerning hallucination in recent reasoning VLMs. The observations and conclusions presented may offer insights for future research in the field. The overall motivation and logical flow of the paper are clear and well-structured.\n- The introduction of RH-AUC and RH-Bench allows for a nuanced quantification of the trade-off between reasoning and visual grounding, enabling a more comprehensive evaluation of recent VLMs on this issue.\n- The paper conducts comparisons across multiple recent models with varying sizes (e.g., 3B and 7B) and training strategies (e.g., SFT and RL), offering insights that could benefit the development of future models.\n\n## Weaknesses\n\n- Some of the conclusions and analyses regarding data domains, training paradigms, and model performance are primarily observational, based on off-the-shelf models, without the support of rigorously controlled experiments (e.g., training models with varied data or training strategies). As a result, certain claims—such as “RL only outperforms SFT+RL” or “data domain matters more than data volume”—may be less convincing.\n- While RH-AUC enables detailed measurement of performance across reasoning chain lengths, it comes with high computational costs, potentially limiting its practical usability. Moreover, enforcing strict control over reasoning length can be rigid, especially given that current reasoning VLMs are not trained with explicitly controllable reasoning depth. For example, a model might naturally prefer shorter reasoning chains to achieve optimal answers. A potentially more realistic evaluation would allow models to generate multiple diverse reasoning chains per question, assessing performance without prescribing specific lengths.\n- A related concern is that more complex questions often elicit longer reasoning chains. However, RH-Bench does not appear to incorporate this factor into its design, which may limit its effectiveness in accurately evaluating model performance across varying question difficulties.\n- The current analysis is primarily conducted on models from the Qwen2.5-VL (as noted in the limitations), which may constrain the generalizability of the findings to broader classes of VLMs.\n\nQUESTIONS:\n## Questions \n\nIn addition to the concerns raised in the weaknesses section:\n- Have you considered incorporating larger VLMs (e.g., GPT-4) into your experiments to assess how their accuracy varies with different reasoning chain lengths? This could provide a broader perspective on the reasoning–hallucination trade-off.\n- In the results presented in Section 5, which length-control methods (e.g., Token Budget Forcing, Test-Time Scaling, Latent State Steering) were used to generate reasoning chains of different lengths? Does the choice of length-control method impact model performance on RH-AUC? Is it possible that a model achieves a low RH-AUC due to the control method rather than the model’s actual capabilities?\n\n## Writing\n- Line 67: missing space after the comma.\n\n\n*Note: I am currently leaning toward a borderline rejection. However, I am open to increasing my score if the concerns outlined above are adequately addressed during the rebuttal phase.*\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the phenomenon that multimodal reasoning models tend to exhibit more hallucinations in perception tasks compared to non-reasoning models. Through empirical studies, they attribute this to reasoning models relying more on language priors and paying less attention to visual input, which causes hallucinations. Furthermore, the paper finds that the optimal reasoning intervals for minimizing hallucinations differs from that for maximizing reasoning performance, and existing metrics fail to capture the balance between hallucination and reasoning ability. To better evaluate this trade-off, the authors propose the RH-AUC metric and the RH-Bench dataset, providing insights based on results from multiple reasoning models.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The paper focuses on an important issue that multimodal reasoning models are more prone to hallucination. Through a systematic analysis across a range of advanced models, it offers valuable insights, including that models trained with SFT followed by RL are most susceptible to hallucination.\n2. The evaluation of multimodal reasoning models should account for both reasoning ability and hallucination severity. This work introduces a metric designed to reflect both factors.\n3. The paper is well-written and clearly structured, making it easy to follow.\n\n### Weaknesses\n1. **Limited Novelty in Attribution Analysis.** Prior work, such as [1, 2], has already highlighted the over-reliance on language priors as a key contributor to hallucinations. The observations in this work—such as weak visual attention and declining visual focus over longer reasoning chains—seem to be natural extensions of these findings rather than fundamentally new insights.\n2. **Lack of a Technical Contribution.** While the paper raises an important and timely question, it does not propose a concrete technical method or solution, which constrains its contribution.\n3. **Limited Empirical Support for Training Paradigms Claims.** The paper’s claim that SFT leads to rigid imitation reasoning and redundant steps would benefit from more direct empirical validation.\n4. **Concerns About the Methodological Soundness of Latent State Steering.** Since the outputs are generated from an intervened latent state rather than the model’s natural reasoning process, it remains unclear how well they reflect the model’s true capabilities. This raises questions about the validity of key analyses (Figure 6 and 7) and the RH-AUC metric that rely on this technique.\n5. **Interpretability and Effectiveness of the RH-AUC Metric.** The RH-AUC metric correlates strongly with both perception and reasoning performance. This suggests it may reflect overall performance rather than specifically measuring the balance between reasoning and perception. More analysis is needed to demonstrate its advantage over using perception and reasoning accuracy directly.\n\n[1] Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding\n[2] Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs\n\nQUESTIONS:\n1. I recommend a deeper discussion on the soundness of the Latent State Steering operation.\n2. It would be beneficial to include additional empirical analysis of the training paradigms to provide more valuable insights into the training of multimodal reasoning models.\n3. A clearer explanation of the RH-AUC metric, along with a comprehensive analysis of its advantages, would greatly enhance the paper.\n4. The results presented in Figure 6a of the appendix appear somewhat counterintuitive: the reasoning model exhibits fewer perception errors but more reasoning errors compared to the non-reasoning model, which seems to contradict the observations highlighted in the main text. Could a plausible explanation for this discrepancy be provided?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a systematic analysis of the phenomenon that as multimodal reasoning models enhance their reasoning abilities, their visual grounding capabilities deteriorate and hallucination issues become more pronounced. By comparing models of different scales and training paradigms, the authors show that longer reasoning chains make models increasingly rely on language priors at the cost of visual evidence, leading to higher hallucination rates. To address this, the paper proposes the RH-AUC metric and the RH-Bench diagnostic set to systematically quantify the trade-off between reasoning and hallucination, and provides an in-depth analysis of how various factors influence this balance.\n\nSTRENGTHS AND WEAKNESSES:\n***Strengths:***\n1. The paper introduces a novel metric (RH-AUC) and a comprehensive benchmark (RH-Bench) for systematically quantifying the trade-off between reasoning and hallucination in multimodal reasoning models.\n2. Through extensive experiments and attention analysis, the study reveals and substantiates the important phenomenon that enhancing reasoning abilities in MLLMs can increase visual hallucinations, especially with longer reasoning chains.\n3. The paper evaluates different training paradigms and model scales, demonstrating that the reasoning-hallucination trade-off is a general phenomenon, and provides useful insights for future model development.\n\n***Weaknesses:***\n1. The size of the RH-Bench benchmark is limited, which raises concerns about whether it is sufficient for a comprehensive evaluation of reasoning-hallucination trade-offs across diverse tasks and settings.\n2. The experiments are limited to a single model series (mainly Qwen2.5-VL), with little evaluation on other mainstream multimodal model families.\n3. The study does not include a sufficient range of model sizes and architectures, making it difficult to assess the generality of the conclusions across diverse model types.\n4. The paper does not provide a more detailed analysis of hallucination types or investigate whether different categories of hallucination are related to specific data characteristics, which could potentially lead to a deeper understanding of the problem.\n\nQUESTIONS:\nThe conclusions drawn in this paper are meaningful, but the current experiments lack evaluation across more model series and sizes (see W2 & W3), raising concerns about the reliability and generalizability of the findings. Additionally, if the authors could further explore different RL methods and examine whether specific types of hallucination are linked to particular data or tasks (e.g., whether hallucinations are more common in counting scenarios), it would significantly enhance the value of this work (W4).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}