Title: MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimodal learning

hallucinations

evaluation
Abstract: Multimodal hallucination in multimodal large language models (MLLMs) restricts the correctness of MLLMs. However, multimodal hallucinations are multi-sourced and arise from diverse causes. Existing benchmarks fail to adequately distinguish between perception-induced hallucinations and reasoning-induced hallucinations. This failure constitutes a significant issue and hinders the diagnosis of multimodal reasoning failures within MLLMs. To address this, we propose the MIRAGE benchmark, which isolates reasoning hallucinations by constructing questions where input images are correctly perceived by MLLMs yet reasoning errors persist. MIRAGE introduces multi-granular evaluation metrics: accuracy, factuality, and LLMs hallucination score for hallucination quantification. Our analysis reveals strong correlations between question types and specific hallucination patterns, particularly systematic failures of MLLMs in spatial reasoning involving complex relationships (\emph{e.g.}, complex geometric patterns across images). This highlights a critical limitation in the reasoning capabilities of current MLLMs and provides targeted insights for hallucination mitigation on specific types. To address these challenges, we propose Logos, a method that combines curriculum reinforcement fine-tuning to encourage models to generate logic-consistent reasoning chains by stepwise reducing learning difficulty, and collaborative hint inference to reduce reasoning complexity. Logos establishes a baseline on MIRAGE, and reduces the logical hallucinations in original base models. Link: \url{https://bit.ly/25mirage}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MIRAGE, a benchmark designed to isolate and evaluate reasoning-induced hallucinations in multimodal large language models (MLLMs), distinguishing them from perception errors—a gap overlooked by existing benchmarks. MIRAGE uses multi-granular metrics (accuracy, factuality, and hallucination score) to assess reasoning failures, revealing that MLLMs struggle particularly with spatial reasoning involving complex visual relationships. To address these issues, the authors propose Logos, a method combining curriculum reinforcement fine-tuning and collaborative hint inference to improve logical consistency in model outputs. Logos reduces hallucinations and sets a baseline on MIRAGE, which will be publicly released.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is the first to cleanly disentangle reasoning-induced hallucinations from the perception errors that dominate prior multimodal benchmarks, motivating a fresh study of logical robustness in MLLMs.\n2. The proposed Logos framework lifts a 7 B Qwen2.5-VL model to nearly match much larger 72 B systems. Ablations show each component's contribution, offering a concrete starting point for future work.\n3. MIRAGE offers 1329 carefully curated questions covering seven reasoning taxonomies and provides three complementary metrics, making it a well-rounded benchmark.\n\nWeaknesses:\n1. The dataset contains only single-image questions; no multi-image, video or temporal reasoning is tested, a limitation the authors acknowledge themselves.\n2. Factuality, hallucination typing, and LHS all hinge on other large language models' judgments, raising concerns about circularity, bias, and stability across evaluator versions.\n3. The authors do not provide the rationale behind picking the seven distinct taxonomies for their dataset or mention which model is fine-tuned for their Logos method.\n\nQUESTIONS:\nQuestions:\n1. The paper states that the dataset was \"constructed through rigorous selection of seven distinct taxonomies, including geometry, algebraic, arithmetic, scientific, spatial reasoning, and statistical reasoning,\" but never explains why exactly these seven lines were chosen.\n2. Section 5 introduces Logos but does not specify which exact model it fine-tunes. The authors should provide the missing information on Logos's reproducibility.\n3. The authors should expand on the reliability of LLM-as-a-Judge.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MIRAGE, a new benchmark designed to assess and diagnose reasoning-induced hallucinations in Multimodal Large Language Models (MLLMs). Unlike existing benchmarks, MIRAGE focuses on scenarios where MLLMs correctly perceive visual inputs but fail in their logical reasoning.\nThe contribution of the paper include:\n- MIRAGE Benchmark: a diagnostic benchmark comprising 1,329 carefully designed questions aimed at isolating reasoning hallucinations. It provides multi-level annotations—including final answers, intermediate reasoning steps, and ground-truth reasoning chains—which enable detailed tracking of hallucination propagation. In addition to accuracy and factuality metrics, MIRAGE introduces a comprehensive LLM Hallucination Score (LHS) to assess model performance.\n- Empirical Analysis and Insights: The paper presents an extensive analysis revealing that model scale, data scale, and training stages impact logical, fabrication, and factual hallucinations. It highlights that current MLLMs show limited improvement in spatial hallucinations, indicating weak visual reasoning capabilities. Additionally, it identifies correlations between question types and distinct hallucination patterns.\n- Hallucination Mitigation Method: To address the identified challenges, the authors propose Logos, a baseline method that combines curriculum reinforcement fine-tuning (CRFT) and collaborative hint inference (CHI). Logos is shown to reduce logical hallucinations and improve answer accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper is well-structured and easy to follow. The problem statement is clearly articulated, and the motivation for MIRAGE is well-justified by highlighting the limitations of existing benchmarks.\n- MIRAGE introduces three levels of evaluation metrics: accuracy (overall answer correctness), factuality (correctness of intermediate steps and claims), and LLMs Hallucination Score (assessment of the entire reasoning chain). This multi-level approach allows for a more precise tracking and diagnosis of hallucination propagation.\n- The paper proposes a cost-effective automated annotation framework combined with human-AI collaborative verification for creating ground-truth reasoning chains. This approach addresses the challenge of obtaining detailed annotations for complex reasoning tasks, which could be followed for future works.\n- The paper conducts thorough experiments on MIRAGE, providing valuable insights into the factors influencing different hallucination types, such as model scale, data scale, and training stages.\n- The authors state that MIRAGE will be publicly available , and they provide detailed experimental setups, including prompts used for construction and evaluation in the appendix.\n\n### Weaknesses\n- The paper could provide a more detailed breakdown of the initial data sources used for MIRAGE, beyond \"publicly available datasets and questions from Internet\", and elaborate on the inherent difficulty of this raw data. While some examples are provided (Figures 12-18), a more explicit discussion on data characteristics and how they contribute to reasoning difficulty would be beneficial.\n- The paper explicitly states that Logos does not significantly reduce spatial or factuality hallucinations, only logical and fabrication hallucinations. This suggests a limitation in the proposed mitigation method's effectiveness across all identified hallucination types.\n- While Logos shows promising results on MIRAGE and MathVista, its performance on a broader range of multimodal reasoning tasks and datasets not specifically designed for reasoning hallucinations could be further explored to establish its wider applicability.\n- While Logos's combination of techniques is novel, the individual components (GRPO, difficulty curriculum learning, Online Reward Filtration) are established concepts. Some recent works has already used these components in LLM/MLLM reasoning tasks, such as \"DAPO: An Open-Source LLM Reinforcement Learning System at Scale\" and \"Efficient Reinforcement Finetuning via Adaptive Curriculum Learning\n\". \n- Although the ablation experiments in appendix already point out the effectiveness of single component, the paper could strengthen its originality claim by further highlighting how the combination and synergy of these components specifically address the unique challenges of reasoning hallucinations in MLLMs beyond what individual components or simpler combinations might achieve.\n\nQUESTIONS:\n1. Could the authors provide a more detailed breakdown of the \"publicly available datasets and questions from Internet\"  that constitute the initial 18K dataset for MIRAGE? Specifically, listing the names of these public datasets would enhance clarity and allow for better understanding of the data's diversity and characteristics.\n2. The paper mentions using an LLM to extract intermediate steps and claims, and comparing them with ground-truth steps and claims for factuality evaluation. What measures are in place to ensure that this evaluation accurately assesses factuality, especially for questions where multiple valid reasoning paths could exist? If a model generates a logically sound but \ndifferent sequence of intermediate steps than the predefined ground truth, how does the current evaluation methodology account for this to avoid penalizing correct alternative solutions as \"hallucinations\" or \"incorrect\" steps? Is there any manual double-checking of the step and claim factuality evaluations to mitigate such issues? \n3. Could the authors provide a more in-depth analysis of Logos's performance in mitigating spatial hallucinations compared to train-free methods like self-reflection and SFT-based methods? Specifically, what are the observed differences in their effectiveness (or lack thereof) for spatial hallucinations, and what specific characteristics of spatial reasoning errors might explain why Logos performs similarly to or differently from these other approaches in this particular area?\n\nLIMITATIONS:\nAlthough the authors discuss some limitations in the appendix, Logos mainly combines existing techniques but lacks a clear analysis of why this combination should work in practice.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
In this paper, the authors proposed MIRAGE benchmark to isolate reasoning hallucinations by questions where inputs are correctly
perceived but reasoning errors persist. The authors also proposed Logos to mitigate these hallucination. Multiple experiments were conducted to justify the value of both the proposed benchmark and mitigation method.

STRENGTHS AND WEAKNESSES:
***Strengths***
1. The proposed MIRAGE benchmark provide a more fine-grained measure of reasoning-induced hallucinations in multimodal hallucination by ruling out perception-induced hallucinations. It provides a more precise measure of MLLMs' capability.
2. Both the data collection process of MIRAGE and the proposed Logos for hallucination mitigation makes sense and are technically sound to me.
3. Extensive experiments and analysis were conducted to justify the proposed benchmark and approach.
4. Writing is good and easy to follow. 

*** Weaknesses***
1. For Table 6, any idea why several of the metrics got worse results for row "+hquestio"?
2. Typo in Line 298: "benefiting" should be "Benefiting".

QUESTIONS:
Please refer to the weakness section and provide more discussions on the performance degradation.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
This paper introduces MIRAGE, the first benchmark specifically designed to evaluate multimodal reasoning hallucinations in Multimodal Large Language Models (MLLMs). By ensuring correct perception inputs, MIRAGE isolates reasoning errors, offering a clearer view into model failures. It features multi-level evaluation metrics, including accuracy, factuality, and a novel LLM hallucination score for comprehensive assessment. The authors conduct an in-depth analysis of hallucination patterns across question types, identifying spatial reasoning as a major weakness in current MLLMs. To address this, they propose Logos, a baseline method that integrates curriculum-based reinforcement fine-tuning and collaborative hint inference to promote logically consistent reasoning. Logos demonstrates reduced hallucinations and improved answer accuracy, indicating its potential for guiding future improvements.

STRENGTHS AND WEAKNESSES:
Strength:
1. dataset contribution with detailed reasoning chain annotation. The effort of benchmarking many existing methods is impressive.
2. The proposed Logos improved the baseline Qwen 2.5-VL.

Weakness:
1. The number of data is very limited. 
2. There are plenty of datasets that already has reasoning chain, like ScienceQA, mathVista, etc. Table 1 is not accurate, or please provide some more explations. 
3. The contribution is limited other than the dataset.
4. The author mentioned hallucination types, but not properly defined.

QUESTIONS:
Please refer to the weakness section.

LIMITATIONS:
Not addressed.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-06