MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM
Authors: Bowen Dong
Minheng Ni
Zitong Huang
Guanglei Yang
Wangmeng Zuo
Lei Zhang
Venue: NeurIPS 2025 poster ()
Keywords: multimodal learning
hallucinations
evaluation
Primary area: evaluation

Abstract: Multimodal hallucination in multimodal large language models (MLLMs) restricts the correctness of MLLMs. However, multimodal hallucinations are multi-sourced and arise from diverse causes. Existing benchmarks fail to adequately distinguish between perception-induced hallucinations and reasoning-induced hallucinations. This failure constitutes a significant issue and hinders the diagnosis of multimodal reasoning failures within MLLMs. To address this, we propose the MIRAGE benchmark, which isolates reasoning hallucinations by constructing questions where input images are correctly perceived by MLLMs yet reasoning errors persist. MIRAGE introduces multi-granular evaluation metrics: accuracy, factuality, and LLMs hallucination score for hallucination quantification. Our analysis reveals strong correlations between question types and specific hallucination patterns, particularly systematic failures of MLLMs in spatial reasoning involving complex relationships (\emph{e.g.}, complex geometric patterns across images). This highlights a critical limitation in the reasoning capabilities of current MLLMs and provides targeted insights for hallucination mitigation on specific types. To address these challenges, we propose Logos, a method that combines curriculum reinforcement fine-tuning to encourage models to generate logic-consistent reasoning chains by stepwise reducing learning difficulty, and collaborative hint inference to reduce reasoning complexity. Logos establishes a baseline on MIRAGE, and reduces the logical hallucinations in original base models. Link: \url{https://bit.ly/25mirage}.

MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MILM
Bowen Dongl2 12 Minheng Ni Zitong Huang' Guanglei Yang Wangmeng Zuo' Lei Zhang²
Harbin Institute of Technology
The Hong Kong Polytechnic University
cndongsky @ com kodenii @outlook.com
deehhag@comp.polyu.edeee Sangguanglei@hit.eddu.
Abstract
Multimodal hallucination in multimodal large language models (MLLMs) restricts the correctness of MLLMs. However, multimodal hallucinations are multi-sourced and arise from diverse causes. Existing benchmarks fail to adequately distinguish between cerception-induced hallucinations and reasoning-induee hallucinations. This failure constitutes significant issue and hinders the diagnosis of multimodal reasoning failures within MILMs. To address this, we propose the MIRAGE benchmark, which isolates reasoning hallucinations by constructing questions where input images are correctly perceived by MILLMs yet reasoning errors persist. MIRAGE introduces multi-ga gulli-granular evaluation metrics: accuracy, factuality, and LLMs hallucination score for hallucination quantification. Our analysis reveals that (1) the model scale, data scale, and training stages significantly affect the degree of logical, fabrication, and factual hallucinations; (2) current MLLMS show no effective improvement spatial hallucinations caused by misinterpreted spatial relationships, indicating their limited visual reasoning capabilities; and (3) question types correlate with distinct hallucination patterns, highlighting targeted challenges and potential mitigation strategies. To address these challenges, we propose Logos, method that combines curriculum reinforcement fine-tuning to encourage models to generate logic-consistent reasoning chains by stepwise reducing learning difficulty, and collaborative hint inference reduce reasoning complexity. Logos establishes a baseline on MIRAGE, and reduces the logical hallucinations in original base models. Link: https://bit 25mmria
Introduction
Multimodal large language models (MLLMs) 23 56 65
achieve advancements in multimodal Recent studies
perception [38] 5 61]. as evidenced by standard benchmarks 660
14
45
42
44
further enhance their reasoning capacities through post-trainin traininiiinggg (57 12
24
62
86). However, two
critical challenges remain, i.e., erroncous visual perception that fabricates non non-existent content, and defective logical reasoning yielding inconsistent conclusions. These multi-source hallucinations (stemming from distinct perceptual and cognitive origins) fundamentally limit the practical utility.
To quantitatively measure hallucination in MLLMs, several multimodal benchmarks have been applied to detect and measure multimodal hallucination in object recognition [14 34 59
20
30) or
academic reasoning [41 aspects. Existing benchmarks [14 59 20 30 attempt measure hallucinations via object recognition or academic tasks. However, two critical gaps remain. First, cur- rent evaluations fail to distinguish between different types of hallucinations, i.e., perception-induced hallucinations stem-
hallucinations caused by inaccurate visual understanding and reasoning-inducee ming from logical flaws, making it difficult to pinpoint errors. Second, most benchmarks focus validating the content of answers or intermediate steps, while lacking fine-grained evaluation of the reasoning process in terms of perception and logic, thereby hindering the ability to trace error
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(6) Dataset Distribution
Algebraic
Geometry
Arithmetic
RP
Num mwayy solutiour hus the equation
If SB Clu thes wkke
point. Hos many
Each piecs cove
129
315
SopMixily 05
angle 54 CS?
compleii
Scientific
Logical
Spatial
Statistical
WAlic
opttoon
Joos mach wore
285
descestles originn of 18fe
conffornc
accurati the
ghves
arrunate
150
accordlig the datt??
pattera
Whick the follon
changes is
foreust by felling
(b) Data Structure
Ground truth Intermediate Results
Aunlaay Information Description
Quessiio
Reasoning Chaina
FinaLAnswer
NTSBC-ADE ADS S4DC3?
bew angor ACU
Since BC AD
Sueu:
Cleter:
The evoer 6Pcac
angler Aabels), Annle
at gronueyy proflees ansoorale stans and armvac dowo
difice AC
aand degle D
Hallucination Type Spatial Hallucination
Logiosi Hallucination
Factual Hallucination
Contect Hallucination
Fabrication Hallucination
C
12 12
Fourd transdiernniinn maal: 50% ACHAD, honcz sngle mangle Anglo 11-90 90 730%icm ADH-AD (Bairr ARRwnn Stage
by enemy
Figure 1: Data distribution and structure of MIRAGE benchmark. (a) shows the classes and amounts of questions. (b) shows the data structure of each question in MIRAGE. And (c) shows the multimodal
reasoning hallucination types we explored, which are clearly defined in Appendix A
propagation patterns. This absence of hierarchical analysis spanning answer-level outputs, step-level intermediate results, and thought-level reasoning logic prevents systematic diagnosis of reasoning failures. Addressing these gaps is essential for building trustworthy MLLMs.
To address these challenges, we propose MIRAGE, a diagnostic benchmark specifically designed to isolate reasoning-induced hallucinations in MLLMs. As shown in Fig. [ MIRAGE contains 1,329 questions where MLLMs demonstrate accurate visual perception but exhibit defective reasoning. Each question provides three-tier annotations: final answers, intermediate reasoning steps and claims, and ground-trun reasoning chains, enabling precise tracking of hallucination propagation in multi- modal reasoning processes. To comprehensively assess reasoning hallucination in MLLMs, MIRAGE proposes three evaluation metrics, i.e., accuracy assessment measuring overall answer correctness, factuality assessment verifying the correctness in intermediate steps and claims, and LLMs Halluci- nation Score to assess hallucination from the whole reasoning chain level. By evaluating MLLMS on MIRAGE from different levels, we aim to answer three critical research questions. First, how reasoning hallucinations compromise MLLM robustness and correlate with answer accuracy. Second, whether specific question types induce distinct hallucination patterns unique multimodal reasoning. Finally, the efficacy of current mitigation methods against reasoning-specife ballucinations.
We conduct extensive experiments on MIRAGE, leading to several key insights. First, the model scale, data scale, and training stages of MLLMs significantly influence the severity of logical, fabrication, and factual hallucinations. Second, these factors offer limited improvements in addressing spatial hallucinations, which are primarily caused by misinterpretations of spatial caationnsips-hghhiiiii the limited visual reasoning capabilities of current MILMs and their inability to benefit from straight- forward scaling. Third, we observe strong correlations between question types and specific patterns of reasoning hallucination, underscoring critical challenges and suggesting targeted mitigation strategies. These findings offer valuable guidance for the future development of more reliable and reasoning- aware MLLMs.
Building on the insight that increasing the probability of logic-consistent reasoning chains reduces specific logical hallucinations, we propose Logos, which integrates curriculum reinforcement fine- tuning (CRFT) for training and collaborative hint inference for testing. During training, CRFT with online reward filtration (ORF) gradually increases question difficulty while dynamically selecting high-reward samples, guiding the model toward accurate and logic-consistent reasoning. At the testing stage, collaborative hint inference provides topic- and question-specifie hints from LLMs, reducing reasoning complexity for optimized models. Experiments demonstrate that Logos significantly reduces reasoning hallucinations and achieves strong performance on both MIR AGE and standard benchmarks [4 In conclusion, the contribution of this paper can be summarized as follows:
We propose MIRAGE, the first benchmark for evaluating multimodal reasoning hallucina- tions in MLLMs. It isolates reasoning hallucinations with tasks where inputs are correctly

---

Table 1: Comparison of MIRAGE with existing benchmarks. "MCQ" means multiple-choice questions, "A" means answers, "D" means multimodal input descriptions, 'R" means full reasoning chains, and "S" means intermediate results. means multiple reasoning chains.
Dataset Properties
Hallucination Assessment
Benchmarks
Taxonomy Scale Annotation Intermediate
Steps Chains
Usage
POPE
Object 18K
Object Hallu
MMVP 59
Object 300
A
MCQ
HallusionBench
Object 1,129
X
Illusion
MME 14
Various
2,374
A
General VQA
SHFDBench 60
Obj+Act
19K
A
MCQ
Math Vista
Math 1,000
X
X
Math Reasoning
OmniBench 35
Various 1.142
A/D
MCQ
MME-CoT 27
Various
1,130
A/D/S
Steps
General CoT
MIRAGE (Ours)
Various
1,329
A/D/R' IS Steps+Hints
Reason Hallu
perceived but reasoning errors persist, and introduces multi-level metrics for comprehensive assessment: accuracy, factuality, and LLMs hallucination score.
Our findings reveal that the model scale, data scale, and training stages of MLLMs, and highlight critical challenges and mitigation for specific hallucination types. These findings will provide insights for future MLLM development.
We propose Logos, a baseline method MIRAGE to encourage model for logic-consistent reasoning via curriculum reinforcement fine-tuning and collaborative hint inference. Logos reduces the multimodal logical hallucination and improves the answer accuracy.
2 Related Work
Reasoning Multimodal Large Language Models. Reasoning MLLMs can be roughly divided into three groups. First is the prompt-based reasoning method 62
83 47 to guide MLLMs by in-context which uses searching methods [55] 82
learning [T]. Second is the plan-base based method [69
85
to explore optimal reasoning chains. And last is learning based method by supervised fine-tuning 46). RL methods generalize better by optimizing with their high-reward predictions instead of fixed ground-truths. Hence, we build Logos on RI. for hallucination mitigation, uniquely focusing on dynamic training difficulty adjustment.
(SFT) 7T] or reinforcement learning (RL) [53 39
Multimodal Hallucination Evaluation. Existing MILMs still suffer from multimodal hallucination, where generated text either contradicts the visual input or deviates from correct logical reasoning. To assess the hallucination and its effect in MLLMs, recent works measure the accuracy degradation
20 mathematic [41 60 80 IQ test 7 17 and general multimodal abilities [30 While existing benchmarks have advanced multimodal evaluation, they
among object perception 34 illusion
often conflate perception-ineee -eccuce induced hallucinations with reasoning- iasouingg-nduuedd ones, making it challenging to diagnose reasoning failures. In contrast, MIRAGE focuses on reasoning hallucinations by isolating reasoning errors from correctly perceived inputs. providing multi-level metrics for assessment.
Multimodal Hallucination Mitigation. Hallucination evaluations provide critical insights for future mitigation strategies, which broadly fall into maining-free and training-based approaches. First is training-free free methods [86] 16 73). They use prompting [66], question decomposition (86), reasoning chain reflection [16 225 and ensembling [63] mitigate perception-inducee hallucination. Second
is training- based mitigation methods 43
71 12
70
79 75). These methods mitigate hallucination
by data curation 75 37. SFT
88 77.. Previous hallucination mitigation methods mainly target perception errors in MLLMs, with only few early efforts addressing multi- modal reasoning hallucinations 886 43). Our proposed Logos focuses on reasoning hallucinations and shows that combining CRFT with CHI effectively reduces hallucinations beyond prior methods.
[1
26 6 and RL
84
3 MIRAGE Dataset
3.1 Data Construction
To evaluate reasoning hallucinations, as in Table we present MIRAGE, emphasizing tasks with accurate perception challenging reasoning. MIRAGE offers multi-level annotations and rich auxiliary data for error diagnosis. As in Fig. 2 the construction involves data collection and curation.

---

Figure 2: The construction and evaluation of MIRAGE. (a) shows the construction of MIRAGE.And (b) shows gaaatlgg evaluation metries: accuracy, factuality, and LLMs hallucination score. Data Collection. To systematically evaluate multimodal reasoning capacities across diverse cognitive dimensions, the MIRAGE is constructed through rigorous selection of seven distinct taxonomies, including geometry, algebraic, arithmetic, scientific, spatial reasoning. and statistical reasoning. Based on above taxonomies, we collect the original benchmark data from both publicly available datasets and questions from Internet. Finally, the size of original dataset is roughly 18K.
Data Curation. To ensure MIRAGE isolates reasoning hallucinations and comprehensively evaluate each topic, we apply two-step curation process, i.e., difficulty curation and balance curation. For difficulty curation, we use three open-source MLLMs [2] [ [8] to generate image descriptions, retaining only questions where these descriptions are consistently accurate (verified by a secondary LLM (36]) but lead to frequent reasoning errors, aligning with our benchmark focus. For balance curation, we sample the resulting data to ensure balanced distribution
[TO 28 33] across seven topics, maintaining small imbalanced rate, and resulting in final dataset of 1,329 questions.
3.2 Data Annotation and Verification
Reasoning Chain Annotation. To address the lack of ground-truth reasoning chains, propose a cost-effective automated annotation framework that optimizes both computational efficiency and output quality. Our approach follows two-stage refinement process. Firstly, we generate initial reasoning chains using the lightweight O3-mini [24]. And then we refine these chains with a strong LLM (21), guided by known ground-truths. We will discuss the annotation cost in appendix.
Collaborative Verification. Next, we conduct annotation verification to ensure the correctness. To improve both verification speed and accuracy. introduce human-Al AI collaborative verifica- tion framework. Specifically, each question is independently assessed by a human expert and an MLLM 65] for potential hallucinations in the reasoning chain. If both assessments reasoning chain is retained as the ground-truth. In cases of discrepancies, the human expert either guides the MILM correct the reasoning chains manually provides reasoning steps if the MILM remains inaccurate. Finally, all newly annotated chains undergo cross-checking by other experts.
accurate, the
Step and Claim Extraction. Finally, given the verified reasoning chain and final answer for each question, we use a state-of-the-art LLM (36] to extract critical intermediate steps and claims via in-context learning. Specifically, for each ground-truth reasoning chain y, we use hand-crafted few-shot prompts guide the L.MM in selecting K, representative reasoning steps $ $1, Sk, and Kg intermediate claims C cl...., ČK.- To ensure reliability, we limit 1 < K, S 10 and 1 < Kg < 10, preventing ever-detailed and unreliable outputs. The final intermediate steps and claims are then parsed using regular expressions for fine- -raingr reasoning hallucination evaluation.
Auxiliary Information Annotation. AGE also uses MI.MM 23 to annotate image descriptions and hints. This information is verified by experts and can help researchers diagnose hallucinations.
4 MIRAGE Benchmark Evaluation
4.1 Accuracy Assessment
The accuracy is fundamental metric since incorrect final answers often indicate reasoning chain hal- lucinations [86] [50] To accommodate different question types, MIRAGE parses the final predictions and matches parsed answers A pred with ground-truths Ag for both multiple-choice and deterministic free-form answers. For questions with approximate answers (e.g., statistical questions without precise annotations on charts), MIRAGE calculates the relative error between predictions and ground-truths, considering answers correct if the error falls below a threshold.
4.2 Factuality Assessment
Step and Claim Factuality Evaluation. For each predicted reasoning chain y, MIRAGE follows the extraction pipeline in Sec. 3.2 to extract intermediate steps S = (s,, } and claims C = [C1,.... ). With given corresponding ground-truth intermidiate steps & and claims C we utilize LLM (36) and use {S as input, and guide the LLM to detect whether a predicted step s;

---

is covered in S and whether ground-trun step š; is stated in S, and then predicts the binary matching results Mesd and M.g. By this formulation, MIRAGE can efficiently match free-form steps and claims for flexible factuality evalution. Finally, we calculate the step factuality score Esee by:
2 Precision, x Recall,
Fstop step
Precision, + Recall,
where Precision MM,
means correctly answered steps and Recall |M means correctly
M geed
M,,
matched ground-truth steps, Similarly, the claim factuality score Fetam is defined by:
Precision, Recall,
F Flaimm Precision, + Recall,
Hallucination Type Detection. Besides, to qualitatively assess which kind of reasoning hallucination does an MILM suffer a specific question, we first clearly define five reasoning hallucination types, including logical, factual, spatial, context, fabricatioon hallucinations. The detailed definitions of each type are provided in App. [ Then, we follow LLM-as-a-Judee [15] 19 and introduce an LLM-based hallucination detector. Specifically, rather than compare two plain reasoning chains directly, MIRAGE detects the reasoning hallucination by examining extracted intermediate steps {S. $ and claims (C, C), then predict the hallucination detection results by in-context learning.
4.3 LLMs Hallucination Score (LHS) Assessment
Finally, we also assess the hallucination from the whole reasoning chains. While entropy- uncertainty estimation methods [13 81 87 58 can identify unreliable reasoning or information- deficient chains, they still face two limitations. First token-level likelihood dependencies. Existing methods rely on token-level likelihood to quantify uncertainty, but it is inaccessible in black-box MLLMs [56] 23 65). And second is high computational cost. Accurate uncertainty assessment typically requires sampling numerous responses per query. escalating evaluation overhead. Therefore, inspired by LLM as judges [32 15 19 31, we propose LLMS Hallucination Score (LHS) simulate uncertainty estimation via Multi-LLMMs and multi-reference ensemble. Specifically, we first define multi-dimension scoring rules to measure the hallucination in the whole reasoning chain rather than extracted steps, including factual accuracy, logical consistency, reasoning completeness, conceptual accuracy, and strategy appropriateness. Above dimensions can be used simulate the uncertainty in responses and formulate the scoring template h Our aim is to predict LHS by M (e.g., 3) LLM judgers. To improve the confidence of LHS, MIRAGE leverage an LLM [36 to rewrite by N variants, thus formulating N = 3 reference chains {rier .... ..fe Then, for each response r'er with corresponding ground i,, both responses are integrated into the template hecore and then generate
the judgement scores ( ....; $5. by each j-th LLM judger. Finally, the LHS & of response r is:
M
N
mean( (5f2 y.. $
By accumulating responses in MIRAGE, specific reasoning MILM. Generally, lower mean indicates higher uncertainty (i.e., hallucination rate), and lower standard deviation means higher confidence of the LHS. We further conduct consistency checks on LHS using human evaluators. We randomly sample 100 responses from Gemini-2-flate and Dwwen2.5-VL-7B. comparing the human evaluation from three experts. The average difference rate is 7.5%, showing the reliability of LHS for measuring reasoning hallucinations.
calcuate the mean and standard deviation of ø for
5 Logos: A Baseline Method of MIRAGE
5.1 Revisit Multimodal Reinforcement Fine-Tuning
As shown in Sec. 6.1 reasoning chains with correct answers generally have lower hallucination rates (e.g., logical hallucination). This suggests that reducing hallucinations in MILMS can be approached by increasing the generation probability of logic-consistent and correct reasoning chains, aligning inherently with Group Relative Policy Optimization (GRPO). To address this, we propose the bascline method Logos for MIRAGE, leveraging GRPO to optimize MLLM (the policy model
5

---

1. Curriculum Reinforcement Fine-tuning
Collaborative Hint Inference
IFS8 C-44 DS, theu what io
A-th Difficulty Fittration
Polley Optimization
Reasoning
T Deapie
Degandin
Prompting
Ouestions
PetkyyMeder (ILLI
Reanninn
Verifiable Reward
Normalization
4
goonsnn probice,
ongle
Training Set
remuentty proctu whhle skliid Sahiii
Ronsoning
Calralaor R.A and BCD... The finel unswer
ANd rakoeec Coscreehh......
G-round Sampling
Online Reward Filtzation
Figure 3: Overview of our baseline Logos. During training, Logos adopts curriculum reinforcement fine-tuning (CRFT) with online reward filtration (ORF) progressively increase data difficulty and filter low- low-impact samples. During testing, Logos introduces Collaborative Hint Inference. leveraging L..--uudded hints to simplify the reasoning process. Logos effectively reduces logical hallucination.
with parameter 0). Specifically, we leverage in-context learning 62 to guide to generate formatted response with "<think>... </think>" and "<answer>... <./.s...r///r"""" blocks, where the former contains the reasoning chain and the latter includes the final answer. Rather than using a separate value model to calculate advantages of responses, GRPO directly samples G different responses {ri,.... } with given multimodal question x. To measure define the reward function R as format reward Rim and accuracy reward Race, where the former binary function to judge whether the i-th response r, follows response format, and the latter binary function judge the correctness of final answer. Then the reward of r, is defined by
relative advantages (A,,..., Ag). we
r, = (r,) + Race (1;). And the advantage A, is:
4 Ti --eam (MI,...,
(4)
std((ri...,,ra))
Finally, we optimize via minimizing GRPO loss LoRPO with above advantages follows:
G
LGRPO = -Byr
Σ
min (Fr,(0) A,, clip((;;. ()))))
G
ra
(5)
x. (rr,a Ti,
where fi.e
(T x, ri.13-1)
We remove the KL-divergence term since reasoning models have non-negligible distribution gap with base models [49 We will investigate the effect in the App.
5.2 Curriculum Reinforcement Fine-Tuning
Note that if responses from specific training sample are all correct or all incorrect, the advantage of each response O, which harmful for GRPO optimization. To reduce the difficulty and improve the training efficiency, we propose curriculum reinforcement fine-tuning (CRFT). Specifically, before the optimization, we first leverage reward Face mean( mean((R(r))).)) ( R(ra)}) ). During the first stage, we keep questions with Face 0 to ensure that π can sample at least one reasoning chain with correct answer and logic-consistent reasoning during training, thus making the advantages non-zero for smooth optimization. Then, during each k-round (A >> 1) curriculun training, we repeat G-round sampling and keep questions with Face 0.5 to ensure æ can face more difficult questions during further CRFT. Our experimental results in See 6.2 illustrate the efficiency of CRFT.
conduct G-round sampling, and calculate average accuracy
5.3 Online Reward Filtration
While CRFT effectively controls data difficulty, it may still encounter training samples where all generated responses receive identical rewards, disrupting the optimization process. To address this without compromising training efficiency, we integrate offline data filtration [74] into our approach,
forming Online Reward Filtration (ORF). each iteration, for given question x with sampled responses rı, ... ra. Logos first computes the corresponding rewards r ..., TG using the predefined reward function R. If all responses share the same reward (r' = rG), the question discarded for that iteration, ensuring only diverse, meaningful samples contribute to optimization.

---

Table 2: Comparison recent state-of-ther the- MLLMs on MIRAGE. Best results are bolded.
Factuality
Model
Type
Accuracy
LHS
Faep f
Fdaim
Black-Bol Box MILMs
Gemini-2- 2--a-2h---ll--------- 2-llha--Thiinn 56
Reasoning
47.6
51.5
50.7
0.7517=00016$
01 2
Reasoning
49.7
41.3
42.7
0.6193110.001
Gemini-22 Flash 56
General
44.1
47.8
47.4
0.6882-E0.0496
GPT-4o
23
General
35.0
39.2
40.6
0.6332+#0.011
Open-sourced 72B MLLMS
Owen2.5-VI-722B-55 Innstrcctt
General
38.8
47.4
44.6
0.7223+0.0339
IntemVL-2.5577 [5]
General
29.6
39.0
36.6
0.6377+:0.0325
Qwen2-V1 -2---11--2BBB- Instruct
61
General
24.5
29.7
26.2
0.492811000322
QuQ-72B-Preview (5
Reasoning
31.0
46.1
45.3
0.5717=0.0597
Virgo-72B 12
Reasoning
37.4
47.1
45.0
0.6328110.0251
Open-sourced 7B MLLMs
Been2..-VL-7BB--msee
General
28.8
34.7
31.7
0.5996100.123
Qwen2-VL. 1-22-VLL--B-Insttte
General
19.5
21.9
18.6
0.363310.01
Qwen2.5- V
80
Reasoning
26.9
22.8
25.2
0.447811000177
Reasoning
26.7
40.1
33.4
0.5826+0.0124
R (5]
Reasoning
22.9
30.7
30.2
0.5099+0.0099
Mulberry-Qwen2- www.n--een222--n
hp
Reasoning
22.6
29.2
24.4
0.4740
VntemVV.-2.5-8B 5
General
20.8
31.9
26.4
0.4838==000056
Llama-3.2 2- Vision-ionnn 11B
18
General
18.7
26.9
22.3
0.4265#100141
Llava-CoT-11 IBB 667
Reasoning
17.4
26.9
22.4
0.4267110.0140
Logos-7B (Ours)
Reasoning
37.1
43.3
38.3
0.6568=0.0119
Open-sourced 3B MLLMs
Qwen2.5-V1.c -nnstrrB-
General
18.8
23.1
18.8
0.3422++0.024
Phi-ses-Instruet 1
General
12.9
16.6
13.8
0.1181=0.0161
Logos-3B (Ours)
Reasoning
29.4
38.9
34.5
0.5840
Sypa
Pearson aden
Alabbaac
Artinttc
22.2%
15.0%
22.1%
16.1%
12.4%
Gecevttry
$1.0%
20.2%
20.1%
26.2%
11.5%
Legica!
6,57
6.50
1::00
Spetiel 25.5%
24.9%
34.8%
23.2%
Muristial
28.0%
14.95
11.05
16.4%
11.7%
6.5
Spaaiial
Dustrgl -Pi
Pakrical
Figure Distribution between question types and reasoning hallucination types.
Figure 5: Pearson correlation among reasoning hallucination types.
5.4 Collaborative Hint Inference
To further reduce reasoning hallucinations beyond training, we introduce Collaborative Hint Inference (CHI), which leverages an auxiliary LLM & to provide context-specifi guidance. Given a question CHI first uses predefined question classification prompt to guide á in predicting the question type c. Based on this type, CHI generates two structured hints: a topic-specific hint htopic, reflecting the general approach for the given type c, and irestion-speciffe hint hquestion, tailored to the particular content x. During inference, we generate response by r = T ((hugice x, hypestion): The optimized MILM can benefit from CHI and generate accurate chains than vanilla MLLMs.
6 Experiments
We state the full experimental setup of MIRAGE evaluation and training of Logos in App. B And in the following, we illustrate the insightful findings MIRAGE and the effectiveness of Logos.
6.1 Empirical Analysis
Overall results. shown in Table 2 achieves the highest accuracy at 49.7, outperforming Gemini-2-flash fiai----faahh-hiiiiiie -hinkiiii by 2.1. However, Ol scores lower in factuality, likely due to generating
7

---

Table 3: Probability of different types of reasoning hallucinations for each model.
Model
Logical
Spatial
Factuality
Context
Fabrication
Gemini-2-ffaa
54.66%
29.33%
39.03%
28.45 48%%%%%
22.32%
Qwen2.5- Sive2..5VV.-1 VT.BB
68. 68.94%
35.35%
51.07%
30.98%
23.91%
Gemini-2- Gemini---flaah-tiinkee t-faiint
47.8 87.
25, 44%
32.85%
22.51%
18.04%
Virgo-72B QvQ-72B-Preview
63.97%
29.18%
40.65%
32.26%
21.32%
73.37%
37.93%
47.91%
47.52%
30.19%
Table Manually fixing reasoning chains exper- imental results on 10% sampled questions with reasoning hallucination.
Table 5: Correlation matrix of all metrics. All
correlations are significant at < 0.001 (***).
Model
Fix Reasoning Accuracy
Metric
Accuracy
Fuep
Fatim
LHS
GPT-4o
12.1
Accuracy
1.000
0.864
0.918
0.889
GPT-40
68.5
Faep
0.864
1,000
0.975 ...
0.915
Qwen2,5-VL-72B
10.4
Fetam LHS
0.918
0.975 *
1.000
0.933***
Qwen2.5-
72,4
0.889
0.915
0.933***
1.000
shorter, summarized reasoning chains that reduce step coverage and recall. For most models except GPT-4o and O1, step scores exceed claim scores, suggesting that generating coarse reasoning steps is easier than detailed calculation steps. The LHS scores align with step scores, confirming the reliability of MIRAGE's metrics. Focusing on open-source Qwen-VL models, increasing parameters from 3B 72B raises accuracy from 18.8 to 38.8. Moreover, better pretraining in Qwen2.5-VI. im- proves both accuracy and factuality/LHS, indicating that enhanced pretraining reduces hallucinations.
Consistency among evaluation metrics. As shown in Table 5 we compute pearson correlation coefficients between Accuracy, Fstep. Fetaim- and LHS across all models. All pairs of metrics exhibit very strong positive correlations (r = 0.86-0.98) with all correlations being highly significant (p < 0.001). These results indicate that the hallucination rate in the reasoning chains has large correlation with final answers, and inspire our hallucination mitigation method.
Correlation between reasoning hallucination and final accuracy. Given correlations between rea- soning chains and answer accuracy, we conduct preliminary study to show the impact of hallucination mitigation. We manually corrected the reasoning chains for about 10% of commonly misanswered questions by GPT-4o and Swen2.5-VL-72B. then prompt the models to reconsider their final answers. As shown in Table this correction significantly improves the answer accuracy to around 70%, confirming that reducing reasoning hallucinations directly enhances overall model performance.
Relation between pretraining data and hallucination types. We also explore relations between pretraining data and hallucination types. Specifically, we keep use Qwen-VL [6 with different pretraining data (i.e., Qwen2-VL and Qwen2.5-VL) and compare the hallucination rates of each hal- lucination type. As shown in Table Qwen2.5-VL. models have less logical, factual, and fabrication hallucination rates than those of Qwen2-VI. VI. models. A possible explanation that pretraining data with higher quality provides more accurate factual knowledge and reasoning chains models, such that models can avoid logical and factuality hallucinations during inference. Nevertheless, the spatial hallucination does not significantly reduced, which indicates that current MLLMs still show weak visual reasoning capabilities.
Relation between model size and hallucination types. We also explore relations between pretraining data and hallucination types. Specifically, we keep use QW 2 with different model sizes (i.e., 3B/7B/72B) and compare the hallucination rates of each hallucination type. As shown Table 7 Larger Qwen2.5-Vi models have less logical, factual, and fabrication hallucination rates than those of smaller models. A possible explanation is that models owning more model parameters have more capabilities for accurate factual knowledge and reasoning chains to models, such that models can avoid logical and factuality hallucinations during inference. Nevertheless, the spatial hallucination does not significantly reduced, which indicates that current MLLMs still show weak visual reasoning capabilities.
Correlation between question and hallucination types. We also analyze the relationship between question types and hallucination patterns, as shown in Fig. 4 Results indicate that logical hallucina- tions are widespread across various question types, while certain hallucination types are more closely associated with specific question types. Specifically, logical and spatial hallucinations are particularly common logical questions, reflecting the high demands for complex reasoning and visual transfor- mations that current MLLMs struggle with 4 In contrast, statistical and scientific questions tend
8

---

Table 6: Hallucination type rates in MIRAGE questions of Qwen-7/7BB with different pertaining
data. Pretraining with higher quality data leads to less logical, fabrication, and factual hallucinations.
Model
Logical
Factuality
Spatial
Context
Fabrication
Qwen2.5-VL-72B
47.7%
33.7%
29.2%
21.6%
16.5%
Qwen2-VL-72B VL-72B
59.3%
45.4%
32.7%
32.6%
26.5%
Qwen2.5-VI.-7B
64.7%
45.7%
33.4%
29.3%
25.5%
Qwen2-VL-7B
74.0%
60.6%
35.6%
42.7%
35.4%
Table 7:
Hallucination type rates in MIRAGE benchmark questions of Qwen-2.5-VL.. Larger Models lead to less logical, fabrication, and factual hallucinations.
Model
Logical
Factuality
Spatial
Context
Fabrication
Qwen2.5-V1.-72B
47.7%
33.7%
29.2%
21.6%
16.5%
Qwen2.5-VL-7B
64.7%
45.7%
33.4%
29.3%
25.5%
Qwen2.5-VL-3B
78.9%
60.1%
36.7%
37.9%
38.1%
to exhibit more factuality hallucinations, likely due to their reliance on precise knowledge retrieval. These findings highlight the specific vulnerabílities of MLLMs in handling diverse reasoning tasks.
Correlation among hallucination types. We further analyze correlations among hallucination types using pearson coefficients. As shown in Fig. logical hallucinations strongly correlate with factuality, context, and fabrication errors, likely because flawed logic often leads to context inconsistency and factual errors. Notably, spatial hallucinations, which arise from complex visual operations, show relatively low correlation with other hallucinations, suggesting they are more independent and unique to multimodal models rather than text-based LLMs. These findings highlight the need for targeted mitigation strategies for hallucination types, particularly for challenging spatial reasoning errors.
Hallucination rate comparison across models. To quantitatively assess the impact of model design and training on reasoning hallucinations, we analyzed five representative MI.LMs, as shown in Table 3 DvQ-72B-Previees exhibits the highest overall hallucination rates, especially in Logical (73.37%) and Context (47.52%) categories, significantly higher than Virgo-72B, which shares the same base model but benefits from more effective tne-tuning. In contrast, Gemini-2 2--iii-- 2-flash-thinking -tinnking consistently shows the lowest hallucination rates, particularly Logical (47.88 (47.88%), Spatial (25.44%) and Fabrication (1 (18.04%) categories, indicating superior robustness.
Existing solutions are not sufficient to mitigate hallucination. Training-free methods like self- reflection [16] and visual inference chain [86] generally degrade both accuracy and LHS on base models without sufficient reasoning capabilities (Table 2). highlighting their limitations. Similarly, SFT-based methods can improve hallucination mitigation on larger models (c.g., Virgo-72B) but often fail enhance smaller models, suggesting that model capacity plays a critical role the effectiveness of external supervision. More detailed analysis can be found in App.[) [D] These findings motivate our integration of CHI into CRFT-optimized MLLMs rather than directly applying to base models.
6.2 Empirical Analysis of Logos
To quantitatively investigate the component of our baseline method Logos, we use Logos-7B as an example and conduct an ablation study on both MIRAGE and a standard benchmark, Mista [4] More in-depth analysis can be found in App.D]
Comparison with previous methods. We compare Logos- 7B with other 7B models to validate its effectiveness. Compared to the base model Swwe2.5-VIL-7BB2 Logos-7B achieves an 8.3 gain in accuracy, and outperforms the base by 8.6 on Fstep and 6.6 on Fetaim- approaching the performance of the larger Virgo-72B [2]. These results, consistent with LHS scores, indicate that Logos effectively reduces reasoning hallucinations, improving reliability across reasoning chains. Similar gains are also observed for Logos-3B, showing the compatibility of our framework across different model scales.
Whether Logos reduces reasoning hallucination or not. Finally we investigate the hallucination mitigate effect on each hallucination type. As shown in Table Logos- 7B reduces logical hallucina- tion by 15.4% and fabrication hallucination by 10%. Similar results can also be found in Logos-3B. Nevertheless, we do not find significant halucination mitigation spatial and factuality hallucination on both Logos models. A possible reason that reinforcement learning does not introduce new knowledge and only refines the logic of reasoning chains.

---

Table 8: Ablation study of Logos-7B, 7B, where CHI means collaborative hint inference.
Method
GRPO CRFT
Hopic
Mqucction
Accuracy
Fup
Fdaim
LHS
Math Vista
Sive2.5-VL-75
X
x
X
28.8
34.7
31.7
0,5996
68.2
+GRPO
x
x
X
33.7
41.0
35,9
0.6180
70.7
+CRFT
X
X
35.7
418
37.3
0.6193
71.9
+hooicc
7
V
X
36,2
42.6
37.6
0.6335
72.2
+iquestion
/
X
V
36,5
42.2
37.6
0.6224
72.2
+full CHI
X
x
37.1
43.3
38.3
0.6568
72.3
Qwen2.5-VL-7B full CHI
29.0
34.9
32,1
0.6011
68.3
Table 9: Hallucination type rates in MIRAGE benchmark questions of Qwen2.5-VIL-3B// and
corresponding Logos-3B/7B. Our proposed method leads to less logical and fabrication hallucinations.
Model
Logical
Factual
Spatial
Context
Fabrication
Qwen2,5-VL-77
64.7%
45.7%
33.4%
29.3%
25.5%
Logos-7B
49.3%
39.7%
29.9%
23.8%
15.6%
Qwen2.5-VI-3B
78.9%
60.1% 1%
36.7%
37.9%
38. 1%
Logos-3B
57.1%
47.4%
36.7%
31.8%
24.0%
Ablation Study of Each Component of Logos. We first investigate the effect of each key component in Logos. The experimental results are shown in Table 8 After adopting reinforcement learning on the base model, the accuracy on MIRAGE and Math achieves 33.7 and 70.7 respectively. Benefiting from RL, the step score and claim score also increase to 41.0 and 35.9, After adopting CRFT, both accuracy and Felaii further increases to 35.7 and 37.3 respectively. By further integrating CHI, Lo achieves 37.1 on MIRAGE and 72.3 on Mathvista. Note that directly adopt CHI on base model does not lead to performance improvement, which further proves the findings in Sec. 6.1
Effect of different hints in CHI. The primary role of Equestio is to guide the model to identify key entities and starting conditions within the question. It excels at grounding the model's 's initial focus but does not provide complete, step-by-step problem-solving template or structure. In contrast, hupic provides a high-level problem-solving schema or template. It offers a clear, structured formulation for the entire reasoning process, ensuring logical completeness from start finish. The difference leads to performance gap in different metrics. For CRFT + hquestion. it has better final accuracy (+0.3) than CRFT hopic but the metrics regarding intermediate steps (i.e., Fstep and LHS) slightly lower. Logos enhanced by CRFT is already optimized to generate complete and precise reasoning chains. When this strong base model is augmented with hoopic. it directly benefits from the clear structural guidance, making it easier to construct a perfectly formed reasoning chain. This synergy explains why hopic achieves slightly better scores on metrics like Faep and LHS compared
Aquestion.
How Logas mitigates reasoning hallucination? We evaluated the impact of CRFT by comparing the accuracy of Logos-7B 8-round sampling across training dataset, before and after training. The accuracy increases from 24.8% to 68.3%, indicating that CRFT effectively guides the model to generate correct reasoning chains. Meanwhile, we calculate the "Logical" hallucination rate in Qwen2.5-VL-7B and Logos-7B, which reduces from 57.1 to 49.3. This result shows that CRFT benefits encourage model learning logic-consistent reasoning chains mitigate hallucination.
Conclusion
We propose MIRAGE, which isolates reasoning hallucinations by questions where inputs are correctly perceived but reasoning errors persist. For analysis of reasoning hallucination, MIR AIR proposes multi-level evaluation metrics, covering different levels of the reasoning chains. Our findings reveal that the model scale, data scale, and training stages of MLLMs: (I) significantly influence the degree of logical, fabrication, and factual hallucinations: (2) show no effective improvement on spatial hallucinations caused by misinterpretations of spatial relationships, suggesting that current MLLMS exhibit weak visual reasoning capabilities and struggle to benefit from simple scaling of training resources; and 3) correlations between question types and specific reasoning hallucination patterns, highlighting critical challenges and mitigation for specific types. These findings will provide insights for future MILMM development. To address this, we propose Logos, method using curriculum reinforcement fine-tuning and collaborative hint inference to reduce logical hallucination for higher accuracy. Logos provides a bascline and offers insights for reducing hallucinations.
10

---

References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
[3] Assaf' Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, and Hadar Averbuch-Elor Mocha: Multi-ojeective reinforcement mitigating caption hallucinations. arXiv preprint arXiv:2312.03631, 2, 2023.
[4] Huanqia Cai, Yijun Yang, and Winston Hu. Mm-iq: Benchmarking human-like abstraction and reasoning in multimodal models. arXiv preprint arXiv:2502. 00698, 2025.
[5] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic vision and pattern recognition, pages 24185 24185-24198, 2024.
tasks. In Proceedings of the IEEE/CVF conference computer
[6] Zhiyang Chen, Yousong Zhu, Yufei Zhan, Zhaowen Li, Chaoyang Zhao, Jinqiao Wang, and Ming Tang. Mitigating hallucination in visual language models with visual supervision. arXiv preprint mrXiv:2311.16477 2023.
[7] Yew Ken Chia, Vernon Toh, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. Puzzlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16259-16273,
2024.
[8] Jieren Deng, Haojian Zhang, Kun Ding, Jianhua Hu, Xingxuan Zhang, and Yunkuan Wang. Zero-shot generalizable incremental learning for vision-language object detection. In The Thirth-eighth Annual Conference on Neural Information Processing Systems.
[9] Bowen Dong, Zitong Huang, Guanglei Yang, Lei Zhang, and Wangmeng Zuo. Mr-gdino: efficient worn- continual object detection. arXiv preprint arXiv:2412. 15979, 2024.
[10] Bowen Dong, Pan Zhou, YAN Shuicheng, and Wangmeng Zuo. Lpt: Long-tailed prompt tuning for image classification. In The Eleventh International Conference on Learning Representations, 2023.
[11] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234 2022.
[12] Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji- Ji-Rong Wen. Virgo: A preliminary exploration on reproducing ol ol-kike milm. arXiv preprint arXiv:2501.01944, 2025.
[13] Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625 630, 2024.
[14] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng. Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.1333 2023.
[15] Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. GPTScore: Evaluate as you desire. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), pages 6556-6576, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
11

---

[16] Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, and Kamile Lukošiute. The capacity for moral self-correction large language models. Parameters, 109(1010):001.
[17] Deepanway Ghosal, Vernon Toh Yan Han, Yew Ken Chia, and Soujanya Poria. Are language models puzzle prodigies? algorithmic puzzles unveil serious challenges in multimodal reasoning. arXiv preprint wxXv:2203303864 2024.
[18] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[19] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411. 2024.
[20] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375-14387 2024.
[21] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12949, 2025.
[22] Yukang Huo and Hao Tang. When continue learning meets multimodal large language model:
A survey, arXiv preprint arXiv:2503.0/887, 2025.
[23] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410..1226. 2024.
[24] Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, al. Openai ol system card. arXiv preprint 2rXiv:2412.16722 2024.
[25] Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating LLM hallucination via self reflection. In Houda Bouamor, Juan Pino, and Kalika Bali, editors. Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827-1843, Singapore, December 2023. Association for Computational Linguisties.
[26] Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye. Ming Yan. Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27036-27046, 2024.
(27] Dongzii Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint arXiv:2502. 09621, 2025.
[28) Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations.
[29] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph F. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large lan- guage model serving with pagedattention. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[30] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal lims with generative comprehension. arXiv preprint 1rXiv:2307.16125. 2023.
12

---

[31] Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Lims-as-ii Llud-as-judges: a comprehensive survey on Ilm-based evaluation methods. arXiv preprint arXiv:2412.05572 2024.
[32] Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. In Findings of the Association for Computational Linguisties: ACL. 2024, pages 3923-3954, 2024.
[33] Mengke Li, Yiu-ming Cheung, and Yang Lu. Long-tailed visual recognition via gaussian clouded logit adjustment. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages
2022.
[34] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing.
[35] Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, et al. Omnibench: Towards the future of universal omni-language models. arXiv preprint arXiv:2409.15272 2024.
[36] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[37] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigat- ing ballucination in large multi-modal models via robust instruction tuning. The Twelfth International Conference Learning Representations.
[38] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[39] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rfi: Visual reinforcement fine-tuning, arXiv preprint arXiv:2503.0178: 2025.
[40] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[41] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
[42] Pan Lu. Swaroop Mishra. Tony Xia. Liang Qiu, Kai-Wei Chang, Song-Chup Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Process- ing Systems (NeurIPS), 2022.
[43] Ruilin Luo, Zhuofan Zheng, Yifan Wang. Yiyao Yu, Xinzhe Ni, Zicheng Lin, Jin Zeng, and Yujiu Yang. Ursa: Understanding and verifying chain-of-thouuht reasoning in multimodal mathematics. arXiv preprint ar0i _04686, 2025.
[44] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/eyf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[45] Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263-2279, 2022.
[46] Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with aul-based large-scale reinforcement learning. arXiv preprint vrXiv:2503.07366 2025.
13

---

[47] Chancharik Mitra, Brandon Huang. Trevor Darrell, and Roei Herzig, Compositional chain-of- thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420-14431, 2024.
[48] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veiito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035, Curran Associates, Inc., 2019.
[49] Yingzhe Peng, Gongrui Zhang, Miaosen Zhang. Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025.
[50] Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. Receval: Evaluating reasoning chains via correctness and informativeness. arXiv preprint arXiv:2304. 10703, 2023.
[51] Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan O Arik, and Tomas Pfister. Mitigating object hallucination in milms via data-augmented phrase-level alignment. In The Thirteenth International Conference on Learning Representations.
[52] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (oo::177706650 707.06347, 2017.
[53] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
[54] Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetuning via adaptive curriculum learning. arXiv preprint arXiv:2504.05520, 2025.
[55] Maciej Swiechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Maádziuk. Monte carlo tree search: A review of recent modifications and applications. Artificial Intelligence Review, 56(3):2497-2562, 2023.
[56] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
[57] Qwen Team. QVQ: To See the World with Wisdom. qvq- rrqg-7bb-preee www.ppeieeee///// 2024.
gith.t io/blog/
[58) Christian Tomani, Kamalika Chaudhuri, Ivan Evtimov, Daniel Cremers, and Mark Ibrahim. Uncertainty-basee abstention in lims improves safety and reduces hallucinations. arXiv preprint arXiv:2404.109660 2024.
(59) Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9568-9578, 2024.
[60] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[61] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint 24Xi:22409.12191, 2024.
[62] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-24831 2022.
14

---

[63] Sangmin Woo, Jachyuk Jang, Donguk Kim, Yubin Choi, and Changick Kim. Ritual: Ran- dom image transformations as universal antt-haluucination lever in Ivims. arXiv preprint urXiv:2405. / /7221, 2024.
[64] Xiyang Wu, Tianrui Guan, Dianqi Li, Shuaiyi Huang, Xiaoyu Liu, Xijun Wang, Ruiqi Xian, Abhinav Shrivastava, Furong Huang, Jordan Boyd-Graber, et al. Autohallusion: Automatic gen- eration of hallucination benchmarks for vision- vision-language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8395-8419, 2024.
[65] XAI. Grok Beta - The Age of Reasoning Agents. https://x.ccm attgrok.aa//ookkk/ 2025.
[66] Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. Mitigating object hallucination via concentric causal attention. Advances in Neural Information Processing Systems, 37:92012-92035, 2024.
[67] Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step, 2024.
[68] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 2024.
[69] Jun Cheng Yang, Zuchao Li, Shuai Xie, Wei Yu, Shijun Li, and Bo Du, poompponn with graph-of-thougie for multi-modal representation learning. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 55024-15036 2024.
[70] Tianyun Yang, Ziniu Li, Juan Cao, and Chang Xu. Mitigating hallucination in large vision- language models via modular attribution and intervention. In The Thirteenth International Conference Learning Representations, 2025.
[71] Yi Yang. Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503. 106., 2025.
[72] Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, al. Mulberry: Empowering milm with ol-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:24 18319, 2024.
[73] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
[74] Yixin Ye. Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025.
[75] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12944-12953, 2024.
[76] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 202225003.466.... 2025.
[77] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han. Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 13807-13816, 2024.
[78] Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. Llama-berry: Pairwise optimization for ol-l olympiad-level mathematical reasoning, arXiv preprint arXiv:2410.02880 2024.
15

---

[79] Jinrui Zhang, Teng Wang, Haigang Zhang, Ping Lu, and Feng Zheng. Reflective instruction tuning: Mitigating hallucinations large vision-lange Sainnnla models. European Conference on Computer Vision, pages 193-213. Springer, 2024.
[80] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang. Yu Qiao, et al. Mathverse: Does your multi multi-modal Ilm truly see the diagrams in visual math problems? In European Conference Computer Vision, pages 169-186. Springer, 2024.
[81] Ruiyang Zhang, Hu Zhang, and Zhedong Zheng. V1-uncertainty: Detecting hallucination in large vision-language model via uncertainty estimation. arXiv preprint arXiv:2411. 11199,, 2024.
[82] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.0730/1 2025.
[83] Zhuosheng Zhang, Aston Zhang, Mu Li, George Karypis, Alex Smola, al. Multimodal chain-of-thought reasoning in language models. Transactions on Machine Learning Research.
[84] Zhiyuan Zhao. Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing Ivlms through hallucination-awann aware direct preference optimization. 2023.
arXiv preprint
[85] Changmeng Zheng, Dayong Liang, Wengyu Zhang, Xiao- Wei, Tat-Seng Chua, and Qing Li. A picture worth a graph: blueprint debate paradigm for multimodal reasoning. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 419-428, 2024.
[86] Haojie Zheng, Tianyang Xu, Hanchi Sun, Shu Pu, Ruoxi Chen, and Lichao Sun. Thinking before looking: Improving multimodal Ilm reasoning via mitigating visual hallucination. arXiv preprint arXiv:2411. 1,1591,,,,,,,,,, 12591, 2024.
[87] Kaitlyn Zhou, Jena Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models' reluctance express uncertainty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3623-32230 (623-3643, 2024.
[88] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have claimed the paper's contributions and scope in both the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitation in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided all the details (including prompts) in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: https://githm: hwt.at/ww....//aaaaga////aaaaaa.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
sttpiiisssoooooooyyy. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have stated the experimental details in both main body of the paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have noted the statistical significance and state the calculation method in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We list the detailed experiments compute resources the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read and confirm the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have stated the broader impacts in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: few samples the proposed dataset come from existing benchmarks (e.g., Math Vista). We have cited related works and have listed the license of related works in the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
22

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLMs are used as data preprocessing tool and an evaluation tool. And we also use LLM to revise the grammar of the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neure wwwpurrppssc.coooeeeeeeeee for what should or should be described.
23

---

Table 10: Categories of multimodal reasoning hallucination investigated in MIRAGE.
Hallucination Type
Hallucination Description
Spatial Hallucination
Errors in reasoning about spatial relationships, shapes, complex visual operations.
Logical Hallucination
Errors in logical consistency or reasoning. even when surface-level facts are correct.
Factuality Hallucination
Factually incorrect claims about scientific principles or established knowledge in input data.
Context Hallucination
Inconsistencies between intermediate reasoning steps and final predictions.
Fabrication Hallucination
Entirely invented values, entities, or relationships not in input data or real world.
This appendix mainly contains:
Hallucination type definition in Section
Detailed experimental settings in Section
Additional quantitative results in Section
Additional in-depth analysis in Section D
Dataset examples in Section
More qualitative results in Section
Statement of limitations in Section
Statement of broader impact in Section
A Hallucination Type Definition
In this section, we summarize the hallucination types mentioned in Sec. which are listed in Table I0 Specifically, the multimodal reasoning hallucinations can be categorized into five distinet types, L.e., spatial hallucination, logical hallucination, factuality hallucination, context hallucination, and fabrication hallucination. The detailed descriptions are summarized in Table
B Experimental Details of MIRAGE and Logos
B.1 Experimental Setup
judge the accuracy of final answers. And for factuality and LLMs hallucination score metrics, to reduce the cost utilize DeepSeek-V3
Implementation Details. During MIRAGE evaluation, leverage GPT-4o
23
while keeping comparable evaluation accuracy,
(36) for both metrics, and utilize 5-2n22.5-72B-nnstruee (68) as well as Jlama-3.1-70B--nsstuue [8] for LLMs hallucination score. For Logos-7B, the pretrained model is Qwwan2255VV--77-IInsrrt and for Logos-3B, the pretrained model is 2wven22.5-L--33 -L-3B--nntrrut. Instruct. The visual encoder is frozen to avoid catastrophic forgetting of visual perception ability (22 D. During training, we collect 13K mathematical questions with K12--level difficulty and ~ text-only math questions from LIMO (44 as training data. The batch size is 128. For each training sample, the rollout samples G is 8 by default. The initial learning rate is 1 x 10-6, both warmup strategy and cosine learning rate scheduler are adopted to stabilize training. We optimize Logos by 10 epochs using Adam (40] during each stage. The number of CRFT stages is set to 1, and we will discuss this choice in Sec. 6.2. Benefiting from the filtration mechanism CRFT and ORF, the total training time is less than 24 hours, Logos-3B
faster. All programs are constructed by PyTorch (48) toolkit and vLL.M 29 framework. All the experiments are conducted on NVIDÍA RTX A6000 GPUs.
B.2 Detailed Breakdown of the Initial Data Sources
Our selection of the seven reasoning taxonomies was principled decision aimed at ensuring comprehensive coverage both broad, general-purpos reasoning and deep, domain-specific skills. For general reasoning capabilities, to evaluate skills applicable to everyday scenarios, we included tasks requiring spatial reasoning (object locations, attributes) and statistical reasoning (interpretation of charts and plots). For domain-specific reasoning capabilities, to probe more complex, multi-step

---

logical deductions, we incorporated challenging problems from mathematics (geometry, algebra). science (physics, chemistry, biology), and formal logic (IQ-test style problems). During data collection, our strategy is to curate a diverse dataset from high-quality, specialized sources, each chosen target specific reasoning skill:
We sourced questions from MathVista and Math Vision, which renowned for their high-quality, complex geometric diagrams and mathematical problems.
Mathematical Reasoning:
Logical Reasoning: To assess formal logie, we drew from the official question banks of national civil service examinations, which feature rigorously designed IQ-style test questions.
Scientific Reasoning: We utilized questions from prestigious science exams (e.g.. USA Biology Olympiad), which require domain-specific reasoning in science subjects.
Spatial Reasoning: Our data comes from established traffic-related datasets and the synthetic SuperCLEVR dataset, which emphasize the reasoning of complex spatial relationships.
We ensure the inherent difficulty of initial data by combining empirical validation with source selection. For public benchmarks (e.g., Math Vision), we confirm their challenge by observing that even powerful MLLMs (e.g., Qwen2.5-VLL 5-VL-7B) achieve relatively low accuracy (25.4%), which provides a strong empirical indicator of their difficulty. For crawled data, we curate questions from sources renowned for their complexity, e.g.. Physics / Biology Olympiads and IQ tests, which are by design created to probe the limits of logical deduction. This approach ensures the initial data poses meaningful challenge to current MLLMs.
B.3 Baseline Models
We evaluate various reasoning and CoT-enhanced general MLLMs, including black-box MLLMs
56
23
and open- sourced MILMs [2 18 II. We also analyze -ehoonnnge methods 72
12
86
16 to explore the hallucination mitigation effecctiveness.
include GPT-4o [23 and Ol (24) from OpenAI, as well as Gemini-2 2-flash and Gemini-2 fommni-2-fllsh-tiiee fash-thinkking (56) from Google. These models have shown state-of-the-ane reasoning or chain-of -ftougghtt thinking capabilities in various tasks.
Black-box MLLMS
O Open-soureed MLLMs
cover both apecifically-deiggneee reasoning MLLMs g.. QvQ-72B
and Virgo-72B (12). and General MILMSs including Qwen2-VL. [1]. SW 22. Intern VI.- 2.5 [5) L.lmm-3.2-Visse inherent or chain-of-thougle reasoning capabilities. Note that the parameter numbers of selected models are largely varied from 3B to 72B, ensuring that models with different scales can be analyzed in our experiments.
[18] and Phi-3.5-Instruct [1. All these models have shown competitive
Reasoning-enhance General MILMss Besides, to comprehensively evaluate the capabilities of reducing reasoning hallucination, we also assess multiple training- free and taining-basce hallucina- tion mitigation methods, including self-refiection [16]. question decomposition (86), and supervised fine-tuning [71]. All these methods have shown effectiveness in improving reasoning capabilities.
B.4 Prompts Used in Construction and Evaluation
For the reproducibility of our work, we release the critical prompts used MIRAGE construction and evaluation. Fig. shows the prompt for extracting intermediate results (i.e., steps and claims). Fig. Ishows the prompt of intermediate results matching results (i.e., factuality evaluation prompt). Fig. 10 shows the prompt used to detect specific reasoning hallucination types. And Fig. II shows the prompt used to calculate the LHS score.
B.5
Computational Resources and Time
During the first stage of C-RFT, Logos-7B uses 8 NVIDIA A6000 GPUs to train 7B model, and the total training time 16 hours. During the second stage (since the optimal stage number 1, as discussed in App. DI, Logos-7B uses the same 8 A6000 GPUs and requires 6 hours to complete
25

---

training. During Inference, benefiting from the optimization of the vLLM [29 framework, Logos- BB only requires one A6000 GPU for inference reasoning chains. And both training and inference of Logos-3B model require fewer computational resources.
B.6 Significance Computation
To calculate the correlation and the corresponding significance value, we leverage scipy package and call pearsonr function to calculate pearson correlation coefficient with corresponding significance (i.e., p-value).
Table 11; Effect of reinforcement learning algorithm. We remove CHI and directly assess the original output of each model.
RL
MIRAGE
Math Vista
29.9
69.3
DAPO GRPO
30.0
69.6
35.7
71.9
Table 12: Effect of online reward filtration.
Table 13: Effect of the KL-divergence in Logos.
KL-Div
MIRAGE
Math
ORF
MIRAGE
Math Vista
lc-2
31.0
67.0
34.2
69.6
le-3
35.0
70.0
37.1
72.3
le-4
36.7
71.1
0 (Logos)
37.1
72.3
Table 14:
Comparison between CRFT and
Table 15: Effect of currieulum learning stage k.
vanilla RL with longer training epochs.
MIRAGE
Math Vista
Method
Total Epochs
MIRAGE Math
35.0
70.7
37.1
72.3
Vanilla RL
20 10+10
35.5
7144
37.2
72.3
CRFT
37.1
72.3
37.2
72.3
C Detailed Quantitative Results
In addition to reporting average accuracy and overall LHS score for each model, we also report accuracy in each question topic and LHS score in each dimension. The detailed per-topic accuracy comparison results are shown in Table 16] and the detailed per-dimension accuracy comparison results are shown in Table [8 Generally, all models suffer from unpromising accuracy on logical and spatial questions, which indicate that existing models still do not obtain sufficient visual pattern and relation reasoning abilities. Meanwhile, mathematic reasoning and statistical question reasoning are usually perform well state-of- oftte-----e-rrr MLLMs. As for the LHS score, the score of logical consistency and reasoning completeness models still struggle in insufficient reasoning capabilities and result in reasoning hallucination. These results reveal the the vulnerability in reasoning MI.MMs.
some unpromising models are relatively low, which indicates that previous
D
More Analysis
D.1 Qualitative Results of Manually Fixing Examples
manually fixing hallucinations in the reasoning chains enhances overall model performance. In addition to the quantitative results, we also illustrate qualitative results from GPT- 4o 23 which is shown in Fig. 6 The corrected reasoning chain (right) can guide MILMs to predict correct answers, and the original reasoning chain with hallucination (left) still results in wrong answers.
As stated in Sec. 6.2
26

---

Original Chain:
Manually Corrected Chain:
The sum of anglos around point Maire The Inside circle, are langto tangle DEA langle EAD 350"icire Vo.
a reguiar pentagon, eech internal angle $108 Naircs, and the central angle betvean $3e0 VairelS 72"Mcirs)).
point
$72"icirsS
StovartineJABjs and $xoverlineDDC5 are separated by two vertices and The sides corresponds $2 Vimmes 72"lrir 144"/aircf.
Since
and langia ane y 90"Neire V, the engll V Vangle DFA
angle
... Whht mousure of atinor are SADS?
What the meceure of witer are SAD5?
Thus, the measure of minor arc AD is the same as the central angle ADD, which is: c4rc
Hence, the measure of minor are AD is 180*)
Figure 6: Qualitative results of manually fixing the reasoning hallucination the reasoning chain and inference the refined answers. The corrected reasoning chain (right) can guide MILLMs to predict correct answers, and the original reasoning chain with hallucination (left) still results wrong answers.
FaaE Conneber betwmen Indiac maries Typee
Legae
Gabroaaeen Ceoees Famtelly
6210
Factandiyy
(a) O1
(b) DDDGmii---ftttsh-hhinnii
(c)
Seemini-2-flash
factallyy Speaa
2411
Spoti
Fastenlly
(d) Virgo-72B
(e) QVQ-72B-Preview
(e) @2wen2.5-VL-7
Figure 7: Pearson correlation regarding hallucination types from six most representative MLLMs. All models tend to represent a similar pattern.
D.2 Pearson Correlation of Hallucination Type Among Single Model
In addition to stating the overall pearson correlation coefficient in Fig. [] we also illustrate corre-
sponding correlation from six most representative models, i.e., O1 24 Gemini-2 2-f-2a2fffhhttt f-thhsht (56),
Gemini-2-flash (56), Virgo-72B [12.. QvQ-72B-Prest Peeevieee [57], and Qwen2.5-VL-7B (2] As shown in
Fig all models show similar correlation pattern, which is consistent with Fig. These results indicate the shared vulnerability in reasoning MI.LMs.
D.3 Training-free Methods in MIRAGE.
We also explore some training- free methods to verify the hallucination mitigation capabilities. Therefore, we evaluate prompt-based self-reflection [16] and visual inference chain [86]. As shown in Table 2 compared to (wee2.55VVL-71 both methods suffer performance degradation on both accuracy and L.H.. These results indicate that for models without insufficient reasoning capabilities and Ballucination--defendie abilities, introducing training- free methods does not help mitigate
27

---

reasoning hallucination. This observation inspires us to integrate CHI into CRFT-enhanced MLLMs rather than the base model.
D.4 Supervised fine-tuning (SFT) Methods in MIRAGE
Intuitively, reasoning hallucination could be solved by further supervised fine-tuning integrate the correct Hence, we select several SFT-based methods on 7B-level [72 67 71 and 72B-level 12
and evaluate MIRAGE. As shown in Table on 72B MILMM,, introducing SFT can lead to better accuracy and partly mitigate reasoning hallucinations. Nevertheless, on 7B MLLMS, only Mulberry surpasses the base model by 3.1 accuracy, while other methods do not lead performance improvement and hallucination mitigation. This contradiction may come from the model's capacity for model sizes. Larger models with more inherent knowledge may be easier to mitigate hallucination via external supervision, while smaller models usually struggle with SFT.
Table 16: Accuracy comparison of each question topic in MIRAGE.
Model
Algebra Arithmetic Geometry Scientiffc Spatial Statistical
Black Box MLLMs
Gemini-2- 2-aini---Flash--- Tlils-hhhiiiinggggg 56
56.1
66.3
53.7
33.3
41.4
26.1
55.0
47.6
01
2
50.9
64.1
60.0
37.7
42.5
37.8
51.9
49.7
Gemini-2-Flan
57.6
50.2
34.1
32.3
26.7
44.1
GPT-4o 23
38.8
40.2
29.5
24.6
28.0
42.2
47.2
35.0
Open-sourced
722 MLLMs
Qwen2.5-VL-72U Instruct
44.9
50.0
37.8
29.0
24.7
32.2
49.6
38.8
InternVL.-2.5-88B
31.1
38.0
31.4
21.7
21.0
27.2
40.3
29.6
Owen2-VL-72B-Instle
61
20.1
30.4
26.0
18.8
19.9
25.0
38.0
24.5
-rQ-ieB-Previcw [57
30.1
44.6
32.1
23.9
28.0
25.0
41.1
31.0
Virgo-72B
37.5
29.0
23.1
41.1
37.4
Open-sourced 7B MILMs
Owen2.5-VVL-7B-Ime
34.8
28.6
26.1
17.2
28.8
Qwwn2-VVL-7--Instruct 2
16.3
13.0
14.6
22.5
15.6
32.8
27.1
19.5
Qwen2.5-VL-7B. Instruct+
86
30.1
31.5
26.0
23.2
17.7
24.4
39.5
26.9
Owenn2.5-VL-7B-Innnnee Refllcctio
16
23.9
31.5
24.8
2.3
17.2
31.7
39.5
26.7
R1-OneVision-77B (5]
20.1
22.8
21.0
26.1
16.1
28.3
33.3
22.9
VL-77B
27.0
22.6
Intem 5
11.4
26.1
22.5
16.7
17.2
30.6
30.2
20.8
Llama-3,2 3.2- 2is-2--iioon IB 18
12.5
21.7
20.0
14.5
16.1
17.2
38.0
18.7
Llava-CoT-11B 67
12.1
14.1
19.7
10.9
16.1
20.0
31.0
17.4
Lo (Ours)
39.1
39.1
38.7
32.6
20.4
34.4
59.7
37.1
Open-sourced
MLLMs
Qwen2.5-V1.cc 3--VI-3BB-nnttmet
10.7
27.2
20.6
25.4
16.1
16.1
27.1
18.8
Phi-3.5-Instruct [I
4.1
16.3
14.0
15.2
14.5
13.9
21.7
12.9
Logos-3B (Ours)
27.3
38.0
31.4
24.6
17.7
27.2
48.0
29.4
D.5 In-depth Analysis of Logos in Mitigating Spatial Hallucinations
We conducted experiments to compare Logos-7B against other methods specifically on spatial hallucination. The results in Table 17 striking. While mitigating spatial deficits remains significant challenge, Logos is the only method that shows a slight improvement, whereas all other methods, including training-free and SFT-based approaches, actually degrade performance. The reasons are two-fold. Firstly, training- free methods (e.g.. Self- Self-Reflection These methods cannot introduce new knowledge. By forcing the base model, which already lacks spatial reasoning ability, to adhere to rigid prompting rules, they can amplify existing errors. Secondly, SFT-based methods (e.g., RI-One RI-OneVision) are often trained on datasets dominated by other domains (e.g., mathematics). This can cause the model to overfit to mathematical reasoning patterns at the expense of its already weak spatial capabilities. In contrast, our Logos avoids these pitfalls. The RL approach enhances reasoning without the overfitting risk of SFT, while our CHI component provides targeted hints that guide the model on difficult spatial problems. Therefore, Logos the only method avoiding performance degradation.
28

---

Table 17: Spatial hallucination error rate (%) comparison across methods. Lower better.
Model
Spatial Hallucination (%)
Owen2.5-VL-7B
33.4
*Reflection
37.4
+VIC
36.7
RI-OneVision-
35.6
Logos-7B
29.9
D.6 Effect of RL algorithms
As mentioned Sec. 5 the multiple sampling pipeline in the GRPO algorithm naturally aligned with our hallucination mitigation proposal, i.e. encouraging models to predict along the correct chain for correct answers. verify effect RL algorithms, we compare Logos-7B using GRPO and that using PPO 12 We remove all CHI stages and directly assess the effect of RI. As shown in Table TT Logos-7B using GRPO surpasses that using PPO by 5.8 and 2.6 on MIRAGE and Math Vista, respectively. These results verify our motivation and provide strong support for the Logos framework design. We also evaluate the effect of recently proposed DAPO 76). a specially-designed GRPO variant. Nevertheless, it does not lead to better performance both benchmarks. A possible explanation that the newly introduced constraints in DAPO lead to overfitting the model and restrict the final performance.
Table 18: LLMs Hallucination Score (LHS) comparison of each dimension in MIRAGE.
Model
Factual Logical Reasoning Conceptual Appropriateness
Overall
Black-Box MLLMs
Semini-2-Flashh Flinh-in 56
0,7182
0.7558
0.7689
0.7349
0.7372
0.7517
01 24
0.6054
0.6427
0.5793
0.6384
0.6306
0.6193
Gemini-2-Flash 56
0.6640
0.7053
0.7265
0.6862
0.7007
0.6882
GPT-4o 23
0,5811
0,6466
0.6777
0,6198
0.6404
0.6332
Open-sourced
MILMs
Qwen2.5-V1.-72B-mmee
2
0.6330
0.7464
0.7912
0.6793
0.7321
0.7233
Ln2e5VL-2..577BB S
0.5830
0.6441
0.6700
0.6088
0.6313
0.6377
Qwen2--11-77B--nssmnne
0.4665
0.5115
0.5339
0.4746
0.4774
0.4928
QvQ-72B-Prent Preview
S
0.5024
0.5495
0.5698
0.5368
0.5168
0.5717
Virgo-72B [2]
0.6094
0.6185
0.6437
0.6252
0.6328
Open-sourced 7>BB MLLMs
Qwen225-VI-77-Instuuu
0.5333
0.6201
0.6765
0.6130
0.5996
0.3512
0.3960
0.4120
0.3573
0.3519
0.3633
Qwem2.5-VL--B--Isstuc---
0.4600
0.4746
0.4336
0.4449
0.4261
0.4478
Qwen2.5-VLL7B-IIm
16
0,5658
0.6242
0.6008
0,5806
0.6117
0,5826
RI-OneVisionn77B [
0.4565
0.5227
0.5809
0.4822
0.5070
0.5098
Mulberry-Qwen--VV LL-BB
0.4545
0.4819
0.5070
0.4605
0,4660
0.4740
V 5
0,4515
0.4967
0,5317
0.4636
0.4757
0.4838
Llama-3.2-Visiona.
0.4014
0.4473
0.4741
0.4030
0.4066
0.4265
Llava-CoT-II IBB 67
0.4050
0.4417
0.4735
0.4116
0,4267
0.4267
Logos-7B (Ours)
0.5841
0.6533
0,7052
0.6233
0.6566
0.6568
Open-soureed
MLLMs
Qwen2.5-VL LL--5---VLLBB-sttrcce
0.3282
0.3593
0,3712
0.3279
0.3242
0.3422
Phi-3.5-Instruet
0.2983
0.3443
0.3459
0.3049
0.2968
0.3181
Logos-3B (Ours)
0.5486
0.5947
0.6411
0.5600
0.5757
0.5840
D.7
Effect of KL Divergence
Since we remove the KL-divergence term Logos training, to analyze the effect, we conduct an ablation study on the KL-divergence weight. As shown in Table 13 when gradually increasing the weight the KL-divergence term, the accuracy on both datasets is gradually decreased. When KL-divergence term is relatively large (e.g.. le-2), the accuracy Math Vista is even slightly lower than the base model (68.2). possible explanation that the distribution of reasoning MI.MM has non-negligible gap with the corresponding base models. To mitigate original reasoning hallucination
29

---

and bring inherent reasoning capabilities, one should disable the KL-divergence term to tolerate the distribution gap between models.
D.8 Effect of online reward filtration.
Next, we explore the effect of online reward filtration and report experimental results in Table 12 After integrating ORF into training, Logos-7B surpasses the counterpart by 2.9 MIRAGE and 2.7 on Math Vista. This improvement proves the effectiveness of ORF and points the future direction for more effective RL algorithms.
D.9 Effect of curriculum learning.
We also conduct experiments to verify the necessity of the CRFT stage. Specifically, we conduct vanilla RL training on all training data and ensure the number of training epochs is equal to the total CRFT. As shown in Table [4| even using longer training, one-stage RL still falls behind Logos-7B with CRFT by 1.6 on MIRAGE accuracy and 0.9 on Math Vista accuracy. These results indicate that, benefiting from multi-stage difficulty filtration, the learning efficiency of Logos is highly improved. Meanwhile, we explore the effect of the curriculum learning stage k Logos. As shown in Table [5] Logos-7B can easily obtain 37.1 on MIRAGE and 72.3 on Math Vista. Further increasing & only introduces marginal improvement. These results prove the effectiveness and efficiency of CRFT. Therefore, we select k to optimize Logos.
D.10 Performance on Multimodal Reasoning Datasets
To show the generalizability of Logos beyond MIRAGE, we evaluate Logos-7B on three general- purpose benchmarks: Mistaista. Math Vision, and Math Verse. As shown in Table 19 Logos-7B not only surpasses its strong base model but also outperforms concurrent state of the-art methods, achieving the highest average score. This shows that Logos to mitigating hallucinations also enhances reasoning abilities, leading to comparable performance on academic benchmarks.
Table 19: Performance comparison multimodal reasoning datasets.
Method
MathVista
Math Vision
Math
Avg
Qwen2.5-V1.n7
68.2
25.4
47.9
47.1
R1-One
64.1
19.7
46.4
46.8
Open
70.2
25.3
47.9
47.8
MM-EUREKA
73.0
26.9
50.3
50.0
Logos-7B (Ours)
72.3
29.8
52.5
51.5
D.11 Comparision with DAPO and ADARFT
Comparison with DAPO 76 The primary distinction lies in the policy gradient loss design. DAPO introduces several strategies (e-g., clip-higher, token-level loss) optimized for text-only math problems, which often feature very long reasoning chains. Our experiments found these strong regularizations were less effective and could lead to overfitting in the multimodal context, where reasoning chains typically more concise (see Appendix D.5). Logos employs simpler, more robust sample-level policy gradient loss that proved more effective for multimodal training, as shown Table 8.
Comparison with ADARFT S4 The key difference is the curriculum learning mechanism. ADARFT uses static offline difficulty annotation and greedy strategy select training batches. This approach does not guarantee that all data is utilized effectively. Logos employs more dynamic brbdi online- offline framework. ORF ensures each batch is filled with high-value samples, while offline difficulty thresholding stages the curriculum effectively. This hybrid approach maximizes data utility and leads to better performance. To provide direct empirical evidence, we compare Logos (w/o CHI) against these methods on MIRAGE and Math Logos demonstrates superior performance both.
30

---

Table 20: Performance comparison between DAPO, ADARFT, and Logos (w/o CHI) on MIRAGE and Math Vista.
Method
MIRAGE
Math
DAPO
30.0 34.0
69.6
ADARFT
70.1
Logos (w/o CHI)
35.7
71.9
D.12 The Quality of Automatic Annotation
We also assess the accuracy of automatically annotated reasoning chains in different phase, which is shown in Table 21] The accuracy of O3-mini initialized reasoning chains achieves 43.9. By incorporating DeepSeek-RR RI and guided by answers, the accuracy of refined reasoning chains achieves 73.7. The relative high accuracy of refined reasoning chains ensures that one reduce the human labor to correct the reasoning chains with reasoning hallucinations.
Annotation cost. Finally also concern the detailed annotation cost. By using the annotation method proposed in Sec. 3.2 the total cost is ~22S. And the total human working hour is 36 hours*person. We also estimate the annotation cost using O1, which is nearly 200$. And if all the questions are annotated by human experts, the total working hour nearly 200 hours*person. These results show the efficiency of our annotation method.
Table 21: The reasoning annotation accuracy in each phase.
Annotation Phase
Reasoning Chain Accuracy
03-mini (init)
43.9
+DeepSeek-RR R1 (refine w/ answer)
73.7
D.13 Human Double Checking for Metric Reliability and Consistency
To verify the reliability and consistency of evaluation metrics, the human double checking procedure is built on three pillars.
First is mitigating circularity. To prevent self-enhancement bias, we intentionally model version (gpt-4o- 20---0024---666 as our primary evaluator, ensuring it has not been trained on the models data it is evaluating. For open-source evaluators, we designed highly structured, specialized prompts (see Appendix) to constrain the evaluation and minimize potential bias.
older, isolated
Second is not rely solely on automated metrics. We conducted manual double-checking for our key metrics (factuality and LHS) representative models. The results in debloo22 between our automated scores and human expert judgment. Specifically, for factuality scores in the following table, the automated scores closely match the manually verified scores, confirming their accuracy.
verification to double-check the validity of LLM-as-a-judge. We do
strong alignment
Table 22: Comparison of automated and manual factuality scores for step and claim metrics.
Model
Step
Claim
Gemini-2-flan Semini-2-flashe (manual)
47.8
47.4
48.5
47.9
Lwvn2.5-VL-7B Dwen2.5-VL-77 (manual)
34.7
31.7
33.9
31.3
And for LHS, we analyze the absolute difference between automated and manual LHS scores. The results T2bel that the metric is highly stable, with over 75% samples having a negligible difference of less than O.
Besides, we further validate our framework by calculating the correlation between all metrics Table 5. The high correlation scores across the board indicate strong internal consistency and robustness.
31

---

Table 23: Absolute difference between automated and manual LHS scores at different percentiles. Over 75% of samples have negligible difference (<0.1).
Model
25%
50
75%
90%
-emini-2-flash
0.02
0.04
0.07
0.11
Qwen2.5-VL-7B
0.04
0.05
0.09
0.13
E
Dataset Examples
To clearly show the structure of MIRAGE, we provide detailed examples of MIRAGE. Fig.
12
shows
an example of geometry questions. Fig. 13 shows an example of algebraic questions. Fig.
shows
an example of arithmetic questions. Fig. 15 shows an example of scientific questions. Fig.
6
shows
an example of spatial questions. Fig. shows an example of logical questions. Fig. 18 shows an
example of statistical questions.
Summary of Topic-specific Hint. We also release the topic-specific hints used in MIRAGE and [9 The topic-specific hints include key concepts and basic rules regarding
Logos. As shown in [ig.[99 the question topies. Meanwhile, the classical reasoning process of corresponding question topics is
also included in the hints.
F More Qualitative Results
We also illustrate couple of outputs from some representative models, i.e., Qwen2.5-VL-77 Instruct, Geminn---ffash--tinning and our Logos-7B. As shown in the Fig. and We find that in this example, Qwen2.5-VI--B has consistent logic in the reasoning chain, but suffers from factual hallucination (only two 90 degree angles should be calculated). In contrast, Gemini-2- 2-flas--thinniing and Logos-7B correctly solve the question.
G Limitation
The limitation of this paper is two-fold. First, MIRAGE does not include multiple images or video gunseinn-answering problems. Hence, the hallucination from the temporal dimension and the hallucination regarding cross-image relations are not fully explored. Our primary goal with MIRAGE was first establish foundational and high-quality benchmark for reasoning chain hallucinations in the prevalent single-image context. This deliberate focus allows for deep and controlled analysis of core challenge in MLLMs. For scenarios that be adapted, concatenating multiple images into single composite image and annotating object indices, effectively converting multi mulli-image problem into a format compatible with the current benchmark. And second, the theoretical analysis of why MLLMs suffer from reasoning hallucination is still insufficient. These limitations motivate us to conduct more in-depth exploration the future.
propose a practical method of
Limitation the Logos's 's effectiveness across all identified hallucination types. The behavior stems from two distinct challenges:
Inherent Gaps in Base Models (spatial hallucination): Our analysis in Table and 2 reveals that even top-performing base models have fundamental weakness in spatial reasoning, Scaling model or data size does not significantly reduce this type of error. Since reinforce- ment learning in CRFT works by rewarding and reinforeing correct behaviors that the model can already sample, it cannot effectively teach a capability that is fundamentally absent. If the base model rarely generates correct spatial reasoning path, there few positive examples for Logos to amplify.
The Nature of RL vs. factual knowledge (factuality hallucination): For strong base models, the remaining factual errors are often subtle and stem from inherent biases in their pre- training data. The goal of Logos's RL framework to optimize the policy for generating correct reasoning chains, not to inject new factual knowledge. If all sampled reasoning paths from the base model contain the same ingrained factual error (e.g., a slightly incorrect constant), Logos lacks ground-truth signal to correct it.
32

---

**ask**
Given a reasoning chain, extract the key **stes*** (critical reasoning or calculation steps) and **laims** (correct intermediate results or derived facts). Follow these guidelines for each problem type:
1. ****eometry** Angles, lengths, area/volume formulas, congruencce/similariyy geometric theorems (e.g., Pythagorean theorem).
**Arithmettic** Critical numerical calculations (e.g., division, fractions) multi-step operations.
3. **Algebra**; Equations, inequalities, variable substitutiinns, function definitions, or computed variable values.
4. **Spatial Reasoning**: Spatial transformations (rotation, reflection), 3D object properties,
coordinate systems.
5. **Logical Patterns (Rule Identifiication)): Explicit rules (e.g., "add 3"), sequence relationships, or logical operators.
**iience (Physiis/Cheemiistt//Blll)))>> Scientific laws, formulas (e.g., F=ma), experimental steps, chemical reactions,, biological processes.
7. "Stttistics (Chart/Graph Problems)"* Data trends, statistical measures (mean, median), chart interpretations, probability rules.
*II the reasoning chain incomplete**, treat *"the answer [answer in the input]"* as both a step and claim.
*IIf the reasoning chain too long, output AT MOST most important (i.e., critical to solve the problem) steps and AT MOST 10 most important (i.e., critical to solve the problem) claims in the chain.
*FFormatting***
Wrap each step with <<step>..<<step>>.
-Wrap each claim with <cllamm>..</olaim>..
**Exmmples (omitted one-page style):**
Now, process the following input:
"Input: (cot) *Output:*
Figure 8: The evaluation prompt used to extract intermediate results.
Hence, these findings show a crucial insight: tackling spatial and factual hallucinations may require architectural innovations
data paradigms beyond the scope of post-training methods. These results also suggest that future work might explore better pertaining data, spatial-enhanced network design and advanced framework to mitigate such hallucinations.
H Broader Impact
The broader impact of this paper lies in advancing the reliability and accuracy of multimodal large language models (MLLMs) by systematically isolating and evaluating reasoning hallucinations. MIRAGE offers targeted benchmark for diagnosing and mitigating reasoning errors, which is essential for applications fields like autonomous systems, medical imaging, and scientific discovery, where accurate multimodal reasoning is critical. By revealing key weaknesses in current MLLMs, such as their struggles with complex spatial reasoning, our work encourages the development of more robust, transparent, and context-aware AI systems, ultimately promoting safer and more trustworthy Al deployment.
33

---

**askk:**
THINK STEP BY STEP and Analyze three components:
1. **Question** (provide the basic judge background information)
2. "Moodel-genrrated steps** (between <steps> tags)
*Grronndtruth steps** (between <steps>" tags)
*Output answer, then provide additional analyiss to enhance the accuracy** **Important No
1. The number of decision <pred_ mateh....</ped > match> should be the exactly same with steps the prediction.
2. The number of decision <gt match?...//t! match should be the exactly same with steps in the ground-truth.
(more detailed rules omitted one-page style).... ...
#*# Output Format: xml <pred_match=MATTHH
Bat.AA/AATCC.RRASOMMALLLLOOOOOOS match>
<gt maachh-MATCCC,CCNFLLLT....C </t.ctthhttthh>> <preessoonooxX</preessio>> <reeaal>o.xK<<recall>> <f1>0.xx</f1>
### Input:
Question: (question)
Prediction from Model:
<steps>
(pred_steps)
</steps>
Ground-Truth:
<steps>
[gt_ ggt_steps)
</steps>
*Output:**
Figure The evaluation prompt used for factuality assessment (e.g., Fsue).
34

---

"Task Requirement:*
You given the output of multi-modie reasoning model, including chain-of- ofthhugh (CoT) and input consisting of question and an accompanying image (represented by the placeholder <image>). You are also provided with a human-verified reference chain-of-thouge (Ref-CoT) that correctly answers the question. Your task is to analyze the predicted CoT and determine whether it contains hallucinations based comparison the Ref-CoT and the input.
*Typees of Hallucinations:**
1. **Spatial Hallucination Misunderstanding the visual structure, shape, angle, or spatial configuration
based on the image.
**Logical Hallucination**: Flawed or missing reasoning steps, incorrect logical deductions, invalid inferences.
3. **Factuality Hallucination**: Statements that contradict known math facts or the image data.
**Cnntett Hallucination Inconsistencies between steps contradictions with the predicted *Faaricatioon Hallucination Reasoning steps that are not derived from earlier steps, image data, question context.
Please output JSON result with the following fields, enclosed within '<resut>>' and </result>> tags:
"result" Either "CONFIRM" ""ALLUCINATION" "reason" " brief justification for your judgment
Few-shot Examples for Reference:**
Examples omitted one-page style
*Now, for the following input, analyze the reasoning chain and provide your final evaluation according to the schema above.**
*Input=**
"Image**: <image>
*Qusstoon*: (question)
** Predicted CoT** (pred_cot)
Referencce Co (ref_cott
*Output **
<result>
{
[Overall':
"result": "CONFIRM" "HALLUCINATION;
"reason":
"Overall explanation here...") )
"Spatial": ("result:: "CONFIRM" or "HALLUCINATION"
"reason": "Detat"eeddeieeeeaaaee explanttio for Spatial hallucination..."
'Logical": :"ggicatt""reeuutt""""C ""ONE""" "HALUCINATTION"""
for haalicaaaaallliiiattii....
"HALLUCINATION"
"reason": "DataaaeeDDDtaileeeeeepaaaaannn for Factuality hallucination..."
"Context": (rrntttr"(Teeuutt:CCCCCNFF or "ALLUUCINTTO:::
"reason": "DrtteneeDDealleeeeeaaaaanniiii for Context hallucination..." "}
("aricattion":""rrsuttt""" CONFIRMM "HALLUCINATION;
"reason": "Detailed explanation for Fabrication hallucination...)
</result>
**Utput**
Figure 10: The evaluation prompt used to detect hallucination types in reasoning chains.
35

---

### Task Requirement:
You are expert reasoning chain evaluator. Your job is model-gemerated Chain of Thought (CoT)
based on six key dimensions that assess its correctness, coherence, and hallucination risk. You provided with:
A problem statement.
model-gonerated Chain of Thought (CoT).
One more reference CoTs (these valid, possibly diverse ground-truth reasoning paths). must evaluate the model's CoT based the following dimensions. For each dimension, assign score between 5, where best. Also provide a concise explanation each score.
Evaluation Dimensions Scoring Criteria:
**1, Factual Accuracy**
5 All stated facts, definitions, and formulas are correct.
4 Minor factual inaccuracies not critical to the reasoning.
3 At least one notable factual mistake, but the main conclusion still follows.
2 Several factual inaccuracies that impact the reasoning chain.
-1 1 Major factual errors or invented knowledge that invalidate the reasoning.
*2 Logical Consistency**
5 Every step logically follows from the previous one; contradictions.
4= Mostly consistent with minor leap or imprecision.
3 Some unclear unjustified transitions between steps.
Multiple reasoning steps logically invalid.
Reasoning incoherent selffccontradictory.
**3. Reasoning Completeness**
Fully complete; key steps included.
Minor omissions; conclusion still follows.
Missing one important step; slightly weakens the argument.
2 Missing multiple key steps; reasoning is hard to follow.
Severely incomplete; the reasoning fails to connect to the answer.
Conceptual Ressoning Accuracy**
Correct use of all key mathematical, spatial, or logical concepts.
4 Small misundesttanding or imprecision.
3 = Misuse of one care concept that weakens reasoning.
2 Multiple concept-level misapplications.
1 = Misuse or hallucination of core principles definitions.
*5.. Strategy Appropriateness*
5 The chosen strategy is valid and optimal or close optimal.
4 The strategy valid but suboptimal.
Strategy is unconventional but valid; minor concerns.
2 The strategy logically flawed poorly suited.
The approach is fundamentally invalid the problem.
Output Format:
Return your evaluation valid JSON object
following format (omitted fit onepage style):
Few-shot Evaluation Example (omitted onepage style):
Now please evaluate the following CoT:
**Problem:**
(prompt)
**Model CoT:**
(model_cot)
**Reference CoT:*
(reference cot)
### Important Notes:
1. Be strict and honest during scoring high scores should be reserved for flawless or near-flawless reasoning.
Pay special attention when final enswers differ between model and reference CoTs.
Use the negative example as reference for identifying serious flaws reasoning.
DO NOT give high scores unless the CoT fully deserves it across all dimensions.
Return your scores and explanations in the JSON format shown above.
**Outpu***
Figure 11: The evaluation prompt used for LLMs hallucination score extraction.
36

---

Geometry
Question:
Three small equilateral triangles of the same size are cut from the corners of a shown. The of the
larger equilateral triangle with sides $6 \mathrm(~cmi$ perimeters of the three small triangles is equal to the perimeter of the remaining hexagon. What the side-length one the small triangles? <image1>
Choices:
$1 \mathem(=om35
$1.2 \mathrm(-cm)s D: $1.5 \mathrm(-cm)$
C: $1.25 \mathrm(-em15
$2 \mathrm(-cm)$
Reasoning Chains:
Intermediate Results:
**Understtodding Prablem:**
<step=Understtnd the problem: equilateral triangle with three smaller equilateral triongles forming hexagon-/step <claim>Sum of perimaters small triangles equals perimeter of hexagon//clamm
large equilateral triangle (side length 6 cm) has three smaller equilateral triangles from corners, forming hexagon.
The sum of the perimeters of the three small triangles
equals perimeter
hexagon.
Gool: Find the side length \(x () of the small triangles.
<step:Calculate perimeter each small triangle: 3x</step>
*"Perimeeer the Small Triangles:**
<claim> Total perimeter for three small triangles: 9xv/clamm>
Each small triangle has perimeter of \( 1(3x1 V). Totaal for three triangles: M Atimes 9x 1).
<step:Determine perimeter hexagon: original side segments (6 2x) and new sides (x)s/step>
**Perimeter of the Hexagon:**
Each original
large triangle contributes
segment of \( 2x \) (after removing V( x from both ends). adds
<claimsHexagon perimeter: (* 2x)
side of the small triongle (length \(x V) to hexagon.
Each corner
18 3x</claim>
Total perimeter of the hexagon:
<step>Set perimeters equal: 9x 18 3x</stepi <claim-Equation:
\times
\times 18 6x 3x 18 3x.1]
3x</claim>
4. *"Equating Perimeters:**
the total perimeter of the small triangles equal the
<step= for 12x 18 222 1.5 ems/step>
hexagon perimeter:
\[9x = 18 3x1]
<claimxx 1.5 cm</claim>
Solling for X(x V):
\[12x : 18 Yquad \Rightarrow Yquad x \frac(188(12)- 1.5 N, (text(cm).]]
<step-the answer is 1.5 cm</stepa <claim-the answer is 1,5 cm</ claim>
Description:
Hints:
green graphic right circular cone with sphere inscribed inside it. The sphere is approximately at center touching curved surface single point. This model is essential for understanding the spatial relationship between the sphere and the particularly focusing their radii, The top and bottom of the truncation (where the large meets the curved surface) should be the key focal points in solving the problem, as the question is about the ratio of these radii given certain volume relationships. The sphere inscribed within the cone suggests that its diameter is equal the height of the truncated part of the specific ic relationships. There are no labels or measurements provided, which reinforces the need to rely visuol proportions nather than exact figures. However, the vivid illustration aids visualizing solving the problem by providing clear geometric shopes necessary for solving the question.
The image
sooopicspecific__em hint:*** To solve the geometry problem, textual data: cross-verify labels, lengths, angles, and relationships, prioritizing explicit textual details over visual estimates. Identify key components (objective, given data) and apply relevant theorems (e.g.. Pythagorean theorem, circle properties. coordinate formulas). Break computations into logical steps, using auxiliary lines algebraic variables needed. Verify unit consistency, answer reasonableness, and tross-check via alternative methods. multiple-choice questions, employ elimination backsolving. Avoid assumptions about scale formula misuse: prioritize textual precision and rigorous validation to mitigate errors.
systematicaly integrate
which could help deriving
"* coouestionsspcifii..m hint:*> Focus on calculating the volumes of the sphere and volume formula for sphere M V \frac(4)(3) \pi r"3 V) and for a truncated cone X( V \frac(1{(3) \pi (R (R_1"2 Find the relationship between these volumes based on the given condition to salve for the ratio of the radii \(R \) and \(R_2 V).
Answer: D
Figure 12: The example of geometry question in MIRAGE.
37

---

Algebraic
Question:
Reasoning Chains:
**Problem Analysis**
State the inequality that describes the region drown the number plane. <image>
The question involves interpreting a graphical region defined by inequality. The image description specifies a shaded area bounded by dashed line (indicating a strict inequality) and references points (- 3, -10) and (5. 10). The goal is to determine the inequality describing this region.
**ey Observations"*
1. BoBonddryy Line**: The dashed boundary suggests
linear inequality (e.g., M(y < + or V(y mx eV)). 2. ** **ritiica Points**; The points (-3, -10) and (5, 10) likely relate to the boundary line or the shaded region.
Intermediate Results:
<STepIIdentify the boundary line linear inequality due to the dashed linev/step <claim-Boundary line suggests linear inequality (eg., or y mx c)//clamm>
3. **xes Interaction"*: The region bounded by the
line and the oxes, narrowing the inequality's form.
*DDrvvving the Boundary Line
<step=Verify boundary line equation y 2x 6 using interceptsr/step
provided answer Vyy 2x 61) implies the boundary line is My = 2x 61). To verify:
<claim>At 0, (y-intercept)//clim>> <claimoAt y 0, x (x-intercept)//llimm
**Intercepts**
At \(x OV), My 61) (y-interceept)
<step-Check the shading direction of inequality/steep <claim-Inequality < 2x 6 implies shading below the lines/claim>
At A(y = OV), Xxx -3V) (x-intercept).
Since the inequality is Myy 2x 61). the region **below** the line is shaded.
*Shadding Direction**;
<step-Test given points
-10) and 10) the
**Reconciling with Given Points**
inequality//steq>
<claim>For (-3, -10) -10 2(-3) > 100 </claim>
The points --0) and (5, 10) not the line M(y : 2x 61) (e.g.. substituting \(x = gives Myy 16\), not 10). However, they may lie within the shaded
<claim>For (5, 10): 10 2(5) > 10 16 </claims
cstepiConclude correct inequality/steep. <claim-Inequalityy 2x correctly describes the shaded regior//clamm>
region:
For \((-3, 110]): M( -10 2(-3) 6 \rightarrow -10
0v)
For M(5, 10)): V(10 2(5) + \rightarrow 10 16V)
<steptthe onswer y 2x 6</step> <claim>the answer is 2x 6x/claim>
** **Conclusion**
The inequality Myy 2x + 61) correctly describes the region below the dashed line Myy = 2x 6V), consistent with the graph description. The initial assumption circle incorrect due
Hints:
""tapic_speccffii specific *i
misinterpreting the
To solve the algebraic prablem, focus extracting and define varidbles,
boundary as curved rather than linear.
interpneting textual information expressions, and equations. Identify the objective (e... solve for variable, simplify an expression, evaluate on equation) and distinguish known values, unknowns, and their relationships, Translate word problems formal digebraic expressions. symbolic image
** **Final Answer** \boxedly 2x 6)
descriptions is present, assess whether provides meaningful cues or serves as a placeholder: prioritize textual clarity when visual content is ambiguous or abstract. Apply appropriate algebraic techniques (eg. substitution, factoring, combining like terms, solving systems) clear logical steps. Clearly define each variable and explain the tronsformations at each stage. Check your final answer for consistency by substituting it back into the original constroints, For multiple- choice questions, alimination, estimation,
Description:
The image shows a graph with axes labeled as x and y, along with shaded region defined by dashed boundary. The graph features two points plotted at coordinates (-3, -10) and (5, 10). This demonstrates that we are looking at a region involving both positive negative values on the and y axes, bounded on the left side by the line segment and on the oxes themselves. The relevant points (-3, -10) and (5, 10) should be noted for understanding the area being described. The boundaries help define the inequality that describes this region.
bock-substinution. Avaid misinterpreting irnelevant visual elements oversimplifying complex expressions prioritize rigorous eediin algebraic reasoning grounded in the problem statement.
** "*Question_ _Qquestio_speciffic specificc_iint:*** hint:** Identify the slope and y- using the points where
intercept of the dashed intersects the axes or notable grid points. Formulate the linear equation V(y mx + c\). Datermine which side of the line the shaded region falls into to establish the inequality, indicated by shading.
considering the
Answer: $y<2x+6$
Figure 13: The example of algebraic question in MIRAGE.
38

---

Arithmetic
Question:
Juan organizes stomps his collection by country the decade which they were issued. prices poid them at stomp shop were: Brazil and Frence, 6 cents each, Peru cents each, and Spoin cents each. (Brazil and Peru are South American countries and France and Spain are in Europe.) <image>
Number of Stampe by Docade
Country
68e
[0e
70s
80s
Braail
4
7
.2
8
In dollors and cents, how much did South stompes issued before him?
Prance
4
22
13
Pern
8
6
10
A: 51textdoller 0.405
5textdolllr 1,065
Spain
3
9
-3
9
C. Sttextdollar 1,805 D: $1textdollar 2.38$
Juan's Stamp Collection
E: >teextdollar 2.64$
Reasoning Chains:
Intermediate Results:
<step-Identify relevant countries and decades: Brazil (6 4 /stomp) and Peru $ /stamp), focusing on 1950s and 1960s=/step>
S*ttep Identify Relevant Countries and Decades** South American countries are Brazil (6 /stamp) and Peru (4 /stamp). "Before the '70s" includes stamps from the 1950s and 1960s.
*claimmRelevant countries: Brozil and Peru: Decades: 1950s and 1960s</claim>
S*Sp Extract Stamp Counts from the Table**
ExteptExtract stomp counts from table: Brazil stamps) and Peru stamps) for the specified decadesv/step
Assuming the table provided the following counts (based the correct answer):
"*"Brazil**; stomps (1950s 1960s)
<claim/8razil: stomps: Peru: 10 stampsv/claim
**Per"" 10 stomps (1950s 1960s)
<step-Cacculate total cost per country: Brazil (11 4) and Peru (10 4//step "claim-Brazil:: 66 cents: Peru: 40 centsv/claim>
**tep 3: Calculate Total Cost per Country** *BBrozii* V( 11 \text(stomps) Vtimes \, \text(cents) 66 \, \text(cents) V) **Peru"": 1(
1, \text(stamps) \times 4 \, \text(cents)- 40 \ \text(cents)
sum costs convert to dollors: 66 cents 40 cents 106 cents $1.06//ttpp
<claim- Total cost: $1.06</claim>
S*Step 4: Sum Costs and Convert to Dollars* Total cost: V( 66 \text(cents) 40 \text(cents) 106 \teet(cntss)= = \textdollar 1,06 V).
<stepithe answer B</step>
<claimtthe answer is Be/claim>
Description:
Hints:
The image shows table titled 'Number of Stamps by becade" that outlines the number of stamps Juan has by country and by the decade in which they
Propic_specifichiin
To solve the arithmetic problem, systematically extract quantitative information from both visual and textual sources. Identify objects, patterns, counts, that can be translated into numerical values. Determine the objective (e.g.. total value, missing term, comparison) and map visual elements algebraic representations or equations. Cross- check for background elements (e... overlapping objects, symmetry. groupings, or hidden features) that may influence the count or operation. Apply basic arithmetic operations (addition, subtraction, multiplication, division) in clear, logical steps, and ensure consistency with visual evidence. Clearly state the inferred numerical value each symbol or object and how they relate through equations. For multiple-choice questions, use elimination or plug-in strategies. Avoid visual misinterpretation similarity: prioritize logical modeling, numerical precision, and verification through reverse calculation or consistency checks.
issued. The columns represent the decades from the 1950s to the 1980s. The rows list the countries deemed relevant to the problem: Brazil, Fronce, Peru, and Spain. The table is essential for solving the problem as it provides the necessory data about the number of stomps Juan has for each decade and country. The relevant part of the table corresponds to the South American countries (Brazil, Peru) issued before the '70s, which the '50s, '60s. and 70s decades, and their respective countries.
spatial relationships
Each cell in the table represents specific infarmation: the count of stamps issued before the '70s particular country. Juan's 's cost the stamps can be computed by multiplying the number of stamps from South American countries (Brazil and Peru) purchased before the '70s by their respective prices (6 cents for Brazil and 4 cents for Peru).
overreliance
** **Question_speee specific *iiffic__iit****** Focus the "Brazil" and "Peru" rows for stamps from the 50s and 60s. Multiply the number of stamps by their respective costs (Brazil: 6 cents, Peru: cents), sum the total cosT, and convert dollars and cents.
Answer: B
Figure 14: The example of arithmetic question in MIRAGE.
39

---

Scientific
Question:
X
20 cm
<image> A uniform meterstick is balanced its midpoint with several forces applied shown below. the stick is in equilibrium, the magnitude of the force X in newtons (N)
30 cm
40 cm
A. 250
150
300 N F. 100 N
b. 200
400
200
6. 500
H. 600 N
I. 50 N
J. 350
300 N
Reasoning Chains:
Intermediate Results:
To solve for X, we need to use the principle of moments (torque equilibrium). The sum of moments (torques) about any point should be zero for equilibrium.
<step-Identify forces and their distances from the midpoint: N (30 cm left), N (40 right), (20 right)//step <claim-Forces and distances: 300 (30 cm left), 200 (40 cm right), X (20 cm right)/cclim>>
We'll take moments about the midpoint the stick. The distonces of the forces from the midpoint are:
The 300 N force is 30 cm to the left of the midpoint.
The 200 force is 40 cm to the right
midpoint.
<steplCalculate torque due 300
force:
The force is 20
the right the midpoint.
30
9000 N
cm
(counterclockwisee///stt
The moment coused by force given by the force multiplied will set up the
<claim> Torque_t= I= 9000 cms/claim>
distonce from the pivot. In this equation for the clackwise and counterclockwise torques.
cstep:alculatee torque due to 200 N force: 200 40 8000 cm(clockwise)//stteo <claim Toiquee- 8000 N cm </claim>
The torque due to the 300 farce will be: Torque = 300 x 30 9000 N cm
counterclockwise moment. The torque due to the 200 force will be: Torque_ 200 40 8000 N cm
<step:Calculate torque due force: 20 = 20X N cm (clockwise)</step. <claim> Torque_3= 20X N cm </claim>
This clockwise moment. The torque due to the force will be:
Sttep Torque equilibrium equation: 9000 8000 20Xx/step> <claim>Equation: 9000 8000 20Xx/claim>
Torque_3 3 X 20 20X N cm
clockwise moment.
For equilibrium, the of torques both directions must be
equal:
<step= for 9000-8000 20X 1000 20X X N:/step <claim-X 50 Nk/claim>
Torque_1 Torque_2 Torque_3
Substitute the values:
9000 = 8000 20X
Solving for
<step=the answer 50 N:/step> <claim>the answer is 50 N:/clamp
9000 8000 20X
1000 20X
= 1000
Descriptio
Thus, the magnitude of force X is 50 N.
The diágrom shows a uniform meterstick balanced at its midpoint (50 em mark) with three forces acting on A 300 downward cm mark (30 cm to the left of the fulcrum), and 200 N downward force is applied at the 90 cm mark (40
Hints:
force applied at
"*tapic__pecific___ specific_hinttee *M To solve scientific problem, identify objectives, key variables, and relationships. Distinguish knowns from unknowns, use diagrams and formulas effectively, and apply relevant principles accurately. Break complex steps into parts, ensure unit consistency, and check result reasenableness. For multiple-choice questions, use strategies like elimination or back-checking, rigor throughout.
**topic_
fulcrum). An unknown upward force, labeled X, acts at the 30 cm mark (20 cm to the left of the fulcrum). The system meaning the sum of clockwise and counterclackwise torques about the fulcrum is
the right of
equilibrium,
common errors and maintain clarity
** **Qusstoon__ "QQustiio_ssecificc midpoint on each side of the meterstick Use the formula torque force distance (from the pivot), ensuring that counterclockwise clockwise torques are balanced. Consider the distances 20 cm, 30 and 40 cm for each relevant force.
** hint:**
Determine torques around
Answer: I
Figure 15: The example of scientific question in MIRAGE.
40

---

Spatial
Question:
<image> Please determine whether the spatial relationships described the captions below are correct based on image.
Caption: The bench in front of the train.
Correct
Error
Intermediate Results:
<step>Analyze the spatial relationship between the train and bench in the imagex/steps <claim-Thh train closer the comera and situated along the trockr/claim>
<step>Determine the pasition of bench relative to the
Reasoning Chains:
trains/step>
the image, train cleanly closer to the camera situated along the track, while the bench is further back the platform. Therefore, the bench is not positioned in front of the train. It located behind the train relative the viewpoint the image. This suggests that the spatial relationship described in the caption incorrect.
<claim- The bench further back on the platform, behind train relative to the viewpoints/claim
Evaepatval accuracy of the caption's claim</step>
<claim> coption's claim that the bench front incorrects/cloimp
train
Hints:
<step-the answer is Be/step> <claimythe answer is B</claim>
"rrepic_speciffc.. hint:** To solve the spatial reasoning problem, carefully interpret both textual and visual inputs to understand object attributes (e.g.. shape, color, size) and spatial relations (e.g., left/right, behind/in front, inside/above/bellw)) Systematically extract the objective (e.g., identify, compare, count, color) and all given relational facts. Construct mental or symbolic spatial map
Descriptio
interrelations. Use consistent reference points (e.g. camera view, object centers) and verify directional logic step-by-step. Pay attention to subtle qualifiers like "exactly," "same," or "eearett"" complex scenes, decompose multi-object relationships into pairwise comparisons. Avaid assumptions about 3D depth without explicit cues. Rigorously validate inferred positions or relations using available evidence. multiple-choice scenarios, use elimination and contradiction checks to isolate the most consistent answer.
object positions
iftlage shows a train station with train passing through. The foreground shows portion of the platfarm with a track, and there benches visible along the platfarm. The train is moving the direction
the camera. The coption claims that the bench is in front of the train. We need to evaluate whether this spatial relationship is accurate based the image.
**Question_ specificc_hittieeeeee hint:** Identify position bench relation the train. Check if the bench is positioned in front of the train's direction of movement or the front end.
Answer: B
Figure 16: The example of spatial question in MIRAGE.
41

---

Logical
1
Question:
fulcrum), <image> From the given four options, select the one that can form 9x2 block combination as shown the figure with the given parts ( (2, 3) and &.
Reasoning Chains:
Intermediate Results:
The problem involves finding the length of side M. BC V) trapezoid
<step:Apply trapezoid area formula: (b1 62)/2 height-/step>
M( (BCD \) with given area, altitude, and lengths of sides V AB \) and M CD
Here's the step-by-step solution:
<claim>164 (b1 62)/2 b1
1. "TTappezoid Area Formula**;
b2 = 41 cm</claim>
The area of a trapezoid is given by:
\[\text(Area)- \ffro((b_] b_2))(2) \times \text(height)V]
<step-Identify non-parollel sides AB and CD</step
Here. V b_1 \) and M b_2 V) are the lengths of the two parallel bases.
The given area is V( 164 \, \text(cm) 2 \), and the height (altitude) is M
<claim>AB cms/claim
cm. CD
(text(cm) Plugging b_2]22) Atimes Vimplies b_1 b_2 =\rrac[164
\[164 \frrc((b__ br_ec(b_11__2)x
\times 2)(8) = 41 >Vtex(ccm)]
<step:Colculate projections of legs CD-/step> sclaimoorriznntl segment for /(10 8²)= 6 cms/claim> sclaimoHHariznnto segment for cD S(17* 8)*)= cm</claim>
horizontal
"*Identify Bases**:
and
V( AB \, Mtext(cm) \) and M CD 17 V. \text(em) \) are the
parallel sides (legs). Let AD \) and \( BC \) be the two bases. Thus:
[AD BC 41 \ttxt(cm])]
3. "Horizontaal Projections of Legs"*:
When dropping perpendiculars from M( \) and M 1) BC 1).
forming right triangles:
For \BBB \):
cstepCompuue total horizontal difference: 15 cms/step> <claimBBC AD 21</claim>
M1textt(arrizonal segment) = \Aqrt(AB"2
\text(height)"2)
\sqri(10"2 8"2) \sgrt(36) \text(cm))]
For V( CD V):
M[textt(Harizontll segment) = \sqrt(CD>2 -\text(height)"2)
Substitubt AD BC AD 21</step> <claim=BC = (41 BC) + 21 - 28C
into
20r"t(1"2-8822) \sqrt(225) 15 v, \text(cmm)] 4. **Relate #ases**:
The total horizental difference between the bases V( 6 15 21 \. (text(cm) V). Thus:
BC /
BC AD 21V] Substituting V( AD
<step answer is Cr/step- <claim>the answer Cx/claim>
\) into equation:
ABCC (41 BC) 21 Vimplies 2BC = 62 Vimplies BC 31 [ttet(cm))]
Final Answer**; \(1boxed[C)))
Descriptio
The diagram shows uniform meterstick balanced at its midpoint (50 mark) with three forces acting on it. A 300 N downward force applied the cm mark (30 to the left and 200 N downward force is applied at the cm mark (40 cm to the right of the fulcrum). An unknown upward force, labeled x. cm mark (20 cm the left of the fulcrum). The system is in equilibrium, meaning clockwise and
Hints:
**oopic_ "*topic_specific___ hint:** *in To solve visual logic problem, analyze image grid systematically by comparing visual components across rows, columns, or diogonals. Identify changes in shope, position, count, color, size, or orientation. Look consistent rules transformations such rotation, reflection, addition, subtraction, XOR, pattern shifts. Describe each pattern evolution step-by-step using symbolic relational expressions (e.g.. "Image C Image A XOR Image B". "Right Left rotated 90* clockwise"), Clearly state the inferred rule and apply it to missing or target positions. When multiple rules are possible, list that best fits all visible patterns. For multiple-choice questions, eliminate inconsistent options by checking each against the rule. Avoid relying visual intuition alone-prioritize rule-based, reasoning, pattern consistency, and explicit justification for your
acts
compare them, selecting the one
the sum counterclockwise torques about the fulcrum is zero.
** "QQustiion__ setertion_secciic____. hint: **
Identify the number arrangement of cubes needed recreate the 9x2 block from the smaller parts (1) (2, (3) and
Answer: C
Compare each option (A, B, C, b) with the target shape to find a matching configuration.
Figure 17: The example of logical question in MIRAGE.
42

---

Statistical
COLOR SURVEY
F 60
Question:
<image> The bar graph shows the results of a survey on color preferences. What percent preferned blus?
40
A: $201 %
B: $241%
20
$301%6 %
D: $361%
$421%6
Reasoning Chains:
Intermediate Results:
To determine the percentage of people who preferred blue in the survey, need to analyze the bar graph and calculate the proportion of frequency for relative frequency.
<step-Identiff the frequencies from the bar graph:/step' sclaim>Red: -50, Blue: Brown: ~60,
total
Pink: -40, Green: ~40:/claim>
### Step Identify the frequencies from the bar graph
calepiCalculate the total frequency//step <claim=50 60 60 40 40 250m/claim>
The bar graph shows the following frequencies for each color: Red: -50 -40
Blue: -60
Brown: -60
Pink: -40 Green:
sstep=Find the frequency for blues/step- <claim-Frequency for blue is approximately 60s/claim>
### Step 2: Calculate the total frequency Add up the frequencies of all the colors:
\50
40 40 250V]
cstepCalculate the percentage for blues/steps <claimsPecenntage for blue (60/250) 100 24%+/claim
So, the total number of responses is 250.
### Step 3:
Find frequency blue
From the graph, the frequency for blue approximately 60.
sstep:Match with the given options</steps <claim>The The calculated percentage is 24%, which matches option B</claim
### Step Calculate percentage for The percentage calculated as:
Mftext(Percentoge for blue) \left( Ifreeuteext(Frequunnyy
of blue]] Textt(Tolll frequency)) \right) Atimes 100N]
<step- onswer 24%x/step <claimythe answer is B: 24%+/claim
Substitute the values: I[texxtferceettge for blue) Meeft( \frac(60X250) \night) Atimes 100 0.24 Vtimes
24\%A]
Descriptio
### Step 5: Match with the given options
The irtáge features bar graph titled "COLOR SURVEY." The horizontal axis lists different color options (Green, Red, Brown, Pink, Blue), and the vertical axis represents frequency, labeled "Frequency." The height each bar represents the frequency of each color mentioned, For instance, the Red color has reaching up to on frequency scale, and the Blue color has bar that reaches 55 on the same scale, This bar graph necessary to solve the problem because visually represents the preferences for each color, and these preferences are used to calculate the percentage that preferred Blue.
The calculated percentage is 24%, which matches option
Hints:
"*topic_ trtopccsspecific_hiie
To solve the statistical problem, carefully examine all visual representations (e.g., bar charts, line graphs, tables, pie charts) and extract precise numerical values lobels. Identify the objective (e.g.. compute, compare, interpret, predict) and determine the relevant data segments. Cross-check axes, scales, avoid misreading quantities. Look for
units, and legends patterns, trends, distributions, and outliers across categories time. Apply appropriate statistical operations (e.g., mean, median, mode, range, percentage change, probability) based strictly on the data provided. For compound questions, decompose tasks and perform step-by-step computations. Use estimation, elimination, or reverse-checking in multiple-choice settings. Avoid assumptions
To further break down the bar groph, each color has an equivalent label at the base of its correspanding
the horizontal color preferences are ordered from left to right as Red (approx. 50). Brown (apprax. 55), Blue (approx. 60), Pink (approx. 25). and Green (approx. 30). The frequencies are annotated with individual bars above each color.
The
grounded the visual/textual evidence. Prioritize accuracy in data extraction, clarity method, and logical coherence in interpreting results.
** **Question_speee cecoo__spcfficccinttte Identify the height of the bar labeled "BLUE" to find the frequency. Then calculate the percentage based total frequency sum colors.
Answer: I
Figure 18: The example of statistical question in MIRAGE.
43

---

Algebralc: To solve the algebraic problem, focus extracting and interpreting textual information define variables, expressions, and equations. Identify the objective (e.g.. solve for variable, simplify expression, evaluate equation) and distinguish known values, unknowns, and their relationships. Translate word problems symbolic descriptions into formal algebraic expressions. If image is present, assess whether it provides meaningful cues serves a placeholder, prioritize textual clarity when visual content is ambiguous or abstract. Apply appropriate algebraic techniques (e.g., substitution, factoring, combining like terms, solving systems) in clear logical steps. Clearly define each variable and explain the transformations at each stage.
solve geometry problem, systematically integrate visual and textual data: cross-verify labels, lengths, angles, and relationships, priaritizing explicit textual details visual estimates. Identify key components (objective, given data) apply relevant theorems (e.g., Pythagorean theorem, circle properties, coordinate formulas). Break computations into logical steps, using auxiliary lines algebraic variables needed. Verify unit consistency, answer reasonableness, and cross-check via alternatiee methods.
Geometry:
Arithmetic: To solve the arithmetic problem, systematically extract quantitative information from both visual and textual sources. Identify objects, patterns, counts, and spatial relationships that can be translated into numerical values. Determine the objective and map visual elements to algebraic representations or equations. Cross-check for background elements (e.g., overlapping objects, symmetry, groupings, or hidden features) that may influence the count or operation. Apply basic arithmetic operations (addition, subtraction, multiplication, division) in clear, logical steps, and ensure consistency with visual evidence. Clearly state the inferred numerical value of each symbol or object and how they relate through equations.
To solve the scientific problem, carefully extract the objective and identify key entities, variables, and relationships described in the prompt. Distinguish between known quantities, assumptions, and what needs to be derived. Integrate textual data with diagrams formulas where provided, and prioritize explicit numerical or symbolic information over vague intuitions. Apply relevant scientific principles (e.g., Newton's laws, conservation laws, reaction equations, biological processes) precisely and in context, Break down complex processes into sequential steps- use dimensional analysis, algebraic manipulation, or system modeling as needed. Ensure unit consistency and check for reasonable magnitudes in your answer.
Scientific:
use
Logical: To solve the visual logic problem, analyze the image grid systematically by comparing visual components across rows, columns, or diagonals. Identify changes in shape, position, count, color, size, or orientation. Look for consistent rules or transformations such as rotation, reflection, addition, subtraction, XOR, or pattern shifts. Describe each pattemn evolution step-by-step using symbolic or relational expressions (e. (e.g., "Image = Image A XOR Image B", ar "Right = Left rotated 90' clockwise"), ") Clearly state the inferred rule and apply to missing or target positions. When multiple rules are possible, list and compare them, selecting the one that best fits visible patterns.
Spatial: To solve the spatial reasoning problem, carefully interpret both textual and visual inputs to understand object attributes (e.g., shape, color, size) and spatial relations (e.g.. left/right, behind/in front,
inside/above/bellww, Systematically extract the objective (e.g., identify, compare, count, color) and all given relational facts. Construct a mental or symbolic spatial map to track object positions and Interrelations. Use consistent reference points (e.g., camera view, object centers) and verify directional logic step-by-step. Pay attention to subtle qualifiers like "exactly," "same," or 'nearest.' For complex scenes, decompose multi-object relationships into pairwise comparisons. Avoid assumptions about 3D depth without explicit cues.
To solve the statistical problem, carefully examine all visual representations (e.g.. bar charts, line graphs, tables, pie charts) and extract precise numerical values and labels. Identify the objective (e.g.. compute, compare, interpret, predict) and determine the relevant data segments. Cross-check axes, scales, units, and legends avoid misreading quantities. Look for patterns, trends, distributions, and outliers across categories time. Apply appropriate statistical operations (e.g.. mean, median, mode, range, percentage change, probability) based strictly on the data provided.
Statistical:
Figure 19: The topic-specific hints used in MIRAGE.
44

---

Geometry
Question:
In the adjoining figure, CDE equilateral triangle and ABCD and DEF6 are squares.
<imagel>
The measure of S'anglle GDA$
Choices:
$90" 190n/h
B: $10rr\
C: Vccirc
b: $135"\iirr
Qwen2.5-V -0-n2.5-----B---- ----VVL----I-srree
Let's s analyze the problem step-by-step to find the measure of V(aaglle GDA
##### Step-by-Step Analysis
1. **Identffy the Geometric Shapes and Their Properties:**
(ABCDV) and (DEFGN) are squares.
MCDEV) on equilateral triangle.
sides of squares are equal, and each angle in a square is (c90")irre)
Each angle in equilateral triangle 1(60"\aire))
** *"Consider the Angles Point \(D))=**
The angles at point ((D)) from the squares M(BCDV)) and V(DFGG) are:
\angle ADC Vein,,,, \quad langle CDE 90" 9cinc,,,, \quad langle EDF 90")circ
The angle at point v(D)) from the equilaterol triangle V(CDEV) is:
langle CDE 60'e
3. Sum the Angles at Point V(D)):** The total the angles point MDV) from squores is:
M
9cie 90"\cir 90"\circ 2cir [Factual Hallucination Here, only 9c"Xcirc angles]
Adding the angle equilateral triangle:
2cire Ncine 3cire [wrong onswers from previous factual hallucination, but logic this step correct.]
4. Determinee the Angle V(Yangle GDA The total angle sum around any paint \air0)cire\), Therefore, the measure of \(Yangle GDA1) the difference between \3r60\ccree) and the sum the other ongles M(D)):
M
3c0rx 330" Ncire 30rr
Given the options provided:
(99"\cirr\)
8: \(105"cirre)
\d20"\circ\)
\crr)))
E: \(150%ccrr\)
The ongle measure for \(Yangle GDAV) \cio"))
#### Final
boxed[30"\cire)
Figure 20: Response example from Qwen2.55-VL-7-Innstee Red font means reasoning hallucination and corresponding judgement.
45

---

Geometry
Question:
the adjaining figure, CDE an equilateral triangle and ABCD and DEF6 are squares.
<imaget> The V'angl 6DA$
Choices: **
A: $90 $120"Verrr5
B: $$155\Mcire" $135"\ccicf
E: 3150"V\cinch
Logos-7B
<thinke that Atriangle CDE is equilateral triangle, know that all interior angles are M (160"\cir Yeirc Therefore, V( Vangle ECD 'angle CDE \angle DEC 60"Xcirc
Since V( ABCD and M( DEFG \) are squares, we know that all their interior angles are Noire V). Therefore, M( langle ACD 90" Voire and langle. EDG 90"\circ V).
We need to find M longle 6DA V). To do this, we can determine the measure of M( \angle 6DC 1). Since M( langle ECD 60"\circ \) and Vangle
angle M kangle 6DC \[ langle 6DC = Vaire 60 teinc 30" Vcirc. \J
9eirein
Since \angle EDS 9cir the angle V( langle M[ langle 6DA 9cinc coic = 120" Vcire. )J
1)
Thus, the measure M langle 6DA \) M Acire V).
Therefore, the answer </thinlo
</onswer>
Gemini-2-fleet Mimmii-2-llash-thiniin
Rationale:
given that $ Vangle GDAS.
square, CDE an equilateral triangle, DEFG square. need find the
Since ABCD square, Saangle ADC 90*\cr
Since CDE is equilateral triangle, $aangle CDE = 60"\ci
square, Vangle 6DE 9ciicss The ongles $ Vangle ADC$, Vangle CDES, $ Vangle 6DE$ consecutive angles around point D.
Since DEFG
The sum of the angles Vangle ADC$, Vangle CDE$, S'angle 6DES is Vangle ADGS (reflex angle).
$ kangle ADG
ADG__refieee (reflex) \angle ADC Vangle CDE 'angle GDE 90"\cire "circ 90"X 240"\circ$ The angle $ Vaangl 6DA$ is the other angle between the lines DA and DS. The sum of the angles around point
Neirc. Thenefore, the of $langle (reffeef)) and Saangle 6DA$ is
\360"\circ$.
kangle 6DA 360"\circ \angle ADG_ [o_[[neflex] 360"Ncinc 240" 1c0"\circf. Thus, the measure of $ $langle 6DAS is N120"Ncirc$.
Answer: The final answer is \boxed[C]5
Figure 21: Response examples from Logos-7B and Gemini-22faash-thinlee
46


Reviews:
[{"review_id": "syNxylu4KQ", "reviewer": "Reviewer_GVdS", "summary": "This paper introduces MIRAGE, the first benchmark specifically designed to evaluate multimodal reasoning hallucinations in Multimodal Large Language Models (MLLMs). By ensuring correct perception inputs, MIRAGE isolates reasoning errors, offering a clearer view into model failures. It features multi-level evaluation metrics, including accuracy, factuality, and a novel LLM hallucination score for comprehensive assessment. The authors conduct an in-depth analysis of hallucination patterns across question types, identifying spatial reasoning as a major weakness in current MLLMs. To address this, they propose Logos, a baseline method that integrates curriculum-based reinforcement fine-tuning and collaborative hint inference to promote logically consistent reasoning. Logos demonstrates reduced hallucinations and improved answer accuracy, indicating its potential for guiding future improvements.", "questions": "Please refer to the weakness section.", "limitations": "Not addressed.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. dataset contribution with detailed reasoning chain annotation. The effort of benchmarking many existing methods is impressive.\n2. The proposed Logos improved the baseline Qwen 2.5-VL.\n\nWeakness:\n1. The number of data is very limited. \n2. There are plenty of datasets that already has reasoning chain, like ScienceQA, mathVista, etc. Table 1 is not accurate, or please provide some more explations. \n3. The contribution is limited other than the dataset.\n4. The author mentioned hallucination types, but not properly defined.", "was_revised": true, "final_justification": "I appreciate the clarifications and justifications of the author. However, my concerns still remains: 1. the number of data in the dataset 2. The unique contribution of this paper. \n\nI decide to change my rating from \"reject\" to \"borderline\" reject.", "rebuttal": "Thanks for your constructive reviews. We hope this response can address your concerns. \n\n### Weakness 1: The number of data is very limited.\n\n|  Dataset  | Annotation | Size |\n|  ---- | ---- | ---- |\n| OmniBench | Answers,Description | 1,142 |\n| R-Bench-m | Answers | 665 |\n| MME-CoT | Answers,Step,Claim,Description | 1,130 |\n| MIRAGE | Answers,Reasoning Chain,Step,Claim,Hint,Description | 1,329 |\n\nThank you for your valuable feedback on our submission. We appreciate your insights and the opportunity to clarify several aspects of our work.\n\nRegarding the dataset size, we acknowledge the comparison to existing benchmarks and recognize that many reasoning \ndatasets in the literature typically have around 1,000 examples. However, MIRAGE has been specifically designed to\noffer a more diverse and comprehensive set of annotations, going beyond just the size of the dataset. As referenced \nin the latest survey [1], our dataset provides significantly more granular annotations, such as ground-truth answers, \nreasoning chains, hints, step-by-step claims, and detailed descriptions, which provide a richer context for evaluating \nreasoning tasks.\n\nIn contrast to previous works that primarily focus on answer-level evaluation, MIRAGE supports a multi-level\nevaluation approach, including assessments at the final answer, intermediate results, and reasoning chain levels.\nThis additional annotation and evaluative capability are integral to improving the robustness of reasoning models by \nexplicitly addressing reasoning hallucinations. The unique emphasis on these detailed annotations is intended to \nenhance the quality of the reasoning process rather than merely increase the dataset size.\n\nWe believe that this holistic approach to reasoning hallucination evaluation makes MIRAGE a valuable contribution to the community, as it provides a more comprehensive and rigorous framework for assessing the true reasoning abilities \nof models, not just their final answers.\n\n[1] Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey, arXiv 2025\n\n### Weakness 2: There are plenty of datasets that already has reasoning chain, like ScienceQA, mathVista, etc.Table 1 is not accurate, or please provide some more explations.\n\nThank you for your thoughtful feedback on the comparison of MIRAGE with other datasets. \nWe would like to provide further clarification to address the concerns raised.\n\nFirstly, regarding the mentioned datasets like ScienceQA and MathVista include reasoning chains, we would like to clarify that, as indicated in the concurrent survey [1], these datasets do not provide the level of detailed reasoning chains that MIRAGE does.\nSpecifically:\n\n1 MathVista does not contain ground-truth reasoning chains. The focus of MathVista is primarily on final answers, \nwithout explicitly providing detailed, step-by-step reasoning processes.\n\n2 ScienceQA includes only high-level lecture hints but lacks comprehensive, fine-grained reasoning chains. \nThe reasoning chains in ScienceQA are not sufficiently detailed or structured to qualify as ground-truth reasoning chains \nfor the purpose of evaluation. The \"lecture\" annotations provided are simplistic hints, rather than the multi-step, \ncomprehensive chains that MIRAGE offers.\n\nIn contrast, MIRAGE stands out by providing rich and multi-level annotations—including ground-truth answers,\nreasoning chains, step-by-step claims, and descriptions—enabling a thorough assessment of reasoning processes across \nmultiple levels. This distinction makes MIRAGE more suitable for evaluating reasoning hallucinations, \nas it allows for a detailed analysis of the intermediate steps in the reasoning process, beyond just the final answer.\n\nMoreover, MIRAGE is explicitly designed for hallucination assessment in multimodal reasoning tasks, \na specific focus that sets it apart from the aforementioned datasets, which primarily judge the correctness of \nfinal answers. This emphasis on hallucination detection at various levels (e.g., final answer, intermediate steps,\nand reasoning chain) provides a more comprehensive framework for evaluating the performance of reasoning models,\nmaking MIRAGE uniquely positioned to address the challenges of reasoning accuracy and hallucinations.\n\nTo clarify the difference with previous works, we will add more benchmarks (including Mathvista and ScienceQA) in the revision. \n\n\n### Weakness 3: The contribution is limited other than the dataset.\n\nThank you for your insightful feedback. We would like to clarify that the contributions of our paper extend well \nbeyond the MIRAGE dataset. While the dataset is a central component of our work, we also introduce several other \nkey contributions that significantly advance the understanding and mitigation of reasoning hallucinations\nin MLLMs. These contributions are outlined as follows:\n\n1. Insights from Experimental Results on MLLMs.\nBeyond the dataset, our experimental results reveal several critical insights into how MLLMs handle reasoning tasks:\n1)We find that model scale, data scale, and training stages significantly influence hallucinations related to logic, \nfabrication, and factuality. This highlights the complexities of scaling MLLMs and the challenges they face with \nreasoning errors; 2)More notably, our experiments show that scaling alone does not improve spatial hallucinations,\nwhere models misinterpret spatial relationships. This suggests that current MLLMs exhibit weak visual reasoning \ncapabilities and cannot simply overcome hallucinations by increasing model size or training data. \nThese findings are crucial for guiding future work in hallucination mitigation strategies for MLLMs.\n\n2. The Logos Training Framework.\nOur third contribution is the proposed Logos training framework, which aims to reduce multimodal logical hallucinations \nand improve answer accuracy. By leveraging curriculum reinforcement fine-tuning and collaborative hint inference, \nLogos provides a novel training paradigm that enhances model reasoning capabilities. \nThis framework addresses key issues identified in our experiments, specifically targeting logical and factual hallucinations, and is a promising approach for future MLLM research.\n\nWe believe that these contributions—ranging from the dataset to experimental insights and a novel training \nframework—collectively offer significant advancements to the MLLM research community. \n\n### Weakness 4: The author mentioned hallucination types, but not properly defined.\n\nThank you for your feedback regarding the definition of hallucination types. We apologize for the oversight and appreciate the opportunity to clarify the terminology used in our paper.\n\nTo address this, we provide the following clear definitions for each hallucination type, as outlined in the table below:\n\n|  Hallucination Type   | Definition  |\n|  ----  | ----  |\n| Spatial Hallucination  | Errors in reasoning about spatial relationships, shapes, or complex visual operations. |\n| Logical Hallucination  | Errors in logical consistency or reasoning, even when surface-level facts are correct. |\n| Factuality Hallucination  | Factually incorrect claims about scientific principles or established knowledge in input data. |\n| Context Hallucination  | Inconsistencies between intermediate reasoning steps and final predictions. |\n| Fabrication Hallucination  | Entirely invented values, entities, or relationships not in input data or real world. |\n\n\nThese definitions aim to provide a clear understanding of the different types of hallucinations we discuss in our work.\nFor further clarification, detailed descriptions of each hallucination type can be found in Appendix A and Table 7 of the paper.\nMeanwhile, we will also mention the hallucination type definition in the revision of the main paper."}, {"review_id": "7VqUPp32Sv", "reviewer": "Reviewer_a3WZ", "summary": "In this paper, the authors proposed MIRAGE benchmark to isolate reasoning hallucinations by questions where inputs are correctly\nperceived but reasoning errors persist. The authors also proposed Logos to mitigate these hallucination. Multiple experiments were conducted to justify the value of both the proposed benchmark and mitigation method.", "questions": "Please refer to the weakness section and provide more discussions on the performance degradation.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "***Strengths***\n1. The proposed MIRAGE benchmark provide a more fine-grained measure of reasoning-induced hallucinations in multimodal hallucination by ruling out perception-induced hallucinations. It provides a more precise measure of MLLMs' capability.\n2. Both the data collection process of MIRAGE and the proposed Logos for hallucination mitigation makes sense and are technically sound to me.\n3. Extensive experiments and analysis were conducted to justify the proposed benchmark and approach.\n4. Writing is good and easy to follow. \n\n*** Weaknesses***\n1. For Table 6, any idea why several of the metrics got worse results for row \"+hquestio\"?\n2. Typo in Line 298: \"benefiting\" should be \"Benefiting\".", "was_revised": true, "final_justification": "The rebuttal addressed my main concerns and thus I will keep my original positive rating.", "rebuttal": "Thanks for your insightful suggestions. We hope this response can address your concerns. \n\n### Weakness 1: For Table 6, any idea why several of the metrics got worse results for row \"+hquestio\"?\n\nBased on the description, we assume that the reviewer compares CRFT + h_question only with CRFT + h_topic only in Table 6. \nThis observation points to the nuanced and complementary roles that different hint types play within our Logos framework. The performance difference between +h_question and +h_topic stems from their distinct functions:\n\nSpecifically, the primary role of h_question is to guide the model to identify key entities and starting conditions within the question. It excels at grounding the model's initial focus but does not provide a complete, step-by-step problem-solving template or structure. In contrast, h_topic provides a high-level problem-solving schema or template. It offers a clear, structured formulation for the entire reasoning process, ensuring logical completeness from start to finish. The difference leads to performance gap in different metrics. For CRFT + h_question, it has better final accuracy (+0.3) than CRFT + h_topic, but the metrics regarding intermediate steps (i.e., F_step and LHS) are slightly lower (-0.4 and -0.01 respectively). \nOur Logos enhanced by CRFT is already optimized to generate complete and precise reasoning chains. When this strong base model is augmented with h_topic, it directly benefits from the clear structural guidance, making it easier to construct a perfectly formed reasoning chain. This synergy explains why +h_topic achieves slightly better scores on metrics like F_step and LHS compared to +h_question. This further validates the design of the full CHI setup that combines both h_topic and h_question to achieve balanced improvement across all dimensions.\n\nCrucially, we want to emphasize that CRFT + h_question still significantly outperforms the CRFT only baseline across all metrics. This confirms that guiding the model's initial attention with h_question is a valuable and effective strategy.\n\nFinally, we appreciate you drawing attention to this table. In reviewing it, we identified a typo in the final row (\"+full CHI\"), which should have all checkmarks as it represents the complete Logos framework. We will correct this in the revision to ensure clarity. We will also add the above analysis of hint functionalities to the paper to further clarify these results.\n\n\n### Weakness 2: Typo of ``Benefiting''.\n\nThank you for pointing this out. We apologize for this and other potential typographical errors. We will perform a thorough proofread of the entire manuscript to polish the writing in the final version."}, {"review_id": "HqLGmnHIE4", "reviewer": "Reviewer_9mGR", "summary": "This paper introduces MIRAGE, a new benchmark designed to assess and diagnose reasoning-induced hallucinations in Multimodal Large Language Models (MLLMs). Unlike existing benchmarks, MIRAGE focuses on scenarios where MLLMs correctly perceive visual inputs but fail in their logical reasoning.\nThe contribution of the paper include:\n- MIRAGE Benchmark: a diagnostic benchmark comprising 1,329 carefully designed questions aimed at isolating reasoning hallucinations. It provides multi-level annotations—including final answers, intermediate reasoning steps, and ground-truth reasoning chains—which enable detailed tracking of hallucination propagation. In addition to accuracy and factuality metrics, MIRAGE introduces a comprehensive LLM Hallucination Score (LHS) to assess model performance.\n- Empirical Analysis and Insights: The paper presents an extensive analysis revealing that model scale, data scale, and training stages impact logical, fabrication, and factual hallucinations. It highlights that current MLLMs show limited improvement in spatial hallucinations, indicating weak visual reasoning capabilities. Additionally, it identifies correlations between question types and distinct hallucination patterns.\n- Hallucination Mitigation Method: To address the identified challenges, the authors propose Logos, a baseline method that combines curriculum reinforcement fine-tuning (CRFT) and collaborative hint inference (CHI). Logos is shown to reduce logical hallucinations and improve answer accuracy.", "questions": "1. Could the authors provide a more detailed breakdown of the \"publicly available datasets and questions from Internet\"  that constitute the initial 18K dataset for MIRAGE? Specifically, listing the names of these public datasets would enhance clarity and allow for better understanding of the data's diversity and characteristics.\n2. The paper mentions using an LLM to extract intermediate steps and claims, and comparing them with ground-truth steps and claims for factuality evaluation. What measures are in place to ensure that this evaluation accurately assesses factuality, especially for questions where multiple valid reasoning paths could exist? If a model generates a logically sound but \ndifferent sequence of intermediate steps than the predefined ground truth, how does the current evaluation methodology account for this to avoid penalizing correct alternative solutions as \"hallucinations\" or \"incorrect\" steps? Is there any manual double-checking of the step and claim factuality evaluations to mitigate such issues? \n3. Could the authors provide a more in-depth analysis of Logos's performance in mitigating spatial hallucinations compared to train-free methods like self-reflection and SFT-based methods? Specifically, what are the observed differences in their effectiveness (or lack thereof) for spatial hallucinations, and what specific characteristics of spatial reasoning errors might explain why Logos performs similarly to or differently from these other approaches in this particular area?", "limitations": "Although the authors discuss some limitations in the appendix, Logos mainly combines existing techniques but lacks a clear analysis of why this combination should work in practice.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The paper is well-structured and easy to follow. The problem statement is clearly articulated, and the motivation for MIRAGE is well-justified by highlighting the limitations of existing benchmarks.\n- MIRAGE introduces three levels of evaluation metrics: accuracy (overall answer correctness), factuality (correctness of intermediate steps and claims), and LLMs Hallucination Score (assessment of the entire reasoning chain). This multi-level approach allows for a more precise tracking and diagnosis of hallucination propagation.\n- The paper proposes a cost-effective automated annotation framework combined with human-AI collaborative verification for creating ground-truth reasoning chains. This approach addresses the challenge of obtaining detailed annotations for complex reasoning tasks, which could be followed for future works.\n- The paper conducts thorough experiments on MIRAGE, providing valuable insights into the factors influencing different hallucination types, such as model scale, data scale, and training stages.\n- The authors state that MIRAGE will be publicly available , and they provide detailed experimental setups, including prompts used for construction and evaluation in the appendix.\n\n### Weaknesses\n- The paper could provide a more detailed breakdown of the initial data sources used for MIRAGE, beyond \"publicly available datasets and questions from Internet\", and elaborate on the inherent difficulty of this raw data. While some examples are provided (Figures 12-18), a more explicit discussion on data characteristics and how they contribute to reasoning difficulty would be beneficial.\n- The paper explicitly states that Logos does not significantly reduce spatial or factuality hallucinations, only logical and fabrication hallucinations. This suggests a limitation in the proposed mitigation method's effectiveness across all identified hallucination types.\n- While Logos shows promising results on MIRAGE and MathVista, its performance on a broader range of multimodal reasoning tasks and datasets not specifically designed for reasoning hallucinations could be further explored to establish its wider applicability.\n- While Logos's combination of techniques is novel, the individual components (GRPO, difficulty curriculum learning, Online Reward Filtration) are established concepts. Some recent works has already used these components in LLM/MLLM reasoning tasks, such as \"DAPO: An Open-Source LLM Reinforcement Learning System at Scale\" and \"Efficient Reinforcement Finetuning via Adaptive Curriculum Learning\n\". \n- Although the ablation experiments in appendix already point out the effectiveness of single component, the paper could strengthen its originality claim by further highlighting how the combination and synergy of these components specifically address the unique challenges of reasoning hallucinations in MLLMs beyond what individual components or simpler combinations might achieve.", "was_revised": true, "final_justification": "Most of my concerns have been addressed in the rebuttal. The authors clarified the dataset sources, improved the explanation of evaluation and method novelty, and also provided additional experiments demonstrating generalizability. I think the paper is acceptable and raised my score.", "rebuttal": "Thanks for your insightful suggestions. We hope this response can address your concerns. \n### Weakness 1: detailed breakdown of the initial data sources\nWe agree that transparency of data source is crucial. Our strategy is to curate a diverse dataset from high-quality, specialized sources, each chosen to target a specific reasoning skill:\n\nMathematical Reasoning: We sourced questions from MathVista and MathVision, which are renowned for their high-quality, complex geometric diagrams and mathematical problems.\n\nLogical Reasoning: To assess formal logic, we drew from the official question banks of national civil service examinations, which feature rigorously designed IQ-style test questions.\n\nScientific Reasoning: We utilized questions from prestigious science exams (e.g., USA Biology Olympiad), which require domain-specific reasoning in science subjects.\n\nSpatial Reasoning: Our data comes from established traffic-related datasets and the synthetic SuperCLEVR dataset, which emphasize the reasoning of complex spatial relationships.\n\nWe ensure the inherent difficulty of initial data by combining empirical validation with source selection. For public benchmarks (e.g., MathVision), we confirm their challenge by observing that even powerful MLLMs (e.g., Qwen2.5-VL-7B) achieve relatively low accuracy (25.4%), which provides a strong empirical indicator of their difficulty. For crawled data, we curate questions from sources renowned for their complexity, e.g., Physics / Biology Olympiads and IQ tests, which are by design created to probe the limits of logical deduction. This approach ensures the initial data poses a meaningful challenge to current MLLMs.\nWe will list these sources in the revision.\n\n### Weakness 2: limitation in the Logos's effectiveness across all identified hallucination types.\nThis is a key finding of our work, and we appreciate the opportunity to elaborate on why Logos shows diferent performance on various hallucination types. The behavior stems from two distinct challenges:\n\nInherent Gaps in Base Models (spatial hallucination): Our analysis in Table 1 and 2 reveals that even top-performing base models have a fundamental weakness in spatial reasoning. Scaling model or data size does not significantly reduce this type of error. Since reinforcement learning in CRFT works by rewarding and reinforcing correct behaviors that the model can already sample, it cannot effectively teach a capability that is fundamentally absent. If the base model rarely generates a correct spatial reasoning path, there are few positive examples for Logos to amplify.\n\nThe Nature of RL vs. factual knowledge (factuality hallucination): For strong base models, the remaining factual errors are often subtle and stem from inherent biases in their pre-training data. The goal of Logos's RL framework is to optimize the policy for generating correct reasoning chains, not to inject new factual knowledge. If all sampled reasoning paths from the base model contain the same ingrained factual error (e.g., a slightly incorrect constant), Logos lacks a ground-truth signal to correct it.\n\nHence, these findings show a crucial insight: tackling spatial and factual hallucinations may require architectural innovations or new data paradigms beyond the scope of post-training methods. These results also suggest that future work might explore better pertaining data, spatial-enhanced network design and advanced framework to mitigate such hallucinations. \n\n### Weakness 3: broader range of multimodal reasoning datasets.\n\nTo show the generalizability of Logos beyond MIRAGE, we evaluate Logos-7B on three general-purpose benchmarks: MathVista, MathVision, and MathVerse.\nAs shown in the table, Logos-7B not only surpasses its strong base model but also outperforms concurrent state-of-the-art methods, achieving the highest average score. This shows that Logos to mitigating hallucinations also enhances reasoning abilities, leading to comparable performance on academic benchmarks.\n\n| Method | MathVista | MathVision | MathVerse | Avg |\n|  ---- | ---- | ---- | ---- | ---- |\n| Qwen2.5-VL-7B | 68.2 | 25.4 | 47.9 | 47.1 |\n| R1-OneVision-7B | 64.1 | 19.7 | 46.4 | 46.8 |\n| OpenVLThinker-7B | 70.2 | 25.3 | 47.9 | 47.8 |\n| MM-EUREKA | 73.0 | 26.9 | 50.3 | 50.0 |\n| Logos-7B (Ours) | 72.3 | 29.8 | 52.5 | 51.5 |\n\n\n### Weakness 4: Discuss Logos with DAPO and ADARFT\nThank you for raising these concurrent works. We analyze them and present a comparison of the key differences:\n\nComparison with DAPO: The primary distinction lies in the policy gradient loss design. DAPO introduces several strategies (e.g., clip-higher, token-level loss) optimized for text-only math problems, which often feature very long reasoning chains. Our experiments found these strong regularizations were less effective and could lead to overfitting in the multimodal context, where reasoning chains are typically more concise (see Appendix D.5). Logos employs a simpler, more robust sample-level policy gradient loss that proved more effective for multimodal training, as shown in Table 8.\n\nComparison with ADARFT: The key difference is the curriculum learning mechanism. ADARFT uses a static offline difficulty annotation and greedy strategy to select training batches. This approach does not guarantee that all data is utilized effectively. Logos employs a more dynamic hybrid online-offline framework. ORF ensures each batch is filled with high-value samples, while offline difficulty thresholding stages the curriculum effectively. This hybrid approach maximizes data utility and leads to better performance. To provide direct empirical evidence, we compare Logos (w/o CHI) against these methods on MIRAGE and MathVista. Logos demonstrates superior performance on both.\n|  Method  | MIRAGE | MathVista |\n|  ---- | ---- | ---- |\n| DAPO | 30.0 | 69.6 |\n| ADARFT | 34.0 | 70.1 |\n| Logos (w/o CHI) | 35.7 | 71.9 |\n\n### Weakness 5: originality claim by further highlighting how the combination and synergy of these components specifically address the unique challenges\nThe originality of Logos lies precisely in the synergy between its two core components, CRFT and CHI, which work together to solve a two-part problem in reasoning.\nThe first part is CRFT. Specifically, an MLLM must learn how to reason. CRFT uses a curriculum-based reinforcement learning approach to transform a standard MLLM into a reasoning-capable model, teaching it to generate coherent, step-by-step thought processes.\nThe second part is CHI. Meanwhile, a reasoning model needs guidance on what to reason about for a specific problem. CHI provides this, offering dynamic hints about both the problem-solving structure (h_topic) and the key information to focus on (h_question).\n\nCrucially, these components are mutually dependent and reinforcing. An MLLM without CRFT lacks the basic ability to follow CHI's guidance. Conversely, a CRFT-enhanced model without CHI's targeted hints still struggles with the complex problems. It is a integrated system—a capable reasoning engine combined with intelligent, dynamic guidance—that allows Logos to directly and substantially reduce complex errors like logical (from 64.7% to 49.3%) and fabrication hallucinations (from 25.5% to 15.6%). This holistic, synergistic framework is the core innovation of our work.\n\n### Question 2: What measures are in place to ensure that this evaluation accurately ? Any manual double-checking of the step and claim factuality evaluations?\n\nFor the first question: Our evaluation is based on the principle that: while the explanatory text of a reasoning chain may vary, the core intermediate factual and numerical results must be consistent across all valid solutions. To accommodate reasoning variations, our evaluation script includes a \"REASONABLE\" tag (Fig. 9), which treats a step as correct if it is semantically equivalent to the ground truth, even if not perfect reasoning path match. This ensures we credit valid alternative reasoning paths.\n\nFor the second question: We conducted manual checking to validate our factuality scores. As the table below shows, the automated scores for two representative models align closely with those from three human experts, confirming the reliability of our evaluation protocol.\n\n|  Model  | step | claim |\n|  ---- | ---- | ---- |\n| Gemini-2-flash | 47.8 | 47.4 |\n| Gemini-2-flash (manual) | 48.5 | 47.9 |\n| Qwen2.5-VL-7B | 34.7 | 31.7 |\n| Qwen2.5-VL-7B (manual) | 33.9 | 31.3 |\n\n### Question 3: In-depth analysis of Logose in mitigating spatial hallucinations compared to train-free and SFT methods\nWe conducted new experiments to compare Logos-7B against other methods specifically on spatial hallucination.\nThe results in the following table are striking. While mitigating spatial deficits remains a significant challenge, Logos is the only method that shows a slight improvement, whereas all other methods, including training-free and SFT-based approaches, actually degrade performance.\nThe reasons are two-fold. Firstly, training-free methods (e.g., Self-Reflection) These methods cannot introduce new knowledge. By forcing the base model, which already lacks spatial reasoning ability, to adhere to rigid prompting rules, they can amplify existing errors. Secondly, SFT-based methods (e.g., R1-OneVision) are often trained on datasets dominated by other domains (e.g., mathematics). This can cause the model to overfit to mathematical reasoning patterns at the expense of its already weak spatial capabilities. In contrast, our Logos avoids these pitfalls. The RL approach enhances reasoning without the overfitting risk of SFT, while our CHI component provides targeted hints (h_topic and h_question) that guide the model on difficult spatial problems. Therefore, Logos is the only method avoiding performance degradation.\n|  Model  | Spatial Hallucination (%) |\n|  ---- | ---- |\n| Qwen2.5-VL-7B | 33.4 |\n| +Reflection | 37.4 |\n| +VIC | 36.7 |\n| R1-OneVision-7B | 35.6 |\n| Logos-7B | 29.9 |"}, {"review_id": "gjC8g40WrT", "reviewer": "Reviewer_bBwB", "summary": "This paper introduces MIRAGE, a benchmark designed to isolate and evaluate reasoning-induced hallucinations in multimodal large language models (MLLMs), distinguishing them from perception errors—a gap overlooked by existing benchmarks. MIRAGE uses multi-granular metrics (accuracy, factuality, and hallucination score) to assess reasoning failures, revealing that MLLMs struggle particularly with spatial reasoning involving complex visual relationships. To address these issues, the authors propose Logos, a method combining curriculum reinforcement fine-tuning and collaborative hint inference to improve logical consistency in model outputs. Logos reduces hallucinations and sets a baseline on MIRAGE, which will be publicly released.", "questions": "Questions:\n1. The paper states that the dataset was \"constructed through rigorous selection of seven distinct taxonomies, including geometry, algebraic, arithmetic, scientific, spatial reasoning, and statistical reasoning,\" but never explains why exactly these seven lines were chosen.\n2. Section 5 introduces Logos but does not specify which exact model it fine-tunes. The authors should provide the missing information on Logos's reproducibility.\n3. The authors should expand on the reliability of LLM-as-a-Judge.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is the first to cleanly disentangle reasoning-induced hallucinations from the perception errors that dominate prior multimodal benchmarks, motivating a fresh study of logical robustness in MLLMs.\n2. The proposed Logos framework lifts a 7 B Qwen2.5-VL model to nearly match much larger 72 B systems. Ablations show each component's contribution, offering a concrete starting point for future work.\n3. MIRAGE offers 1329 carefully curated questions covering seven reasoning taxonomies and provides three complementary metrics, making it a well-rounded benchmark.\n\nWeaknesses:\n1. The dataset contains only single-image questions; no multi-image, video or temporal reasoning is tested, a limitation the authors acknowledge themselves.\n2. Factuality, hallucination typing, and LHS all hinge on other large language models' judgments, raising concerns about circularity, bias, and stability across evaluator versions.\n3. The authors do not provide the rationale behind picking the seven distinct taxonomies for their dataset or mention which model is fine-tuned for their Logos method.", "was_revised": true, "final_justification": "Thank you to the authors for the detailed and thoughtful rebuttal. Their responses have addressed my primary concerns comprehensively. The rebuttal has successfully addressed all the weaknesses I raised. The new evidence, particularly regarding the validation of your evaluation metrics, has substantially strengthened the paper. Given that my primary concerns have been fully addressed, I am increasing my score.", "rebuttal": "Thanks for your insightful suggestions. We hope this response can address your concerns. \n\n### Weakness 1: The dataset contains only single-image questions; no multi-image, video or temporal reasoning is tested.\n\nThank you for this valuable suggestion regarding the scope of our benchmark. We agree that multi-image and temporal reasoning are important future directions for hallucination research. And we acknowledge that no multi-image, video or temporal reasoning data is the limitation of our paper, as stated in the Appendix G (Limitation). \nOur primary goal with MIRAGE was to first establish a foundational and high-quality benchmark for reasoning chain hallucinations in the prevalent single-image context. This deliberate focus allows for a deep and controlled analysis of a core challenge in MLLMs.\nFor scenarios that can be adapted, we propose a practical method of concatenating multiple images into a single composite image and annotating object indices, effectively converting a multi-image problem into a format compatible with the current benchmark. Looking ahead, we are committed to extending MIRAGE to include dedicated multi-image and video data in future work to explore hallucinations in more complex temporal and multi-view scenarios.\n\n### Weakness 2 and Question 3: LLM-as-a-judge raises concerns about circularity, bias, and stability across evaluator versions.\n\nThis is a critical point, and we appreciate the opportunity to elaborate on our comprehensive, multi-pronged strategy to ensure the reliability and validity of our LLM-based evaluation. Our approach is built on three pillars.\n\nFirst is mitigating circularity. To prevent self-enhancement bias, we intentionally use an older, isolated model version (gpt-4o-2024-08-06) as our primary evaluator, ensuring it has not been trained on the models or data it is evaluating. For open-source evaluators, we designed highly structured, specialized prompts (see Appendix) to constrain the evaluation and minimize potential bias.\n\nSecond is human-in-the-loop verification to double-check the validity of LLM-as-a-judge. We do not rely solely on automated metrics. We conducted manual double-checking for our key metrics (factuality and LHS) on representative models. The results demonstrate a strong alignment between our automated scores and human expert judgment. Specifically, for factuality scores in the following table, the automated scores closely match the manually verified scores, confirming their accuracy.\n\n|  Model  | step | claim |\n|  ---- | ---- | ---- |\n| Gemini-2-flash | 47.8 | 47.4 |\n| Gemini-2-flash (manual) | 48.5 | 47.9 |\n| Qwen2.5-VL-7B | 34.7 | 31.7 |\n| Qwen2.5-VL-7B (manual) | 33.9 | 31.3 |\n\nAnd for LHS, we analyze the absolute difference between automated and manual LHS scores. The results show that the metric is highly stable, with over 75% of samples having a negligible difference of less than 0.1.\n\n|  Model  | 25% | 50% | 75% | 90% |\n|  ---- | ---- | ---- | ---- | ---- |\n| Gemini-2-flash | 0.02 | 0.04 | 0.07 | 0.11 |\n| Qwen2.5-VL-7B | 0.04 | 0.05 | 0.09 | 0.13 |\n\nBesides, we further validate our framework by calculating the correlation between all metrics in Table 5. The high correlation scores across the board indicate strong internal consistency and robustness.\n\nWe believe this rigorous, three-tiered validation process effectively addresses concerns about circularity and stability, ensuring that our findings are reliable. We will add a summary of this validation strategy to the main paper.\n\n### Weakness 3 and Question 1: The authors do not provide the rationale behind picking the seven distinct taxonomies for their dataset.\n\nThank you for this question. Our selection of the seven reasoning taxonomies was a principled decision aimed at ensuring comprehensive coverage of both broad, general-purpose reasoning and deep, domain-specific skills. For general reasoning capabilities, to evaluate skills applicable to everyday scenarios, we included tasks requiring spatial reasoning (object locations, attributes) and statistical reasoning (interpretation of charts and plots). For domain-specific reasoning capabilities, to probe more complex, multi-step logical deductions, we incorporated challenging problems from mathematics (geometry, algebra), science (physics, chemistry, biology), and formal logic (IQ-test style problems). \n\nThis structured approach, which balances breadth and depth, is well-aligned with the categorization frameworks used in recent comprehensive surveys on multimodal reasoning [1, 2], confirming the relevance of our chosen taxonomies.\n\n[1] Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey, arXiv 2025\n\n[2] Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models, arXiv 2025\n\n### Weakness 3 and Question 2: Section 5 introduces Logos but does not specify which exact model it fine-tunes.\n\nThank you for pointing out this omission. We apologize for not including these details in the main text. For Logos-7B, the pretrained model is Qwen-2.5-VL-7B-Instruct; and for Logos-3B, the pretrained model is Qwen-2.5-VL-3B-Instruct. The visual encoder is frozen to avoid catastrophic forgetting of visual perception ability. During training, we collect 13K mathematical questions with K12-level difficulty and \\sim 1K text-only math questions from LIMO as training data. The batch size is 128. For each training sample, the rollout samples G is 8 by default. The initial learning rate is  1 \\times 10^{-6}, both warmup strategy and cosine learning rate scheduler are adopted to stabilize training. We optimize Logos by 10 epochs using AdamW during each stage. The number of CRFT stages is set to 1, which is discussed in the ablation study. Benefiting from the filtration mechanism in CRFT and ORF, the total training time of Logos-7B is less than 24 hours, Logos-3B is faster. We will add these crucial implementation details to the main paper for clarity and reproducibility, while a complete list of hyperparameters has been stated in Appendix B."}]

Author rebuttal:

Meta-review: MIRAGE, a benchmark that isolates reasoning-induced hallucinations in multimodal LLMs, and a method called Logos, a method to mitigate them. The dataset’s fine-grained annotations and analysis provide new insights into model weaknesses, and the experiments show clear improvements. Although concerns about dataset size and evaluation were noted, the rebuttal addressed them convincingly. Overall, this is a timely and valuable contribution. I recommend accept.
Decision: Accept (poster)