Title: Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: hallucination

Chain-of-Thought

reasoning

Metacognition
Abstract: The development of Reasoning Large Language Models (RLLMs) has significantly improved multi-step reasoning capabilities, but it has also made hallucination problems more frequent and harder to eliminate. While existing approaches address hallucination through external knowledge integration, model parameter analysis, or self-verification mechanisms, they fail to provide a comprehensive insight into how hallucinations **emerge** and **evolve** throughout the reasoning chain. In this work, we investigate hallucination causality under constrained knowledge domains by auditing the Chain-of-Thought (CoT) trajectory and assessing the model's cognitive confidence in potentially erroneous or biased claims.
Analysis reveals that in long-CoT settings, RLLMs may iteratively reinforce biases and errors through flawed reflective processes, ultimately inducing hallucinated reasoning paths.
Counterintuitively, even with interventions at hallucination origins, reasoning chains display pronounced ''chain disloyalty'', resisting correction and sustaining flawed trajectories.
We further point out that existing hallucination detection methods are *less reliable and interpretable than previously assumed*, especially in complex multi-step reasoning contexts.
Unlike circuit tracing that requires access to model parameters, our auditing **enables more interpretable long-chain hallucination attribution in black-box settings**, demonstrating stronger generalizability and practical utility.
Our code is available at [this link](https://github.com/Winnie-Lian/AHa_Meta_Cognitive).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates how hallucinations arise and persist in Reasoning Large Language Models (RLLMs), especially during complex multi-step reasoning. While prior methods attempt to reduce hallucinations via external knowledge or self-verification, they lack insight into the progression of errors across reasoning chains. By auditing Chain-of-Thought (CoT) trajectories and analyzing cognitive confidence, the authors show that RLLMs often reinforce initial biases, leading to persistent hallucinations—despite corrective interventions—due to a phenomenon they call \"chain disloyalty.\" Their black-box auditing method offers interpretable and generalizable hallucination attribution without requiring access to model parameters, outperforming existing detection techniques in complex reasoning scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper is well-written and thoroughly examines a significantly relevant area, particularly given the limited understanding of the reasoning abilities of reasoning models. It addresses the challenges posed by long-CoT hallucinations, which remain an unresolved barrier to the establishment of reliable RLLMs. Concentrating on metacognition, specifically the model's confidence in its assertions, addresses a notable research gap. \n2. Implementation of four RFC-grounded subsets (Type I / II plus controls) characterized by verifiable truths and a thoroughly documented construction workflow facilitates reproducibility and accurate attribution.  \n3. The case studies effectively illustrate the \"chain disloyalty\"; once an error is introduced, reflection may inadvertently reinforce it instead of rectifying it. The authors candidly discuss the scope, biases, and societal implications, thereby advocating for responsible future research endeavors. \n\nWeaknesses: \n1. Experiments use DeepSeek-R1 (plus GPT-4o for annotation); cross-model generality is untested. \n2. While the paper addresses hallucinations, it does not propose a new mitigation technique to enhance the manuscript further.\n\nQUESTIONS:\nQuestions:\n1. All current experiments rely on DeepSeek-R1; thus, it is unclear whether the documented \"metacognitive drift\" is model-specific quirks or general tendencies. It would be best to test with other models.\n2. The paper convincingly diagnoses the problem but stops short of actionable remedies. Showing even a modest reduction in hallucination persistence would turn the work from purely diagnostic to partially prescriptive.\n3. Some baselines require days on dual A100s, yet the cost/benefit discussion is brief. The authors should expand on this.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate how hallucinations emerge and propagate in Reasoning Large Language Models (RLLMs), particularly in multi-step Chain-of-Thought (CoT) reasoning. The authors construct a controlled knowledge environment using RFC documents to systematically audit hallucination trajectories. They identify that hallucinations often originate from the model's overconfident use of incorrect or unlearned knowledge, and are amplified through flawed reflective reasoning, where the model increases confidence in false claims due to semantic alignment with user prompts. Moreover, they show that despite editing interventions, hallucinated reasoning paths show \"chain disloyalty\", resisting correction. In sum, this study highlights the limitations of current hallucination detection and mitigation methods in long-CoT settings, emphasizing the need for future models with explicit metacognitive capabilities to ensure more reliable and interpretable reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n-The paper tackles a timely issue with the massive use of reasoning models, and tries to provide interesting descriptive statistics with respect to CoT-driven hallucinations.\n\nWeaknesses:\n\n-My main concern with this paper is the lack of clarity in the writing. The introduction on its own assumes knowledge that is either not present in the paper or comes much later in the manuscript. Figure 1 is extremely cryptic, and I cannot make sense of any of the subgraphs in this figure as I read along in the paper. The rest of the paper follows a similar pattern, poor clarity. \n\n-The paper would strongly benefit from an incremental description of the concepts, as well as spending more effort in grounding the concepts in a way that is clear for the reader.\n \n-Modelling assumptions seem to be crucial to the results, and thus the validity of the interpretability method proposed by the authors. It seems that the proposed method which gains in interpretability of the CoT, comes at the cost of robustness (an issue that is avoided by deeper level methods).\n  \n-The paper would benefit from a native English person proof-read.\n\nQUESTIONS:\n1. What does it mean for a model to exhibit high confidence, i.e., high confM(k)? How is this formalized by the authors? Equation 3 is also very cryptic and hard to interpret, how is conf(cq) − conf(cp) reflected on the right side of equation 3 (also what is f in eq. 3, I imagine “feedback”?).\n\n2. Have the authors thought about other metacognitive confidence modelling approaches, that could be intuitive but generate diverging results? Regardless of whether the proposed modelling was based on previous work.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors address important recent research issues: emerging & evolving hallucination in the reasoning chains of frontier RLLMs. By auditing model behaviors during the CoT trajectory (controlled knowledge environment with RFC datasets), the authors provide several insights and observations to understand underlying mechanisms of hallucinations during the test-time reasoning process.\n\nSTRENGTHS AND WEAKNESSES:\nThe reviewer enjoyed to read this paper, which addresses an important and timely topic (hallucination in test-time CoT based RLLMs).  Despite of the research importance, there are several points to discuss:  \n\n1. Due to the missing references in the manuscript (line 105, 121), the readers hard to follow terms used in this paper. Also the lack of implementation details make it difficult to understand the overall flow (especially section 2.3 provides high-level explanations, some of details should be explained more detailedly - such as “how to obtain confidence for each claim” or “how to explicitly model g function to capture prompt-aligned bias”, etc).\n\n2. (continued) To provide better understanding to potential readers, Fig. 1 should be improved to better align with the main text (section 2). Although it is briefly discussed in lines 41-52, it looks currently disconnected from the core explanations - some of terms used in the figure are not referenced in the main body (thus may confuse readers). Providing well-matched descriptions in section 2 along with Fig.1 can enhance clarity.\n\n3. Many concepts and terminologies make this paper hard to follow (such as metacognitive drift, flawed reflection, chain disloyalty, prompt-aligned bias, etc). To understand such abstract concepts, the reader should put a lot of work into uncovering what they actually are. In addition, the current controlled knowledge environment with using of RFC document datasets may raise concerns about the generalizability of the paper’s main insights and observations to more open-ended scenarios typically encounter by current RLLMs\n\nQUESTIONS:\nQ1. For the Obs 2, the observation suggests that the longer reasoning chain indicates the metacognitive drift due to the initial uncertainty. But the reviewer think that it is unclear whether the increased claim length indeed reflect of uncertainty - possibly due to task complexity (may be more claims are needed to answer the question) or the need to fill out the answer for the query instance. As shown in Table 2, the proportion of hallucinated claims is not notably high, which naturally raise questions about the loose link between the claim lengths and hallucination (originated from uncertainty).\n\nQ2. The reviewer thinks Type 2 control part is missing, and also analysis of Part C is skipped in the manuscript. Accordingly, the obs 3 seems somewhat speculative and do not provide enough insights for the given listed performances.  Q3. In section C.1, the authors provide extended explanations for each taxonomy of CoT trajectories. The reviewer wonders whether the example cases in Fig. 2 are curated instances or representative samples following certain procedure. Moreover, it remains unclear how the specific reasoning trajectories can be traced or identified (such as dropped path or conditioned branch (“if” cases)) from the models’ generation branch. Providing more concrete examples can be helpful to readers to understand this analysis.\n\nQ3. In section C.1, the authors provide extended explanations for each taxonomy of CoT trajectories. The reviewer wonders whether the example cases in Fig. 2 are curated instances or representative samples following certain procedure. Moreover, it remains unclear how the specific reasoning trajectories can be traced or identified (such as dropped path or conditioned branch (“if” cases)) from the models’ generation branch. Providing more concrete examples can be helpful to readers to understand this analysis.\n\nLIMITATIONS:\nyes\n\nThe reviewer leans to BR for current version, and will finalize the final rating score after the discussion period.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the critical problem of hallucinations in RLLMs, going beyond simple detection to investigate the underlying mechanisms from the emergence to the evolution of hallucinations in RLLM's reasoning chain. The authors identify that models can iteratively reinforce biases and errors through flawed reflective processes, even with interventions at hallucination origins. They do this by auditing the CoT trajectory and assessing the model's cognitive confidence in potentially erroneous or biased claims. The authors then define four key research questions and, through extensive experiments, demonstrate that existing hallucination detection methods are less reliable and interpretable than previously assumed, especially in complex reasoning contexts in RLLMs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper is well-structured, and the writing is clear. I appreciate the authors framing their entire investigation from modeling to the four research questions. This provides a logical backbone for the paper that makes the argumentation easy to follow from start to finish.\n\n2. The experimental parts are both detailed and comprehensive. The authors conduct a deep and convincing audit of the model's behavior. The experiments, especially the controlled CoT editing, are thoughtfully designed and go far beyond simple performance metrics to reveal the *process* behind the hallucinations in RLLMs.\n\n3. Finally, some insights and findings of this paper would be valuable to the community. Concepts like \"chain disloyalty\" offer a new and useful vocabulary for describing error propagation. Furthermore, the paper's critical analysis of existing hallucination detection methods serves as an essential reality check, pushing the field to develop more robust solutions.\n\n**Weaknesses**\n\nI do have some concerns with the paper that I believe should be addressed:\n\n1. The theoretical model of confidence updates presented in Section 2.3 is insightful. However, the analysis that follows functions more as an interpretive lens for the qualitative results rather than a hypothesis that is quantitatively tested. The conclusions would be more convincing if the authors could bridge this gap, perhaps with experiments that attempt to measure or approximate the variables in their proposed equations (2) (3).\n\n2. Regarding Observations I & III, the paper attributes the model's tendency to accept and elaborate on false premises to a \"prompt-aligned bias\". This is a plausible interpretation, but it warrants more evidence. An alternative explanation is that these errors stem from a simple knowledge deficit; the model may not be *willfully* aligning with a user's error but simply doesn't know the information is incorrect. Additional experiments to disentangle a knowledge gap from a genuine alignment bias would make this claim more robust.\n\n3. The study's reliance on a single model, DeepSeek-R1, raises questions about generalizability. The observed phenomena, such as \"chain disloyalty\" and the specific mechanisms of error amplification, could potentially be artifacts of this particular model's architecture or training. The paper's claims would be more effective if the core findings were validated on other major model families, like the GPT or Claude series.\n\n4. A detailed description of the experimental setup and dataset construction is located in Appendix B. As a reader, I looked for this context before diving into the results in Section 3. The paper would be easier to follow if a summary of the setup, or at least a clear pointer to the appendix, were included at the beginning of Section 3.\n\nQUESTIONS:\n- Regarding the \"prompt-aligned bias\" claim, could you provide further justification for why this is a more likely cause of error than a simple knowledge deficit? Any discussion or analysis to disentangle these two possibilities would significantly strengthen this core claim.\n- To improve readability, please adding a brief overview of the experimental setup to the beginning of Section 3, or at least a clear forward reference to the full details in Appendix B?\n\nLIMITATIONS:\nYes. The authors discussed limitations and broader impact in Appendix A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}