Title: Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Factual implication

Transformer

Implicit bias
Abstract: Large language models (LLMs) can acquire new knowledge through fine-tuning, but this process exhibits a puzzling duality: models can generalize remarkably from new facts, yet are also prone to hallucinating incorrect information. However, the reasons for this phenomenon remain poorly understood. In this work, we argue that both behaviors stem from a single mechanism known as out-of-context reasoning (OCR): the ability to deduce implications by associating concepts, even those without a causal link. Our experiments across five prominent LLMs confirm that OCR indeed drives both generalization and hallucination, depending on whether the associated concepts are causally related. To build a rigorous theoretical understanding of this phenomenon, we then formalize OCR as a synthetic factual recall task. We empirically show that a one-layer single-head attention-only transformer with factorized output and value matrices can learn to solve this task, while a model with combined weights cannot, highlighting the crucial role of matrix factorization. Our theoretical analysis shows that the OCR capability can be attributed to the implicit bias of gradient descent, which favors solutions that minimize the nuclear norm of the combined output-value matrix. This structure explains why the model learns to associate facts and implications with high sample efficiency, regardless of whether the correlation is causal or merely spurious. Ultimately, our work provides a theoretical foundation for understanding the OCR phenomenon, offering a new lens for analyzing and mitigating undesirable behaviors from knowledge injection.


Peer reviews:
comments: SUMMARY:
This work presents an theoretical analysis which shows that a factorized, one-layer attention-only Transfoer is enough to obtain a generalizable deduction ability from learned facts, while a simpler model like a single-head attention or a non-factorized model is not. Meanwhile, such a deduction ability also also causes LLMs to hallucinate easily by deducing false implications when two events are not causally related but only co-occur several times, which provides a possible explanation of hallucination in LLMs. While experiments are conducted on small-scale datasets, they have covered several recent LLM famlilies.

STRENGTHS AND WEAKNESSES:
I believe this is a theoretically solid work that provides a well-motivated explanation for Transformers' deductive abilities, and why it sometimes lead to non-factual hallucination. Several supported claims in this work are meaningful and interesting. On the other hand, the main shortcome of this work is that its experiments only used a very small sythetic dataset. There are plenty of large, real-world datasets for deductive reasoning that could have been tested on, such as ProntoQA-OOD [1], EureQA [2], or any of the KGC datasets [3]. The current small-scaled experiments provides a decent explanation to the claims, but leave the generalization of these claims uncertain.


[1] Saparov et al. Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples. NeurIPS 2023  
[2] Li et al. Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination? NAACL 2024  
[3] Petroni et al. Language Models as Knowledge Bases? EMNLP 2019

QUESTIONS:
I would like to hear the authors' opinion on whether the experiments can be extended to large-scale, real data.

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-03

comments: SUMMARY:
This paper focuses on the mechanisms underlying generalization and hallucination in large language models (LLMs), exploring the out-of-context reasoning (OCR) capabilities of Transformers through theoretical analysis and empirical studies. The core contributions lie in revealing that a single-layer attention-based Transformer can achieve fact-related reasoning through parameter decomposition and explaining the unified mechanism of generalization and hallucination

STRENGTHS AND WEAKNESSES:
-Strengths: 1) Synthesis of theory and experiments unifies explanations of generalization and hallucination, contributing to theoretical understanding 2) Hallucination is a challenging problem in LLMs; this paper gives a deep analysis of this question from the view of OCR. 
- Weakness: 1) Idealized synthetic data and lack of benchmark model comparisons limit conclusion; 2) Limited analysis of practical implications; I mean the authors should step forward to investigate how to design anti-hallucination techniques from the view of OCR; 3) More different model structures should be tested in the experiments.

QUESTIONS:
1) From the view of OCR, how can we design models to improve the generalization rather than hallucination?
2) Can the authors construct a benchmark to test the OCR problem?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper empirically and theoretically demonstrates that a single-layer linear attention can perform out-of-context reasoning (OCR). Specifically, the transformer can predict $s',r_2\\rightarrow b$ given $s,r_1\\rightarrow a$,   $s,r_2\\rightarrow b$,  and $s',r_1\\rightarrow a$. Further experiments on synthetic tasks prove that real-world LLMs also exhibit such OCR capability. However, such capability can cause hallucinations once the associations in training data are not causal relationships, revealing a potential cause of the hallucinations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. This work provides novel insights into the formation of hallucinations in LLMs by theoretically proving that a single-layer linear attention is capable of achieving OCR.\n\n2. This work theoretically reveals the defect of the parameterization of the output-value matrix widely adopted by previous works.\n\n3. The paper is well-written and easy to follow.\n\n**Weaknesses**\n\nThe theory and experiments consider a very simple case where only two facts are correlated. While I acknowledge that such a setup may reflect certain real-world situations, it fails to capture the complexity of most reasoning patterns. In more realistic cases, the \"facts\" can be modeled as a causal graph with multiple variables and relationships. For instance, in your OCR training data, since $s,r_1\\rightarrow a \\Rightarrow s,r_2\\rightarrow b$, we can imagine a causal graph in which there exists a confounder $C$ such that $C\\rightarrow A$ and $C\\rightarrow B$, where event $A$ represents  $s,r_1\\rightarrow a$ and event $B$ represents $s, r_2\\rightarrow b$. $A$ and $B$ are correlated through $C$. However, this represents only a rudimentary causal graph and offers limited applicability to more complex scenarios.\n\nMoreover, according to the definition in your paper, whether the model \"generalizes\" or \"hallucinates\" depends on whether the correlation between $A$ and $B$ is causal. Specifically, if $A\\rightarrow B$ across training and test data, then the model \"generalizes\"; otherwise, the model produces hallucinations. In my opinion, however, \"generalization\" should refer to the model's ability to learn generalizable features or relationships, such as capturing a causal link in a causal graph, so it may not be suitable to regard the capability of successfully solving the tasks in Fig. 5 as \"generalization\".\n\nQUESTIONS:\nCould you extend the identified capability of transformers to perform the OCR task defined in your paper to more general reasoning modes? I think either empirically or theoretically extending your findings to explain the failure or capability in more general reasoning patterns will be helpful to improve your work.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores how transformer models can both generalize from new information and hallucinate incorrect facts. Specifically, it investigates a phenomenon called Out-of-Context Reasoning (OCR) where a model makes inferences based on partial patterns seen during training. The authors ask whether this same reasoning mechanism is responsible for both useful generalization and harmful hallucination. They theoretically analyze a very simple model and surprisingly show that even this basic model can perform OCR, learning patterns from just a few examples. Their analysis shows that this ability comes from the way the model is trained, specifically how gradient descent favors certain types of solutions that can link facts together. However, this also means the model may make false connections when two things only appear together but aren’t truly related.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n- Interesting topic that investigates why LLMs can both generalize well and hallucinate.\n- Shows that even a one-layer attention-only transformer can perform OOC reasoning \n- Connects model behavior to gradient descent’s implicit bias.\n\n\nWeaknesses\n- Focuses only on one-layer transformers; unclear whether the conclusions fully apply to complex models. \n- It is fine to use synthetic data for evaluation, but it would be helpful to see and discuss how the model performs on tasks such as question answering. \n- Identifies the cause of hallucination that is a good result, but how can we reduce or prevent it, based on the analysis provided in the paper? \n- The proposed optimization as well as the use of SVMs are difficult to follow, could the authors provide some clarifications?\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nThe authors adequately discuss the technical limitations of their work, particularly the focus on one-layer transformers and the use of synthetic data.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}