How do Transformers Learn Implicit Reasoning?
Authors: Jiaran Ye
Zijun Yao
Zhidian Huang
Liangming Pan
Jinxin Liu
Yushi Bai
Amy Xin
Liu Weichuan
Xiaoyin Che
Lei Hou
Juanzi Li
Venue: NeurIPS 2025 spotlight ()
Keywords: Implicit Reasoning
Multi-hop Inference
Transformer Models
Interpretability
Primary area: deep_learning

Abstract: Recent work suggests that large language models (LLMs) can perform multi-hop reasoning implicitly---producing correct answers without explicitly verbalizing intermediate steps---but the underlying mechanisms remain poorly understood.
In this paper, we study how such implicit reasoning emerges by training transformers from scratch in a controlled symbolic environment.
Our analysis reveals a three-stage developmental trajectory: early memorization, followed by in-distribution generalization, and eventually cross-distribution generalization.
We find that training with atomic triples is not necessary but accelerates learning, and that second-hop generalization relies on query-level exposure to specific compositional structures.
To interpret these behaviors, we introduce two diagnostic tools: cross-query semantic patching, which identifies semantically reusable intermediate representations, and a cosine-based representational lens, which reveals that successful reasoning correlates with the cosine-base clustering in hidden space.
This clustering phenomenon in turn provides a coherent explanation for the behavioral dynamics observed across training, linking representational structure to reasoning capability. 
These findings provide new insights into the interpretability of implicit multi-hop reasoning in LLMs, helping to clarify how complex reasoning processes unfold internally and offering pathways to enhance the transparency of such models.

How do Transformers Learn Implicit Reasoning?
Jiaran Ye
Zijun Yao Zhidian Huang*
Liangming Pan 9i Jinxin Liu
Yushi Bai
Amy Xin
Weichuan Liu
Xiaoyin Che Lei Hou
Juanzi Li
DCST, BNRist; KIRC, Institute for Artificial Intelligence, Tsinghua University, China MOE Key Lab of Computational Linguistics, Peking University, China Siemens AG, China
(yejr23, [oozj20]0maill tsinghua, edu. (houlei, lijuanzi) @tssinhuu. edu cn
Abstract
Recent work suggests that large language models (LLMs) can perform multi-hop reasoning implicily-produuvee intermediate butps-but the underlying mechanisms remain poorly understood. this paper, we study how such implicit reasoning emerges by training transformers from scratch a controlled symbolic environment. Our analysis reveals a three- stage developmental trajectory: early memorization, followed by in-distribution generalization, and eventually cross-distribution generalization. We find that train- ing with atomic triples is not necessary but accelerates learning, and that second-hop generalization relies on query-level exposure to specific compositional structures. To interpret these behaviors, introduce two diagnostic tools: coos--uery seman- tic patching, which identifies semantically reusable intermediate representations, and cosine-based representational lens, which reveals that successful reason- ing correlates with the cosine-base clustering in hidden space. This clustering phenomenon in turn provides coherent explanation for the behavioral dynamics observed across training, linking representational structure to reasoning capability. These findings provide new insights into the interpretability of implicit multi-hop reasoning in LLMs, helping to clarify how complex reasoning processes unfold internally and offering pathways to enhance the transparency of such models.
correct answers without explicitly verbalizing
https://gitai com/ Jiaran /iiran-Ye/Impliiee wet.mm/iiiiteassoniiiiiio
Introduction
Large language models (LLMs) demonstrate strong performance complex, multi-step reasoning tasks [9, 7, 29, 1, 32, 19, 14]. Typically, these reasoning abilities are elicited using chain-of- thought (CoT) prompting. which encourages models to explicitly articulate intermediate reasoning steps [28, 35, 6, 33, 30]. Beyond CoT, recent studies indicate that LLMs can also engage in implicit reasoning [37, 11, 15], producing correct answers without verbalizing intermediate steps.
While implicit reasoning is widely acknowledged, the internal mechanisms that empower this ability remain unclear. In this paper, we aim to uncover the internal processes of implicit reasoning by examining a concrete, structured scenario: multi-hop implicit reasoning, where the model must answer compositional queries (e.g., (c1, r1, P2) cx) by implicitly traversing intermediate entity 62, without explicitly verbalizing it. A fundamental question in this scenario is that: does the model genuinely conduct step-by-step reasoning internally, is it merely recalling the answer from its memorized knowledge? Although both behaviors can produce correct outcomes, they reflect
Equal Contribution. Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Training Process
Memory Phase
Implicit Reasoning Phase
explicitly decoded
Still explicitly decoded but caiine-onnverged
input taken
but cosine-scattered
target token
Second hoo.
hidden states
decodable via unembedding
Late layers
cosine-based projection
layers
Shortcut reasoning
First
shared-tail triples
early layers
early
0.4
1.3
e.8
layers
shared entity
2-hop
bridge
2-hop queries
bridge
shared- head tripies
We track the evolution of bridge entity representations phase (left), intermediate entities are explicitly decodable but geometrically scattered. In the implicit reasoning phase (right), representations converge in cosine space, supporting structured multi-hop reasoning through early-to-late layer transitions. Queries that share an bridge entity are composed for cosine-based representation analysis.
Figure 1:
training. In the memory
fundamentally distinct cognitive processes. This observation motivates our central research question: How do LLMs acquire and perform implicit reasoning during training and inference?
Existing studies that investigate implicit reasoning often rely on pretrained LLMs whose training data lacks precise experimental control, making it challenging
conclusively determine whether models have genuinely learned implicit multi-step reasoning or instead rely on its prior knowledge shortcut solutions [12, 34, 5]. Symbolic datasets [25, 27, 26] partially alleviate this concern by training models from scratch, yet they still lack the fine fine-graing -e-ggriindd experimental control and behavioral granularity necessary for deeper analysis. To address these limitations, we construct an extended symbolic environment, featuring targeted omissions and query-level variations, to precisely identify whether implicit reasoning and generalization truly emerge.
To facilitate the analysis under our symbolic environment, we introduce two diagnostic tools that overcome specific limitations of prior methods: (1) cross-query semantic patching, which enhances causal interpretability by locating intermediate entity representations based on their semantic trans- ferability across queries rather than solely their impact final outputs; and (2) cosine-based representational lens, which avoids assumptions inherent decoding- based probing by examining structural consistency of internal representations across reasoning contexts. Together, these tools enable precise examination of the internal processes driving implicit reasoning.
Our empirical analysis begins with a behavioral study conducted under fine-grained experimental control (Section 2). Under a complete training configuration, we observe that multi-hop implicit reasoning emerges in three distinct stages; memorization, in-distribution generalization, and finally cross-distribution generalization. Through ablation studies, we further demonstrate that while expo- sure to in-distribution (ID) triples is not strictly necessary for achieving in-distribution generalization, its absence significantly delays the onset of this behavior. Additionally, we find that generalization to second-hop queries fails unless the model encounters exact compositional structures during training, revealing a strong dependency on query-level exposure.
These behavioral insights reveal previously unreported patterns, motivating us to revisit and probe the internal mechanisms of implicit reasoning. In Section 3, we first use cross-query semantic patching to localize intermediate entity representations, typically identifying them within the middle layers

---

corresponding to the tokens. We then test the common assumption that intermediate entities are explicitly decodable from internal states and find this assumption inconsistent with our observed reasoning behavior. This disconnect leads us to adopt a geometric perspective, wherein successful reasoning strongly correlates with consistent clustering of intermediate representations within cosine similarity space (Figure 1).
In Section we close the loop by explicitly connecting these internal representational mechanisms to external behavioral patterns. We demonstrate that successful generalization robustly correlates with the clustering structure of intermediate representations across diverse queries and training distributions. Although in-distribution (ID) triple supervision is not required to induce this clustering, it substantially accelerates its emergence by constraining the representational space early in training. Finally, we identify that what appears to be first-hop generalization to out-of-distributin triples is actually an artifact arising from representational alignments induced by ID exposure, highlighting the fragile and data-dependent nature of implicit generalization.
(OOD)
Collectively, our results provide comprehensive account of how implicit multi-hop reasoning emerges within LLMs grounded in observable behaviors, elucidated through mechanistic analyses, and offering foundational insights for future studies on model interpretability.
2 Behavioral Signatures of Implicit Reasoning under Fine-Grained Control
Existing studies implicit reasoning fall into two broad categories, each with notable limitations. (I) Analyses based on pretrained LLMs operate in an uncontrolled setting where the training data are opaque-making it difficult distinguish genuine reasoning from memorization.
(2) In contrast, recent works adopt symbolic datasets with synthetic training from scratch [25], but primarily focus on dataset-level trends, without isolating what specific training signals are necessary for solving each compositional query.
We argue that idcal analysis setting should satisfy three key properties: (1) compositional struc- ture, to support multi-step inference; (2) fine-grained control, to support query-level ablations and conditionally constructed variants; and (3) behavioral resolution, to distinguish between memo- rization, generalization, and reasoning. With these goals in mind, we construct a symbolic training environment that extends prior datasets with new configurations and targeted omissions, and reveals several behavioral phenomena not captured in prior work.
To achieve this, we adopt GPT-2 our base model due to its balance of capacity and tractability, and verify the scalability of results using larger models. Full training details are provided in Appendix G.
2.1 Data Construction: Fine-Grained Control for Compositional Reasoning
To enable fine-grained behavioral analysis, we extend the symbolic reasoning setup of Wang et al. [25] with expressive query-level control configurations. The data comprises atomic triples and compositional queries:
Each atomic fact is represented as triple (c1,r1) + c2. This formulation mimics simple factual relations such as (Alice, mother-of) - Beth and (Beth, sister-of) + Carol, serving the atomic unit of the reasoning environment. The triples are partitioned into two subsets: In-Distribution (ID) Triples are used in both standalone form and as components of multi-hop training queries;
Atomic Triples.
(OOD) Triples appear in training data only in standalone form, and are excluded from multi-hop composition, enabling the creation of test queries involving out-of-distribution reasoning. Note that ID and OOD triples share the same set of entities and relations.
2-Hop Queries. Each reasoning task takes the form of a compositional chain (c1, F1, r2) + ey. where the model performs implicit reasoning over an bridge entity 62- For instance, the model receives only the compositional query (Alice, mother-of. sister-of) and is expected to predict the correct target Carol, implicitly reasoning through the intermediate entity Beth. We distinguish: Test-Ol: test queries where the first hop comes from an OOD triple and the second hop from an ID triple: Tllin-Il: queries with both hops from ID triples used during training. Other query types, such as Test-Il, Test-I0, and Test-OO, follow similar definitions.
3

---

Training Configurations. Our base configuration (Figure 6a) includes all atomic triples (ID and OOD) and the full set of Train- queries, and evaluate on Test-II, Test-OI, Test-OO, and Test-IO, allowing comprehensive generalization assessment. To isolate the conditions for generalization, we define a flexible family training variants that omit specific triples, restrict compositional roles, or remove entire subsets, allowing targeted query-level ablations. This design supports controlled investigations into the functional dependencies behind implicit reasoning behaviors.
Extension to 3-hop Reasoning, Although our main analyses focus on 2-hop queries, the same construction framework naturally applies 3-hop settings, exhibiting consistent behavioral and mechanistic patterns. For more details, refer to Appendix C.
Together, these configurations details and illustrations are available in Appendix B.
as the foundation for study. Further dataset construction
2.2 Ttree-Stage Generalization
Leveraging the base configuration introduced in Section 2.1, we track model performance throughout training and observe striking behavioral trajectory that unfolds in three distinct phases (Figure 2):
Triples
Train-11
Test-Ol
Test-00
Test-10
OOD Triples
Test-Il
ID Cohesion
00D All
Phase III
Phase
Phase
0.8
Accuracy
Avarag Cos
40
0.4
10
16
10
10'
0.0
Checkpoints (Training Steps)
Figure 2: Training dynamics under the base configuration, revealing three distinct phases. Accuracy curves track model performance on different query types, while dashed lines plot the Cohesion and OOD Alignment scores (Section 3.3).
Phase Memorization. The initial stage involves quickly fitting the training data, including atomic facts and 2-hop compositions. The model memorizes these facts, but generalization to unseen queries remains minimal.
Phase II: ID Generalization. After memorization saturates, the model begins to generalize to
Test-ll queries (unseen IID-ID compositions), marking shift from memorization to compositional generalization within ID), akin to the grokking phenomenon described by Wang et al. [25].
Phase III: Cross-Distributton Reasoning. The model next learns generalize across distributions, gradually incorporating OOD triples in the first hop while maintaining the ID in the second. This transition is slower than Phase II and requires more training. Building on the grokking phenomenon, our analysis uncovers this additional phase of generalization across distributional boundaries.
Interestingly, generalization fails consistently when the second hop is from OOD triples, revealing a stronger bottleneck in the second relational step. These phases show that reasoning develops in structured stages, each with distinct patterns of success and failure, highlighting the need to treat reasoning not as a monolithic ability, but as a set behaviors with separable developmental conditions.
2.3 ID Triples Are Not Required for ID Generalization Accelerate It
Prior work has repeatedly observed that while models can correctly answer individual atomic triples, they often fail to generalize 2-hop queries constructed by composing those same triples[3, 31, 42]. Additionally, in Section 2.2 we observe that our model quickly memorized atomic triples (Phase took longer to generalize to Test-ll queries. These findings raise a natural question: Are atomic ID Triples actually necessary for learning ID-based 2-hop reasoning?

---

100
Train-Il
100
Test-ll
Accuracy (%6
Accuracy (%)
75
75
50
50
ID Tripees
25
25
Train-Il
Test-ll
0
10a
10*
10
10
0
10*
10
10
Checkpoints (Training Steps)
Checkpoints (Training Steps)
(a) Train-II only
(b) Train-II + ID triples
Figure 3: Training trajectories under two configurations used in Section 2.3. (a) Only Train-II queries are used, with no exposure to atomic ID triples. (b) Both Train-Il queries and atomic ID triples are included in training.
To investigate this, we test minimal training configuration that excludes both ID and OOD Triples, using only Train-II queries. Surprisingly, the model still generalizes to unseen ID combinations Test-ll (Figure 3a). That is, training with Train-Il alone sufficient for ID-based generalization.
However, when comparing this minimalist setting to a variant where ID Triples are included! alongside Train-II, we found that generalization Test-II occurs significantly earlier (Figure 3b), suggesting that while atomic facts are not required for generalization, they accelerate learning.
2.4 Second-Hop Generalization Requires Query-Level Training Match
While the model can generalize when OOD triples appear in the first hop (Section 2.2), it consistently fails when the OOD component is in the second hop. This raises a natural follow-up question: What training data is necessary to enable second-hop generalization within the ID) domain?
Building on Section 2.3, where training with only Train-Il queries still led ID generalization, we perform targeted ablation to isolate the role of second-hop coverage. Specifically, we remove subset of atomic triples (e.g., (EB, rs, EF)) from being used as second hops in any Train-II query. We then test whether the model could correctly answer Test-Il queries that involved these excluded triples as second-hop (Figure 6b).
We find that the model consistently fails to answer these Test-Il queries, while performance other queries remained unaffected, even when the same atomic triples were used in the first hop. This confirms that second-hop generalization requires query-level training match: the model must encounter the specific second-hop composition during training to generalize over it. Exposure the same facts in other structural roles is not sufficient.
To further validate this finding, we replicat the ablation under the full base configuration (Ap- pendix D.1) and observe the same failure. Additionally, we analyze how second-hop exposure frequency impacts query acquisition order (Appendix D.2). We found that Test-ll queries involving particular atomic triple as the second hop were answered correctly earlier when that atomic triple appeared more frequently second hop during training.
3 Locating and Characterizing Reasoning Representations
In Section 2, we observed surprising behavioral phenomena that offer new insights into implicit reasoning in Transformers. These findings challenge prevailing assumptions, such the necessity of exposure to atomic facts [40, 36] or the impossibility of OOD generalization [25].
To better understand these behaviors, we shift focus from what the model does to how the model achieves internally. Specifically, we examine the intermediate entity that connects the two relational steps. Any correct solution, least implicitly, passes through this latent bridge, making it a key target for probing the model's reasoning process. To this end, we first introduce the causal
'ee exclude OOD Triples from both configurations to ensure comparability: since the "Train-ll only" configurations contains OOD facts, generalization to Test- Best-Olt is inherently impossible.
5

---

probing method Cross Cross-Query Query Semantic Patching (Section 3.1) to locate these intermediate entities in representations. We then revisit whether internal states are decodable using logit lens (Section 3.2). We finally explore how geometric regularity in these representations supports the model's l's ability to generalize across queries (Section 3.3).
3.1 Locating Intermediate Entity Representations via Cross-Query Patching
To analyze how transformers internally represent intermediate reasoning steps, a crucial first step is to identify where such representations encoded in the model's hidden states.
Existing methods such as linear probing and causal patching offer only partial insight. Linear probing reveals correlations between hidden states and output tokens, but not their causal role in reasoning. Causal patching assesses causal influence, typically measures whether a random source activation affects the target's output, but doesn't 't assess what the activation semantically represents [25].
Cross- Moos-Querry Semantic Patching. To go beyond correlation or superficial causal influence, we introduce cross query semantic patching. a method designed to test whether a hidden representation encodes semantically valid intermediate entity. Specifically, given a source query (e,,'1, r2). ;1,)), we test set of candidate hidden states from different layers and positions (e.g., layer at the r position) that may contain the bridge entity representation. For each candidate, we insert its hidden vector into structurally similar target query (es, To, (7) at the same position, replacing the original hidden state.
If the patched model's prediction changes from the original reasoning path r7 (r6((e))) to (r(r (e1)), this indicates that the inserted represen- tation carries transferable semantic information corresponding the bridge entity.
e1
100
Layer
We apply this patching procedure across multiple layers and token positions, with three settings that differ in both the source of the in- termediate entity and the model's training stage: (1) Phase II with ID-derived intermediate entities, (2) Phase III with ID-derived interme- diate entities, and (3) Phase III with OOD-derived intermediate entities. This alignment ensures that patching is conducted under conditions where the model is capable of reasoning over the relevant intermediate entity type. For completeness, we also report Phase I results, where patching yields negligible success, confirming that reasoning-relewe representations emerge only after generalization.
Layer 2
80
Patching Success Rate (%))
Layer
60
Layer 4
Layer 5-
Layer 6
Layer 7
Layer 8
Detailed per-setting results are presented in Appendix F. We report the average patching success rate across these three settings in Figure 4. This aggregated result shows that effective patching occurs primarily at the ri token position in the middle layers. In the following analyses, use layer 5 at the r token position of our 8-layer GPT-2 model as the reference point, denoting the hidden state as h
Figure 4: Average patching success rate across layers and token positions.
3.2 Explicitly Decodable * Implicitly Informative
Having located the positions encoding intermediate entities, we next ask whether their internal role can be explained using existing interpretability tools. A key assumption in prior work is that hidden state encodes an intermediate entity, it should be decodable into a human- iimaan-iterrpretable token, for example via the logit lens [18, 16, 33, 22, 40]. We test this assumption by measuring the decodability of ha and examining whether decodability aligns with the emergence of reasoning behavior across training phases.
Setup. We adopt the logit lens to evaluate decoding performance in two modes: (1) Immediate probing: projecting the extracted hidden state onto the output vocabulary directly. (2) Full-run probing: patching the extracted hidden state into randomly selected query at the corresponding position, and decoding it after processing through the model's layers. These methods assess whether the hidden state contains token-level signal whether the model itself can internalize and recognize

---

Table 1: Success rates (%) of explicitly decoding intermediate entities from h across reasoning phases and decoding methods.
Immediate Probing
Full-run Probing
Source
Phase!
Phase
Phase III
Phase
Phase II
Phase III
ID-derived
92.1
98.8
99.9 99.8
97.1
99.9
99.9
OOD-derived
67.7
81.3
83.7
98.6
99.7
it. For each phase, we compute the decoding success rate for intermediate entities grouped by their vithnneithher from ID or OOD triples".
Result 1: Decodability does not correlate with reasoning emergence. As shown Table 1, decoding success remains high and stable for ID-derived representations across all phases. However, implicit reasoning capabilities only emerge after Phase II, suggesting that decodability alone does not explain reasoning emergence.
Result No decodability gap between ID and OOD sources during dittriiistii gener- alization. In Phase II, while the model generalizes to ID-ID (Test-ll) queries but fails on ID-OOD (Test-OI) queries, there no significant difference decoding success between ID-derived and OOD- derived representations. This further demonstrates that representations can be equally decodable yet differ whether they are functionally recognized and utilized by the model.
Implication. These results indicate that explicit decodability alone cannot explain when or how representation contributes reasoning. Even when representation can be decoded correctly, the model may not rely on it for reasoning.
To further probe the role of explicit decoding in reasoning, constructed controlled setting where the model was incentivized to represent intermediate entities a decodable form. Interestingly, the model initially attempts this strategy but quickly abandons it, indicating that the model prefers non-explicit representations for generalization. We provide details of this experiment in Appendix E.
3.3 Geometric Regularity of Intermediate Representations via Cosine Lens
The gap between explicit decodability and actual usage motivates a different approach. Instead of asking "Can we decode what this hidden state?", we ask "How is this representation organized across different contexts?"
Most prior work focuses on decoding representations, but we take the reverse approach: given a known intermediate entity, contexts to achieve consistent representations [24]? This reverse mapping is enabled by our earlier analysis in Section 3. which identifies the position h encoding the intermediate entity. At this anchor, we collect hidden states from queries sharing the same intermediate entity (Figure 1), and examine whether these vectors reflect consistent internal pattern.
identify recurring structure in how the model represents it across
To assess consistency, we focus on structural aligmment in the model's 's embedding space, using cosine similarity- common metric for semantic proximity in high-dimensional representations(8. 21, 13]. This allows us to examine whether the model reuses internal abstractions through representational geometry, instead of relying on explicit decodability.
Case Study: Visualizing Representational Clustering. To gain an initial sense of the representa- tional patterns that emerge during training, we visualize the hidden states of randomly selected intermediate entity that appears in multiple two-hop queries. For this entity, we extract h. across relevant queries instances (where the intermediate entity is either ID-derived or OOD-derived) and compute pairwise cosine distances (defined as 1 cosine similarity) between them. We then project these high digh-siieniii vectors into dimensions using Multidimensional Scaling (MDS)
As shown Figure 5, hidden states for a common intermediate entity form distinct geometric patterns across the phases. In Phase I, both ID-derived and OOD-derived representations are scattered; in Phase II, ID-derived representations form tight cosine-space clusters, marking the transition to
'Notably, the origin of an intermediate entity depends solely on the first hop and is independent of the second-hog hop configuration.
7

---

ID-derived
O0D-derived
ID-derived
O0D-derived
10-derived
O0D-derived
1.3
1.91
Comperent
0.2
Component
-0.25
-0.25
-2.9
-0.9
-039
4.7
-4.7
4.7
-1.00
1.50
558
0.5
192
-1.0
1.03 4.05
0.30
8.2 0/00
1.2
550
6.5
-1.0
0.0. 191
Component
Componem:
Component
(a) Phase!
(b) Phase II
(c) Phase
Figure 5: Cosine-space projection of a random intermediate entity across three training phases.
in-distribution reasoning; In Phase III, OOD representations also begin to align with the ID-bbaeed cluster, signaling cross-distribution generalization. This suggests that latent variables are reused not by explicit decoding, but through the emergence of consistent geometric structure.
Quantifying Representational Convergence. Encouraged by this observation, we next quantify the consistency of entity-level representations across the full dataset. We define two metrics: (I) ID Cohesion Score: the average cosine similarity between ID-derived representations and their centroid, reflecting in-distribution consistency. (2) OOD Alignment Score: the average cosine similarity between OOD-derived representations and the same ID centroid. reflecting how well the model unifies cross coss-distribbttin representations. These scores computed on a per-entity basis and averaged across all intermediate entities.
Tracking these scores, we find that the ID Cohesion Score rises steadily, aligning with Test-ll generalization, while the OOD Alignment Score starts increasing later, following the rise in Test-OI performance (Figure 2). This suggests that successful implicit reasoning relies on representational consistency across diverse contexts: only when hidden states for the same entity align closely in cosine space can they be consistently reused for multi-hop inference. Thus, cosine-space clustering emerges as the model's internal mechanism for semantic abstraction and generalization.
Closing the Loop: Explaining Behavioral Phenomena
Drawing the mechanistic evidence in Section 3.3, we revisit our empirical observations in Section 2 and explain how they emerges from the internal dynamics of representation formation.
4.1 Clustering of OOD-Derived Representations Driven by ID Supervision
In Phase III, where the model successfully performs OOD reasoning at the first hop, the clustering of OOD-derived representations in cosine space plays a crucial role. However, while the clustering of ID-derived representations is expected due to direct supervision from Train-ll queries, the alignment of OOD-derived representations with ID clusters is less intuitive. OOD-derived entities are never explicitly supervised as intermediate steps in multi-hop queries, making their eventual clustering with ID entities surprising.
We hypothesize that frequent exposure to atomic triples is the driving factor behind the observed alignment. This exposure leads the model assimilate the OOD representations into the existing ID clusters, thereby stabilizing them. To validate this hypothesis, we designed an ablation study where we varied the ID/OOD ratio across three configurations: 0.8/0.2, 0.5/0.5, and 0.3/0.7. Only the 0.8/0.2 2 ratio demonstrates successful generalization into Phase III, effectively answering Test-OI queries, while the others failed to achieve this stage. Detailed results are provided in Appendix H.
4.2 ID Triple Queries Accelerate Generalization by Constraining the Representation Space
Knowing that generalization relies on the clustering of intermediate representations, we link the acceleration effect (Section 2.3) brought by ID triple to the property of autoregressive Transformers:
8

---

both ID triple queries (f1, r1) and two-hop queries (E1, r2) produce the same hidden state at the P1 token position due to causal masking, where the model only attends tokens preceding r1-
Since our mechanistic analyses anchor at r1 position (h ). the hidden states we study are indis- tinguishable across both query formats. This means ID triple supervision directly shapes the same representations used in multi-hop reasoning. While the ID triple task optimizes for explicit decod- ing-mappinng (e1,r1) to eg- doesn't 't guarantee functional reasoning (Section 3.2). However, plays a crucial role by constraining the Ti hidden state to lie within a subspace that supports entity decoding, thereby limiting the model's search space during generalization
smaller region.
To validate this mechanism, we construct an ablation configuration that removes a subset of ID triple (e.g., (cA. rl, r1, }) from training, while still including their corresponding two-hop compositions (e.g., (eA, r1, r2)) in Train-II. We then test whether the model can recover the held-out ID) triples at test time (e.g., given input (eA, P1), predict e The results align with expectations: the model is able to correctly predict é (see Appendix for details). This outcome demonstrates that the rı hidden state associated with (EA. r1) lies in the same region as other Train-Il queries sharing latent (eg) a region already shaped by remaining ID triples involving eB (e.g., (ex, (7, (B)):
(IA-TI)
share in position with
(cA. F1, (1, (2) T7, (EB),
constrained by
(ex,
decodable eB
beld-out triple query
retained ID supervision
Train with latent (cB)
This supports the claim that ID triple supervision constrains the ri hidden state a decodable region, facilitating clustering. Consequently, as the model enters Phase II and learns two-hop reasoning, refines representations within an already structured subspace, speeding up representational conver- gence and behavioral generalization compared to a without ID configuration where clustering must be learned from scratch.
4.3
Why the First Hop, and Only the First Hop, Generalizes to OOD?
The intended supervision signal from Train-II queries is to teach the model structured two-step
_ reasoning process: token(e1) " latent(ez) '2n token (s), where both ri and r2 22hpp are relations applied in purely compositional context, independent of atomic triple learning (Section 2.3).
In contrast, atomic triples expose the model to direct mappings of the form: token(ei) y te2) which train a shallow predictive behavior over observed fact pairs.
is therefore surprising that models can correctly answer Test-OI queries, suggesting that the mapping p somehow transfers into v1 allowing the model to reuse OOD-derived token (e2) representations for implicit reasoning. However, this apparent generalization is in fact side-effect of representational alignment induced by ID triples.
As established in Section 4.2, the hidden state the T1 token position is shared across atomic and 2-hop queries, encouraging the model to align latent(e2) with teken(e2) for ID triples. Separately, as shown in Section 4.1, the model gradually pulls OOD-derived representations into the same cosine- space cluster as ID-derived representations. These two mechanisms together enables OOD-derived latent(e2) (2)) to match the expected format of 2hop:
2hop:
doD-der token(ez) pulled by ID-derived token (2)
align with
latent(e2);
In this sense, the model doesn't truly generalize OOD ri alomic into r "cheats" by reusing shared representation scaffolds shaped by ID triples. The success on Test-OI queries is thus illusory: what appears to be a generalization is in fact a misalignment between model structure and supervision.
Viewed from this perspective, the failure of second-hoo hopp generalization is not an exception. Unlike the first hop, the model cannot rely on representational anchoring from shared prefixes and must learn behavior through direct query-level supervision. In contrast, first-hop OOD generalization is an exception, made possible by incidental alignment from overlapping input contexts, hence this does not extend to deeper reasoning, A similar pattern holds in 3-hop reasoning: generalization is only

---

observed when the reasoning path beyond the first hop remains within ID (Test-lll and Test-Oll), highlighting the need for explicit supervision later steps (Appendix C).
We hypothesize that without the representational anchoring effect induced by ID supervision, OOD triples fail to form functionally useful intermediate representations. To test this, we construct configuration with only OOD triples and Train-II queries, removing ID triples. In this setup, the model fails on Test-OI generalization, confirming that in the absence of ID-based anchoring, OOD triples alone cannot support implicit multi-hop reasoning, See Appendix J for details.
5
Discussion
Our study, conducted in a controlled symbolic dataset environment, reveals key insights into the mech- anisms of implicit reasoning in transformers, highlighting specific patterns and behaviors that clarify how multi-hop implicit reasoning emerges. These findings may provide valuable answers to existing questions about the implicit reasoning capabilities of LLMss. For instance, our observation regarding the requirement for query-level match offers a potential explanation for why knowledge learned from single-hop tasks does not easily transfer to multi-hop reasoning in L.MS [3, 39, 31]. However, it is important to note that LLMs operate with far richer and more complex knowledge bases, and their internal knowledge interaction mechanisms likely differ from those in our controlled environment. Therefore, while our findings offer useful insights, they should be regarded as preliminary guidance rather than a complete explanation of the reasoning dynamics in LLMs.
Acknowledgement
This work is supported by Beijing Natural Science Foundation (L243006), National Natural Science Foundation of China (62476150), and the Tsinghua University ( (Department of Computer Science and (achholloyy-Simmeene Ltd., China Joint Research Center for Industrial Intelligence and Internet of Things (JCIIOT).
References
[1] Anthoopic. Claude 3.7 sonnet, 2025. URL 2025-05-15.
hwtpp//cc....ccccaaac/////oeeeee Accessed:
Anthropic. Reasoning models don't always say what they think, 2025, URL htwp://eeaarcc/reaoninn---- models- dont soynt-syy-teee say-think. Accessed: 2025-05-15.
[3] Mikita Balesni, Tomek Korbak, and Owain Evans. The two-hop curse: Lims trained on a-> b, b-> c fail learn a-> c. arXiv preprint arXiv:2411. 16353, 2024.
[4] Aryasomayajula Ram Bharadwaj. Understanding hidden computations in of-in-of-thought reasoning. arXiv preprint arXiv:2412.04533 2024.
[5] Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models multi-hop queries. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14113 14130, 2024.
[6] Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long tfahn-of-thought for reasoning large language models. arXiv preprint Xrvvv 09567, 2025.
[7] Google DeepMind. Gemini 2.5 pro: Our most intelligent ai model, 2025. URL httpp://bbplgg.: google/techee cessed: 2025-05-15.
wewpeddlltiiniignnnnppe mapdates-march-20000 Ac-
[8] Valeriya Goloviznina and Evgeny Kotelnikov. I've got the "answer"! interpretation of Ilms hidden states in question answering. International Conference on Applications of Natural Language to Information Systems, pages 106-120. Springer, 2024.
[9] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. ree:seek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint 2011 2025.
10

---

[10] Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models reason in continuous latent space. arXiv preprint arXiv:2412.06769 2024.
[11] Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of of oftthutght of Iims via reinforcement learning. arXiv preprint #:Xiv2504.01290 2025.
[12] Tianjie Ju. Yijin Chen, Xinwei Yuan, Zhuosheng Zhang, Wei Du Yubin Zheng, and Gongshen Liu. Investigating multi-hop factual shortcuts in knowledge editing of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 8987-9001, 2024.
[13] William Jurayj. William Rudman, and Carsten Eickhoff. Garden path traversal in gpt-2. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 305-313, 2022.
[14] Mikail Khona, Maya Okawa, Jan Hula, Rahul Ramesh, Kento Nishi, Robert P. Dick, Ekdeep Singh Lubana, and Hidenori Tanaka. Towards an understanding of stepwise inference in transformers: A synthetic graph navigation model. In Forty-first International Conference Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024, URL https: / http///pperreee/eeeee aepeerevieu.net/forrre www./eruu/oreidee 8VEGkphQaK.
[15] KimiTeem, Angang Du Bofci Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:250. 12599, 2025.
[16] Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, and Ying Wei. Understanding and patching compositional reasoning in llms. In Findings of the Association for Computational Linguistics ACL 2024, pages 9668 9688-96888882 2024.
[17] Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking. arXiv preprint arX11:2504.09838 2025.
[18] nostalgebraist. Interpreting gpt: The logit lens, 2020. URL https: htress//ww..essrrrnng hwtpswwwww.ooooo/opeeeeeeeeeee/e www/posts/ AckaBBoDDpaalvvvrr/nnnett htt.::/rrrrnttteeeeennne gpt- too-ttheelgttttoo
Accessed: 2025-05-01.
[19] OpenAL Introducing openai o3 and 04-mini, 2025. introducing- o3- and- o4-mini/. Accessed: 2025-05-15.
URL /tttp:://penna.coo//ii index/
[20] Jacob Pfau, William Mcrrill, and Samuel R Bowman. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprint ar arXiv:2404.15758 2024.
[21] Ella Rabinovich, Samuel Ackerman, Orna Raz, Eitan Farchi, and Ateret Anaby-Tavor. Predicting question- answering performance of large language models through semantic consistency. In The 2023 Conference on Empirical Methods in Natural Language Processing, volume 10, page 138, 2023.
[22] Mansi Sakarvadia. Towards interpreting language models: A case study in multi-hop reasoning. arXiv preprint arXiv:2411.05037 2024.
[23] Yuval Shalev, Amir Feder, and Ariel Goldstein. Distributional reasoning in Ilms: Parallel reasoning processes in multi-hop reasoning. arXiv preprint 24Xi:22406.13888 2024.
[24] Boshi Wang and Huan Sun. Is the reversal curse a binding problem? uncovering limitations of transformers from a basic generalization failure. arXiv preprint arXiv:2504.0192. 011988 2025.
[25] Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokking of implicit reasoning in transformers: A mechanistic journey the edge of generalization. Advances in Neural Information Processing Systems, 37:95238-9528 952639952552299 2024.
[26] Siwei Wang, Yifei Shen, Shi Feng. Haoran Sun, Shang-Hua Teng, and Wei Chen. ALPINE: un- veiling the planning capability of autoregressive learning in language models. In Amir Glober- sons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances Neural Information Processing Systems 38: Annual Confer- ence on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, De- cember 10- 15, 2024, 2024. URL http: //tppe//ppp/ee//e . nips, cc/paper__eceeeeeeeeeeeeeeee fateeeffilsspppprreereee (aapper//224// /2024/2ahhh 4848cb2c8410bbal77 wtt.eeaaawa.1111122233332333 Abstract- Conference html.
11

---

[27] Xinyi Wang, Alfonso Amayuelas, Kexun Zhang, Liangming Pan. Wenhu Chen, and William Yang Wang. Understanding reasoning ability of language models from the perspective of reasoning paths aggregation. In International Conference on Machine Learning, pages 50026-50042. PMLR, 2024.
[28] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of- thoughttt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[29] xAL Grok beta the age of reasoning agents, 2025, URL htLLtts:///x.aabbog/ggoo/ 2025-05-15.
Accessed:
[30] Amy Xin, Jinxin Liu, Zijun Yao, Zhicheng Lee, Shulin Cao, Lei Hou, and Juanzi Li. Atomr: Atomic operator-empooered large language models for heterogeneous knowledge reasoning. arXiv preprint arXiv:2411. 16495, 2024.
[31] Ruoxi Xu, Yunjie Ji. Boxi Cao. Yaojie Lu, Hongyu Lin, Xianpei Han, Ben He, Yingfei Sun, Xiangang Li. and Le Sun. Memorizing is not enough: Deep knowledge injection through reasoning. arXiv preprint arXiv:2504.
2025.
[32] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang. Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang. Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang. Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenhiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan. Yang Su. Yichang Zhang. Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505 258508850 2025. URL https: : //trx/aavvvv///////////ee//aaava org/ abs/2505. 09388.
[33] Sohee Yang. Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi hop reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 10210-10222 2024.
[34] Sohee Yang. Nora Kassner, Elena Gribovskaya, Sebastian Riedel, and Mor Geva. Do large language models perform latent multi-hop hopp reasoning without exploiting shortcuts? arXiv preprint arXiv:241 16666, 2024.
[35] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths. Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:1 1809-1889111000 11822, 2023.
[36] Yunzhi Yao, Jizhan Fang. Jia Chen Gu, Ningyu Zhang, Shumin Deng, Huajun Chen, and Nanyun Peng. Cake: Circuit- 2025.
arcuit--ware aware editing enables generalizable knowledge learners. arXiv preprint arXiv:2503.16355
[37] Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system arXiv preprint arXiv:2407. 06023, 2024.
[38] Mengqi Zhang. Bowen Fang, Qiang Liu, Pengjie Ren, Shu Wu, Zhumin Chen, and Liang Wang. En- bancing multi-hop reasoning through knowledge crasure in large language model editing, arXiv preprint arXiv:2408. 12456, 2024.
[39] Xiao Zhang, Miao Li, and Ji Wu. Co-occurrence is not factual association in language models. In The eight Annual Conference on Neural Information Processing Systems, 2024.
[40] Zhuoran Zhang, Yongxiang Li, Zijian Kan, Keyuan Cheng, Lijic Hu, and Di Wang. Locate-then-edit for multi-hop factual recall under knowledge editing. arXiv preprint arXiv:2410.06331, 2024.
[41] Zexuan Zhong, Zhengxuan Wu, Christopher D Manning. Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. In Proceedings of the 20233 Conference on Empirical Methods in Natural Language Processing, pages 15686-15702, 2023.
[42] Hanlin Zhu, Baihe Huang, Shaolun Zhang, Michael Jordan, Jiantao Jiao, Yuandong Tian, and Stuart J Russell. Towards a theoretical understanding of reverralll curse" via training dynamics. Advances in Neural Information Processing Systems, 37:90473-90513,, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction introduces the central focus the study, which is understanding how implicit multi-hop reasoning emerges in transformers during training.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We acknowledges in the Discussion section that the results in this paper should viewed as preliminary and may not fully generalize to more complex, real-world large language model settings.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
Justification: The paper primarily discusses empirical observations and provides mechanistic analyses through diagnostic tools (like cross-query semantic patching and cosine-based representational analysis) rather than formal theorems or mathematical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The paper outlines a detailed experimental setup, including a symbolic training environment and specific configurations for data generation. Information such as the exact model parameters, hyperparameters, or training steps necessary is provided in the Appendix G.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
14

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The data and code will be attached in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
Suuides/CodeSumiisse
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper provides sufficient experimental details understand the results. Details of data construction are mentioned in Appendix B; hyperparameters like the learning rate and optimizer type are explicitly mentioned in Appendix G.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: There are too many configurations our experiments, error bars are thus not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Information on the computer resources is provided in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics ttttps://nat http://:////.p.///eeee http://uurrrssccppullieee//////e eetpubiie//hhissuuieellllllll ines?
Answer: [Yes]
Justification: We did reviewed the NeurlPS Code Ethics and the research conducted in this paper adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
16

---

While the paper mentions potential positive implications for LLM interpretabil- ity (improved transparency of reasoning processes), it does not explicitly discuss negative societal impacts.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We utilizes symbolic dataset that does not contain any real sensitive informa- tion, and therefore poses no risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper uses open-source models, and the creators or original owners of these models have been properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
17

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will provide detailed documentation for all new assets, including datasets, models, and code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper doesn't involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper doesn't 't involve crowdsourcing research with human subjects.
18

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https: pettpp://neet /tt///eeers..//e/ee theuurpps.cc/Coofreeeeee eeee/ooeeeeecce/2225//////////// for what should or should not be described.
19

---

A Related Work
Mechanistic Exploration of Implicit Reasoning. The initial focus on the mechanisms of implicit reasoning arose from the discovery that traditional single-hop knowledge editing methods are inef- fective in the context of multi-hop implicit reasoning [41, 38, 12]. This phenomenon since gained more significant attention, prompting researchers to investigate the mechanisms underlying implicit reasoning. Some works suggest that the failure of implicit reasoning is due to the intermediate entities not being properly processed [22, 33]; Other works suggest that the reason lies in the intermediate entities being processed, but not being passed to the correct position [16, 36, 5]. The parallel explo- ration paths within the model are also research direction [23, 27]. Recent works have found that the different knowledge composed as different hops used in implicit reasoning is stored in different layers of the model [40, 36]. Additionally, some studies propose that models rely shortcuts to successfully complete implicit reasoning [34].
Conditions for Learning Implicit Reasoning. Some works have found that the conditions for models to learn implicit reasoning are quite demanding. Some have discovered through custom symbolic datasets that models generalize implicit reasoning abilities only after grokking, and that they require highly compositional data [25]. Other works have found that simply providing optical knowledge is insufficient for enabling models to perform implicit reasoning, it requires training with corresponding multi-hop reasoning data [31, 39, 42]. Moreover, it has been observed that when the two pieces of foundational knowledge used for 2-hop implicit reasoning appear in different paragraphs of the training corpus, the model struggles to combine them for implicit reasoning. However, when these pieces appear within the same paragraph, the model's accuracy in answering the corresponding 2-hop queries significantly improves [3].
Probing Intermediate Entities. As for the probing tools, most of the work uses decoding- -aasdi methods to probe intermediate entities. Many works assume that if a model encodes an intermediate entity, it should be able to explicitly decode it. They use the logit lens [18]. evidence of the presence of intermediate entities in implicit reasoning [22, 33, 16]. A few studies have also trained a linear transformation layer to decode intermediate entities [23].
Although current Shain-of-Thought (CoT) based reasoning models have shown impressive performance, some perspectives argue that the thought chains do not truly reflect the model's reasoning process [17, 2]. Some works have begun to explore Chain of Thought (CoT) methods that do not verbalize intermediate steps. One approach is to replace the Chain of Thought (CoT) with dots, adjusting the number of dots based on the original length of the thought chain [20, 4]. Another approach avoid verbalizing is to bypass decoding the tokens in the CoT process and directly use the last hidden state from the previous step as the input vector for the next step [10]. These approaches have achieved results comparable traditional CoT on specific tasks.
Latent COT.
B Dataset Details
Our data construction pipeline is adapted from the open-source code released by Wang et al. [25], with modifications to support fine-grained query-level control.
Entity and Relation Vocabulary. In all configurations, we construct a symbolic environment consisting of 2000 entities and 200 relations, each assigned a unique token with no inherent semantics. Since the model is trained from scratch, it has no prior knowledge about the symbols, and learning depends entirely on compositional supervision.
Atomic Triple Generation. We generate 40000 atomic triples of the form (e1, -> e2, where each entity é is randomly assigned 20 outgoing relations, and for each such relation r', the tail entity e2 is sampled uniformly from all entities. These triples are then randomly partitioned at the triple level into ID and OOD subsets, with a default OOD ratio of 5%, while entities and relations remain globally shared across all subsets.
2-Hop Query Construction. Two-hop queries are compositional chains of the form (e1, 71, r2) -> E3, where the model must implicitly reason through intermediate entity e2. These queries
20

---

constructed by pairing atomic triples: if both (e1,T1,ey) and (e2, r2, e3) exist, the corresponding query is eligible. Training queries (Train-II) are sampled from the of ID-ID chains. The remaining queries are categorized as Test-I1, Test-10, Test-O1, or Test-O0 depending on whether the first and second hops are or OOD.
We construct the training set with a 7.2:1 ratio of Train-II queries to ID atomic triples to ensure compositional supervision dominates, and sample fixed test of 3,000 examples for each type. Table 2 summarizes the key dataset statistics.
Illustrations of Data Construction Configurations. To aid in understanding the dataset construc- tion process, we provide two representative configurations as illustrations (Figure 6).
Second Hop Ablation
Base Configuration
Train
Train
Test
Train-Il
OOD Triples
Triples
Test-il
A
Test-O1
Test
Test-Il
Train-ll
Test-OO
Test-1O
B
(a) Basc configuration
(b) Sccond-hop ablation
Figure 6: Two representative data construction configurations. (a) Base configuration: Contains all atomic triples and Train-Il queries, with evaluation covering all test query types. (b) Second-hop ablation configuration: A targeted setup where a subset of atomic triples (e.g., (eg, To, (F) are excluded from appearing as second hops in any training query, while corresponding second-hop queries remain present the set. set.
Table 2: Dataset Statistics
Data Type
Split
Count
Entities
2000
Vocabulary
Relations
200
In-Distribution (ID)
38000
Atomic Triples
Out-of-Distribution (OOD)
2000
Train-II (ID - ID)
273600
Test-Il (ID + ID)
3,000
2-hop Queries
Test-IO (ID - 00D)
3,000
Test-Ol (OOD ID)
3,000
Test-OO (OOD + OOD)
3,000
Design Choices and Assumptions.
In designing our dataset, we made two key parameter choices to support our analysis focus and the intended analogy real-world LLM behavior:
First, we fix a relatively high ratio of Train-II to ID Triples (7.2:1), ensuring that compositional supervision dominates over direct fact learning. While prior work [25] has shown that high Train-II ID ratio is necessary for generalization on Test-ll queries, we do not treat this ratio as variable of interest. Instead, maintain it at a sufficient level to ensure the emergence of in- distribution reasoning, and shift our focus to more challenging phenomena such as distriiistintttion generalization and the underlying mechanisms that support generalizations.
Second, we choose high ratio of ID to OOD atomic triples (95% ID), which critical for the representational alignment effects discussed in Section 3 and further validated in Appendix H. In
21

---

particular, Phase III generalization relies on OOD -0D-veeived intermediate representations aligning with the subspace formed by ID-derived ones an effect that only arises when ID triples are sufficienlly dominant.
We acknowledge that these high-ratio settings may appear biased. However, they reflect plausible properties of real-world pretraining corpora: (1) compositional chains (analogous to our Train-II queries) are likely more common than isolated fact triples; and (2) most knowledge entities are learned in richly connected contexts (analogous to our ID triples), while only a minority are sparsely anchored or appear in isolation (analogous to our OOD triples). From this perspective, our symbolic setup not only enables controlled analysis, but also approximates realistic data imbalance patterns observed in LLM pretraining. Accordingly, we treat both ratios as default conditions rather than ablation variables.
C Extension to 3-Hop Reasoning
We extend our symbolic framework support 3-hop queries and verify that the behavioral and mechanistic patterns observed in the 2-hop setting also emerge in deeper compositional regimes.
C.1 3-hop Dataset Construction
The overall construction methodology mirrors that of 2-hop queries, as described in Appendix B, with an additional relational step. These queries take the form (c1, f1, F2, r3) - c4, where the model must implicitly traverse three relational steps through two intermediate entities.
However, the increased compositional depth introduces the combinatorial nature of 3-hop chaining (i.e., atomic composition scales cubically) leads to test set sparsity Test-O00) if the OOD triple proportion is too small. To mitigate this, we reduce the vocabulary size to 1000 entities and 100 relations, and increase the OOD ratio to 20%, while still maintaining majority of ID supervision discussed in Appendix B. These adjustments ensure sufficient coverage across all evaluation regimes, including out-of- offf----distittiinn settings. Table summarizes the key statistics of 3-hop dataset.
data balancing challenges. In particular,
Table 3: 3-hop Dataset Statistics
Data Type
Split
Count
Entities
1000
Vocabulary
Relations
100
In-Distribution (ID)
8000
Atomic Triples
Out-of-Distribution (OOD)
2000
Train-IIf (ID + ID) - ID)
120000
Test-Ill (ID ID ID))
1,000
3-hop Queries
Test-IIO (II) > +ID - -0DD)
1,000
Test-IOI (ID -
1,000
Test-100 (ID + OOD + OOD)
1,000
Test-Oll (OOD - ID
1,000
Test-O10 (OOD + ID O0D)
1,000
Test-OOI (OOD OOD
1,000
Test-O00 (OOD + OOD - OOD)
1,000
C.2 Training Dynamics and Generalization Patterns
We evaluate model behavior the 3-hop dataset using the same model, training regime, and diagnostic tools as in the 2-hop setting. Figure summarizes the accuracy trajectories across all test query types and clustering metrics of intermediate representations.
Compared to 2-hop queries, 3-hop reasoning introduces an additional intermediate entity, resulting in two latent steps: e1 M, e2 /3 6g Ti 64. analyze how the model internally represents these latent entities, extend our diagnostic metrics accordingly.
22

---

Trilles
Test-100 Test-Oll
Test-00l
00D Alignment
000 Triples
Test-11O
Test-000
e3 Cohesion
Train-IIl
Test-101
Tast-Oio
e2
1.0
100
0.8
80
Cosine Similarity
Accuracy (%)
0.6
0.4
40
20
0.2
0.0
10a
104
10a
109
107
Figure 7: Training dynamics in the 3-hop setting. Accuracy and representation metrics exhibit consistent generalization patterns with the 2-hop case.
Specifically, we extract: (1) h the hidden the ri token in layer 5, to represent the internal encoding of e2 (same as in the 2-hop setting); (2) h the hidden state the 72 token in layer 6, to represent the encoding of e3.
Based on these, compute the following representational clustering metrics: (1) ID 62 Cohesion: average cosine similarity among representations of the same ID-derived 62 entity; (2) ID cg Cohesion: the analogous metric computed over ey representations. (3) OOD e2 Alignment: cosine similarity between OOD-derived 62 representations and the corresponding ID-derived centroid. We not compute alignment for eg because only the first hop (e2) can successfully generalize from OOD inputs.
We observe two key generalization patterns consistent with the 2-hop results:
(1) In-distribution generalization emerges reliably: The model successfully generalizes to Test-III queries (ID + ID + ID), with accuracy rising steadily during training.
(2) ditt---disttibttionn generalization remains constrained to the first hop: Among all OOD- containing query types, only Test-Oll (OOD > ID > ID) shows significant improvement. Other configurations where OOD triples appear in the second third hop (e.g., Test-11O, IOI, 000) fail to generalize. This reinforces the bottleneck observed in the 2-hop case: query-level exposure is necessary for downstream relational generalization.
Furthermore, OOD e2 Alignment closely tracks Test-OlI performance, while ID e2 and e3 Cohesion metrics rise in concert with Test-Ill accuracy. Together, these results highlight that representational clustering at multiple relational depths serves as the internal mechanism enabling successful multi-hop reasoning.
While we do not repeat behavioral ablations from the 2-hop setting (e.g., acceleration from ID triple exposure, second-hop query-level matching), we note that these phenomena are well explained by the clustering dynamics reported above. particular, similar representational bottlenecks and alignment requirements arise at each reasoning step, such that second- and third-hop generalization depend on the same clustering dynamics as first hop. As result, the cohesion and alignment metrics report suffice to capture the core generalization behaviors in the 3-hop case.
23

---

D Additional Validation of Query-Level Requirements in Second-Hop Generalization
D.1
Verifying Second-Hop Failure under Full Supervision
To ensure the observed failure of second-hop generalization not an artifact of minimal training configurations, we repeat the second-hop ablation experiment under the full base setting described in Section 2.1. Specifically, we exclude a subset of atomic triples (e.g., (EB, Ts, ep)) from appearing in any second-hop hopp positions during training, while allowing them in atomic queries or first-hop usage (Figure 8).
Despite the model being exposed to these triples in other structural roles, it fails to generalize to Test-Il queries that require them as second-hop compositions. Importantly, performance on all other query types remains unaffected. This confirms that second-hop generalization requires direct query-level supervision: exposure fact in other contexts is insufficient for enabling role in compositional reasoning.
Base Configuration with Second-Hop Ablation
Train
Test
OOD Triples
Triples
Test-ll
x
B
Y
Z
Test-OI
Train-ll
Test-OO
Test-10
Figure 8: Illustration of the second-hoo hop ablation under the full base configuration. A subset of atomic triples (e.g., (eB, eF)) is excluded from appearing as second hops during training. The rest of the data remains unchanged, allowing controlled evaluation second-hop generalization.
D.2 Accuracy vs. Second-Hop Exposure Frequency
To further investigate the role of query-level exposure in second-hop generalization, we examine whether second-hop triples that appear more frequently in training queries are learned earlier, using the full base configuration.
We select fixed training checkpoint in the early part of Phase II spescciffcaally when Test-OI accuracy reaches approximately 50%-and we group second-hop triples by their frequency of occurrence in Train-ll queries. For each frequency group k. we compute the average accuracy over all Test-Il queries whose second-hop triple appeared exactly i times in the training set.
The results reveal a clear trend: higher second-hop exposure frequency during training leads to greater accuracy corresponding queries this intermediate phase (see Figure 9), reinforcing the causal role of second-hop participation in enabling generalization.
E Probing Preference for Explicit Decoding
To test whether the model prefers to encode intermediate entities an explicitly decodable form, we construct new configuration that reveals the model's decoding preference behaviorally.
This configuration (Figure 10a) removes triples from the training data, but retains (i) all Train-II 2- h-hop queries, which require reasoning through ID-derived intermediate entities, and (ii all OOD

---

100
100
Test-Ol Accuracy at Early Phase II (%)
90.918
05,56%
73.80%
61.77%
40
27.88%
20
0.00%
10
14
Second-Hop Triple Frequency in Train-ll Queries
Figure 9: Average Test-Ol accuracy early Phase II (approx. 50%) grouped by second-hop triple frequency in Train-II queries.
triples, which still provide supervision for decoding the position. Crucially, tail entities in both ID and OOD triples share the same tail entity vocabulary, encouraging the model to apply similar decoding strategies across domains.
Decoding Preference Configuration
100
ID Triples
Train
Test
OOD Triples
OOD Triples
ID Triples
Accuracy (%)
80
Test-ll
40
20
Train-Il
Test-1l
10a
10
10a
Checkpoints (Training Steps)
(a) Decoding Preference Configuration
Accuracy dynamics under this configuration.
Figure 10: Decoding Preference Experiment. (a) Experimental setup where ID triples are excluded from training and only used for testing., while the model trained on OOD triples which share the same tail entity vocabulary shows that the model initially recovers held-out ID triples, suggesting an attempt at explicit decoding, but later abandons this strategy.
the ID triples, and all Train-ll queries. (b) Accuracy over training steps
This setup leverages the shared hidden state structure between atomic and two-hop queries discussed in Section 4.2, implying that if the model encodes the intermediate entity in a decodable form during 2-hop reasoning, it should be able to recover ID triples (e.g., given a ID query (e r1), predict eB) even if these triples were never seen during training.
Figure 10b shows the result. Initially, the model answers some triples correctly, suggesting an early attempt for explicit decoding. However, this accuracy soon declines, while the performance on Test-ll continues to rise. This indicates that the model abandons decodable representations favor of internal ones that support reasoning but cannot be directly decoded. Explicit decoding, while initially attempted, not sustained-suggee available.
it not the model's preferred solution when alternatives are
25

---

Additional Results for Cross-Query Causal Patching
We provide the individual patching success rates for each of the three settings: Phase II (ID- (D-derived), Phase III (ID-derived), and Phase III (OOD-derived). These results complement the averaged trend shown in the main text (Figure 4) and confirm the consistency of intermediate entity localization across training stages and generalization regimes.
To provide full developmental picture, we also include Phase patching results in Figure Ila. These results exhibit near-zero success across positions and layers, consistent with the absence of any generalization behavior at this stage. This reinforces our claim that meaningful intermediate representations emerge only after compositional reasoning capabilities begin to develop.
F
2
Layer
Layer
Layer
Layer
Layer 2-
Fatching Success Rate (%)
Layer
40
Patching Success Palle (%)
Layer
Patching Success (%)
Layer 2
Layer 3
Layer 3
Layer 3
Layer
Layer
44
Layer
Layer 4
Layer
Layer
Layer
Fatching
Layer
Layer
Layer
Layer
Layer 7-
Layer 7-
Layer 7-
Layer 7-
Layer 8
Layer 8
Layer 8
Layer 8
(a) Phase I: ID-derived
(b) Phase II: ID-derived
(c) Phase III: ID-derived
(d) Phase III: derived
OOD-
Figure 11: Patching success rate across layers and token positions for different phases and data sources.
G
Training Details and Bigger Models
G.1 Training Details
Our training procedure largely follows the public implementation provided by Wang et al. [25], with few modifications to accommodate our specific experimental setting.
The model is a decoder-only Transformer, identical in architecture to GPT-2, with 8 layers, 768 hidden dimensions, and 12 attention heads. Optimization performed using AdamW with learning rate of x 10-1, 2000 warm-up steps, weight decay 0.1, and a batch size 1024. All models are trained significantly beyond convergence allow observation of late-stage generalization behavior (Section 2.2).
Training is conducted NVIDIA RTX 3090 GPUs, and the maximum training duration is extended to weeks to ensure stable cross-distributions generalization. All experiments are implemented using the same PyTorch and Huggingface Transformers framework in the original codebase.
G.2 Scaling Analysis: Dynamics and Alignment in Larger Models
We extend our main experimental setup by training larger model, Qwen2.5-1.5B. under the same base configuration. Our goal is to examine whether the developmental trajectory of multi-hop reasoning observed smaller models persists scale, and further investigate the alignment between behavioral accuracy and representational metrics.
The training progresses successfully through Phase I (memorization) and Phase II (in-distribution generalization), and reaches Phase III distriiiittiitoonn generalization). However, we observe increased instability during Phase III: although the model demonstrates the ability generalize to Test-Ol queries, the performance exhibits significant fluctuations. We report the Test-Il and Test-OI accuracies, alongside the ID Cohesion and OOD Alignment metrics (Figure 12).
Interestingly, we find that the Test-ll accuracy does not rise in lockstep with the ID Cohesion metric, in contrast to the strong correlation observed in smaller models (Figure 2), We interpret this
26

---

Test-ll Test-Ol
OOD Allgnment
100
1.0
Average Cosine Similarity
80
Accuracy
60
40
0.4
0.0
250000
500000
Checkpoints (Training Steps)
Test-ll and Test-Ol accuracy and ID Cohesion and OOD Alignment metrics over training steps in the Qwen2.5-1.5B model. Although the model reaches Phase III generalization, substantial variance is observed in both Test-OI accuracy and OOD Alignment, which nonetheless remain tightly coupled. In contrast, Test-II aceuracy does closely track the ID Cohesion metric, suggesting representational bottleneck at the second relational step.
Figure 12:
decoupling as evidence that ID Cohesion is a necessary but not sufficient condition for successful Test-ll generalization. While a coherent latent space required to support compositional reasoning, achieving high Test-ll accuracy also depends on the model's ability to utilize these representations in executing the second relational step. In other words, beyond aligning representations, the model must also learn to map from aligned intermediate state to the correct final output via the second-hop relation.
In smaller models, these two prpects-reepreenttationn alignment and second-hop reasoning- to emerge together as part of single learning phase, leading tight coupling between ID Cohesion and Test-ll performance. In contrast, larger models appear to decouple these processes: representational clustering may occur early, while second-hop reasoning capabilities require additional training to fully mature. As result, the second relational step becomes the dominant bottleneck in Phase II generalization.
This interpretation is further supported by the close alignment between the OOD Alignment metric and Test-OI accuracy. Because second-hop reasoning over ID triples is already well established by Phase II, generalization on Test-OI becomes predominantly constrained by whether OOD-derived intermediate representations have successfully aligned with the ID-centric latent space. This tight correlation holds across model scales: in both the main 2-hop setup with smaller models and the 3-hop results in Appendix C (e.g., alignment between Test-Oll and OOD-derived clustering), the emergence of Phase III generalization closely tracks OOD Alignment. In our mroee-model experiment, although the Test-OI accuracy exhibits high variance, its fluctuations are closely mirrored by the OOD Alignment metric, reinforcing our hypothesis.
We leave the optimization of Phase III training strategies for larger models to future work. Our findings suggest that alignment-basee representational diagnostics may serve as useful guides for tuning training schedules or data exposure in this regime, and we encourage future work explore these directions further.
H Validation of Phase III Emergence via ID/OOD Ratio Ablation
To validate our hypothesis in Section 4.1 that the emergence of cross-distribution generalization (Phase III) depends on the dominance of in-distribution (ID) supervision, we conduct an ablation study by varying the ID/OOD ratio of atomic triples under the base configuration.
27

---

Experimental Setup. We begin with the full base configuration, which includes all atomic triples (both ID and OOD) and the complete set of Train-Il queries. We fix the total number of atomic triples and vary the ID/OOD ratio while keeping all other components of training unchanged. Specifically, 20% OOD, 50% ID / 50% OOD, and 30% ID 70% OOD. In all cases, the Train-II / ID ratio is held constant, as prior work Wang et al. [25] identifies this as critical factor for Phase generalization.
we test three settings: 80%
Results. All three ID/OOD configurations unsurprisingly reach Phase I and Phase II, to focus on the emergence of Phase III. we report the Test-Ol accuracy and OOD Aligmment Score, which capture the model's ability to reason across distributions and align OOD- intermediate representations with the ID-induced cluster structure.
As shown in Figure 13, in the 0.8/0.2 setting, both Test-QI accuracy and OOD Alignment Score increase together during training, indicating that the model successfully assimilates OOD-derived intermediate representations into the ID-induced subspace and is able to reuse them for cross- distribution reasoning. In contrast, in the 0.5/0.5 and 0.3/0.7 settings, both metrics remain consistently low, suggesting that the model fails to form aligned representations for OOD triples and consequently cannot generalize to Test-OI queries. This divergence across configurations highlights that strong ID supervision is essential for enabling Phase III generalization.
Tost 01
0OD Aignmmnt
T:st 01
00D Aligmment
Tost 01
000 Aignment
160
1.0
Similanty
100
Average Cosine Similanty
160
1.0
Average Similanty
Accuracy (%)
80-
Accuracy (%)
60
0-
60
0.6
60
0,6
40
Average
20
20
3002000 6606000
0.0
0-
2500000
Sc0000n
0.0
2500000
5000000
0.0
Checkpaints (Training Steps)
Checkpaints (Training Steps)
Checkpaints (Training Steps)
(a) ID/OOD = 0.8/0.2
(b) ID/OOD 0.5/0.5
(c) ID/OOD = 0.3/0.7
Figure 13: Test-O1 accuracy and OOD Alignment Score under different ID/OOD splits. All configurations successfully reach Phase I and Phase II. Only the 0.8/0.2 setting supports Phase III generalization, as indicated by joint increases in Test-Ol accuracy and OOD alignment. Lower-II) settings fail to align OOD-derived bridge entities, preventing cross-distribution reasoning.
I Evidence for Representation Clustering from the Decodable Subspace
To validate the representational mechanism discussed in Section 4.2 that ID triple supervision constrains the representation to lie in decodable subspace, we design ablation experiment to test whether held-out ID triples can be recovered solely through shared compositional contexts in Train-Il queries.
We randomly select a subset ID triples (e.g.. (cA, eB) to exclude from the atomic triple training set. These held-out triples are removed from all atomic query contexts but remained involved in Train-Il queries (e.g., (e,,7), (1, r2) - ec, where en serves as the intermediate entity). Crucially, the model retains exposure to other atomic triples that sharing the same tail entity, such as (ex, r7) - εB, which appear in both atomic and corresponding compositional queries (e.g., Tex,T7,T2 + ey), Figure 14a illustrates the configuration. This configuration enables us to validate whether ID atomic triples supervision constrains the ri hidden state to a decodable region by testing whether these held-out triples could be recovered.
Specially,
As illustrated in the Figure 14b, The model successfully recovers held-out triples despite their absence from atomic training. Crucially, this recovery capability emerges concurrently with Test-ll generalization, confirming that the model leverages the same intermediate representation subspace for both atomic and compositional reasoning. The results indicate that ID triple supervision accelerates generalization not by providing explicit factual memorization, but by structurally constraining the model's 's representational space to align atomic and compositional reasoning pathways.
28

---

Held-out ID Triple Configuration
0O0 Tpples
Retaieed "rples
Iranal Test
nwidoout Irppls
Train
100
80
Test
Held-out Triples
60
OOD Triples
40
Test-Il
Train-Il
20
A
107
16a
16"
10a
Checkpoints (Training Stops)
(b) Training curve showing accurate prediction of the held-out triples.
(a) Illustration of construction.
Figure 14: Verification of ID triple constraint effect. The model successfully recovers held-out ID triples by leveraging representational constraints from multi-hop supervision and structurally related retained triples, supporting the claim that ID triples accelerate generalization by shaping a decodable representational subspace.
Removing ID Triples Breaks First-Hop OOD Generalization
To test whether ID supervision is necessary for cross-distribution (Test-OI) generalization, we constructed simplified configuration that removes all IID triples from training. The model is trained only on OOD atomic triples and Train-II 2-hop queries, as illustrated in Figure 15a.
Despite having access to OOD facts and multi-hop supervision, the model fails to generalize to Test-OI queries where the first hop is from an OOD triple. As shown in Figure 15b, Test-O1 accuracy remains near chance throughout training. This validates claim in Section 4.3: without representational anchoring from ID triples, OOD-derived entities cannot support implicit multi-hop reasoning.
Fipiss
Unanchored OOD Configuration
100
Train
OOD Triples
Test
Accuracy (96)
Test-Ol
60
Test-ll
Train-Il
20
Checkpoints (Training Steps)
(a) Illustration of the Unanchored 0OD Configuration. (b) Training dynamics: Test-Ol accuracy fails improve
Figure 15: Validation experiment under ID-removed configuration. Without ID triples, the model fails to reach Phase III, confirming that representational anchoring from ID supervision is essential for OOD-base generalization.
29


Reviews:
[{"review_id": "BOiW5ySU2s", "reviewer": "Reviewer_1ywD", "summary": "This work studies multi-hop implicit reasoning in Transformer models. The task structure builds on Wang et al. [24]: given a set of entities $e_i$ and relations $r_j$, there exists a set of \"atomic facts\" $(e_i, r_j) \\to e_k$. A 2-hop query takes the form $(e_1,  r_1, r_2) \\to e_3$, requiring the atomic facts to be composed: $r_2(r_1(e_1))$. This work studies the ability of Transformers to learn such a task, including the ability to generalize out-of-distribution. The paper begins by presenting an evaluation of in-distribution and out-of-distribution generalization throughout training (*what* the model does), followed by a \"interpretability\" analysis (*how* the model achieves this behavior internally). Among the results, this work finds that the model first learns to memorizes, then generalizes in-distribution, then generalizes out-of-distribution, in three phases, similar to the grokking behavior described by Wang et al. [24]. In analyzing the internal representations, the authors find that the ability to decode entities from internal representations does not explain when the representation contributes to reasoning. In response, they study the geometry of the model's embedding space using cosine similarity.", "questions": "* In Figure 5, why are there so few OOD-derive points? Is the MDS projection computed using only the points displayed in the figures?\n\nOther minor feedback:\n* The title sounds a bit grammatically awkward to me. Perhaps \"How do Transformers Learn Implicit Reasoning?\" would be better.\n* When stating “we adopt GPT-2 as our base model,” please clarify that this refers to the GPT-2 architecture, not the pretrained model, and specify the model size (e.g., GPT-2 small) explicitly. This will help avoid confusion.\n* Some sentences in the paper would benefit from simplification. Certain phrasings appear overly complex or indirect.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n* The interpretability portion (Section 3) is a highlight and offers novel insights. In particular, the observation that entity decodability does not predict reasoning behavior is interesting and important, given the prevalence of decodability-based analysis in recent work. Similarly, the finding of cosine-space clustering aligning with when generalization capabilities emerge is a satisfying insight.\n* The work tackles interesting and well-formed questions that are highly relevant for understanding how Transformer models generalize and reason compositionally, which aligns with ongoing interest in grokking and systematic generalization.\n\n**Weaknesses**\n\n* There are some presentation issues, especially in earlier sections of the paper, which can be improved. For example, the introduction uses jargon and terminology (e.g., \"ID triples\") that are only defined later in the paper. Section 2 appears to be written in a way that assumes strong familiarity with Wang et al. [24], making the paper less self-contained. In particular, it does not spend enough time motivating and clearly defining the task, instead jumping to the description of different generalization splits. I would recommend revising this section to include a motivation and a clearer explanation of the task, including examples.\n* Additionally, since this work builds heavily on Wang et al. [24], some aspects of the technical contribution are incremental. For example, the design of the symbolic task underlying this investigation and observations of grokking-like behavior have been studied in previous work.\n* Finally, the experiments contain a single trial per configuration. This can be an important limitation for Figures 2 & 3 especially which concern the timing of the emergence of different behaviors, since randomness can play a role here. I would recommend running several trials and including error bars. This seems feasible given the use of relatively small models on synthetic data.", "was_revised": true, "final_justification": "Overall rating maintained, but increased significance rating. \n\nThe authors responded to each point raised in the review and clarified their position. These concerns were not fully addressed since they pertain to core aspects of the work, but the authors' response is appreciated and does help communicate their position. The authors promised to address some of the presentation issues, acknowledged overlap in contribution with Wang et al. and clarified where their primary contributions lie, and acknowledged the limitations regarding single-trial experiments.", "rebuttal": "We thank the reviewer for the thoughtful feedback and for highlighting the strengths of our work.\nIn the following, we address the reviewer’s questions and concerns by grouping related points.\n\n## [Q1] Justifying the Low OOD-Derived Point Ratio in Figure 5\nWe thank the reviewer for pointing this out.\nIt is correct that the number of OOD-derived points in Figure 5 is smaller than ID-derived points.\nThis stems from our intentional data imbalance.\nAs justified in Appendix B (line 543), this **high ID/OOD ratio reflects the realistic distributional skew seen in LLM pretraining corpora**:\n>...they reflect plausible properties of real-world pretraining corpora...most knowledge entities are learned in richly connected contexts (analogous to our ID triples), while only a minority are sparsely anchored or appear in isolation (analogous to our OOD triples)...\n\nThis design enables our symbolic environment to approximate realistic structure.\n\nTo further address the potential concern about this imbalance, **we also conduct an ablation study in Appendix H**.\nWe show that only high ID ratios lead to successful cross-distribution generalization.\n\n## [W2] Novelty Beyond Wang et al. [24]\nWe appreciate the reviewer’s assessment regarding the relationship with Wang et al. [24].\nWhile our work indeed builds on their symbolic environment, its **novelty lies in substantial extensions in both breadth and depth**, as detailed below.\n\nOur contributions cover four key dimensions of multi-hop implicit reasoning:\n1. **Data Requirements.** Our findings in Sections 2.3 and 2.4 challenge the prevailing assumption that LLMs perform implicit reasoning by merely combining known atomic triples (see Appendix A).\nWe identify what data actually supports generalization, and further uncover fine-grained query-level requirements.\n2. **Training Dynamics.** We uncover a three-phase progression of learning and explicitly characterize the boundaries of OOD generalization under diverse training configurations, including the 3-hop setting.\nIn contrast, Wang et al. [24] concluded that OOD generalization does not emerge.\n3. **Interpretability Tools.** We introduce new tools to probe internal representations.\n4. **Mechanistic Insight and Disproof of Prior Assumptions.** As the reviewer noted, our analysis directly challenges the widely held assumption—present in Wang et al. [24] and others—that decodability implies reasoning utility.\nInstead, we demonstrate that successful reasoning aligns with representational clustering in hidden space.\n\nTo be transparent, we do overlap with Wang et al. [24] in two respects:\n1. Dataset Construction. We adopt their symbolic setup, but we **extend it with new training and test configurations** that enable fine-grained ablations.\n2. Grokking Observations. Our Phase II aligns with their grokking transition. However, while Wang et al.[24] emphasized that grokked models exhibit implicit reasoning capabilities, we focus on **how these capabilities emerge and how they generalize—including to OOD settings**.\n\n**Together, these overlaps represent only a small part of our contribution.** The core of our work centers on fine-grained data requirements, generalization boundaries, and mechanistic interpretability—areas not well-addressed in Wang et al. [24].\n\n## [W3] Regarding Single-Trial Experiments\n> \"the experiments contain a single trial per configuration...I would recommend running several trials and including error bars...\"\n\nWe appreciate this insightful suggestion.\nDue to the limited time available in the rebuttal period, we are unfortunately unable to rerun full training trials.\n\nThat said, we have already conducted experiments under several variant configurations (e.g., with 1000 vs. 2000 entities), which introduce variation in data scale while preserving task structure.\n**These results consistently show the same behavioral trajectory and representational patterns, suggesting that our findings are robust to data-level perturbations.**\n\nWe will include these results in the appendix of the revised version as additional support for the stability of our observations.\n\n## [W1, Q2] Writing and Presentation Improvements\nWe appreciate the reviewer’s helpful suggestions on improving the clarity and readability of our paper.\n**We will make the following revisions in response:**\n+ Title Revision:\nWe agree that \"How do Transformers Learn Implicit Reasoning?\" is more natural and grammatically correct. We will adopt this revised title.\n+ Terminology Clarification:\nWe will ensure that all technical terms (e.g., ID triples) are clearly defined at their first mention.\n+ Improved Motivation and Task Definition in Section 2:\nWe acknowledge that the current version of Section 2 assumes prior familiarity.\nIn the revision, we will provide a clearer motivation and explicitly define the reasoning task with illustrative examples.\n+ Model Description Clarity:\nWe will clarify that “GPT-2” refers to the architecture only and specify the model size explicitly.\n+ Simplifying Complex Phrasings:\nWe will revise selected portions of the text to make the language more concise and direct, while preserving the intended technical content.\nWe agree that some sentences are unnecessarily complex in the current form.\n\nWe thank the reviewer again for thoughtful feedback and hope our responses have addressed the concerns raised."}, {"review_id": "wPIGb3eFp4", "reviewer": "Reviewer_XYMx", "summary": "The paper investigates how LLMs execute multi-hop relational reasoning; specifically, the authors focus on studying the properties of the intermediate/implicit entity representation. The authors construct an artificial dataset and train their model from scratch. They conduct a number of probing and intervention (patching) techniques, obtaining a number of insightful observations.", "questions": "## Questions/Suggestions:\n\n### Deliberation tokens\nOne experiment that'd be very natural in this setting is with using the deliberation or planning tokens (https://arxiv.org/html/2310.05707v4). I.e. if we offer the model an extra token to \"think\" for longer, it might use this extra token to come up with the implicit entity representation.\nFor example, the dataset for such an extra experiment could take the form of e1, r1, <deliberation1> r2 <deliberation2> answer.\n\nTo clarify: I am not demanding that the authors should do this. Nevertheless, I think such an experiment would substantially strengthen the paper. Especially since in practice, chain-of-thought-like applications are common; so LLMs can implicitly reason not only layer-by-layer, but also by using intermediate tokens to refine their representations.\n\n### Re: clustering:\nFigure 5 is generally good and very interesting, but I suggest changing the color of OOD derived entities. The pale-yellow points are extremely hard to see. I didn't realize they were there at all before I read the legend. \n\nAlso, in this particular case I feel it'd be good to display an extra control sample: perhaps the entity embedding for the first, explicitly given entity? This is just to better understand if, maybe, all representations are shrinking/aligning in general.\nThe same applies to the experiment in 4.1. Are the clustering results specific to the \"implicit\" entity? Alternatively, it could be that any class of stimuli the model is regularly exposed to during training will see this clustering behavior.\n\n### Clarity\n\nAlthough the paper is generally well-written, at times, the language is very heavy and difficult to parse. \nFor example, consider this sentence:\n\"Cross-query semantic patching, which enhances causal interpretability by locating intermediate entity representations based on their semantic transferability across queries rather than solely their impact on final outputs.\" \nFor me personally, it was extremely hard to understand what this means. The sentence is very long, and it also uses a lot of terms that were not yet clearly introduced in the paper.\nI think it'd greatly benefit the paper if the authors go through it again, rephrasing and simplifying such overly long sentences.\n\nI also strongly suggest reworking the definition of Cross-Query Semantic Patching. Right now the phrasing is confusing. The part \"we extract the hidden representation believed to encode the bridge entity\" suggests that the authors already know which hidden representation is encoding the bridge entity. Only later the authors clarify that they are trying a number of different options.\n\nI suggest phrasing the approach in a more sequential manner. I.e. the order should be:\n1) we want to find which hidden representations encode... \n2) for each potential representation, we do ... \n\nIt doesn't actually need to be a list, but the order is important. Otherwise, it reads a little backwards and can be very confusing to the reader.\n\n## Minor:\nI initially read \"We distinguish: Train-II\" as \"Train 2\". It might be good to start with Train OI, for clarity.", "limitations": "The limitations are briefly but adequately discussed in the discussion section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths:\n- The paper addresses a highly relevant problem\n- The experimental design and execution are solid\n- The results regarding the training dynamics split into different phases are quite insightful.\n\n# Weaknesses:\n## Clarity:\n- The paper is generally well-organized, but I believe it'd be beneficial to have an extra round of editing, focusing on 1) simplifying/breaking up overly long sentences. Making sure that, on paragraph level, new concepts are introduced clearly before their usage. \nIt's especially important for everything on Page 2. I believe that the authors are trying to precisely define everything right there, but it's impossible to do so. The resulting definitions are too condensed and abstract to be useful. \n\nWhen these newly introduced terms are re-used again, it only creates more confusion.\nE.g. line 58: \"In Section 3, we first use cross-query semantic patching to localize intermediate entity representations, typically identifying them within the middle layers corresponding to the r1 tokens.\" At this point, the reader only briefly saw one mention of the \"r1\" tokens, many paragraphs earlier. The definition of \"cross-query semantic patching\" was given, but it was too abstract to understand what exactly the \"cross-query\" method does. So it might be better to just drop these specifics, and give a general idea of Section 3. Something like:\n\"In Section 3, we show that intermediate entity representations typically occur in the middle layers.\" There will be time to introduce the \"cross-query\" method later, as well as give more details.\nIf it is crucial to mention which tokens are involved, something like this could work to remind the reader of the problem structure: \"In Section 3, we show that intermediate entity representations typically occur in the middle layers and correspond to \\\\( r_1 \\\\) tokens in \\\\( e_1, r_1, r_2 \\\\) prompts.\"\n\nOverall, this weakness is not disqualifying, but I believe that an extra round of editing will greatly enhance the reader experience and, therefore, make the paper more impactful.\n\n## Experimental support\n\nGenerally, the experimental support is quite strong. There are a few extra controls that would be beneficial to add in section 4.\n\nAdditionally, it seems that a natural thing to look at would be to study whether/how intermediate \"deliberation\" tokens can be used to \"store\" the implicit entities. It's unfortunate that the authors did not look into it, because in practice, Chain of Thought - like methods allow the model plenty of room to come up with intermediate computations not only through explicit chain of thought reasoning, but also by using the intermediate tokens to refine their implicit representations.\n\nI give more detail on this in the Questions section.\n\n## Applicability\n\nAs any study with artificial data, this one also suffers from a decreased practical applicability. It's not clear whether and how well the results would generalize to LLMs \"in the wild.\" I don't think this is a disqualifying issue, but it would, of course, greatly benefit the paper if the authors included some experiments on, for example, pre-trained LLMs, but with new entities, etc.\n\nThe deliberation token experiment I mentioned above would also, indirectly, connect the paper closer to practical applications.\n\n## Exploratory-ness\n\nWhile the results are valuable, I believe it'd be good to spend more time to integrate them into the existing literature. I.e. the obtained conclusions are sound and interesting, but I sometimes felt that the authors were pursuing interesting results they stumbled upon by chance, rather than testing hypotheses that were formed based on previous research. This is, again, not disqualifying, but it limits the impact of the paper, as it's not clear how to integrate the obtained knowledge into the existing research.\n\nI.e. the authors obtained many interesting observations, but it's not 100% clear how impactful they are and whether they substantially change our understanding of LLM dynamics.\n\n# Conclusion\nGenerally, the paper is quite strong as is. Although I listed more weaknesses than strengths, some of these weaknesses are essentially different sides of the same issue, and none of them are disqualifying. Overall, I think the paper is sound enough and interesting enough to benefit the community.", "was_revised": true, "final_justification": "After considering the author's rebuttal and other reviews, I decided to increase my initial score by one point.\n\nWhile some of the issues remain, they are largely in the \"desiderata\" space. E.g. my suggestion about deliberation tokens. I do think some experiments on them could be added quite easily and would improve the paper, but it's also acceptable to just acknowledge this in the limitation section; I trust the authors will do that.\n\nAll in all, I feel that \"borderline accept\" doesn't do this paper justice, so I have increased my score by one point, to \"5, Accept\".", "rebuttal": "We sincerely thank the reviewer for the thorough reading of our paper and for the detailed and constructive feedback.\nIn the following, we address the reviewer’s questions and concerns by grouping related points.\n\n## [W2, Q1] Deliberation tokens\nWe thank the reviewer for the insightful suggestion regarding the use of deliberation tokens.\nOur goal in this paper is to investigate the emergence of implicit multi-hop reasoning under the maximum architectural constraint—the transformer’s ability to reason without any explicit space or token dedicated to intermediate computation.\nWe therefore focus on the standard autoregressive setting, where the model must internally encode and reuse intermediate entities.\n\nThat said, we agree that introducing deliberation tokens, as proposed by the reviewer, opens up a natural and practical extension to our study.\nIt could allow the model to externalize intermediate representations, which would bridge implicit reasoning and practical CoT-style explicit processes.\n**We will explicitly incorporate this direction into our discussion of future work in the paper.**\n\n## [Q2] Clustering Generality\nWe thank the reviewer for pointing out both the visualization and interpretability issues regarding clustering.\n\nWe agree that the pale-yellow in Figure 5 is difficult to distinguish.\nWe have already **updated the figure** with more saturated colors, and slightly increased the marker size for better visibility.\nThese changes will be reflected in the revised version.\n\nRegarding whether the observed clustering is a global effect, we appreciate the reviewer’s insightful question.\nTo address this, we randomly selected a set of different intermediate entities and computed the average pairwise cosine similarity among their representations over training.\n\nWe report the average similarity across Phases I–III in the following table:\n| Phase | ID Cohesion ↑ | OOD Alignment ↑ | Cross-Entity Similarity ↓ |\n|-------|----------------|------------------|-----------------------------|\n| I     | 0.407           | 0.188             | **0.068**                    |\n| II    | 0.765           | 0.115             | **0.011**                    |\n| III   | 0.871           | 0.651             | **0.004**                    |\n\nAs shown, the cross-entity similarity consistently decreases.\nThis trend strongly argues against a global representational collapse and confirms that the observed clustering is **entity-specific**, not a byproduct of general compression or training dynamics.\nWe hypothesize that this decreasing trend results from entity representations being actively **pulled toward their own ID-derived cluster centers**, which increases intra-entity similarity while decreasing cross-entity overlap.\n\n**This additional result provides even stronger support for our original conclusion.**\nWe will incorporate the findings into the revised version of the paper, and we are grateful to the reviewer for the suggestion that inspired this analysis.\n\n## [W3] Applicability\nWe thank the reviewer for raising this important point regarding the practical applicability of our findings.\nWhile our main experiments are conducted in a controlled symbolic environment, we believe our conclusions offer useful insights into how implicit multi-hop reasoning may operate within pre-trained LLMs as well.\n\n**Regarding applicability to LLMs \"in the wild\":**\n\nAlthough we do not directly evaluate pre-trained models, **our findings align well with several recent empirical results on LLMs and provide coherent explanations for their observed behaviors**:\n1. Our finding in Section 2.3—that models don't rely on atomic triples to learn implicit reasoning—helps explain why single-hop knowledge editing fails to transfer to multi-hop queries ([Zhong et al., 2023](https://arxiv.org/pdf/2305.14795)).\nConversely, Section 2.4 shows that such reasoning is acquired from compositional data, which aligns with recent findings that editing LLMs using multi-hop prompts leads to better generalization on multi-hop tests ([Yao et al., 2025](https://arxiv.org/pdf/2503.16356), [Zhang et al., 2024](https://www.arxiv.org/pdf/2410.06331)).\n2. Our finding in Section 2.4 also sheds light on why exposing LLMs to richer compositions improves multi-hop reasoning ([Zhang et al., 2024](https://arxiv.org/pdf/2409.14057), [Xu et al., 2024](https://arxiv.org/pdf/2504.00472)).\n3. Our Section 4.3 result—that generalization to OOD triples only occurs in the first hop—offers a mechanistic interpretation of the observation in [Feng et al., 2024](https://arxiv.org/pdf/2412.04614), where second-hop OOD queries are significantly more difficult than first-hop OOD ones.\n\nIn short, our findings offer a mechanistic lens through which to understand a range of puzzling behaviors already reported in pre-trained LLMs.\n\n**Regarding fine-tuning on pre-trained models with new entities:**\n\nWe agree that fine-tuning LLMs on new knowledge could be a path to validate our findings.\nHowever, multiple recent works have found that knowledge acquired during fine-tuning can **behave quite differently** from knowledge learned during pretraining, and **may even distort pre-existing structure** ([Gekhman et al., 2024](https://arxiv.org/pdf/2405.05904), [Zucchet et al., 2024](https://arxiv.org/pdf/2503.21676)).\nAs our primary goal is to understand the mechanism through which implicit multi-hop reasoning emerges, we chose to train models from scratch in a controlled environment.\n\nNonetheless, we acknowledge that incorporating experiments with pre-trained LLMs and injected new entities would be a valuable future direction, and we are actively exploring such extensions to test the generality of our conclusions.\n\n## [W4] Clarifying Motivation and Connection to Prior Work\nWe thank the reviewer for pointing out that parts of our study may appear exploratory and not fully integrated with prior research.\nWe acknowledge that this impression likely arises because our motivation was not stated clearly enough in the current version.\n\nAn important motivation is the **lack of consensus in prior works** regarding how LLMs perform implicit multi-hop reasoning and whether they can reliably do so, existing studies report conflicting conclusions on key aspects (see Appendix A).\nAs described in line 34 of the paper, we believe that a key reason for this inconsistency is:\n> ...training data lacks precise experimental control, making it challenging to conclusively determine...\n\nTo address this gap, we designed a **highly controlled symbolic setup** to systematically revisit these questions.\n\nWe appreciate the reviewer’s suggestion to better connect our findings to the prior works.\nIn the revised version, we will **make this motivation explicit in revised version and clarify how our study aligns with and challenges prior assumptions**.\n\n## [W1, Q3] Clarity\nWe appreciate the reviewer’s helpful suggestions on improving the clarity and readability of our paper.\n**We will make the following revisions in response:**\n+ Simplify and break down long sentences. We will break down long sentences to make the language more concise and direct, while preserving the intended technical content.\n+ Terminology Clarification.\nWe will ensure that all technical terms (especially on page 2) are clearly defined at their first mention.\n+ Rework the definition of Cross-Query Semantic Patching.\nWe will present the procedure in a clear and sequential manner, consistent with the reviewer’s suggestion.\n+ We will avoid the “Train-II vs. Train 2” ambiguity by first introducing Train-OI as recommended by reviewer.\n\nAgain, we sincerely thank the reviewer for the careful reading and thoughtful suggestions.\nWe hope that our responses have addressed your concerns and clarified the paper.\nWe would be happy to provide any additional clarifications if needed and look forward to further discussion."}, {"review_id": "65p24iRcDn", "reviewer": "Reviewer_mwv1", "summary": "This paper studies the dynamics of how Transformers learn to perform implicit compositional reasoning, based on the setups in prior work. Specifically, the authors conduct a more detailed examination on top of previous studies, including 1) a more fine-grained classification of out-of-distribution (OOD) scenarios in terms of the sources of constituting facts, where the authors find novel generalization phenomena; 2) a set of further ablation studies and analysis techniques towards confirming and explaining these phenomena, such as intervention methods that integrate causality and semantics, and clustering/visualization approaches which provides statistics that correlate well with generalization behaviors.", "questions": "None", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- This paper reports a set of novel findings on top of previous work in similar directions, including how Transformers can actually generalize in compositional reasoning under some out-of-distribution scenarios (in particular, cases where the first hop is out-of-distribution and the second hop is in-distribution, or 'test-OI'), and also how ID atomic triples are not strictly required for ID generalization, yet, they accelerate ID generalization and encourage the bridge entity representations under OOD scenarios to converge onto those under ID scenarios. These findings are novel and provide interesting insights into the dynamics of how Transformers learn implicit compositional reasoning.\n- The authors conduct a series of ablation studies and analysis on the intermediate representations, which provide further consolidation of the results and support for the explanation of the new observations, including the effect of ID triples. These also point to potential ways of analyzing the geometry of internal representations beyond simple projection/probing as done in prior work, and could inspire future studies in related directions.\n\nWeaknesses\n- The overall structure/writing could be improved to focus more on the central findings/hypothesis. Details on some ablation studies that are closely relevant are left in the Appendix, while some sections which are not well-connected to the main thesis are unnecessarily detailed. This makes the overall logic flow a bit scattered. I would suggesting moving some non-critical results to the Appendix in exchange for more concrete ablation studies in the main content. For example, I believe the results in Section 2.4 are not well-connected with the rest, and are also mostly reassurance of prior findings.\n- While the experiments and analysis provide decent insights into the roles of ID triples, I believe one critical question of why/how they help shape the OOD bridge entity representations during Phase III is still not fully explained, e.g., what could be the reason that test-OI generalization starts to improve at around 10^6 optimization steps (Figure 2)? It would add more depth to the insights if some experiments could be done to characterize this observation and other factors such as regularizations.", "was_revised": true, "final_justification": "The rebuttal phase satisfactorily addressed my concerns, and I will maintain my original positive assessment of the paper.", "rebuttal": "We thank the reviewer for the thoughtful feedback and for highlighting the strengths of our work.\nIn the following, we address the reviewer’s questions and concerns by grouping related points.\n\n## [W1] Paper Structure\n\nWe understand the reviewer’s concern about the organization of the paper, and we clarify our reasoning below.\n\n### **1. Importance of Section 2.4**\n\nWe would like to clarify the role of Section 2.4 in our study.\nSection 2.3 and 2.4 jointly answer the core question of **what data signals are truly required** for implicit reasoning:\n+ Section 2.3 demonstrates that **ID atomic triples are not required** for ID generalization, challenging the prevailing assumption in prior works.\n+ Section 2.4 goes one step further, **identifying what the model actually depends on** at the query level, revealing that second-hop generalization requires query-level exposure.\nThis extends the understanding of data dependency from a macro level (type of supervision) to a fine-grained query level, which is essential to answering how LLMs learn.\n\nTherefore, Sec. 2.4 is not redundant but a key part of the data-dependency analysis.\n\n### **2. Multi-faceted Structure is Intentional**\n\nOur study is organized to systematically answer “how LLMs learn implicit reasoning” from **multiple complementary perspectives**:\n1. the role of data (Sec. 2.3–2.4),\n2. training dynamics and observed phenomena (Sec. 2.2–2.4),\n3. analysis tools (Sec. 3), and\n4. internal mechanisms (Sec. 4).\n\nThis design is deliberate: phenomena alone cannot explain how learning occurs, and mechanistic insights require the context of observed phenomena.\n\nWe also note that different reviewers valued different aspects:\n\n+ Both reviewers **mwv1** and **4Y73** found the new phenomena observed in Section 2 interesting and informative.\n+ Reviewer **1ywD** highlighted that “The interpretability portion is a highlight.”;\nReviewer **mwv1** also praised our mechanistic analyses.\n+ Reviewer **4Y73** expressed interest in the patching tools.\n\nTogether, this indicates that the current structure naturally supports a comprehensive answer to the central question.\n\n### **3. Open to Further Discussion on Presentation**\n\nWe acknowledge the reviewer’s concern about the placement of certain results between the main text and Appendix.\n\nWe are **happy to further adjust the presentation**, whether by bringing key results into the main text or by moving less critical details to the Appendix, if the reviewer feels this would improve clarity, and we are open to continued discussion to make the structure clearer for the final version.\n\n## [W2] Deeper Mechanism Behind Test-OI Generalization\nWe appreciate the reviewer’s insightful question on the deeper cause of Test‑OI generalization.\n\nOur study is scoped around how LLMs learn implicit reasoning, and we provide an empirical mechanistic explanation—as also noted by the reviewer in the strength section:\n> \"...ID atomic triples...encourage the bridge entity representations under OOD scenarios to converge onto those under ID scenarios..\"\n\nWe intentionally stopped at this level of analysis for two reasons:\n1. **Scope alignment with the main research question.**\nOur goal is to characterize how implicit reasoning emerges through data, training dynamics, and internal representations.\nAs Test‑OI generalization constitutes only one component of the broader question, our current explanation is both intuitive and sufficient for supporting this specific behavior, without over-extending the paper into a deeper causal analysis.\n2. **Ensuring rigor and generality.**\nWe believe that a rigorous investigation requires a more general study (e.g., connecting to representational collapse), **which cannot be convincingly achieved within the current symbolic setup alone.**\nTherefore, we treat this as an exciting direction for future work rather than something we aim to fully resolve in the current paper.\n\nWe will clarify this scope in the revised version and leave a deeper investigation of the timing and underlying dynamics of Test‑OI generalization as a natural direction for future work.\n\nWe thank the reviewer again for thoughtful feedback and hope our responses have addressed the concerns raised."}, {"review_id": "MQmmTdZVQo", "reviewer": "Reviewer_4Y73", "summary": "This paper investigates how Transformer models learn implicit reasoning by training them on symbolic \"fact\" triples: in-distribution (in training and in multihop examples) and out-of-distribution (in training but not in multihop). Memorization of in-distribution triples and multihop appears first during training, then novel in-distribution multihop, then out-of-distribution multihop. They find that entity representations appear in middle layers, and in-distribution entity representations begin to cluster earlier in training, then out-of-distribution entity representations begin to align with the in-distribution representations. The multihop reasoning capability of the model for OOD entities depends on the clustering of the OOD entity representations with the in-distribution entity representations. This suggests that multihop abilities for in-distribution entities drives the multihop abilities for OOD entities.", "questions": "1. How does the \"Cross-Query Semantic Patching\" method differ from causal patching in other work (e.g. https://arxiv.org/pdf/2304.14767)? If there are key differences, this would improve my originality score.\n2. In the \"full-run\" probing, why do we expect the logit lens to extract the original entity after passing the patched representation through all model layers? Might the model layers transform the patched entity into some other target entity if a relation is described in the query?\n3. In the abstract, the representation results for multihop reasoning are not made clear. E.g. \"which reveals that successful reasoning correlates with the cosine-base clustering in hidden space. This clustering phenomenon in turn provides a coherent explanation for the behavioral dynamics observed across training, linking representational structure to reasoning capability. These findings provide new insights into the interpretability of implicit multi-hop reasoning in LLMs, helping to clarify how complex reasoning processes unfold internally and offering pathways to enhance the transparency of such models\" is a bit vague, and does not mention any of the key findings, e.g. that alignment of OOD and in-distribution entity representations drives multihop reasoning abilities for OOD entities. I think these results are quite interesting, and it would be good for the abstract to highlight these contributions!", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Overall, the results are quite interesting and could have substantial implications for how Transformer language models learn multihop reasoning.\n2. The results in Section 2, showing learning of different types of multihop reasoning are interesting and informative.\n3. The \"Cross-Query Semantic Patching\" methodology appears sound, although it is not entirely clear how it differs from causal patching in previous work (e.g. https://arxiv.org/pdf/2304.14767).\n\nWeaknesses:\n1. The dataset (synthetic fact triples) generation process could be described more fully in the main text. The task is quite synthetic, although this makes sense to reduce the complexity of the phenomenon that the paper is studying.\n2. The two new approaches (\"cross-query semantic patching\" and the \"cosine-based representational lens\") do not seem particularly novel (e.g. see https://arxiv.org/pdf/2304.14767; https://arxiv.org/abs/2403.05440; https://arxiv.org/abs/2205.05092). Framing these as main contributions of the paper (e.g. in the abstract) detract from potentially more novel contributions, such as the implications for model learning of multihop reasoning.\n3. The finding that OOD entity representations and in-distribution representations increase in cosine similarity throughout training is not entirely surprising given that language model representations usually concentrate in a narrow cone throughout training (e.g. https://arxiv.org/abs/2401.12143). However, it is interesting that this can be linked to multihop reasoning ability to some degree.", "was_revised": true, "final_justification": "Maintaining my currently positive score. Specifically:\n* The author response addresses my question about the comparison of their method to causal patching; based on this, I have increased my originality rating to 3.\n* The author response also addresses my concern about the \"narrow cone\" phenomenon, demonstrating that the result isn't just a by-product of model anisotropy. However, because my overall score is already 5 (recommending acceptance), I do not feel that this addition raises my score further.", "rebuttal": "We thank the reviewer for the thoughtful feedback and for highlighting the strengths of our work.\nIn the following, we address the reviewer’s questions and concerns by grouping related points.\n\n## [W1] Dataset Description:\nWe thank the reviewer for the suggestion. We acknowledge that the main text currently provides only a brief description due to space limitations, and many details were placed in the appendix. In the revised version, we will:\n1. Move **key details** (e.g., ID/OOD ratios) from the appendix into the main text.\n2. Add illustrative **examples** to help readers better understand how the symbolic dataset mimics realistic multi-hop reasoning scenarios.\n\nWe believe these changes will improve clarity.\n\n## [Q1] Cross-Query Semantic Patching vs. Standard Causal Patching\nWe thank the reviewer for the question regarding the originality of our Cross-Query Semantic Patching (CQSP) method. We clarify the key differences below:\n\n**1. Same general family, but different purpose.**\n\nOur method is indeed inspired by the general causal patching paradigm, but CQSP explicitly targets **semantic reusability** of intermediate entity representations across queries, which is essential for studying multi-hop reasoning.\n\nStandard causal patching primarily asks “does this activation (in target query itself) affect the target output?”, whereas our method asks “does the representation from a source query carry reusable intermediate-entity information that can correctly drive reasoning in a target query?”.\n\n**2. Difference in how source activations are treated.**\n\nIn standard causal patching, source activations are often treated as **perturbations** (sometimes even randomized) to measure causal effect on the target prompt.\n\nIn contrast, CQSP leverages the source prompt representation as a meaningful semantic candidate, inserted into the target prompt’s context, and we evaluate whether it produces the correct 2-hop answer (see line 191).\n\n**3. Stricter criterion**\n\nStandard causal patching does not have a well-defined “success” output—it can only evaluate the **magnitude of output probability shifts** at the target prompt to estimate causal influence at different positions.\n\nIn contrast, CQSP has **a known expected answer** (see line 191), only producing this correct answer counts as success.\nThis makes our metric goal-directed and semantically meaningful.\n\n\n**4. Concrete example vs. standard patching.**\n\nIn our extended setup of [Wang et al. [24]](https://arxiv.org/pdf/2405.15071), standard causal patching was applied to locate circuits encoding intermediate entities.\nTheir approach compared Phase I and Phase II causal graphs and subtracted them, which produced **noisy results** (see Figure 9 in their paper).\n\nBy contrast, our CQSP method achieves **nearly 100% success** (Figure 4) in pinpointing the positions that encode the correct intermediate entities, providing a far cleaner and functionally interpretable signal.\n\nIn short, CQSP is a cross-query, semantically driven extension of causal patching.\nWe hope this clarification addresses the reviewer's concern about our methodological novelty and supports an improved originality assessment.\n\n\n## [W3] Narrow Cone vs. Entity-Specific Clustering\nWe thank the reviewers for raising the concern that the observed OOD–ID cosine alignment might be a byproduct of the global “narrow cone” phenomenon (e.g., 2401.12143), where all representations become more similar during training.\n\nOur key distinction is that the alignment we report is entity-specific and functionally predictive:\n1. We track hidden states of **specific bridge entities**, not global embeddings.\n2. Their clustering dynamics are tightly **linked to functional behavior**: ID cohesion rises with Test-II accuracy (Phase II), and OOD alignment rises with Test-OI accuracy (Phase III).\n3. To further rule out global collapse, we computed the average pairwise cosine similarity among **different intermediate entities** over Phases I–III:\n\n| Phase | ID Cohesion ↑ | OOD Alignment ↑ | Cross-Entity Similarity ↓ |\n|-------|----------------|------------------|-----------------------------|\n| I     | 0.407           | 0.188             | **0.068**                    |\n| II    | 0.765           | 0.115             | **0.011**                    |\n| III   | 0.871           | 0.651             | **0.004**                    |\n\nAs shown, cross-entity similarity consistently **decreases**, confirming that the clustering is highly entity-specific rather than a byproduct of general compression.\n\nThis result directly **refutes** the “all representations become similar” hypothesis and **strengthens our original claim**:\nfunctional multihop reasoning emerges when bridge-entity representations form tight, entity-specific clusters, not from global embedding collapse.\n\nWe will incorporate the findings into the revised version of the paper, and we are grateful to the reviewer for the suggestion that inspired this analysis.\n\n## [Q2] Full-Run Probing as Stricter Complement\nWe appreciate the reviewer’s question regarding why we expect the logit lens to extract the original entity after passing the patched representation through all model layers.\nOur reasoning is as follows:\n\n**1. Empirical observation of early-layer decodability.**\nWhen we perform cross-query patching at the r1 token position, we observe that:\n+ Patching at very **early layers** (1–2) already leads to the final layer decoding the correct intermediate entity with high success.\n+ In contrast, direct immediate decoding of hidden states (without full-run propagation) only becomes stable from **layers 4–5** onward.\n\n**2. Role of later layers.**\nThis observation indicates that the decodability of the intermediate entity is already established in early layers, and the downstream layers **refine and stabilize** the decodability.\n\nWe therefore introduce full-run probing as a **stricter complement to immediate decoding**, ensuring that representations which cannot be immediately decoded, but already contain sufficient information for correct decoding, are not overlooked.\n\n## [W2, Q3] Contribution Framing and Highlighting Key Findings\n**1. Adjusting the framing of our analysis methods**\n\nWe appreciate the reviewer’s comment regarding the novelty of our analysis tools.\n\nOur intention is **not to claim standalone methodological novelty**.\nThese tools were designed primarily to **challenge** the prevailing decodability-based assumption in multi-hop reasoning analysis, which assumes that intermediate entities must be explicitly decodable (Section 3.2), and to enable the **mechanism discovery** that forms the core contribution of our paper.\n\nWe will revise the final version to clearly present these tools as supporting analysis, with the core contribution framed as the mechanism discovery.\n\n**2. Highlighting the key OOD-ID alignment finding**\n\nWe thank the reviewer for emphasizing that the alignment of OOD and ID entity representations is a particularly interesting and impactful finding, which another reviewer also noted as a central and impactful contribution.\n\nWe fully agree and will revise the abstract and contribution statements to make this finding explicit.\n\nWe will clarify this motivation for the full-run probing setup in the revised version."}]

Author rebuttal:

Meta-review: This paper studies the dynamics of how implicit reasoning successfully arises in language models, based on a symbolic multi-hop task from prior work. All reviewers unanimously appreciated the contributions as significant and interesting, the experiments providing solid support for the claims. Multi-hop reasoning is an important problem to understand -- there's both theoretical and empirical literature pointing out the challenges of multi-hop reasoning; thus it is instructive to see a positive result and a clean analysis of its emergence in a symbolic setting. The community will find the insights here useful.

There are writing issues that multiple reviewers pointed out, which I hope the authors will fix.

Here are a couple of suggestions for pre-2025 related work that would bolster the context given in this paper. [1] present a negative result, whereas [2] present a positive result in path-finding over the weights of a model, which involves a form of implicit reasoning. 

- [1] Siwei Wang, Yifei Shen, Shi Feng, Haoran Sun, Shang-Hua Teng, and Wei Chen. ALPINE: unveiling the
planning capability of autoregressive learning in language models. In Advances in Neural Information
Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS
2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024.

- [2] Mikail Khona, Maya Okawa, Jan Hula, Rahul Ramesh, Kento Nishi, Robert P. Dick, Ekdeep Singh
Lubana, and Hidenori Tanaka. Towards an understanding of stepwise inference in transformers: A
synthetic graph navigation model. In Forty-first International Conference on Machine Learning,
ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Decision: Accept (spotlight)