A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoning
Authors: Guan Zhe Hong
Nishanth Dikkala
Enming Luo
Cyrus Rashtchian
Xin Wang
Rina Panigrahy
Venue: NeurIPS 2025 spotlight ()
Keywords: Mechanistic interpretability
circuit analysis
logical reasoning
propositional logic
large language model
transformer
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We mechanistically analyze how large language models solve synthetic propositional logic problems.

Abstract: Due to the size and complexity of modern large language models (LLMs), it has proven challenging to uncover the underlying mechanisms that models use to solve reasoning problems. For instance, is their reasoning for a specific problem localized to certain parts of the network? Do they break down the reasoning problem into modular components that are then executed as sequential steps as we go deeper in the model? To better understand the reasoning capability of LLMs, we study a minimal propositional logic problem that requires combining multiple facts to arrive at a solution. By studying this problem on Mistral and Gemma models, up to 27B parameters, we illuminate the core components the models use to solve such logic problems. From a mechanistic interpretability point of view, we use causal mediation analysis to uncover the pathways and components of the LLMs' reasoning processes. Then, we offer fine-grained insights into the functions of attention heads in different layers. We not only find a sparse circuit that computes the answer, but we decompose it into sub-circuits that have four distinct and modular uses. Finally, we reveal that three distinct models -- Mistral-7B, Gemma-2-9B and Gemma-2-27B -- contain analogous but not identical mechanisms.

A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoning
Guan Zhe Hong"!
Nishanth Dikkala² Enming Luo² Cyrus Rashtchian
Xin Wang² Rina Panigrahy²
Purdue University 2 Google Research
hong288@purdue
(nishanthd, enming, cyroid, wanxin, rinap)@google.com
Abstract
Due to the size and complexity of modern large language models (LLMs), it has proven challenging to uncover the underlying mechanisms that models use to solve reasoning problems. For instance, is their reasoning for specific problem localized to certain parts of the network? Do they break down the reasoning problem into modular components that then executed sequential steps as go deeper in the model? To better understand the reasoning capability of LLMs, we study minimal propositional logic problem that requires combining multiple facts arrive at a solution. By studying this problem on Mistral and Gemma models, up illuminate the core components the models such logic problems. From a mechanistic interpretability point view, we use causal mediation analysis uncover the pathways and components of the LLMs' reasoning processes. Then, we offer fine-grained insights into the functions attention heads in different layers. We not only find sparse circuit that computes the answer, but we decompose it into sub-circuits that have four distinct and modular uses. Finally, we reveal that three distinct models Mistral-7B, Gemma- 2-9B and Gemma-2-27B contain analogous but not identical mechanisms.
27B parameters,
solve
Introduction
LLMs can solve many tasks in a few-shot manner (Brown et al.) 2020 Radford et al. (019b). One emergent ability of these models solving problems that require planning and reasoning. In particular, LLMs seem to have internal components that judiciously process and transform the provided information. Unfortunately. it is not clear whether these models implement step-by-step algorithms or whether they perform opaque computations to arrive at their answers.
The area of circuit analysis has emerged components. While the definition a circuit varies across different works, in this paper, our definition follows Waag et (2023a). A circuit within transformer is collection of model components (attention heads, neurons, etc.) with the edges in the circuit indicating the information flow between the components in the forward pass. Circuit analysis inform how to improve models, how to debug errors, and how to explain patterns in their behavior. Despite the importance of these goals, there has been limited number of large-scale studies of interpreting LLM reasoning. Within mechanistic interpretability (Geva et al. 2021: | Vig et al.) 2020; Hou et al., 2023), existing studies either only provide partial evidence for the underlying circuits (Meng al.) 2022 or are limited to small models like 3-layer transformer or GPT-2 sized models (Merullo et al. 2024a Rauker et al.) 2023
way understand how transformers use their internal
Wang et al. (023a). Our goal go beyond this, uncovering the distinet roles of different
Part of this work was done a student researcher at Google Research.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

sets of components for solving canonical propositional reasoning problem in frontier models with as many as 27B parameters. Specifically,, we aim identify separate sub-circuits for distinct steps like retrieving information from the context, processing this information, and arriving at an answer
1.1 A Suitable Set of Propositional Logical Reasoning Problems
Performing circuit analysis for LLMs requires careful selection of what problem to study. If is too simple (e.g., just copying part of the input), then we will learn little about reasoning. If it is too complex (e-g., generic math word problems), then we will have way to systematically apply causal analysis and we will struggle to localize and delineate the model's internal components. The problem also needs be sufficiently natural, such that is a common subproblem of more complex reasoning problems in the wild, such as those found in natural logic (MacCartney & Manning 2014).
Keeping the above considerations in mind, we study the following propositional logic template, which requires reasoning over distinct parts of the input. The problem involves five boolean variables. Given two propositions unknown truth value of a different variable (which we call the "query"). " For concreteness, here are two possible instantiations:
"Rules" and truth values of three variables "Facts" we wish to infer the
Rules: A or B implies C. D implies E. Facts: A is true. B is false. D is true. what is the truth value of C??
Rules: V and W implies X. Y implies Z. Facts: V is false. W true. is false. what is the truth value of Z?
Question:
Question:
Consider the left problem, which queries the LogOp (disjunction) chain. One concise solution the question is "A is true. A or B implies C; C is true." This problem, while simple-lookin on the surface, requires the model to perform actions that are essential to more complex reasoning problems. Even writing down the first answer token (A in "A is true") takes multiple steps the ambiguity of which rule is being queried. In this case, it is "A or B implies C" not "D implies E". Then, the model needs to determine which facts are relevant and to process the information "A is true" and "B is false". Finally, it decides that "A is true" is relevant and implics that "C is true" due to the nature of disjunction. In contrast, the problem the right queries the linear chain, requiring the model to locate different rules (Y implies 2) and process the facts (Y is false).
3 The model must resolve
Our logic problem is also inspired by some of the ones in reasoning benchmarks such as GSM8k (Cobbe et al., (02T) or ProofWriter (Tafjord et al.] (02I). The issue with using exist- ing benchmarks directly is that there is too much syntactic and semantic variation in the questions. This makes it challenging to use the tools of mechanistic interpretability, which require changing parts of the input in systematic way to minimize confounding factors. One can amplify the difficulty of our problem by increasing the number of variables or the length of the reasoning chain. However, even on this simple problem, Mistral and Gemma models only achieve 70% to 86% accuracy in writing the correct proof and determining the query value, even with few-shot prompting.
1.2 Interpreting How LLMs Solve the Logic Problems
Given the above problem template, we design systematic causality-based experiments to understand how LLMs solve it. We aim to localize and delineate circuits, as a way interpret how contemporary LLMs solve propositional logic problems (e.g., how they determine relevant parts of the input, how they move information around, how they process facts, and how they make decisions). Finding an interpretable circuit amounts to identifying a small number of components that map to natural operations and together implement an intuitive reasoning algorithm. It is important to note that, a priori, a non-interpretable outcome is also possible, where the LLM internally merges steps in reasoning or distributes the steps across all its parameters. One of our contributions is strong evidence that the mechanisms for LLMs to solve such reasoning problems are indeed modular and localized.
Figure 1 shows the main ideas of our approach, including the data model, the interpretability tools, and some findings that are shared across all three models: Gemma-2-27B, Gemma-2-9B (Gemma Team al. 2024) and Mistral-7B-v0.1 (Jiang et al. 2023). Our main findings are:
2 "Our code is available at mt/ttpp:///giiii com coom//gannzhhhon//rro--oooooo hrapoo/goorrroooooie-ttaaarrreer twwassrrrrr-ecaeeeeeee circuit Each word (or capital letter representing boolean variable) gets tokenized an individual token.

---

Problem Set-up
Mechanistic Interpretability for Logical Reasoning
Model Comparison
Causal Data Model
Necessity (Patching)
2. Sufficiency (Verification)
3. Fine-Grained (intereention))
Mistral 7BB
Gemma 9B
Logical
Operator
out!
out2
Output
W
Make Decision
Linear Chain
Process
Gemma 278
Layere
Facts
Facts: W. are (true, false)
Query:.. truth value of (X, 21?
Constructing Counterfactuals
Rules: and W implies X. Imples V
Customized
Capture signals from model components after moditying the input (eg, changing the
i fasse.
Caussal Mediation Analysk
Question: what the truth xf
New Flindings from Analysis:
query variable to Z)
Necessity Test:
Sufficiency Test:
These LLMs contain the same core reasoning functionality
Swap activations with counterfactual signals and
Replace the complement of corjectured circuit with counterfactusl signats and verify the model maintains its behaviar
truth value of X?
They contain separate parts that locate rules, move info, process facts, make dlecisions
check for ouiput degradation (e.g. logit differences)
truth value of 2?
Figure I: Left: Problem set-up. Top left shows the data model. The causal structure has two chains: one with a logical operator (LogOp, "and" or "or") and the other with linear causal chain. Bottom left shows how we get counterfactual signals for the model components (e.g.. attention heads). We vary parts of the input, such as the query variable, truth values, and order of rules. Middle: Key steps in our interpretability approach. We use causal mediation analysis to localize components in LLMs that are used to solve the logic problem (circles represent attention heads, with layers stacked vertically). We derive evidence about the functionality of components using multiple methods. For necessity, we patch with counterfactual signals (represented by blue and red nodes). Then we check the impact of each component by looking at the output of the network (e.g., checking it outputs very different distributions "outl" vs. "out2" after patching). For sufficiency, we replace the complement of the circuit with counterfactuals, and we verify that the output does not change too much. For fine -en-ggrined analysis, we use sely---aueuuuryy interventions to identify the role each component plays. Right: New findings. Three LLMs use similar reasoning steps when solving these problems. They use four sparse and distinct sets of attention heads in step-by-step fashion to execute rule locating, rule moving, fact processing, and decision making (depicted by the colored nodes).
All three models contain four families of attention heads with specialized roles in solving different steps in the problem: queried-rule locating heads, queried-rul mover heads, fact- processing heads, and decision heads ($3). This is surprising, as it indicates the model solves the logic problem in a sequential manner, without merging the steps. We further verify the roles of these four families by analyzing their behavior separately by patching after various types of counterfactual prompts, e.g., swapping rule locations or fact values ($3.3).
Intriguingly, the three LLMs all perform type of lazy reasoning. The core circuit does not immediately pre-process the Rules and Facts in the input. Rather, it primarily activates after seeing the Question and the ultimate query variable ($3). Moreover, the LLMS tend to reuse some of their sub-circuits for different parts of the argument when possible, consistent with Merullo et al. (2024a). We verify this for Gemma-2-9H and Mistral-7B ($.44).
We provide a partial analysis of Gemma-2-27B. which somewhat limited due to compu- tational contraints ( (33.4) While all three models contain these four families components, Demma-2-2275 also possesses certain logical-operator heads that do not seem to exist or have strong causal roles in the 9B version. Gemma-2-2771 2-27B's circuit appears to be slightly more parallel than the 9B version, which is more sequential. This adds nuances to a study showing that circuits are consistent across scale by Tigges et al. (2024). Indeed, from Gemma-2-9B to Gemma-2-27B. while the core algorithm appears to stay the same, there are mechanical differences and additional functional components which likely contribute to the larger model's higher proof accuracy.
Finally, we contrast pre-trained LLMs with 3-layer models trained on our task. The small models perform intermingled reasoning, linking together their attention blocks and residual streams in subtle ways. These small models have nearly perfect accuracy, but their reasoning mechanisms are less modular, as the functional roles of model's 's internal components tend
3

---

to be srtuation-depennee ($3.5) In contrast, Mistral and Gemma seem to consistently use specialized components for different parts of the task.
Our analysis is, to our knowledge, the first to characterize the circuits employed by LLMs in the wild for solving a logic problem that involves distracting clauses and requires multi-step reasoning. While this reasoning is limited compared to the abilities of today's ' thinking models (Guo et al. 2025al Arcuschin et al.] 2025). we still go beyond single-hop tasks like token copying or addition from recent studies, e.g., Feucht et al. (2025); Hu et al.] (2025). As another outcome, we have uncovered fundamental differences between small (3-layer), medium (7B/9B), and large (27B)
models, indicating only some mechanistic insights generalize across scales.
2 Preliminaries and Methodology
We explain our data model and then give an overview of the methodology we use for our analysis.
Input Prompts. We query pre-trained LLMs in a few-shot manner on propositional logic problems defined in the introduction. We show 4 6 examples of questions and their minimal proofs (see Appendix [ for details). Then, we append a new problem that asks for the truth value of one variable, in the "Question" part. We refer to this variable the QUERY token. A minimal reasoning chain consists of invoking the relevant fact(s) and rule to answer the query. Given the in-context examples, the models we consider always output a proof followed by the value of QUERY from (true, false, undetermined). Interestingly, for the disjunction problem (c.g., invoking A or B implies C)) when the model makes errors, it always starts with an incorrect first token (c.g., saying B is true, where relevant fact is that A is true). Hence, we can analyze the model's components when predicting the first token. In other words, to produce the first answer token in the minimal proof, the model must execute the causal chain "QUERY Relevant Relevant -act(s)-DDeission over the context.
Causal Mediation Analysis. Following prior work, we primarily rely on Causal Mediation Analysis (CMA) (Pearil (001) to characterize the reasoning circuits. In our setting, CMA is primarily concerned with measuring the (natural) direct and indirect effects (DE and IE) of model component. Consider the following classic causal diagram of CMA, in Figure
Suppose we wish to understand whether certain mediator M plays an important role in the causal path from the input X the outcome Y. We decompose the "total effect" of X on Y into the sum of direct and indirect effects, as shown in the figure. The indirect effect (IE) measures how important a role the mediator M plays in the causal path X + Y. To measure it, we compute Y given X, except that we artificially hold M's output to its "corrupted" version (called an intervention), which is obtained by computing M on a counterfactual ("corrupted")) version of the input. significant change in Y indicates strong IE, which implies that M is important in the causal path. On the other hand, a weak IE (so strong DE) implies that M is unimportant.
M
Indirect
x
Y
Direct effect
Figure 2: CMA causal graph.
Using CMA for Circuit discovery in LLMs Necessity and Sufficiency. We can apply the above CMA methodology to identify the reasoning circuits in LLMs. We look at both the proof and the final answer output by the model. We perform our interpretability analysis before the model has produced the first token. We believe the first token demands the greatest number of latent reasoning steps by the model, so it is the most interesting place to examine. Consider the conjunction problem in $2 The first step in our analysis is the construction of a 'counterfactual" prompt given a base prompt (we refer to the base prompt as the normal prompt in the rest of the paper). For the first answer token, when we flip the QUERY token from C E, the LLM must execute a fully different causal chain, from "C-+A and B implies C-AA is true, B is false-B" to "E-+ implies E(-D is true)-tD". In other words, flipping the QUERY token alone generates a "counterfactual" prompt which will help reveal the how the LLM latently executes the causal chain of "QUERY-<Relevan" rule-+Relevant faceis)=-Decision" The outcome Y measure is also naturally defined here: the softmax layer's 's logit difference between the two possible answers, B and D. Therefore, by generating nornal-counnerractual prompt pairs this fashion, LLM components with strong IEs, i.e., those pushing the LLM from the normal to the counterfactual answer, must belong to the reasoning circuit. To verify the sufficiency of the circuit, we just need to set the complement of the circuit as the mediator, and show that the DE is strong.

---

A or implies C
Component Legend
Koyhvauun
implles
Query
Attention head type (Layer
Outout
is true
is false
Queried-rule locating heads (W11), (112). (22.6).
Query:
Queried-rule mover heads (20.7. (23,)) (2455)
Fact-processing heads (24.5). (26.0:12)
Decision (28,120. (30.9)
Figure 3: Gemma-2-9B's reasoning circuit for the first answer token: The (chunks of) input tokens are on the left, which are passed into the residual stream and processed by the attention heads. We use (e, h) to denote an attention head. When referencing multiple heads in the same layer, we write (E, h1; h2; hn) for brevity. We illustrate the information flow manipulated by the different types of attention heads identified be vital to the reasoning task.
Fine-grained analysis: Understanding sub-component functionality. After obtaining the full circuit, can theorize about the role each component in the circuit plays in the reasoning chain. However, we need to verify such interpretations with empirical causal evidence. To do so, we again perform CMA with specific sub-components of the circuit held as the mediator, and we measure their IEs. For instance, if we hypothesize that an attention head uses the QUERY token to locate the rule being queried (the first reasoning step), then perform two experiments. First, set the mediator as the key activations of this head in the Rule section, and generate the counterfactual prompt by swapping the position of the rules, without touching anything else in the prompt. We should then observe a strong IE. Second, we generate the counterfactual prompt by flipping QUERY. Then, the query activation of this head, when held as the mediator, should have strong IE. In general, testing hypothesis of a reasoning circuit component's role requires careful control on where the input prompt "corrupted", where we need control over how the answer changes or does not change.
For the model solve our logic problem, it has to perform multiple steps of reasoning. going from the Question to the Rules to writing down the proof. This leads to more complicated process for systematically investigating the internal mechanisms the LLMs. Other tools commonly used in mechanistic interpretability such activation patching, causal tracing (Vig et al. 2020 Meng et al.
2022
Hase et al., 2024 Heimersheim & Nanda 2024 Zhang & Nanda 2024 are types of CMA.A
Discovering Modular Reasoning Circuits in LLMs
To make the problem easier for the smaller models, only examine Gemma-2-99 and Mistral-7B on the disjunction version (the LogOp is "or"); but, we study Gemma-2-27B on the full problem. For the "or" problem, Gemma-2-9B and Mistral-7B have proof accuracy around 83% and 70% respectively. Their accuracy with "and" is lower, while Gemma-2-27B has accuracy 86% on the full problem.
We focus on Gemma-2-99 in the main text, which illustrates our analysis process. This example elucidates the common procedure we apply to the other models as well. Appendix B contains the full results for Gemma-2-99, Mistral-7B and Gemma-2-27B models.
3.1 Discovering the Necessary Circuit: Qased Search for Model Components
We discover the core reasoning circuit by performing QUERY-based intervention experiments. After performing attention head output patching and measuring logit differences, we locate a small set of attention heads that are central to the reasoning circuit of the LLM. The MLPs have much lower intervened logit difference (mostly < 0.1) showing they play a limited role the reasoning process. However, we observe MIP-0 has a higher intervened logit difference. Prior work has observed that MLP--0 acts more "nonlinear token embedding" than complex high-level processing unit (Wang al. 2023a). Hence, in the rest of this section, we focus analysis on the attention heads.
5

---

18
Gueried-rut locating
315 CISEERR
19
20
08
726 2212:
20
21
(30L0Hg)
22
Querled-rule mover
24
Fact processing
0.4
26
27
02
29
Decision
50
00
fread Index
FNC
Head Indox
(a) Query
(b) Typical attention pattern
(c) Value
Figure 4: Patching of query and value actívations in (a) and (c). We show in (b) the typical attention patterns of representative
of the attention heads, which are important in the intervention experiments shown in (a) and (c). We make several observations from (b). Queried-rule locating head (19,1 11): observe that it correctly locates the queried rule that ends with Q. Queried-rule mover head (23,6): :): it primarily focuses on the QUERY token Q, with some small amount of attention on the tokens following Fact processing head 12): attention concentrates in the fact section. Decision head (30.9): attention focused on the correct first answer token U. In this experiment, intervening on "key" activations yields trivial scores, so we focus on "value" and "query.".
Remark. Unless otherwise specified, we adopt a calibrated version of the logit difference (see Appendix [ the closer to 1, the stronger the indirect effect of that component in the causal chain, and the greater a role that component plays in the reasoning circuit.
Attention-head sub-component patching (QUERY-based patching). We now aim to understand why the attention heads identified the last sub-section are important. For now, we continue with altering the QUERY in the prompt as our causal intervention. We intervene on the sub-components of each attention head, namely their value, key, and query, and we examine details of their attention weights. We find that there are four types of attention heads. We show the results in Figure 4
Queried-rule locating head. Attention head (19,11)'s query activation has a large intervened logit difference according to Figure a). Therefore, its query (and attention patterns) are QUERY- dependent and have strong IE. Furthermore, at the QUERY position, we find that on average, its attention weight is above 85% at the "conclusion" variable of the rule being queried. It is responsible for locating the queried rule, and storing that rule's information at the QUERY position !
Queried-rule mover head. Attention head (23,6)'s value activations have large intervened logit difference, and intriguingly, its query and key activations do not share that tendency. This already suggests that its attention pattern performs fixed action on both the original and altered prompts, and only the value information is sensitive QUERY. Furthermore, within the relevant context (excluding the 4 in-context examples given), (23,6) assigns around 50% attention weight to the QUERY position, and its attention weight at QUERY is about times larger than the second largest one on average. Recalling the role of the shallower layers, we find evidence that (23.6) moves the QUERY and queried-rule information the "." position
Fact processing heads. Attention heads (26,12)'s query activations have large intervened logit differences. Within the relevant context, the "." token position, it attention weight the sentence stating the correct fact is around 60% on average
Decision head, Attention heads (28,12) and (30,9)'s query activations have large intervened logit differences. Their attention patterns suggest that they are "decision" heads. Within the relevant context, each head's 's attention weight focuses the correct Fact variable (when the model is correct). The single token position occupies about than 80% of its total attention in the relevant context on average. In other words, locates the correct answer token.
*(17,9), (21,0), (21,7), (22,5), (23,12) 12) exhibit similar tendencies.
$(20,7), (24,15) also belong to this type.
((445)) (25.7). (26,0) also belong this type.

---

We delay detailed inspection and visualization of the attention statistics to Appendix [.4.2 Figure illustrates the reasoning circuit we identify. We remark on two intriguing properties:
1. Compared to the attention blocks, the MI.Ps are relatively unimportant to correct prediction.
2. There is sparse set of attention heads that found to be central to the reasoning circuit: the queried-rule locating heads, queried-rule mover heads, fact-procssiing heads, and decision heads. We discuss circuit discovery in $3.1 and verification $3.2 This indicates that very small fraction of the 9B parameters are taking part in solving specific question.
3.2 Verifying Sufficiency of the Discovered Circuit
A natural question now arises: is C. sufficient to explain the (QUER--sensttvee reasoning actions of the LLM? We prove sufficiency by measuring the direct effect of the input prompt on the difference in the final logits, with the complement of the identified circuit treated as the mediator (defined in $2). More concretely, we run the model on the normal prompts and only allow normal information flow in every attention head in the circuit C {(19, 11), (23, 6), (26, 12), (30,9) .)..)) (on or after QUERY), while freezing all the other attention heads their counterfactual activations obtained by running on the counterfactual prompts. We expect the average
peruut-inteevened logit difference A normal approaching or surpassing the average logit difference of the (un-intervened) model run on the normal prompts, A Anormat. We confirm this hypothesis in Table 1
C³ hormal normal
Caull
C
C-QRLH C C C-QMM C C-FPH C-DH
AC
-1.0
0.94
-0.97
-0.40
0.17
-1.11
Table 1: AC
normal/ A normat for Gemma-2-9R, with different choices of Ct. Cralt denotes the empty circuit, i.e. the case where no intervention performed. We abbreviate the attention head families: QRLH- = queried-rule locating heads, QRMH = queried-rule mover heads, FPH fact-processing heads, DH decision heads; removed.
DH = full circuit but with the decision heads
Including all 13 attention heads in C. AC Aorrma about 94% of Anormal on the normal samples. Removing any one of the four families of attention heads from C renders the direct effect almost trivial. Therefore, every head family is critical to the circuit. Additionally, for Mistral-7B's circuit which we obtained via the same search process, we were able to recover about 98% of the average logit-difference, indicating transferability of the procedure; please see B.5 for more details.
Please refer to Appendix B.8.3 (B...3 for deeper discussions of the experimental procedure and nuances of the results, including different versions of the circuit which we performed sufficiency tests on for Gemma-2-9B, the complexities of this verification process, and how we consider certain aspects of circuit verification major open problem in mechanistic studies of LLM reasoning.
3.3 Additional Evidence: More Fine-Grained Circuit Analysis
In this sub-section, we discuss example experiments how we further verify the functionalities of the attention head families. We present the full analysis in Appendix B.8
Queried-rule locating heads. We use the queried-rule locator heads, the "first step" in the reasoning circuit, to demonstrate how we perform finer "causal surgery" on the LLM to understand sub- components of the full circuit better.
First, based on our full-cireuit CMA experiments, we have formed an interpretation the queried-ruie locator heads: they rely the QUERY information to locate the queried rule in the Rules section. This is further corroborated by the attention statistics of these attention heads, shown in b) these attention heads place significant amount of attention weight on the correct position of the queried rule.
To further verify the "look-up" functionality, we create counterfactual prompts by swapping the location of the rules while keeping everything else untouched in the normal prompt. For instance, we would have the following normal-coumterfactue pairs:
7

---

1.0
0.2
16
posn
0.6
17
0.1
Aitertion
0.4
18
19
0.22
0.2
20
0.0
11.99
11. 11
(22.5)
(23.12)
0.0
(23.6)
(20,))
(24.15)
(b)(i) Dueried-rule locators
(b) Queried-nie rule movers
22
1.0
tok
23
Attentin woith
0.8
24
64
2 3
0.4
Head Index
62
0.2
(a) Example fine- grained intervention: swap rule locations, then patch key activations in the Rules positions
(24.51
(b)(i Fact procesoors
(b)(iv) Decision
Figure 5: Finer-gre gener-grained examination of circuit components in the Gemma-2-98 model. (a) shows the (zoomed-in) result of a finer-grate finer-grained activation patching experiment, aimed at providing further causal evidence of the rule-locating heads' functionalities. (b)(i) (iv) show the attention statistics of the core attention head families in the circuit.
Rules: A or B implies C. D implies E. Facts: A is true. B is false. D is true. Question: what is the truth value of C?
Rules: D implics E. A or B implies C. A is true. B is false. D is true. Question: what is the truth value of C?
altered
Facts:
prompt
Clearly, this rule location swap does not cause any change to the answer (here, C is true in both versions). However, the queried-rule locator heads' functionality suggests that they heavily rely the key activations (in the Rules section) to perform their role correctly. To provide further causal evidence for this interpretation, we should set the mediator as the key activations of the attention heads in the Rules section, and observe their indirect effects on the model's output logits, namely the drop in the logit difference between the answer of the queried rule and that of the alternative rule.
As expected, heads with the largest IE are the queried-rule locator heads, as shown in Figure 5] The keys with strong indirect effects are (17,4), (19,5), (22,2) and (23,6). By noting that Gemma-2-99 uses Grouped Query Attention which results in key and value activations shared per two heads, we see that keys with strong indirect effects indeed correspond the attention heads (17,9), (19,11), (22,5) and (23, 12). In addition, we perform CMA experiments where we swap the Fact value assignments. The facc-processing heads indeed exhibit the strongest IE.
possible limitation in generalizing this specific "causal surgery" to contexts with a large number of clauses is that, how distant the two clauses we interchange positions with becomes a possible confounding factor. Careful control on how distant the two interchanged clauses are and their absolute positions in the context is then required.
also show the weight of four attention head families in Figure F to (iv). The main takeaway is correlational evidence showing that heads attend to the expected part of the input (blue bars), which varies by type of head, rather than other positions.
Attention statistics.
3.4 The Reasoning Circuit in Gemma-2-27B
summarize a few interesting observations about Gemma-2-22 2-22BB here (Appendix B.9 has details).
Mechanical circuit differences between 27B and 9B models. Even though the 27B model also possesses attention heads which can be roughly divided into the families in the case of the model, there still are certain finer differences between their circuits: the 27B model's circuit appears have a greater degree parallelism.
particular, a number of fact-processing heads have direct effects the model's output: they tend to bypass the "decision" heads. On the other hand, the 9B model's is circuit more sequential: the fact processing heads are mostly "connected" to the decision heads and exhibit weak direct effects the model's logits.

---

Logical-operaten heads. As the 27B model is required to solve the full version of the problem, on top of the four generic families of attention heads, we find that there are certain flggcal-operatee heads which pay particular attention to the words "and" and "or" in the rules, and exhibit strong causal influence on the model's 's reasoning actions. particular, if we generate counterfactual prompts by flipping the logical operator from "and" to "or" (or vice versa) while keeping everything else the same as the normal prompt, then we find these logical-operator heads to have strong indirect effects. This is further discussed in Appendix B.9
3.5 Further Mechanistic Observations
Subcircuit reuse. Both Gemma-2-9B and Mistral-7B reuse their queried-rule locator heads and decision heads. Specifically, when the model needs to invoke a relevant rule (from the input) later to construct their output proof, it uses the same heads (but other input positions) to retrieve the location of the rule. Similarly, the decision heads may perform their function multiple times. In other words, this pair of components serve as the first and last reasoning step in both places of the model's proof. Please refer to Appendix 6.8.6 for details.
Prompt-format variation. To provide insights on how generalizable the sub-circuits we found are, we perform CMA Gemma-2-9B our logic problem, but with the prompt- formmtt changed to "Facts: ... Rules: Question: Answer: ..." (Rules and Facts sections are swapped in position). The queried-rule locators and decision heads remain almost the same in the discovered cireuit as before. The mover and fact-processing heads experienced some changes, however. For example, some of the fact-processing heads focus both on relevant premise variables in the Rules section and facts in the Facts section now. This indicates that the intermediate heads are more "fluid" in their functional roles in the circuit. We discuss these results in detail in Appendix B.8.7
Contrast against small transformers. To contrast against LLMs which are highly overparameterized and trained on large variety of data, we also conducted mechanistic analysis of small GPT-like transformer trained from scratch on a slightly more complex version of our logic problem. We present the analysis of the 3-layer 3-head transformer in Appendix Cand
and D with the high-level reasoning strategy employed by the model illustrated in Figure 30 it is the smallest model capable of achieving 100% test accuracy the problems.
The most pronounced difference we found is the less interpretable and modular nature of the attention heads in the small transformer. We show that, when a linear chain is queried, the second layer attention heads already arrive at the answer and the third layer stays "dormant", but when logical- operator-chain is queried, layer-3 attention heads need to actively participate to resolve the right answer. In other words, the small model does not have specialized "decision heads" for all situations in the problem, unlike the LLMs. Furthermore, when the logical opperatr chain is queried, even though the final-layer attention heads are proven to be the ones resolving the correct answer, their attention patterns unstructured, scattered across token positions in the Rules and Facts sections. These observations suggest a (relative) lack of modularity and interpretability in the small model.
4 Related Work
Mechanistic Interpretability. This area explores the hidden mechanisms that enable language-
modeling capabilities and other underlying phenomena from LLMs (Olsson et al. [022 Wang et al.)
2023b Feng & Steinhardf| 2024 Wu et al.
2023
Hanna et al.
2024
McGrath et al. 2023 Singh
al. 2024| 2424))2244geep Geiger al. [224]] Feng al.
2024
Marks et al.
2024
Engels et al. 2024| Csordás
et al. 2025] Baroni et al.] |025:
Merullo et al.
2024b)
Yin & | 2025:Lindssy Lindsey et al. 2025)) Some
works show how models retrieve knowledge and facts (Ferrando et al. [025]] Geva et al., [2023 Sun
iet al., 22255) Yao et al. [2024] Wang et al. 2024b) Lu et al.) 2025), encode concepts (Todd et al. 2024]
Yin & Steinhardi 2025 Hong et 2025 Beaglehole et |025 Li et al. (225cc employ certain
features in CoT reasoning (Dutta et al. [024]]] Troiskii et al. 2025 Wang et al. 2025a Venhoff et al.
2025b Chen al.) [2025] Venhoff et al. 2025a| Back Tegmark [2025] Li al.] 2025b; Huang
et al. 2025: Dai et al. [2025] Lee et al. (025). These works primarily provide causal evidence of
their claims through variants of causal medictionn analysis (Vig et al. 2020 Meng et al.
2022 Hase
et al. 2024 Heimersheim & Nanda 2024) Zhang & Nanda 2202 Mueller et al. 2025 Geiger et al.
2025
Our paper is in the sub-area of circuit- based mechanistic interpretability. e.g., (Elhage et al.
2021
Wang et al. 2023a). The novelty of our work is that we study larger LLMs on a compact logic

---

problem requiring hidden intermediate reasoning steps, which affords the analysis of how retrieval, processing, and decision components operate in conjunction, distinct from aforementioned works which identify certain steering features for CoT reasoning.
Coarse-Grained Evaluation of Reasoning in LLMs. There has been flrrry of work on training
and evaluating reasoning LLMs Muennighoft et al. 2025 Ye et al. 2025 Wang ct al., 22025c] |iinaa
ct
al.,
2025;
Zhang et al.
2025
Shojace et al. 2025). This extends the work on evaluating the
reasoning abilities of LLMs across different tasks (Xue et al. 2024] Taljord et al. 202 1 Hendrycks
ct al.
2021; Chen et al.
2024 Patel al. 2024 Berglund et al. 2024 Morishita et al. al.] [0023] Liu
it al.
2023 Fu et al. 2024| Seals & Shalin 2024 Joren et al. [2025] Zhang et al.
(aaaarov
& He| [203]] Saparov et al. 2024: Sun et al. 2023] Luo al. al.]| [024|] [Sahh et al. 2024 Arora et al.
2024]
Han et al. (224) et al. 2024 Yang et al. [2024] Loo et al. 2026 Li et 2025aa. While
these studies primarily benchmark their performance on sophisticated tasks, our work focuses on
understanding "how" transformers reason on logic problems accessible to fine-grained analysis (e.g..
mechanistic circuit analysis).
Fine-Grained Analysis of How LLMs Reason. There are fewer studies that provide fine-grained analysis of how pretrained LLMs reason latently. We build on ideas from mechanistic interpretability
for arithmetic (Stollo et al. 2023 Yu & Ananiadou 22024, Kantamneni & Tegmark] 2025 Zhou
ct al. 2024, Wang et al. 2025b; Mamidanna et 2005). syllogistic reasoning (Kim al. 2024).
graph connectivity (Saparov et al. 2022),, and indirect object identification (Wang et al. 2023a).
However, none studies full propositional -loic problems with [Variable relationships }+[ Variable value assignment]+[uueet
aspects while considering modern L.MSs. We complement existing work
on mechanistic analysis for such symbolic reasoning that considers small models trained on the
task (Brinkmann et al. 2024 Guo et al. 2025b Wang et al.) 2024a Zhu et al. 2024 Lin et al. 2025).
5 Discussion and Conclusion
We studied the reasoning mechanisms of three pre-trained models, ranging from 7B to 27B pa- rameters. To do so, we introduced simple propositional logic problem that was amenable to mechanistic interpretability tools. We characterized the LLMs' reasoning circuits, showing that they contain four families of attention heads. These components implement the reasoning pathway of "QUERY Relevvntt Eulevate the inner workings of LLMs on in-context logical reasoning problems, going beyond prior work. The fact that we found similar circuits in three distinct LLMs suggests that certain components organically arise from pre-training (even though it is unlikely that the LLMS were trained on identical problems).
" Our findings provide valuable insights into
A priori, it was not clear whether we could find isolated reasoning components in LLMs. In fact, we had a different conjecture based on studying the logic problem for small models, such as the 3-layer transformers (Appendix C and D). As mentioned before, for these small models, we found less modularity their components, and in particular, they are less lazy. This behavior on Transformers trained specifically on a synthetic task is also supported by the work of Ye et al. (2024). Real- world L.LMS seem to behave in different manner, where not much processing happens until the QUERY token is seen, and their attention patterns tend to be sparser and exhibit more consistent and specialized structures. Plainly, LLMs not only ingest facts from the input prompt (via in-context learning), but they also have specialized internal components that process the provided information before downstream computation.
Limitations. An area of improvement is to extend to more complex problems. With more computa- tional resources, we should aim to understand what structures are consistently employed by LLMs in implementing longer logical chains. A possible direction is analyzing latent circuits at critical token positions in the CoT trace of (thinking) LLMs, building on works such as Bogdan al. (025). but in more formalized settings, such as on longer propositionalllogii problems. This would strengthen and generalize our claims about the existence and robustness of reasoning circuits. Moreover, there are unanswered questions about how model size and family impact the types of reasoning circuits. We compare the results on Gemma with Mistral-7B in Appendix [B.4. providing some evidence that model size may be more indicative of capabilities than model family (given certain similarities between Gemma-2-9R and Mistral-7B). Embedding- analysis and steering are other important future directions.
10

---

Acknowledgements
The authors wish to thank Fred Zhang for the fruitful discussions around circuit analysis of L.Mss, and the anonymous reviewers for their helpful comments and suggestions for improving the paper.
References
Areuschin, I., Janiak, J., Krzyzanowski, R., Rajamanoharan, S., Nanda, N., and Conmy, A. Chain-of- thought reasoning in the wild is not always faithful. In ICLR Workshop on Reasoning and Planning for Large Language Models, 2025.
Arora, A., Jurafsky, D., Potts, C., and Goodman, N. D. Bayesian scaling laws for in-context learning. 2024. URL https: wtwiv...vvvv. org/ 2bs00 16531
Ba, J. L.., Kiros, J. R., and Hinton, G. E. Layer normalization, 2016. URL httpp:::////axiv..oo 0bs/1607.06450 06450
Back. D. D. and Tegmark, M. Towards understanding distilled reasoning models: A representational approach, 2025, http://t/////xiixorr/////////
03733
Baroni, L., Khara, G., Schaeffer, J., Subkhankulov, M., and Heimersheim, Transformers don't need layernorm at inference time: Scaling layernorm removal to gpt-2 xl and the implications for mechanistic interpretability, 2025. https//ts:// www.orrggaaaaaaoooeeeeeeeeee 02559
Beaglehole, D., Radhakrishnan, A., Boix-Adsera, E., and Belkin, M. Toward universal steering and monitoring of ai models, 2025. URL hwtpttttsss:s.....ooo org/ htwpb//252220.000000
Berglund, I.., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., and Evans, O. The reversal curse: L.LMs trained on "a b" fail to learn "b is a". In The Twelfth International Conference on Learning Representations, 2024. URL https: http://ppppreeee.eeeeeeeeeee nww. net/forumiid GPKTIktAOK
Bogdan, P. C., Macar, U., Nanda, N., and Conmy, A. Thought anchors: Which Ilm reasoning steps matter?, 2025.
//axii: http///www....oraaaaaaaa/a /abb///506/ 19143
Brinkmann, J., Sheshadri, A., Levoso, V., Swoboda, P., and Bartelt, C. A mechanistic analysis of transformer trained on symbolic multi-step reasoning task. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics; ACL 2024, pp. 4082-4104 Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10.18653/v1/2024, htw.i//...../.....--.o..a ac1.242. Bttlhttss///ccaattoooll
2024 www.or///.....ni........... findings- acl
242/
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G., Askell, A., Agarwal, S., Herbert-Vost A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D.
Language models are few-shot learners. In NeurIPS, 2020.
Chen, R., Zhang, Z., Hong, J., Kundu, S., and Wang, Z. Seal: Steerable reasoning calibration of large language models for free, 2025. URL htRps::tt://// phttps://arxiis //tpxv/aavvvvv//a org abs/a/as/////// 07986
Chen, X., Chi, R. A., Wang. X., and Zhou, D. Premise order matters reasoning with large language models, 2024. URL https::/aaxiv..ortt
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton,
J., Nakano, R., and Schulman, J. Training verifiers solve math word problems. arXiv preprint 20 2021.
Csordás, R., Mamning, C. D., and Potts, C. Do language models use their depth efficiently?, 2025. 20255. URL http:tttts/sxii..ott wwwpexivv..rreeommmm wtt.:rgggaa......aa.//////// 13898
Dai, X., Guo, K., Lo, C.-H., Zeng, S., Ding, Luo, D., Mukherjee, S., and Tang. J. Graphghost: Tracing structures behind large language models, 2025. URL URL https::///// //tp:///a.v..///////// wwwxiv.org/aaie. www./or/////aeeee 2510 08613

---

Dutta, S., Singh, J., Chakrabarti, S., and Chakraborty, T. How to think step-by-step: A mechanistic understanding of chain-of- chain-of-thought reasoning. Transactions on Machine Learning Research, 2024.
ISSN 3335-8856. URI.bttps: http://perrevveeeeeeeeeerrrrr/// net./Eorumm.d-ullDDDVC....
Dziri, N., Lu, X.., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., Welleck. S., West, P., Bhagavatula, C., Le Bras, R., Ettinger, A., Harchaoui, Z.,, and Choi, Y. Faith and fate: Limits transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024.
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Haffield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L.., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/transformmr circuits. pttp://0////aammooot/i/eeee
Engels. J., Riggs, L.. and Tegmark. M. Decomposing the dark matter of sparse autoencoders, 2024.
URL attps://aaxiii: org/ abs/2410 14670
Feng, J. and Steinhardt, J. How do language models bind entities in context? In The Twelfth International Conference on Learning Representations, 2024. URL https:://peenre net/ "oorumidd=23bbokk0)
Feng, J., Russell, S., and Steinhardt, J. Monitoring latent world states in language models with propositional probes, 2024. URL httpp:://axii..rrr abs wwwpaa/6..... /400 19501
Ferrando, J., Obeso, O., Rajamanoharan, S., and Nanda, N. Do know this entity? knowledge awareness and ballucinations in language models. In ICLR, 2025.
Feucht, S., Todd, E., Wallace, B., and Bau, D. The dual-route model of induction. arXiv preprint Mrxii:2504.03022 2025.
Fu, D., Guo, R., Khalighinejad, G., Liu, O., Dhingra, B., Yogatama, D., Jia, R., and Neiswanger, W. Isobench: Benchmarking multimodal foundation models on isomorphic representations. In
First Conference on Language Modeling, 2024. httpppp://oppppruuu.ueeet/rrrura KZd1EErRJ1
Geiger, A., Wu, Z., Potts, C., Icard, T., and Goodman, N. D. Finding alignments between interpretable causal variables and distributed neural representations, 2024. / 2303. 02536
Geiger, A., lbeling, D., Zur, A., Chaudhary, M., Chauhan, S., Huang, J., Arora, A., Wu, Z.. Goodman, N., Potts, C., and Icard, T. Causal abstraction: A theoretical foundation for mech- anistic interpretability. Journal of Machine Learning Research, 26(83):1-64, 2025. URL http://jmlr org/ wwtpootwrg/////eeeet.oo.. pwppps/ppsseeee 22/ 0058. html
Gemma Team, Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momehev, N., Hoffman, M., Thakoor, S., Grill, J.-B., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot, B., Wu, B., Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C., Choquette-Choo,, C. A., Sinopalnikov, D., Weinberger, D., Vijaykumar, D., Rogozińska, D.. Herbison, D., Bandy, F., Wang, F., Noland, F.,, Moreira, F.,, Senter, F.,, Eltyshev, E.,, Visin, F. Rasskin, G., Wei, G., Cameron, G., Martins, G., Hashemi, H. Pimmczak-llucinska, H., Batra, H. Dhand, H., Nardini, I., Mein, J., Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, J. P., Carrasqueira, J., Djjzi, J., Becker, J., Fernandez, J., van Amersfoort, J., Gordon, J., Lipschultz, J. Newlan, J., yeong Ji, J., Mohamed, K., Badola, K., Black, K., Millican, K., McDonell, K., Nguyen, K. Sodhia, K., Greene, K. Sjoesund, L.. Usui, L.., Sifre, L., Heuermann, L., Lago, L., McNealus, L., Soares, L. B., Kilpatrick, L.. Dixon, L., Martins, L., Reid, M., Singh, M., Iverson, M., Görner, M., Velloso, M., Wirth, M., Davidow, M., Miller, M., Rahtz, M., Watson, M., Risdal, M., Kazemi, M., Moynihan, M., Zhang, M., Kahng, M., Park, M., Rahman, M., Khatwani, M., Dao, N., Bardoliwalla, N., Devanathan, N., Dumai, N., Chauhan, N., Wahltinez, O., Botarda, P., Barnes, P., Barham, P., Michel, P., Jin, P., Georgiev, P., Culliton, P., Kuppala, P., Comanescu, R.,
12

---

Merhej, R., Jana. R., Rokni, R. A., Agarwal, R., Mullins, R., Saadat, S., Carthy, S. M., Cogan, S., Perrin, S., Arool, S. M. R., Krause, S., Dai, S., Garg, S., Sheth, S., Ronstrom, S., Chan, S., Jordan, T., Yu, T., Eccles, T., Hennigan, T., Koeisky, T. Doshi, T., Jain, V., Yadav, V., Meshram, V., Dharmadhikari, V., Barkley, W., Wei, W., Ye, W., Han, W., Kwon, W., Xu, X., Shen, Z.,, Gong, Z., Wei, Z., Cotruta, V., Kirk, P., Rao, A., Giang. M., Peran, L.., Warkentin, T., Collins, E.,, Barral, J., Ghahramani, Z., Hadsell, R., Sculley, D., Banks, J., Dragan, A., Petrov, S., Vinyals, O., Dean, J. Hassabis, D., Kavukcuoglu, K., Farabet, C., Buchatskaya,, E., Borgeaud, S., Fiedel, N., Joulin, A., Kenealy, K., Dadashi, R., and Andreev, A. Gemma 2: Improving open language models at practical size, 2024. URW.ttps: /tttps://arxtt org/ abs /ww.////.... 001188
Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. -494,, 2021.
Geva, M., Bastings, J., Filippova, K., and Globerson, A. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216 12235, 2023.
Guo, D., Yang, D., Zhang, H., Song. J., Zhang, R., Xu, R., Zhu, Q.. Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint MrXiv:2501.12944 2025a.
Guo, T., Zhu, H., Zhang, R., Jiao, J., Mei, S., Jordan, M. I., and Russell, S. How do llms perform two-hop reasoning in context? arXiv preprint arXiv:2502.1391 2025b.
Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Zhou, W., Coady, J., Peng, D., Qiao, Y., A., Szabo, H., Zubova, E., Burtell, M., Fan, J., Liu, Y., Wong, B., Sailor, M., Ni, A., Nan, L., Kasai, J., Yu, T., Zhang, R., Fabbri, A. R., Kryscinski, W., Yavuz, S., Liu, Y., Lin, X. V., Joty, S., Zhou, Y., Xiong, C., Ying. R., Cohan, A., and Radev, D. Folio: Natural language reasoning with first-order logic, 2024. URL httts:tts:///rra
Benson, L., Sun, L.,
org/ abs http:/bb/22000.000 008000
Hanna, M., Liu, O., and Variengien, A. How does gpt-2 compute greater-than? interpreting mathematical abilities in a pre-trained language model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Hase, P., Bansal, M., Kim, B., and Ghandeharioun, A. Does localization inform editing? surprising differences in causality- -bsee localization vs. knowledge editing in language models. In Proceed- ings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024, Curran Associates Inc.
Heimersheim, S. and Nanda, N. How to use and interpret activation patching, 2024. URL https //arxiv. waw.ii.org/aaeo.ee...o..aoooolae aba//ba//244. 15255
Hendrycks, D., Burns, C., Kadavath, S., Arora, A.. Basart, S., Tang, E., Song, D., and Steinhardt, J. Mcasuring mathematical problem solving with the math dataset. arXiv preprint StV:2203303874. 2021.
Hong, G. Z.,, Vasudeva, B., Sharan, V., Rashtchian, C., Raghavan, P., and Panigrahy, R. Latent concept disentanglement in transformer-based language models, 2025. URL https:t//tsxiiii org/ /206 16975
Hou, Y., Li. J., Fei, Y., Stolfo, A., Zhou, W., Zeng, G., Bosselut, A., and Sachan, M. Towards a mechanistic interpretation of multi-step reasoning capabilities of language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4902-4919, 2023.
Hu, X., Yin, K., Jordan, M. I., Steinhardt, J., and Chen, L. Understanding in in-context learning of addition via activation subspaces. arXiv preprint arXiv:2505.0511 2025.
Huang, Y., Chen, H., Ruan, S., Zhang, Y., Wei, X., and Dong, Y. Mitigating overthinking in large reasoning models via manifold steering, 2025. /tt/tttss://rxxivorrr
22411
13

---

Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C.. Chaplot, D. S., de las Casas, D., Bressand, F. Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. I.., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL. https
//arxiv. wwrriv.org//e abs 2a10./2331.00m .211000.68555
Joren, H., Zhang. J., Ferng, C.-S., Juan, D.-C., Taly, A., and Rashtchian, C. Sufficient context: A new lens retrieval augmented generation systems. In ICLR, 2025.
Kantamneni, S. and Tegmark, M. Language models use trigonometry do addition. arXiv preprint arXiv:2502.00873, 2025.
Kim, G., Valentino, M., and Freitas, A. A mechanistic interpretation of syllogistic reasoning in auto-regressive language models, 2024, URL http:///axxxvv...aabbb/ wtt./og///aaaa///oeeee 08590
Lee, A., Sun, L., Wendler, C., Viégas, F., and Wattenberg, M. The geometry of self-verification in task-specific reasoning model, 2025. URL URL http:::tttssrrrss
hatgg/bb//25000 14379
Li, A., Wang, C., Fu, D., Yue, K., Cai, Z., Zhu, W. B., Liu, O., Guo, P., Neiswanger, W., Huang, F, Goldstein, T., and Goldblum, M. Zebra-cot: A dataset for interleaved vision language reasoning. 2025a. httpsts::://att aariv.iiiivv//a///eee org/ wwwp/rg////aaeeeeeee aa/////s//// 2561/
Li, Y., Dong, Z., Sun, Y., Wang. W., Xiong, S., Luo, Y., Liu, J., Lu, H., Wang. J., Su, W., Zheng, B., and Yan, Attention illuminates IIm reasoning: The preplan-and-anchon rhythm enables fine-graines policy optimization, 2025b. URL /etttp://arxiii www.ooggg///baabeeeeoooeeee absp/510.... 13554
Li, Y., Michaud, E. J., Back, D. D., Engels, J., Sun, X., and Tegmark, M. The geometry of concepts: Sparse autoencoder feature structure. Entmopy, (7),,, 2025c. ISSN 1099-4300. doi: 10.3390/e27040344. Utthttps::/ http://pppp..../000333//////////
Lin, T., Xie, J., Yuan, S., and Yang, D. Implicit reasoning in transformers is reasoning through shortcuts, 2025. Utt.pttps://aaxi org/ 2aabb///5500000 07604
Lindsey, J., Gurnee, W. Ameisen, E., Chen, B., Pearce, A., Turner, N. L., Citro, C., Abrahams, D., Carter, S., Hosmer, B., Marcus, J., Sklar, M., Templeton, A., Bricken, T., McDougall, C., Cunningham, H.. Henighan, T., Jermyn, A., Jones, A.. Persic, A., Qi. Z.. Thompson, B.. Zimmerman, S., Rivoire, K., Conerly, T., Olah, C., and Batson, J. On the biology of a large language model. Transformer Circuits Thread, 2025, http::tttt:::tsttttete -ireerriiirrrttt pub/2025/ attr ieiiiiuuiionnoo wwwuuiionngrasss//i www.ps//aalloo.ooooooee html
Liu, H. Teng, Z., Cui, L., Zhang, C., Zhou, Q., and Zhang, Y. LogiCoT: Logical thain-of-thougle instruction tuning. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 20233, pp. 2908-228822 2921, Singapore, December 2023. Association for Computational Linguistics. doi: 2011653331/22233 http2222...innnngemmm..9.9999... URL https://
aclanthology.org//000
eerddings-mmlpp 191/
Loo, A., Pavlick, E., and Feiman, Lims model how humans induce logically structured rules. Journal of Memory and Language, 146: 146675 2026. ISSN 9749-596X. doi: https://doi.org//
jol. 104675. rtt. 50749596X25000683
Lu, M., Zhang, R., Eickhoff, C., and Pavlick, E. Paths not taken: Understanding and mending the multilingual factual recall pipeline, 2025. Wttthttss///axxtt wwtpr/iwwwwr..bbss///eoesboeae 2505. 20546
Luo, M., Kumbhar, S., shen, M., Parmar, M., Varshney, N., Banerjee, P., Aditya, S., and Baral, C. Towards logiglue: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models, 2024. URL httlhhtts://eee http:/iwwwwrr/abb/////eae 00836
MacCartney, B. and Manning, D. Natural logic and natural language inference. In Computing Meaning: Volume 4, pp. 129-147. Springer, 2014.
Mamidanna, S., Rai, D., Yao, Z., and Zhou, Y. All for one: Lims solve mental math at the last token with information transferred from other tokens, 2025. URL 09650
org/ warggabs//20 2509
14

---

Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models, 2024. URL 2403. 19647
McGrath, T., Rahtz, M., Kramar, J., Mikulik, V., and Legg, S. The hydra effect: Emergent self self-repair in language model computations, 2023. URL http::tttt:/atae org/ aba/a/aaa///00 15771
Meng, K., Bau, D., Andonian, J. and Belinkov, Y. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, 2022. BRKpitttps//opperrrriee net het/forum?id= h6WAS6eE4
Merullo, J., Eickhoff, C., and Pavlick, E. Circuit component reuse across tasks in transformer language models. In The Twelfth International Conference on Learning Representations, 2024a.
URL
Merullo, J., Eickhoff, C., and Pavlick, E. Talking heads: Understanding lnter-layer communication in transformer language models, 2024b.
wwtp:///gg//bbaaa/e///ooese 2406. 09519
Morishita, T., Morio, G., Yamaguchi, A., and Sogawa, Y. Learning deductive reasoning from synthetic corpus based on formal logic. Krause, A., Brunskill, E.,, Cho, K., Engelhardd, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 25254-25274. PMLR, 23-29 Jul 2023. URL
Mueller, A., Brinkmann, J., Li, M., Marks, S., Pal, K., Prakash, N., Rager, C.,, Sankaranarayanan, A., Sharma, A. S., Sun, J., Todd, E., Bau, D., and Belinkov, Y. The quest for the right mediator: Surveying mechanistic interpretability for nip through the lens of causal mediation analysis. Computational Linguistics, pp. 1-48, 09 2025, ISSN 0891-2017. doi: 10.1162/COLLa..72. URL https:://doi.rrg// 10/ 11w1//1111...... wtt.:////....111/........
Muennighoff, N., Yang. Z., Shi, W., Li, L., Fei-Fei, L.., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., and Hashimoto, T. sl: Simple test-time scaling. arXiv preprint arXiv:2501. 19393, 2025.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodee Z., Hernandez, D., Johnston, S., Jones, A., Kermion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. In-context learning and induction heads. Transformer Circuits Thread, html..
2022.
Patel, N., Kulkarni, M., Parmar, M., Budhiraja, A., Nakamura, M., Varshney, N., and Baral, C. Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models, 2024. URL //tp:iaaaiivvavaaaa org/ abs /20//2466..0 17169
Pearl, J. Direct and indirect effects. In Proceedings of the Seventeenth Conference Uncertainty and Artificial Intelligence, 2001, pp. 411 411-420. Morgan Kaufman, 2001.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. In OpenAl blog. 2019a. /ttttttps://ap. httpht/ps://pp..mmaatteeeeeee semanticscholar org/CorpusID: 160025533
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019b.
Rauker, T., Ho, A., Casper, S., and Hadfield-Menell, D. Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML.,, pp- 464-483, Los Alamitos, CA, USA, February 2023. IEEE Computer Society, doi: 10. 20t.aa..55555.5222...0. //doi. www.://cpppeeerroccctr.ooor////e htt.:/c..ct....ooo...ooo////a/aa 200.::..2222.000 2023.00039
URL https
Saparov, A. and He, H. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In The Eleventh International Conference on Learning Representations, 2023.
URL http:::/oppnrrevies http://www......eeeeeoooooooaaaa
15

---

Saparov, A., Pang, R. Y., Padmakumar, V., Joshi, N., Kazemi, S. M., Kim, N., and He, H. Testing the general deductive reasoning capacity of large language models using ood examples. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Saparov, A., Pawar, S., Pimpalgaonkar, S., Joshi, N., Pang, R. Y., Padmakumar, V., Kazemi, S. M., Kim, N., and He, H. Transformers struggle to learn to search. In ICLR, 2025.
Seals, S. and Shalin, V. Evaluating the deductive competence of large language models. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 8614 8630, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. http5///1//2222mmalllaooaa baacl-long.o. 476
BRLLhttps:.
Shah, K., Dikkala, N., Wang. X., and Panigrahy, R. Causal language modeling can elicit search and reasoning capabilities on logic puzzles. In Globerson, A., Mackey, L., Bel- grave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang. C. (eds.), Advances Neu- Curran Associates, 67b31 https://prroceseinngs mewpeeeddinnnnnnunsss.eeee newrips......e cc ccppaper_... files/ httee//appe//22//11.e.. .pff
ral Information Processing Systems, volume 37, pp. Inc., 2024. URL https:
Shojace, P., Mirzadeh, I., Alizadeh, K., Horton, M., Bengio, S., and Farajtabar, M. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity, 2025. URI.https //wrxxvvvvvv///////////////xii org/ abs/abss///00/ 050./00006000111111 06941
Singh, C., Inala, J. P., Galley, M., Caruana, R., and Gao. J. Rethinking interpretability in the era of large language models, 2024. URL https://axxii org http://rrggaaaaa///oab//e /btp//bbbb/22221111.1
Sinha, A., Arun, A., Goel, S., Staab, S., and Geiping, J. The illusion of diminishing returns: Measuring long horizon execution in Ilms, 2025.
/209//55...o 2509.09677
Stolfo, A., Belinkov, Y., and Sachan, M. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7035-7052, 2023.
Sun, J., Fu, D., Hu, Y., Wang, S., Rassin, R., Juan, D.-C., Alon, D., Herrmann, C., van Steenkiste, S., Krishna, R., and Rashtchian, C. Dreamsyne: Aligning text-to-image generation with image understanding feedback, 2023. httt:httss://rrrt
org/ 2211. 17946
Sun, Z., Zang, X., Zheng, K., Song, Y., Xu, J., Zhang, X., Yu, W., and Li, H. Redeep: Detecting hallucination in augmenel- generation via mechanistic interpretability. In ICLR, 2025.
Tafjord, O., Dalvi, B., and Clark, P. ProofWriter:
Generating implications, proofs, and abductive statements over natural language. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 3621-3634, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 Linddiigs-n..n...333 URL http:::/acllanthoooogy org 2021 w0..2...innnini findings- ac1.317
Tigges, C., Hanna, M., Yu, Q. and Biderman, S. Llm circuit analyses are consistent across training and scale. In Advances in Neural Information Processing Systems, volume 37, pp. 40699-40731. Cur- ran Associates, Inc., 2024. URL https: //ttce/eecc/ccceeeeeeeee neurips cc/ soc/paper_fiiie saapperffiee/paare Paper Conference. pdf
Todd, E., Li, M., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function vectors in large language models. In The Twelfih International Conference Learning Representations, 2024. URL httpps:/opeereriieee http://pwoorreeeeeeeee/o//aaaaaa Awyyyyyyaa
Troitskii, D., Pal, K., Wendler, C., McDougall, C. S., and Nanda, Internal states before wait modulate reasoning patterns, 2025. URL tattps://arxxx org/ abs/ abs/2510. 0t////2110.0122888
16

---

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez. A. N., Kaiser, L. u. and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems. volume 30. Curran Associates, Inc., 2017. URL https://pproceeddngg neuri wwwwrip..ccppppp..eeeeeeee.e cc/paper
Venhoff, C., Arcuschin, I., Torr, P., Conmy, A., and Nanda, N. Base models know how to reason, thinking models learn when, 2025a. URL
07364
Venhoff, C., Arcuschin, I,, Torr, P., Conmy, A., and Nanda, N. benderrstandinng in thinking language models via steering vectors, 2025b, URL httpps///aarrvv..rr/////////// 18167
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieher, S. Investigating gender bias in language models using causal mediation analysis. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12388-12401. Curran Associates, Inc., 2020. URL paper_files wttp://awrr.......22222222222210
iet.eeerediinnnnnnnp.p......p neurips cc/ pdf
Wang, A., Engels. J., Clive-Griffi, O., Rajamanoharan, S., and Nanda, N. Simple mechanistic explanations for out-of-context reasoning, 2025a. URL httph:tttt//iiii// arggabs//5007.085 08218
Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference Learning Representations, 2023a. URL MpsVSS604ul
net/ forum?id
Wang. L., Li, L., Dai, D., Chen, D.. Zhou, H., Meng. F., Zhou, J., and Sun, X. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9049-9949555 2023b.
Wang. X., HU, Y., Du, W., Cheng, R., Wang. B., and Zou, D. Towards understanding fine-tniin mechanisms of Ilms via circuit analysis, 2025b. URL httpt:ttt////iiii
htrgg/abs/2502/ 11812
Wang, Y., Yang, Q., Zeng, Z., Ren, L., Liu, L., Peng, B., Cheng, H., He, X., Wang, K., Gao, J., et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.205771 2025c.
Wang, Z., Wang, Y., Zhang, 7., Thou, Z., Jin, H., Hu, T. Sun, J., Li, 7., Zhang, Y., and Xu, Z.-Q. J. The buffer mechanism for multi-step information reasoning language models. arXiv preprint 2rXiv:2405.15302, 2024a.
Wang, Z., White, B., and Xu, C. Locating and extracting relational concepts in large language models. arXiv preprint arXiv:2406.1333 13184, 2024b.
Wu, Z., Geiger, A., Icard, T., Potts, C., and Goodman, N. Interpretability at scale: Identifying causal mechanisms in alpaca. In Teirrt-seventhh Conference on Neural Information Processing Systems, 2023. URL
Xue, A., Khare, A., Alur, R., Goel, S., and Wong, E. Logicbreaks: framework for understanding inference, 2024. URL https: org/ 0tbs//240.0000
subversion of
Yang, S., Gribovskaya, E., Kassner, N., Geva, M., and Riedel, S. Do large language models latently perform multi-hop reasoning? In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 10210-10229, Bangkok, Thailand, August 2024. Association for Computational URL https: ptttps://aclaathollee org/ 2024
Linguistics. doi: 1/06553/1/20244aclllogggg acl 2a1-long.555
Yao, Y., Zhang, N., Xi, Z., Wang, M., Xu, Z., Deng. S., and Chen, H. Knowledge circuits in pretrained transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL htttsp:///ppnreviie
Ye, T., Xu, Z., Li, Y., and Allen-Zhu, Z. Physies of language models: Part 2. 1, grade-school math and the hidden reasoning process, 2024. http:t www.:/org//bb/////. 20311
17

---

Ye, Y., Huang. Z., Xiao, Y., Chern, E., Xia, S., and Liu, P. Limo: Less is more for reasoning, arXiv preprint arXiv:2502.03387, 2025.
Yin, K. and Steinhardt, J. Which attention heads matter for in-context learning? arXiv preprint 1400000502.44000, 2025.
Yin, Y. and Wang, Z. Are transformers able to reason by connecting separated knowledge in training data?, 2025.
15857
Yu, Z. and Ananiadou, S. Interpreting arithmetic mechanism in large language models through comparative neuron analysis. arXiv preprint arXiv:2409. 14/44 2024.
Zhang. and Nanda. N. Towards best practices activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreviee net httr://oommidddff////
Zhang. H., Li, H., Meng, T., Chang, K.-W., and Van Den Broeck, On the paradox of learning to reason from data. In Proceedings of the Thirly-Second International Joint Conference on Artificial Intelligence, IJCAI '23, 2023, ISBN 978- 978-1-956792--0--4
doi: 00.24966/jcaai..2223/77 URL
https://doi. org/ 10. 24963 24963/1jcas iacii 2023/375
Zhang. J., Juan, D.-C., Rashtchian, C., Ferng, C.-S., Jiang, H., and Chen, Y. Sled: Self logits evolution decoding for improving factuality in large language models. In Advances in Neural Information Processing Systems, pp. 5188-5209, 2024.
Zhang, Q., Fu, Y., Wang, Y., Yan, L., Wei, T., Xu, K., Huang, M., and Qiu, H. On the self-awareness of large reasoning models' capability boundaries, 2025. URL wttphhttt///xiii/ http//awwwv..g.aaa/a////.a/ hta//bbb/22000 24711
Zhou, T., Fu, D., Sharan, V., and Jia, R. Pre-trained large language models use fourier features com-
pute addition. In Advances in Neural Information Processing Systems, volume 37, pp. 25120-25151. Curran Associates, Inc., 2024. URL httpp:://proceeddagggg wwwrips.e.ppp...e. cctpuurppssccppppppppp ccppaper...ppp....ppp.oe wwtpc//ppppaeeeeeeeeeeeeeeeeeeee larerrrfiles_ files/
oore pdf
Zhu, H., Huang, B., Zhang, S., Jordan, M., Jiao. J., Tian, Y., and Russell, S. Towards a theoretical understanding the 'reversal curse" via training dynamics. The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. htllhtts///opereviieee fermmiom QoWf31o6m7
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims in the abstract are present the main paper or appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
18

---

Answer: [Yes]
Justification: We have a specific limitation section in them main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Clear details of all experiments and set-up in the appendix.
Guidelines:
19

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are not able to release code. All data generation processes are clearly described and all models used open source and cited.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nis.cc/public/guust http://i/cgiiieee/eeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
20

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All details of the models and experiments are in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We mention running more experiments / examples every time the significance is in question.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We details the compute resources (GPUs) used to run the models, which follow the standard set-up for the open source models in this work.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
No ethics concerns.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We study synthetic problems, without ties potential applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
22

---

Justification:
We only use existing pre-trained LLMs.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All models and ideas are cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: Everything detailed clearly, and we do not release new assets. Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
23

---

Answer:
[NA]
Justification:
No crowdsourcing or data release.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No human involvement.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We study LLMs and clearly specific how to use them. We do not use LLMs for other purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanndaddrrd components.
Please refer to our LLM policy (https://neet /cC//aareeecccc/2222//////////// for what should or should not be described.
24

---

Appendix
A Methodology and Problem Details
A.1 Appendix Table of Contents and Outline
In this section, we provide an overview our methodology that we presented at a high level in Section We then give forward pointers to our additional experimental evidence. We aim to provide an outline the LLM analysis, guiding the reader through the rest the appendix.
A.2 gives representative examples of the propositional logic problem that we study, In general, we vary the variable names, the logical operator ("or" vs. "and"), the truth values of the variables (i.c., the "Facts"), and the QUERY variable.
BB.T provides details of the few-shot examples and input format, and some initial results about model accuracies. In particular, we engineer the in-context examples to ensure that all models perform much better than random guessing.
B.2 is an expository, background section that shows how activation patching can surface circuit components. For completeness, we illustrate the main techniques on toy 2-layer model, with figures. The main technique to run the LLM on altered (alt) prompts, while freezing certain components to the activations on the original (orig) prompt. Throughout, we also define some notation, such as AC arig-aale which is the intervened logit difference. The relationship between the orig and alt prompts depends on the specific experiment, where we may change the query variable, the rule locations, or the truth values.
B.3 defines the metrics for QUERY-basee activation patching.
B.4 through [B.7 the full analysis of the Mistral 7B model. We break this down into multiple subsections present the details of the experiments, where we use the same approach for the Gemma models later on. In particular, $B.5 has the sufficient test, $B.6) contains results for varying the rule locations, and $B.7 has results for varying the facts.
$ B.8connaiis the experiments for Gemma 2 9B. This provides full details for the experiments that we report in the main paper.
B.9 similarly contains results for Gemma 2 27B, including attention head analysis, and a discussion of the mechanical differences between Gemma 2 9B and 27B.
$C presents full mechanistic analysis of small transformer models (e.g.. 3-layer). We present the main insights in şC then supplement this with the full details in $D We train several models autoregressively on the propositional logic task. Then, explore the mechanisms through combination of experiments (patching, probing, classification, etc), focusing a 3-layer model. We present the high level strategy employed by the 3-layer model to solve the logic problems in Figure 30 providing a contrast to the modular and more interpretable circuit in the pre-trained L.Mss.
A.2 Propositional logic problem and examples
In this section, we provide more detailed description of the propositional logic problem we study in this paper, and list representative examples of the problem.
At its core, the propositional logic problem requires the reasoner to (1) distinguish which chain type is being queried (LogOp or linear), and (2) if it is the LogOp chain being queried, the reasoner must know what truth value the logic operator outputs based on the two input truth values.
Below we provide comprehensive list of representative examples of our logic problem at length 2 (i.e. each chain formed by one rule). We use [Truth values] denote the relevant input truth value assignments (i.e. relevant facts) to the chain being queried below.
1. Linear chain queried, [True]
Rules: A or B implies C. D implics E.
Facts: A is true. B is true. D is true.
25

---

Question: what is the truth value of C?
Answer: D is true. D implies E; E True.
2. Linear chain queried, [False]
Rules: A or implies C. implies
Facts: A is true. B is true. D is false.
Question: what is the truth value of C?
Answer: false. implies E; undetermined.
3. LogOp chain queried, LogOp OR, [True, True]
Rules: A or implies C. D implies
Facts: A is true. B is true. D is true.
Question: what is the truth value of C?
Answer: is true. A B implies C True.
Remark A.1. In this case, the answer "A true. A or implies C; True" is also correct.
4. LogOp chain queried, LogOp= = OR, [True, False]
Rules: A or B implies C. D implies E.
Facts: A is true. B is false. D is true.
Question: what is the truth value of C?
Answer: true. or B imples C; C True.
5. LogOp chain queried, LogOp= = OR, [False, False]
Rules: A or B implies C. D implies E.
Facts: A is false. B is false. D is true.
Question: what is the truth value of C?
Answer: false false. A or B implies C: C undetermined.
6. LogOp chain queried, LogOp = AND, [True, True]
Rules: A and B implies C. D implies E.
Facts: A is true. B is true. D is true.
Question: what is the truth value of C?
Answer: A is true. B is true. A and B implies C; C True.
7. LogOp chain queried, LogOp = AND, [True, False]
Rules: A and B implies C. D implies E.
Facts: A is true. B is false. D is true.
Question: what is the truth value of C?
Answer: B is false. A and B implies C; C undetermined.
8. LogOp chain queried, LogOp = AND, [False, False]
Rules: A and B implies C. D implies E.
Facts: A is false. B is false.
true.
Question: what is the truth value of C?
Answer: A is false. A and B implies C; C undetermined.
Remark A.2. In this case, the answer "B is false. A and B implies C; C undetermined" is also correct.
Importance of the first answer token. As discussed in T correctly writing down the first answer token central to the accuracy of the proof. First, it requires the model to process every part of the context properly without CoT due to the minimal-prood requirement of the solution. Moreover, it the answer token position which demands the greates! number of latent reasoning steps in whole answer, making it the most challenging token for the model to resolve. Therefore, this token the most interesting place to focus our study on, as we are primarily interested in how the model internalizes and plans for the reasoning problems in this work.
26

---

B
The reasoning circuit in LLMs: experimental details
B.1 Problem format
We present six examples of the propasitional-loge problem in context the Mistral-7B model, and four examples to the Gemma-2-9B model. We then prompt them for their answer to newly appended problem. To generate the problem, we use the template and then sample variables and truth values. The two chains have independent proposition variables (leading to five distinct boolean variables). We randomly choose variables from capital English letters (A-Z) and truth values (true or false).
Remark B.I. To ensure fairness, for the disjunction problem, we provide the LLM with equal number of in-context examples for each query type (the OR chain and the linear chain) in random order. Please note that, while the premise variables in the linear chain examples could be set FALSE when not queried, the actual question (the seventh example which the model needs to answer) always sets the truth value assignment of the linear chain to be TRUE, preventing the model from taking a shorteut and bypassing the "QUERY -elevant Rule" portion of the reasoning path.
Additionally, the models perform much better than random on these problems, with high accuracy for Mistral (above 70%) and Gemma-2-9B (above 83%). Gemma-2-278 achieves nearly perfect accuracy. This reflects a balanced evaluation, with the models queried for the LogOp and linear chains at a 50% probability each. Specifically, we tested the models on 400 samples. Mistral achieved 96% accuracy when QUERY is for the linear chain, and 70% accuracy when QUERY is for the OR chain (so they average above 70% accuracy). Similar trends hold for the other models.
B.2 Causal mediation analysis: further explanations
This subsection complements the causal mediation analysis methodologies we presented in $ in the main text. In particular, we illustrate how the interventions are done in the circuit discovery and verification processes, by using 2-layer attention head transformer
example for simplicity.
Figure o fillusttates the activation patching procedure in circuit discovery. We investigate how internal transformer components causally influence model output. In the specific example, we show how we would examine the causal influence of attention head (0,2)'s activations on the correct inference of the model.
Circuit verification, on the other hand, goes through a somewhat more complex process of interven- tions, as illustrated in Figure 7 Recall our main procedure (discussed in the main text).
1. We run the LLM on the original prompts, and cache the activations of the attention heads.
2. Now, we run the LLM on the corresponding altered prompts, however, we freeze all the attention heads' activations inside the model to their activations on the original prompts, except for those in the circuit which we wish to verify (i.e. only the attention heads in C are allowed to run normally). We record the (circuit-intervened) altered logit differences on the altered prompts.
3. We average the Σm= AC
altered logit differences across the samples, namely Aorig-all and check whether they approach the "maximal" altered logit differ-
ence, namely Ant
Remark B.2. As the reader can observe, we do not freeze the MLPs in our intervention experiments. We note that the MLPs do not move information between the residual streams at different token positions, as they only perform processing of whatever information present at the residual stream. Therefore, similar to Wang et al. (2023a), we consider the MLPs as part of the "direct" path between two attention heads, and allow information to flow freely through them, instead of freezing them and disrupting the information flow between attention heads.
Thiis specific term reflects, on average, how much the model favors outputting the answer token for the altered prompts over the original prompts after the circuit interventions.
*Recall that this term obtained by running the LLM on the altered prompts without any modification to its internal activations at all. This specific term reflects, on average, how much the (un- (un-intervened) model favors outputting the answer tokens for the altered prompts over those of the original prompts, when it is run on the altered prompts.
27

---

Altered activations
lngin Anssa) < An8s
[ogit[Aniiaa
Ans
Original activetions
Classifter
Classiffer
Causal influence /
Patching influnced activattons
(1.33
(1.23
(1.21
MLP1
MLP1
10.11
(0:2)
(0.1)
(0.2)
atered
MLPO
MLPO
MLPO
Altered Input
Original Input
Original Input
Figure Illustration of how activation patching is performed in the cireuit-discovery process (necessity-based patching), using a 2-layer, 2-head transformer as a simplified example here. An attention head denoted as h).
In this specific illustrated example, activation on the correct inference of the model. After caching the altered activations of (0,2) (shown on the left), we run the model on the original prompt and cache the activations (shown in the middle), then replace the original activation of head (0,2) by its altered activations, and let the rest of the layers be computed normally (shown on the right) they now operate out of distribution, and are colored in green.
studying the causal influence of attention head (0,2)'s
In this specific example, the intervened outputs logits which reflect "belief altering": that is, the probability for the answer token the original prompt now is lower than the answer token for the altered prompt. This indicates that head (0,2) has causal influence on the corrent inference of the model.
B.3 Finer details of activation patching
Activation patching experiments: metrics. We rely on a calibrated version of the logit-difference metric often adopted the literature for the QUERY-based activation patching experiments (aimed at keeping the score's 's magnitude between and 1). In particular, we compute the following metric for head (E, h) token position
EnE(Ny (orig-aal.. A A
(1)
Aatt A
orig
where Arrig
1 EnEIN logit( X orig.n) [B
logit( X orig.n) (Boriggn)) and Aett
N EnElNy Dogit(Xatta, (yaten) logit( X alt, [.rrig..] The closer to this score is, the stronger the model's s "belief" is altered; the closer to 0 it is, the closer the model's "belief" is to the original unaltered one.
Fach of our experiments are done 60 samples unless otherwise specified - repeat some experiments (especially the attention-head patching experiments) to ensure statistical significance when necessary.
28

---

Aftered
Original
Patshingg infuunncd
0
Soulh
orig Dele
Nulichcuttnn
Sufticlnncy attering
Cleaciter
Classifior
"celaet altering offect
Classifer
(1.20
Creait
(
Feeece every headiccitpin sacept
MLP1
MLP1
MLP1
MIPO
MLPD
Set of ateredinputts
Figure 7: Illustration of how activation patching is performed in the circuit- verificatio process (sufficiency- -baseed patching). We use 2-layer 2-head transformer as a simplified example here. We use (f, h) to denote an attention head.
In this specific illustrated example, we are verifying whether the circuit C = {(0,,) (1, consisting of the two attention heads sufficient for altering the "belief" of the model.
We first obtain Aatt. the average altered logit difference, by running the model on the altered prompts without interventions. We also run the model on the original prompts and cache the attention heads' activations (these "original" activations The naive "baseline" for sufficiency verification is the null circuit Caull = 9 (shown in the middle): freeze all the attention heads to their original activations when running the model the altered prompts. This null circuit, as shown in this example, barely alters the model's "belief" from the original, as A Cruit aalt ~ - -alts i.c. the model still strongly favors outputting the answer tokens for the original prompts over those of the altered prompts on average.
colored in blue in the figure).
In contrast, if we unfreeze the attention heads in the circuit C' when running the model (shown the right), we observe that the model's 's circuit-inerrvened logit difference approaches the "maximal" altered logit difference Aett- This indicates that the attention heads in C are sufficient for correctly manipulating the information flow (and processing the information) for reaching the right answer.
B.4
Reasoning circuits in Mistral-7B
B.4.1 Attention head group patching
Starting in this subsection I and ending at $B.7 we present and visualize the attention heads with the highest average intervenes logit differences, along with their standard deviations (error bars). Furthermore, we provide further causal evidence for sub-families of attention heads in the circuit.
We note that Prrupped-ueen Attention used by Mistral-7B adds subtlety the analysis of which attention heads have strong causal influence on the L.LM's correct output. (In Mistral-7B--O. 1, each attention layer has key and value activations, and 32 query activations. Therefore, heads (f, h x 4) to (e, x 4 + 3) share the same key and value activation.) Patching single head might not yield high logit difference, since other heads in the same group (which possibly perform a similar function) could overwhelm the patched head and maintain the model's previous "belief". Therefore, we also run a coarser- grained experiment which simultaneously patches the attention heads sharing the same key and value activations, shown in Figure b). This experiment reveals that heads belonging to the group (9, 24 - 27) also have high intervened logit difference. Combining with the observation that (9,25:26) have somewhat positive scores in the single-head patching experiments, and by examining these two head's s attention patterns (which shall be discussed detail in the immediate next subsection), we determine that they also should be included in the circuit.
29

---

(9.24.27)
0.4
(12.9)
(13,11)
10
(14,24/28)
Layer Index
(15.8)
(16,12:4)
1.5
0.0
16,00
20
!17.25)
0.2
(19,833116)
25
30
-066
10
20
30
10
20
30
(a) Single-head patching
(b) Head group patching
Figure 8: Attention head patching, highlighting the ones with the highest intervened logit difference; rraxiis the head index. (a) shows single-head patching results (same as the one shown in the main text, repeated here for the reader's convenience), and (b) shows coarser-grained head patching in groups. In (b), we only highlight the head groups that are not captured well by (a).
B.4.2 Attention patterns of QUERY-sensitive attention heads
In this subsection, we provide finer details on the attention patterns of the attention heads we discovered. Note that the attention weights percentage dividing the observed attention weight at a token position by the total amount of attention the head places in the relevant context, i.e. the portion of the prompt which excludes the 6 in-context examples.
present in this section calculated by
the average attention weight the queried-rule locating heads place the "conclusion" variable and the period "." immediately after the queried rule the QUERY token position (i.e. the query activation the heads come from the residual stream the QUERY token position)- (12,9) is exception this recording method, where only record its weight on the conclusion variables alone, and already observe very high weight on average. The heads (12,9), (14,24), (14,26), (9,25), (9,26) indeed place the majority of their attention the correct position consistently across the test samples. The reason for counting the period after the correct conclusion variable "correctly" locating the rule is that, it is known that L.MSs tend certain "register tokens" to record information in the preceding sentence.
Queried-rule locating heads.
Attention weight of the queried-rule locating heads
Attention weight
0.4
0.2
0.0
(12.9)
(9,25)
(9,26)
(14.24)
(14,26)
(Layer
Figure 9: Mistral 7B. Average attention weights the queried-rule locating heads, along with the standard deviations. The weights are calculated by dividing the actual attention weight placed on the correct "conclusion" variable of the rule and the period "." immediately after, by the total amount of attention placed in the relevant context (i.e. the prompt excluding the 6 in-context examples). Head (12,9) is an exception: we only record its attention right on the conclusion variable, and still observe 93.0 + 9.4% "correctly placed" attention on average.
30

---

We can observe that head (12,9) has the "cleanest" attention pattern out of the ones identified, placing on average 93.0 = 9.4% of attention on the correct conclusion variable alone. The more diluted attention patterns of the other heads likely contribute to their weaker intervened logit difference score shown in $3. the main text.
Queried-rule mover heads. Figure shows the attention weight of the queried-rule mover heads. While they do not place close 100% attention on the QUERY location consistently (when the query activation comes from the residual stream from token ".. right before the first answer token), the top-1 attention weight consistently falls on the QUERY position, and the second largest attention weight is much smaller. In particular, head (13, 11) places 54.2 + 12.5% attention on the QUERY position on average, while the second largest attention weight in the relevant context is 5.2 1.1% on average (around 10 times smaller, this ratio is computed per sample and then averaged).
Extra note about head (16,0): it does not primarily act like "mover" head, its attention statistics suggest that it processes an almost even mixture of information from the QUERY position and the "!" position. Therefore, while we present its statistics along with the other queried-rule mover heads here since it does allocate significant attention weight on the QUERY position on average, we do not list such in the circuit diagram of Figure Furthermore, do not include it part of the circuit C in our circuit verification experiments.
Attention statistics of the queried rule mover heads
0.7
Attention weight
0.6
Max attention weight other positions
Attention weight
0.5
0.4
0.3
0.2
0.1
0.0
(13, 11), QUERY (16,0), QUERY (16,0), :
(15,8), QUERY (Layer index, Head index)'s attention weight on token position
Figure 10: Mistral 7B. Average attention weights of the queried-rule mover heads, along with the standard deviations. The raw attention patterns are obtained at token position ":" (i.e. the query activation comes from the residual stream at the "." position). right before the first answer token, and the exact attention weight (indicated by the blue bars) is taken at the QUERY position; for head (16,0), we also obtain its attention weight at the 4." position, as we found that also allocates large amount of attention weight to this position in addition to the QUERY position. Note: for (15,8), we found that it only acts as "mover" head when the linear chain is being queried, so we are only reporting its attention weight statistics in this specific scenario; the other heads do not exhibit this interesting behavior, so we report those heads' statistics in all query scenarios.
Fact processing heads. Figure 11 below shows the attention weights of the fact processing heads; attention patterns are obtained the "." position, right before the first answer token, and we sum the attention weights in the Fact section (starting at the first fact assignment, ending on the last "." in this section of the prompt). is clear that they place significant attention on the Fact section of the relevant context. Additionally, across most samples, we find that these heads exhibit the tendency to assign lower amount attention on the facts with FALSE value assignments across most samples, and nontrivial portion of the samples, they tend to place greater attention weight the correct fact (this second ability is not consistent across all samples, however). Therefore, they do appear to perform some level of "processing" of the facts, instead of purely "moving" the facts to the "." position.
Decision heads. Figure 12 shows the attention weights of the decision heads samples where the model outputs the correct answer (therefore, about 70% of the samples). The attention patterns are obtained at the s.n position. We count the following token positions as the "correct" positions:
31

---

Attention statistics of the fact processing heads
0.8
Attention weight
0.6
0.2
0.0
116, 12)
(16,14)
(14,26)
(Layer index,
Figure 11: Mistral 7B. Average attention weights of the fact processing heads computed at the ":" token position (last position before the answer), along with the standard deviations. The weights are calculated by dividing the actual attention weight placed in the Fact section by the total amount of attention placed the relevant context (i.e. the part of the prompt excluding the in-context examples).
In the Rules section, we count the correct answer token and the token immediately following it as correct.
In the Facts section, we count the sentence of truth value assignment of the correct answer variable as correct (for example, "A true.").
Note: the only exception is head (19,8), where we only find its attention on exactly the correct tokens (not counting any other tokens in the context); we can observe that it still has the cleanest attention pattern for identifying the correct answer token.
Attention statistics of the decision heads
Attention weight answer tokens
Max attention weight other positions
0.7
Attention weight
0.6
0.5
0.4
0.2
0.0
(19, 8)
(19.9)
(19.16)
(17.25)
(Layer Head index)
Figure 12: Mistral 7B. Average attention weights of the decision heads, along with the standard deviations. The weights are calculated by dividing the actual attention weight placed the correct answer tokens by the total attention the model places in the relevant context.
An interesting side worth pointing out is that, (17,25) tends only concentrate its attention in the facts section, similar to the fact-processing heads. The reason which we do not classify it as decision head that, in addition finding that their attention
fact-processing head and instead patterns tend to concentrate on the correct fact, evidence presented in $B.7 suggest that they are not directly responsible for locating and moving the facts information the "." position, while the heads (1;11244) exhibit such tendency strongly.
B.5 Sufficiency tests for circuit verification
In $3.2 we presented sufficiency test of the circuit. Here, we elaborate further on the experimental procedures and finer details of the experiment.
32

---

The circuit which we perform verification on is the union of the four attention head families, C =
QRLH QRMH FPH DH, with
QRLH = Queried-Rule Locating Heads = {{9, 25; 26), (12, 9), (14, 24; 26) patched at
token position QUERY;
QRMH H = Queried-Rule Mover Heads = {(13, 22), (15, (1),8)) patched at the ":" posi-
tion (the last position of context);
FPH = Fact-Processing Heads = {(16, 12; 14), (14, ) patched at the ":" position;
DH Decision Heads {(19, 8; 16); (17, 25)) patched at the ":" position.
We obtain the following results.
C
Caull
C
- QRLH C C-QRMH C-FPH C
DH
A Rormal/ Anorma!
C
-1.0
0.98
-1.02
-0.99
-0.25
-0.89
Table 2: Mistral 7B. A Anormat/ Anormat, with different choices of Ct, Coust denotes the empty circuit, L.e the case where every attention head output is frozen to its counterfactual version. We abbreviate the attention head families, for example, DH = decision heads; C - DH = full circuit but with the decision heads removed.
An exception is that the queried-rule locating head (14, 24) is also patched at the ":" position, as we observed that it tends concentrate attention at the queried rule at this position: it does not locate the queried rule as consistently as it does at the QUERY position, however. We still chose to patch it this position as found that it tends improve the altered logit difference, indicating that either the model relies on this head to pass certain additional information about the queried rule to the "." position, or certain later parts of the circuit do rely on this head for queried-rule information. The exact function of this attention head remains part of our future study in the reasoning circuit of Mistral-7B. We likely need examine this head's role in other reasoning problems clearly understand what its role is at different token positions, and whether there is deeper meaning behind the fact that, their apparently redundant actions at different token positions all seem have causal influence on the model's ' inference.
Challenges of reasoning circuit sufficiency verifications. From what sufficiency of a reasoning circuit is a major open problem. Part of the root of the problem lies in what exactly counts as a circuit that is truly relevant to reasoning: attention heads and MLPs responsible for lower-level processing such as performing change of basis of the token representations, storing information at register tokens (such as the periods "* after sentences), and on, do not truly belong to a "reasoning" circuit in the narrow definition of the term. In our considerations, a "narrow" definition of a reasoning circuit one which is QUERY sensitive and has strong causal influence on the correct output of the model on the reasoning problems. The first condition of QUERY sensitivity is justified by noting that the QUERY lies at the root of the reasoning chain of HOUBBYY~Relevar Rule(s)-RReevvan circuit/internal processing the "QUERY", "Relevant Rule(s)" and "Relevant Fact(s)" underwent from token level to representation level (notice that the reasoning circuit we identified starts at layer 9: it is entirely possible for the token embeddings these important items to have undergone significant processing by the attention heads and MLPs in the lower layers). Simply setting the lower layers' embeddings to the zero vector, to their mean activations or some fixed embeddings which erase the instance-dependent information could completely break the circuit.
see, verifying the
-act(ss)=Decision)
We do not analyze through what
B.6 Queried-rule location interventions: analyzing the queried- rule locating heads
In this experiment, only swap the location of the linear rule with the LogOp rule in the Rule section of the question, while keeping everything else the same (including all the in-context examples). As an example, we alter the rules as follows. The prompts have the same answer.
(... in-context examples J..] Rules: A or B implies C. D implies E.
[... in-context examples ...]
altered prompt
Rules: D implies E. A or B implies C. Facts: true. false. D is true. Question: what is the truth value of C?
Facts: true. false. D) is true. Question: what is the truth value of C?
33

---

Increase in loss after intervention Average loss betore intervention 0.463
0.4
0.6
0.3
2.4
indox
intervention
Average intervention
0.0
Inceasee toss (velun)
22
0.4
-0.2
0.2
-0.3
01234567 01234567
-0.4
(12.2)
Key
(34.6)
(9.6)
(a) Key patching loss increase Left: QUERY lin chain; Right: QUERY OR chuin
(b) Average increase in loss Top: QUERY lin chan; Bottor: QUERY OR chain
Figure 13: Mistral 7B. Key activations patching results. In this experiment, we swap the location of the linear rule and the LogOp rule in the Rule section and keep everything else in the prompt the same; we patch the key activations of the attention heads the Rule section only. (a) visualizes average increase in the cross-entropy loss with respect to the true target (the true first token of the answer) for all key indices, and (b) shows average and standard deviation of top three key indices with the highest loss increase. Observe that these are the keys for the queried-rule locating heads (12,9), (14,24:26) and (9,25;26) identified $3.1
If the queried-rule locating heads (with heads (12.9), (14,25;26), (9,25:26) being the QUERY- sensitive representatives) indeed perform their functions as we described, then when we run the model on the clean prompts. patching the altered key activations these heads (within the Rules section) should cause "negative" change to the model's output, since it will cause these heads to mistake the queried-rule location the altered prompt be the right one, consequently storing the wrong rule information at the QUERY position. In particular, the model's i cross- cross-entropy loss with respect to the original target should increase. This indeed what we observe.
The average increase in cross-entropy loss exhibit a trend which corroborate the hypothesis above, shown in Figure [3] While the average cross-entropy loss the original samples is 0.463, patching (12,9), (14,24:26) and (9,25:26)'s keys (with corresponding key indices (12,2), (14,6) and (9,6) in the Rule section.
Patching the other QUERY- sensitive attention heads' keys in the Rule section, in contrast, show significantly smaller influence on the loss average, telling that their responsibilities much less involved with directly finding or locating the queried rule via attention.
Note: this set of experiments was run on 200 samples instead of 60, since we noticed that the standard deviation of some of the attention heads' loss increase is large.
Remark B.3. While attention heads with key index (15,5) (i.c. heads (15, 20-23)) did not exhibit
nontrivial sensitivity QUERY-based patching (discussed Section [. in the main text), patching this key activation does result in a nontrivial increase in loss. Examining the attention heads belonging this group, we find that they indeed also perform the function of locating the queried rule similar to head (12,9). We find them to be less accurate and place less attention on the exact rule being queried on average, however: this weaker "queried-rule locating ability" likely contributed their low scores in the QUERY-based patching experiments presented in the main text.
B.7 Facts interventions: analyzing the fact-processing and decision heads
In this section, we aim to provide further validating evidence for the fact-processing heads and the decision heads. We experiment with flipping the truth value assignment for the OR chain while
34

---

0.2
0.1
14
15
18
18
15
19
22
22
23
23
75
26
76
-0.2
29
29
30
30
-0.3
2
3
Hoad Index
Head Index
Head Index
(a) Key
(b) Value
(c) Query
Figure 14: Mistral 7B. Key, value and query activation patching in the Facts section, with the metric being the calibrated intervened logit difference. The truth value assignments for the OR chain is flipped (while keeping everything else the prompt the same), and the OR chain is always queried. Observe that only the key activations at index (16,3) obtain high intervened logit difference score of approximately 0.34 (this key index corresponds the attention heads (16, 12 15)). Also observe that the value and query activations in the facts section do not exhibit strong causal influence on the correct inference of the model.
keeping everything else the same in the prompt (we always query for the OR chain in this experiment). As an example, we alter the truth values as follows.
[... in-context examples ...]
(... in-context examples ...]
Rules: A or B implics C. D implics E. Facts: A is true. B is false. 1) is true. what the truth value of C?
attered
Rules: A or B implies C. D implies E. Facts: A is false. B is true. D is true. Question: what is the truth value of C?
prompt
Question:
In this example, the variable that the answer depends on flips from to B (since after altering, only true). The (calibrated) intervened logit difference is still good choice in this experiment, therefore we still rely on it to determine the causal influence of attention heads on the model's inference, just like in the QUERY-basee patching experiments.
the fact-p procescccssnng heads (with (1;,12;14) being the QUERY-sensitive representatives) indeed perform their function as described (moving and performing some preliminary processing of the facts as described before), then patching the altered key activations the Facts section of the problem's context would cause these attention heads to obtain nontrivial intervened logit difference, i.e. it would help bending the model's "belief" in what the facts are (especially the TRUE assignments in the facts section), thus pushing the model to flip its first answer token. This is indeed what we observe. In Figure [4] we see that only the key activations with index (16,3) (corresponding to heads (16, 12 15)) obtain much higher score than every other key index, yielding evidence that only the heads with key index (16,3) rely on the facts (especially the truth value assignments) for answer. Moreover, notice that patching the key activations of the decision heads does not yield a high logit difference on average, telling us that the decision heads do not directly rely on the truth value assignment of the variables for inference (we wish to emphasize again that, the positions of the variables in the Facts section are not altered, only the truth value assignments for the two variables of the OR chain are flipped).
Finally, for additional insights on the decision heads (19,8;9;16) and (17,25), we find that by patching the query activations of these decision heads at the "." position yields nontrivial intervened logit
35

---

0.2
13
0.0
17
17
18
19
22
22
23
25
76
76
27
27
-0.2
28
30
5
15 Head Index
20
25
31
31
(a) Key
(b) Value
(c) Query
Figure Mistral 7B. Key, value and query activation patching at the ":" position (last token position in the context, right before the answer token), with the metric being the calibrated intervened logit difference. The truth value assignments the prompt the same), and the OR chain is always queried. Observe that only the query activations at index (19,8) obtain a high intervened logit difference score of approximately 0.28; the other decision heads (19,9;16) and (17,25) also obtain nontrivial scores when their queries are patched. Also observe that the key and value activations at the ... position do not exhibit strong causal influence on the correct inference of the model when only flip the truth value assignments for the OR chain.
the OR chain flipped (while keeping everything else in
difference, as shown in Figure c)) ((19,) has an especially high score of about 0.27). In other words, the query activation at the ":" position (which should contain information for flipping the answer from one variable the OR chain the other, as gathered by the fact-processing heads) being fed into the decision heads indeed have causal influence their "decision" making. Moreover, patching the value activation of these heads at "." does not yield nontrivial logit difference, further suggesting that it is their attention patterns (dictated by the query information fed into these heads) which influence the model's 's output logits.
36

---

B.8 Reasoning circuit in Gemma-2-9B
In this section, we present an analysis of the reasoning circuit Gemma-2-99 in solving the same reasoning problem which Mistral-7B was examined on from before. We find that the discovered attention heads' attention patterns inside Gemma-2-98 bear surprising resemblance to Mistral-7B's according to their highly specialized attention patterns, they can also be categorized into the four families of attention heads which Mistral-7B employs to solve the problem, namely the queried-rule locating heads, queried-rule mover heads, fact-processing heads, and decision heads. While it is too early to draw precise conclusions on how similar the two eircuits in the two LLMs truly are, the preliminary evidence suggests that the reasoning circuit we found in this work potentially has some degree of universality.
B.8.1 QUERY- -based attention head activation patching
We perform activation patching of the attention head output of Gemma-22 2-99, by flipping the QUERY in the prompt pairs. This the same procedure we used to discover the attention head circuit for Mistral-7B as discussed in A and B.2 We highlight the attention heads with the strongest causal influence on the model's (correct) inference Figure T6
0
0.4
5
5
0.2
(20,7)
(19.1))
15
15
(21.7)
15
Val. (20.7)
20
20
20
0.0
(24.5)
25
25
25
(25.7)
(26.00
(26,12)
Val. (24.5)
-0.2
(20.2)
(2116)
(30.9)
35
-0.4
40
40
5
(a) Query
(b) Head output
(c) Value
Figure 16: Gemma 2 9B. QUERY-basee activation patching results of Gemma-2-90 2-99,, with sub- component patching on the query and value activations. We highlight the attention heads with the highest calibrated intervened logit difference.
B.8.2 Attention patterns of QUERY-sensitive attention heads in Gemma-2-9B
Queried-rule locating heads. The queried rule locating heads inside Gemma-2-9B, namely {(19, 11), (21, 7), (22,5), (23, 12)}. are very similar their attention patterns to those in Mistral-7B. At the QUERY position, their attention concentrates the ." which follows. Interestingly, heads (21.7), (22,5) and (23,12) also tend to place some attention on the "implies" token of the queried rule. Another intriguing difference they exhibit is redundant behavior: these attention heads are often observed to have almost exactly the same attention pattern at the and "Answer" token positions following the QUERY token. We visualize their attention statistics at the QUERY position in Figure 17
the conclusion token of the queried rule, and
Queried-rule mover heads. When the query activations of the queried-rule mover heads {(20, 7), (23, 6), (24, 15)} come from the ".* residual stream, they have fixed attention patterns
which focus large portion of their attention weights on the QUERY token and two token positions
37

---

Attention statistics of the queried-rule locating heads
1.0
0.8
Attention weight
0.2
0.0
(19, 11)
(21.7)
(22.5)
(23,12)
(Layer
index)
Figure 17: Gemma 9B. Average attention weights of the queried-rule locating heads in Gemma-2- 9B, along with the standard deviations. The attention pattern is obtained the QUERY position (i.e. query activation of the attention head is from the residual stream at the QUERY token position). We record the attention weight on the queried rule.
following it, namely the " and "Answer" token. Their attention weights are slightly more diffuse compared to their counterparts in Mistral-7B, likely due to queried-rule locating heads performing similar functions at the and "Answer" positions. Furthermore, shown in Figure Ioje), note that these attention heads are the only ones where patching their value activations results a large intervened logit difference, further suggesting their role in performing fixed "moving" action. We record their attention weights in Figure 18
Attention statistics the queried-rul -rule mover heads
Attention weight on QUERY
weight positions
Attention welg
0.4
0.3
0.2
(Layer index. Head indexx
Figure 18: Gemma 2 9B. Average attention weights of the queried-rule mover heads in Gemma-2-9B, along with the standard deviations. The attention pattern is obtained at the ":" position, and we sum the attention weights at the QUERY position and the "* and "Answer" token positions which immediately follow QUERY.
Fact- pact-processing heads. The fact-processing heads {(24,5) 5), (25, 7), (26, 0), (26, 12) attention patterns at the "." position tend place larger weight on the correct fact for the answer, similar to the fuct-processing heads in Mistral-7B. An interesting difference does exist though: heads (24.5) and (25,7) also tend to place nontrivial amount of weight on the QUERY and 5." token positions, indicating that these heads are relying on some form mixture information present at those positions for processing. While it is reasonable to hypothesize that these heads are likely relying the queried-rule information present in the QUERY and 5:" residual streams, we have not confirmed this hypothesis in our current experiments. We visualize the statistics of these heads in Figure 19
Decision heads. The decision heads {(28, 12), (30, 9)) attention pattern are obtained at the "." position. They bear strong resemblance to those in Mistral-7B: they place significant attention on the correct answer token (in both the rules and facts sections, same as Mistral-7B's decision heads), and little attention weight anywhere else. This is shown in Figure 20
38

---

Attention statistics of the fact processing heads
Attention weight fact
0.7
Attention weight QUERY
attention weight positions
Attention weight
0.5
0.4
0.2
0.1
(24,5)
(25,7) (Layer index, Head index)
(26,0)
(26,12)
Figure 19: Gemma 2 9B. Average attention weights the fact-processing heads in Gemma-2-9BB, along with the standard deviations. The attention pattern is obtained at the "." position. We record the attention weights at the correct fact, QUERY and ":" positions, and the maximum weight on any other position.
Attention statistics the decision heads
1.0
Attention weight on answer Max attention weight other positions
Attention weight
0.6
0.4
0.2
(28,12)
(30,91
(Layer index, Head index)
Figure 20: Gemma 9B. Average attention weights of the decision in Gemma-2-9H. along with the standard deviations. The attention pattern obtained at the ".." position. We record the attention weights at the correct answer token positions.
B.8.3 Circuit verification of Gemma-2-9B
In this sub-section, we present two versions the circuit we performed verifications on. Version 1 was older and coarse-grained due to limited resources available at the time. Version 2 is finer- grained and performed on a larger number of samples. We leave the older version here to ensure transparency to circuit analysis process and show the complexities of circuit verifications.
Version 1 of Circuit Verification. This is the old version the Gemma-2-90 circuit originally found in the first version of our work, which we observed to already have a high faithfulness score.
The cireuit which perform verification is the union of the four attention head families, C QRLH U QRMH FPH U DH, with
QRLH Queried-Ruil Rule Locating Heads = {(19, 11), (21, (22, 5), (23, 12)]: QRMH Queried-Reee Rule Mover Heads {(20, 7), (23, 6), (24, 15)):
39

---

FPH = Fact-Processing Heads = {(24, 5), (25, (26. 0). (26, 12));
DH = Decision Heads = {(28, 12), (30, 9))
Remark.
This version of circuit verification is performed in coarse-grained manner, we patch the output of the attention heads in C from the QUERY position to the ":" position, instead of clearly distinguishing the token positions which each head primarily focuses on.
Ct
Cault
C
C --QLLH T-QRMH C-FPH C-DH DH
C orig-ratii Datt
-1.0
0.94
-0.97
-0.40
0.17
-1.11
Table 3: Gemma 2 9B. A -aig Datt for Gemma-2-90 2-99,, with different choices of Ct. Crull denotes the empty circuit, i.e. the case where no intervention performed. We abbreviate the attention head families, for example, DH decision heads; C DH full circuit but with the decision heads removed.
We find that by patching all 13 attention heads in C. A oott is about 94% of the "maximal" average logit difference Agtt on the altered samples. Moreover, removing any one of the four families of attention heads from C in the circuit interventions renders the "belief altering" effect of the intervention almost trivial.
Version 2 of Circuit. With more resources available, we conducted a finer-grained test of the circuit, covering a larger number of attention heads which have nontrivial causal scores in our circuit search process, and restricted the token positions which the families are allowed to operate on, similar to our Mistral experiments before. We also run the verification test on a larger number of samples (240 samples, instead of 80). The following updates are made.
1.
Token position restrictions. For the four families of heads, we now restrict the token positions on which they are allowed to operate normally be (i.e. the token positions on which they are unfrozen):
QUERY for queried-rule locating heads;
QUERY and '." for the mover heads, since they have diffuse attention on these positions.
The last token position 5." for the fact-processing and decision heads, since we observed their primary processing for writing down the answer token to take place at the last token position.
2. Additional QRLHS. We add heads (17,9), (21,0) to the QRLH family. (17,9) did not exhibit strong causal score in the initial search when we set the mediator to be the output of the attention head, but surfaced with high causal score in our rule-location-wwer experiment (where the key activation in the Rules section is the mediator), and its attention statistics strongly suggest that it is locating the queried rule. (21,0) included for completeness this time, since it has moderately high causal score in the circuit search, and its attention pattern resembles those of the QRLH family closely.
3. The "post-processing" heads. With these token position restrictions, the 4 families of heads alone recover 86% of the logit difference - high score, but still lower than before by nontrivial margin. We find that for two more attention heads deeper in the model which have high causal scores in the search process. (27,15) and (33,4), un-freezing them at the QUERY and ".." positions (along with the rest of the circuit as we described above) helps us recover 95% of the un intervened average logit difference.
We term these heads the "post-processing" heads. Their attention patterns are somewhat similar to the mover heads" consistently heavy attention at the last token position "." with small amount of attention on QUERY - yet they are deeper in the model, and patching their value activations alone does not yield high causal scores. This suggests that they are not merely moving information, but boosting the answer signal given by the core circuit (from the 4 families of heads). The exact role of these two post-processing heads remains mysterious, and requires future investigations.
These two heads were not included in the Version I circuit as they have less interpretable functional roles in the circuit, and lower causal scores than the dominant heads in the four core circuit families (especially head (33,4)). Another reason to include them now
40

---

is that they seem to have counterparts in the larger Gemma-2-27B model: some of the heads in the deeper layers in that model also exhibit the tendency to only focus on the last token position, and have nontrivial causal scores.
Further remarks.
1. To see the nuances with the sufficiency tests of these circuits more clearly, we encourage the interested reader to examine, and ideally work with our open-sourced Jupyter notebook "L.LM Analysis Part 1 Circuit search, interpretation, and verification. ipynb" in our GitHub repository, which covers the full workflow from environment setup to the details of circuit tests. By ablating different combinations of the circuit components and token positions (and with/without the post- processing heads), the reader should be able to gain a more concrete understanding of the reasoning circuits.
We find it surprising that two LLMs (Mistral-7B and Gemma-2-9B) which are trained with different procedures and data ended up relying on attention- head circuits which bear strong resemblance to each other's. In the current literature, it unclear how one can rigorously quantify the similarity of two nontrivial circuits inside different LLMs, however, this subsection does yield preliminary evidence that, the reasoning circuit we discover potentially has some degree of universality to it, and is likely an emergent trait of LLMs.
B.8.4 Analysis of queried-rule locating heads
Similar to the Mistral-7B experiments in verifying the queried-rule locating heads, we only swap the location of the linear rule with the LogOp rule in the Rule section of the question, while keeping everything else the same (including all the in-context examples). As an example, we alter "Rules: or B implies C. D implies E." to "Rules: D implies E. A or B implies C." while keeping everything else the same. The two prompts have the same answer. As the results have already been visualized in the main text, we do not repeat the results here.
The basic intuition is that, if the queried-rule locating heads indeed perform their functions described, then when we run the model on the clean prompts, patching in the altered key activations at these heads (within the Rules section) should cause "negative" change the model's output, since it will cause these heads to mistake the queried-rule location in the altered prompt to be the right one, consequently storing the wrong rule information at the QUERY position. In particular, the model's logit difference between the two possible answers (to the LogOp chain and linear chain) should decrease. Indeed, that what we observe.
B.8.5 Analysis of fact-processing heads and decision heads
Similar to the Mistral-7B experiments, in this section, we aim to provide further validating evidence for the fact-processing heads and the decision heads. We experiment with flipping the truth value assignment for the OR chain while keeping everything else the same in the prompt (we always query for the OR chain in this experiment). As an examplc, we alter "Rules: A or implies C. D implies E, Facts: A is true. B is false. D is true. Query: please state the truth value of C." to "Rules: A or B implies C. D implies E. Facts: A is false. C.". In this example, the answer is flipped from A to B. The (calibrated) intervened logit difference between the two variables in the OR chain is still a good choice in this experiment.
true. D is true. Query: please state the truth value of
If the fact-processing heads indeed perform their function as described (selecting the correct fact to invoke, and moving such facts to the ".* position for next-word prediction). then patching the altered key activations in the Facts section of the problem's 's context would cause these attention heads to obtain a nontrivial intervened logit difference, i.c. it would help in bending the model's 's "belief" in what the facts are (especially the TRUE assignments in the facts section), thus pushing the model to flip its first answer token. This is indeed what we observe. In Figure 2T we see that only the key activations with index (23,6), (25,3), (26.6) and (28,4) (corresponding to heads (23,13), (25.6), (26,12) and (28,8) obtain a higher score than every other key index, yielding evidence that they are sensitive to the truth values assignments. Interestingly, head (23,13) did not exhibit strong causal influence in our CMA experiment in the main text, thus was not included in the circuit. We suspect that this is due to its inconsistent ability in locating the correct fact: we find that when QUERY is for the linear chain, this head tends to be correct, and allocates a large amount of attention to the sentence of the correct fact, yet, when QUERY is for the OR chain, its performance is inconsistent.
41

---

Moreover, by patching the attention head output and after the QUERY token (up to ":"), we gain understanding which heads output important truth- value-sensitive information for the model's inference. We find that the fact-processing heads (25,7), (26,12), (28, and (30,9) indeed have large intervened logit differences. Interestingly, head (27.15) also has large score: examining its attention pattern, we find that it resembles a mover head, focusing attention on "Answer" and ":" positions. We suspect that this head is moving truth-value-sensitiv information sent out by the shallower layers (e.g. from the heads (25,7), (26,12)). Due to resource constraints, we were not able to verify this hypthesis.
15
15
15
16
16
16
17
17
17
18
18
0.4
18
0.6
23
23
0.2
23
24
24
24
25
25
25
0.2
30
30
-0.2
32
32
0.2
-0.6
38
38
-0.4
38
19
39
39
40
40
40
7.5 Heed index
12.5 15.0
Heed Index
Head Index
(a) Key
(b) Value
(c) Output, on and after QUERY
Figure 21: Gemma 2 27B. Key, value and output intervention experiment results. We note that the key and value activations are intervened in the Facts section, while the output intervention is done token positions on and after QUERY. The former two helps us understand which attention heads are muuh-valuc-sensitte and rely truth value information for inference. The output intervention helps us know which attention heads' output are dependent information for the model's 's reasoning actions.
and send out seth---alu--sssggmenn
B.8.6 Rule invocation: circuit reuse in proof writing
We localize the attention heads which invoke the correct rule for the argument. In particular, we still generate the counterfactual prompt by flipping the query token: clearly, this not only flips first answer token (the correct fact invocation), but also the "first rule token". Consider the normal-counteffactual pairs:
[.. in-context examples ]
[... in-context examples ...]
Rules: A or B implies C. D implies E. Facts: A is true. B is false. D is true. Question: what is the truth value of C? Answer: A true.
Rules: A or implies C. D implies E. Facts: A is true. B is false. D is true. Question: what is the truth value of E? prompt
altered
Answer: D is true.
What follows would be the correct rule invocation, For the normal prompt, would be "A or B implies C", and for the counterfactual prompt, it would be "D implies E". We perform patching again the first answer token position to localize components responsible for retrieving the correct rule to invoke; we show the results in Figure 22 We intervene on the token positions starting at QUERY, and ending at the final token position
Note that we perform patching accounting for GQA for Mistral-7B we know from before that this coarser-grained patching actually helps us locate more attention heads which perform the role of rule-locating (recall Figure 8). As we see, the models reuse the aule-locator and decision heads.
42

---

10
11
12
13
Queried-rule locator heads
14
15
16
Queried-rule locator heads
11
10
19
20
14
15
21
22
g
23
18
3 22
28
23
29
Decision heads
33
30
31
32
-0.2
28
27
26
-0.3
34
Decision heads
-0.4
20
25
30
-04
37
36
39
40
-066
41
10
15
Head Incex
(a) Gemma-2--00
(b) Mistral-7B
Figure 22: We observe subcireuit reuse when the Gemma-2-99B and Mistral-7B models invoke the correct rule.
Remark. With finer examination of the rule-locator heads in Gemma-2-9B, we found that the rule- locator heads' activations causally implicate the answer both the QUERY token position and the first token of fact invocation (right after "Answer:"): indeed, both positions store information about the queried rule (reflected in their attention patterns). The decision heads, on the other hand, causally implicate the answer primarily at the last token position, right before invoking the rule their precise functionality is more similar to that for fact invocation.
B.8.7 Prompt format ablation
In this subsection, aim understand how robust discovered subcircuits (the 4 families of heads) are to prompt format changes. In particular, we switch the Facts and Rules section of our problems. Our problem pairs now have the following form:
(.. in-context examples J
(... in-context examples ...]
Facts: true. is false. D is true. A or B implies C. D implies E. Question: what the truth value of C?
Facts: A true. false. D is true. Rules: A or B implies C. D implies E. Question: what the truth value of E? Answer: D is true.
altered
Rules:
prompt
Answer: is true.
We show the CMA results, along with representative attention patterns of the surfaced attention heads. Please refer Figure 23 for detailed visualization and explanations of our observations.
B.9 Gemma-2-27B reasoning circuit
In this section, we analyze Gemma-2-27B reasoning circuit. We first show the CMA result below, again relying on QUERY-based activation patching.
Additionally, we caution the reader that the experimental study in this subsection less exhaustive in nature compared to our study of Mistral-7B and Gemma-2-9B due to limitations in computation budget.
B.9.1 Attention head analysis
Most of the attention heads' attention patterns in this model's reasoning eircuit (for writing down the first answer token) resemble those in Gemma-2-9B. therefore omit another set of visualizations
43

---

0.4
0.1
0.0
Querled rule locating
28
25
-00
Moving
30
Relevent premise
-03
locating Focts agpregating
10 22131415
Head IOoe
3
(Best viewed when zoomed in) CMA results on logic problems presented in the format "Facts: Rules: Question: ...", along with representative attention patterns of the discovered attention heads. Overall, the attention heads which have strong IEs remain almost the same as before. However, some of their functional roles changed. We elaborate on what stayed the same and what changed below.
Figure 23:
First, the majority of the queried-rule locating and decision heads remain the same as before (with head (34,12) being a new hybrid decision head + fact-procsssing head which was not present in circuit before). In other words, the "starting" and "ending" sub-circuits in the model remain virtually unchanged. However, the "middle-layer heads", namely the fact paot-ppocessing and mover heads experienced some changes. In particular, significant portion of these heads shifted to locating the relevant premise variables and aggregating relevant facts, instead of only performing moving actions only focusing on the correct fact-sentence.
showing almost identical results to the 9B subsection from before (besides different layer-head indices). We focus on attention heads which exhibit significantly new behavior which had not been observed in the smaller models. This includes two intriguing points mentioned in the main text, (1) analyzing the logical-operator heads, and (2) showing that a portion of the fact-processing heads exhibit strong direct effect on the model's logits.
Logical-operator heads. We discuss how we localize these attention heads, and then show examples of their attention patterns: find these heads" attention pattern not so easy to capture by simple bar plots showing summarized statistics.
First, to localize these attention heads, we perform the following set of experiments. We set the following restrictions to the problem context:
We always query for the oogical-operatoe (LogOp) chain
We always set the truth value assignments for the two premise variables of the LogOp chain to be ["true", "false"]. We do not allow ["true", "true"] or ["false". "false"] in this analysis. The reason is that for both the situations of LogOp=AND and LogOp=OR, there is a unique first answer token (i.e. unique fact that should be invoked in the minimal proof).
Given a normal prompt such as "Rules: A or B implies C. D implies E. Facts: A is true. false. D true. Query: please state the truth value of C.", we generate the counterfactual prompt "Rules: A and B implies C. D implies E. Facts: is true. B is false. D is true. Query: please state the truth value of C." This can also be done the other way around of course.
We then hold the attention head output on or after QUERY as the mediator, and search for attention heads with strong indirect effects. In this setting, measure the logit different between the two
44

---

possible answer tokens, namely the two premise variables in the LogOp chain. We show the results Figure 24
While a significant portion of the attention heads with nontrivial indirect effects are mover and decision heads, find that heads (23,13), (23,31) place significant attention not only the rule being queried, but also specifically on the logical operator of the queried rule, whenever that rule has logical operator. Figure 25 contrasts their behavior on samples querying for the LogOp chain versus linear chain.
14
0.3
15
16
17
18
19
0.2
20
21
22
23
24
0.1
25
26
Layer Index
27
29
0.0
30
31
32
33
34
-0.1
35
36
37
38
39
-0.2
40
42
43
-0.3
5
10
15
20
25
30
Head Index
Figure 24: Gemma 2 27B. Output intervention experiment results of the logicalloperraoo--fiiiing experiment. We note that the outputs intervened and after QUERY.
Mechanical differences in Gemma-2-9B and Gemma-2-27B: -ectssnnis We illustrate in the Figure 26
45

---

0USRY
OU/OFT
Log5p
Logtp
135
processing Logical-operator
Sugical-operate
heeds
3.00
6.0
Foctior00es5ing
-005
(a) Path patch to decision heads
(b) Path patch to (23,13:31)
Figure 25: Please zoom to view the detailed path patching results and attention patterns. (a) shows the query-based path patching result to the decision heads of Gemma-2-27B, and (b) shows the query-based path patching results to the heads (23, 13;311, which we found to place attention not only the right fact(s), but also the logical operator word "and" or "or". Furthermore, we show the typical attention patterns of the logical-operator related heads on top. The path patching results are done by flipping the logical operator, as detailed in text.
The two attention heads (23,13) and (23,31) 1) are sensitive to the change in the logical operator, and exhibit the strongest direct effects the decision heads. For the four rows showing the example attention patterns of these two heads, the top two rows are the attention patterns of the attention heads obtained on samples querying for the linear chain, the bottom for the LogOp chain. Each row is obtained from the same sample. Observe that the bottom two rows, the two heads consistently place attention the word "and" and "or" in addition placing attention on the queried rule.
rows from samples querying
We also show typical attention patterns of (20,16) and (22,13), which have strong direct effects (23,13;3) 13:313 and also place significant attention on the logical operator. Moreover, note that they are not very precise fact-processing heads, even though they do place attention on certain facts: they do not place attention on the correct fact consistently. We hypothesize that these heads' role has more to do with informing later attention heads (particularly (23,13:31)) the logical operator present this problem instance.
C The reasoning circuit in small transformer
In this section, study how small GPT-2-like transformers, trained solely the logic problem, approach and solve it. While there are many parts of the answer of the transformer which can lead to interesting observations, in this work, primarily focus the following questions:
1. How does the transformer mentally process the context and plan its answer before writing down any token? In particular, how does it use "mental notes" to predict the crucial first token?
2. How does the transformer determine the truth value of the query at the end?
We pay particular attention to the first question, because as noted in L the first answer token reveals the most about how the transformer mentally processes all the context information without any access to chain of thought (CoT). We delay the less interesting answer of question to the Appendix due to space limitations.
46

---

0.20
20
Logical-operator or fact-processing heads
0.4
0.15
23
24
26
0.10
27
15
28
Layer Index
30
0.05
20
0.0
Decision heads
Decision heads
0
34
25
35
Movinglpast-poocessing
38
30
39
35
-0.4
42
43
0..5
45
10
20
25
-0.20
10 Head Index
Head Index
(a) Gemma-2 2-9B, direct effects on logits
(b) Gemma- 2-27B, direct effects on logits
Figure 26: Direct effects of attention heads on the model logits for Gemma-2-4 and Gemma-2-27B. The latter appears to have more parallel processing components than the 9B model. More specifically, for the 9B model, only the decision heads exhibit strong direct effects on the logits, the earlier ones (including fact-processing heads) do not. In contrast, the 27B has logical-operator and fact-processing heads exhibiting strong indirect effects on the logits, in addition to actual decision heads. Rather interestingly. if we recall the results in the previous Figure 25 we know that the logical-operator and fuct-processing heads also have direct effects on the decision heads. This tells us that these intermediate heads are "connected" to both the decision heads, and also have direct influence on the logits. This is not observed the smaller 9B model!
C.1 Learner: decoder-only attention-only transformer
In this section, we study decoder-only attention-only transformers, closely resembling the form of GPT-2 (Radford al. 2019a). We train these models exclusively on the synthetic logic problem. The LogOp chain is queried 80% of the time, while the linear chain is queried 20% of the time during training. Details of the model architecture are provided Appendix D.2
Architecture choice for mechanistic analysis. We select a 3-layer 3-head transformer to initiate our analysis since it is the smallest transformer that achieve 100% test accuracy; also show the accuracies of several candidate model sizes in Figure 28 in Appendix[D] for more evidence. Note that model's 's answer of the correct answer. Please refer to Appendix D.2 for an illustration of the model components.
problem is considered accurate only if every token in its answer matches that
C.2 Mechanism analysis
The model approximately follows the strategy below to predict the first answer token:
1. (Linear vs. LogOp chain) At the QUERY position, the layer-2 attention block sends out special "routing" signal to the layer-3 attention block, which informs the latter whether the chain being queried is the linear or The third layer then acts accordingly.
2. (Linear chain queried) If QUERY is for the linear chain, the third attention block focuses almost 100% of its attention weights on the QUERY position, that is. it serves a simple "message passing" role: indeed, layer-2 residual stream QUERY position already has the correct (and linearly decodable) answer in this case.
3. (LogOp chain queried) The third attention block serves more complex purpose when the LogOp chain is queried. particular, the first two layers construct partial answer, followed by the third layer refining it to the correct one.
47

---

We illustrate the overall reasoning strategy and core evidence for it in Figure 30 in Appendix D.3
C.2.1 Linear or LogOp chain: routing signal at the QUERY position
The QUERY token is likely the most important token in the context for the model: it determines whether the linear chain is being queried, and significantly influences the behavior of the third attention block. The transformer makes use this token in its answer in an intriguing way.
Routing direction at QUERY. There exists "routing" direction hrouee present in the embedding generated by the layer-2 attention block, satisfying the following properties:
1. on [X hroute is present in the embedding when the linear chain is queried, and 0r2( X hroute is present when the LogOp chain is queried, where the ax(X)'s
sample dependent, and satisfy the property that on (X) > 0, and (=2 (X) < 0.
2. The "sign" of the Proute signal determines the "mode" which layer-3 attention operates in at the ANSWER position. When a sufficiently "positive" frrouee is present, layer-3 attention as if QUERY is for the linear chain by placing significant attention weight the QUERY position. A sufficiently "negative" Proute causes layer-3 to behave as if the input is the LogOp chain: the model focuses attention on the rules and fact sections, and in fact outputs the correct first token of the LogOp chain!
We discuss empirical evidence below support and elaborate the above mechanism.
Cosine similanty matrix between embboddings
1.0
25
0.8
50
Sample Index
75
0.4
200
125
150
0.0
175
-0.2
150
175
Sample indox
Figure 27: Small transformer (trained from scratch). Cosine similarity matrix between output embeddings from layer-2 attention block. Samples 0 to 99 query for the linear chain, samples 100 to 199 query for the LogOp chain. Observe the in-group clustering in angle (top left and bottom right), and the negative cross-group cosine similarity (top right and bottom left).
Evidence la: chain-type disentanglement at QUERY. We first observe that, at the QUERY position, the layer-2 attention block's output exhibits disentanglement its output direction depending on whether the linear or LogOp chain is being queried, as illustrated in Figure 27
To generate Figure 27 we constructed 200 samples, with the first half querying the linear chain and the second half querying the LogOp chain. We then extracted the layer-2 self-attention block output the QUERY position for each sample, and calculated the pairwise cosine similarity between these outputs.
Evidence 1b: distinct layer-3 attention behavior w.r.f. chain type. When the linear chain is queried, the layer-3 attention heads predominantly focus on the QUERY position, with over 90% of their attention weights on the QUERY position average (based on 1k test samples). In contrast, when the LogOp chain is queried, less than 5% of layer-3 attention is on the QUERY on average. Instead, attention shifts to the Rules and Facts sections of the context, shown in Figure 31 in Appendix D.4
Observations and 1b suggest that given a chain type (linear or LogOp), certain direction(s) in the layer-2 embedding significantly influences the behavior of the third attention block in the
48

---

aforementioned manner. We confirm the existence and role of this special direction and reveal more intriguing details below.
Evidence Ie: computing hroute. and proving its role with interventions. To erase the instance- dependent information, average the output of the second attention block over 1K samples where QUERY is for the linear chain. We denote this estimated average as hroute which effectively preserves the sample-invariant signal. To test the influence of hroutes we investigate its impact on the model's 's reasoning process, and we observe two intriguing properties:
1. (Linear->LogOp intervention) We generate 500 test samples where QUERY is for the linear chain. Subtracting the embedding ftyoute from the second attention block's ' output causes the model consistently predict the correct first token for the LogOp chain on the test samples. In other words, the "mode" in which the model reasons is flipped from "linear" to "LogOp".
2. (LogOO-~lineat intervention) We generate 500 test samples where QUERY for the LogOp chain. Adding froote to the second attention block's s output causes the three attention heads in layer to focus on the QUERY position: greater than 95% of the attention weights are on this position averaged over the test samples. In this case, however, the model does not output the correct starting node for the linear chain on more than 90% of the test samples.
It follows that hroute indeed exists, and the "sign" of it determines the attention patterns in layer 3 (and the overall network's output!) in the aforementioned manner.
C.2.2 Answer for the linear and LogOp chain
Linear chain: answer at layer-2 residual stream at QUERY position. At this point, is clear to us that, when QUERY is for the linear chain, the third layer mainly serves simple "message passing" role: it passes the information in the layer-2 residual stream at the QUERY position the ANSWER position. One natural question arises: does the input to the third layer truly contain the information to determine the first token of the answer, namely the starting node of the linear chain? The answer is yes.
Evidence 2: linearlyddecodable linear-chain answer at layer 2. We train affine classifier with the same input as the third attention block at the QUERY position, with the target being the start of the linear chain; the training samples only query for the linear chain, and we generate 5k of them. We obtain test accuracy above 97% for this classifier (on 5k test samples), confirming that layer already has the answer linearly encoded at the QUERY position. To add further contrasting evidence, we train another linear classifier with exactly the same task as before, except it needs to predict the correct start of the LogOp chain. We find that the classifier achieves low test accuracy of approximately 27%, and exhibits severe overfitting with the training accuracy around 94%.
LogOp chain: partial answer in layers 1 & 2 + refinement in layer 3. To predict the correct starting node of the LogOp chain, the model employs the following strategy:
1. The first two layers encode the LogOp and only a "partial answer". More specifically, we find evidence that (1) when the LogOp is an AND gate, layers 1 and 2 tend to pass the node(s) with FALSE assignment to layer 3, (2) when the LogOp an OR gate, layers and 2 tend to pass node(s) with TRUE assignment to layer 3.
2. The third layer, combining information of the two starting nodes of the LogOp chain, and the information in the layer-2 residual stream at the ANSWER position, output the correct answer.
We provide full technical explanation of this high-level overview in Appendix D.4 Our argument mainly relies on linear probing and causal interventions different layers and token positions in the model.
D Length-3 small transformer study: further technical details
In this section, we provide further technical details of our three-layer transformer experiments.
49

---

D.1 Data definition and examples
As illustrated in Figure the propositional logic problem always involve one logical-operator (LogOp) chain and one linear chain. In this paper, study the length-3 case for the small-transformer setting, and length-2 case for the Mistral-7B-vO. case.
The input context has the following form:
BULES_STARY implies D. implies E. or E implies A. P implies T. T implies S. RULES END FACTS_ START TRUE. V FALSE. P TRUE. FACTS_EN
QUERY_ START A. QUERY_EN ANSWER
and the answer written as
K TRUE. K implies D; D TRUE. D E implies A; A TRUE.
In terms the the English-to-token mapping, DULES_START, RULES _ED, FACTS_STARI FACTS_ENN END, QUERY_STARTS QUERY_EM END ANSWER, and ; are all unique single tokens. The logical operators and and or and the connective implies are unique single tokens. The proposition variables are also unique single tokens.
Remark D.I. The rules and facts are presented in a random order in the respective sections of the context in all of experiments unless otherwise specified. This prevents the model from adopting position-basd based shortcuts in solving the problem.
Additionally, for queried, LogOp = OR and the two relevant facts both have FALSE truth values (or LogOp = AND and both relevant facts are TRUE), in which case the answer is not unique. For instance, if in the above example, both K and V are assigned FALSE, then both answers below are logically correct:
clarity, it is entirely possible to into the scenario where the LogOp chain is
FALSE V FALSE. K implies D; D UNDETERMINED, V implies E; E UNDETERMINED. D or E implies A; A UNDETERMINED.
and
FALSE FALSE. V implies E; UNDETERMINED. K implies D; D UNDETERMINED. D E implies A; A UNDETERMINED.
Problem specification. In each logic problem instance, the proposition variables are randomly sampled from pool of 80 variables (tokens). The truth values in the fact section also randomly chosen. In the training set, the linear chain is queried 20% of the time; the LogOp chain is queried 80% of the time. We train every model on 2 million samples.
Architecture choice. Figure indiicats the reasoning accuracies several candidate model variants. We observe that the 3-layer 3-head variant is the smallest model which achieves 100% accuracy. We found that 3-layer 2-head models, trained of some random seeds, do converge and obtain near 100% in accuracy (typically above 97%), however, they sometimes fail converge. The 3-layer 3 3-head variants we trained (3 random seeds) all converged successfully.
D.2
Small transformer characteristics, and training details
D.2.1
Transformer definition
The architecture definition follows that of GPT-2 closely. We illustrate the main components of this model in Figure 29 and point out where the frequently used terms in the main text our paper are in this model.
The following the technical definition of the model. Define input (#1, T2, .... F.) No. a sequence of tokens with length t. It converted into a sequence of (trainable) token embeddings X token (e(x1), ..., e(x,))T dx where we denote the hidden embedding dimension of the model with de. Adding to it the (trainable) positional embeddings P (p,, P2. P2..... "T e R
50

---

100
90
Reasoning accuracy
80
70
50
L=2, H=1
L=2, H=12
L=3, H=1
L=3, H=3
L=3, H=12
Figure 28: Small transformers (trained from scratch), varying sizes. Reasoning accuracies of several models on the length-3 problem. x-axis: model architecture (number of layers, number of heads); y-axis: reasoning accuracy. Note that the 3-layer 3-head variant is the smallest which obtains 100% accuracy on the logic problems.
Output
Linear classifter
Attention block embedding
Layer residual
Rep'n position
Layer-3 attention block embeddinng
Attention www.ghttspprotstiiiiii
Repin & position
Attention
L-3
Repin
restrul streem residual
Rep'n position?
Layer-2 attention block
ombaddinn
Projection
Velue
Attention
L=2
residual
embedding at token position
Layer-1 attention block
embeddiing
Attention
L-1
Input
Figure 29: Illustration of the major components of a 3-layer attention-onle decoder-only transformer on the left, and a rough "sketch" of what is computed inside an attention block (2 attention heads for simplicity of the sketch).
form the zero-th layer embedding of the transformer
X X token P (e(#1) + Pi.... e(x,) + p.).
(2)
This zero-th th layer embedding is then processed by the attention blocks as follows.
Let the model have L layers and H heads. For layer index é E [L] and head index j E [H], attention head Ar, is computed by
Ae, (X = causall
1
(Rea
T K Kes 1) X._VV% Bttx
(3)
Vdh
where d, d
51

---

We explain how the individual components are computed below.
Let us begin with how the ((...) term is computed.
Rexd, where LayerNorm denotes the layer normalization X 4-1 LayerNorm( X f-1)
operator (Ba et al. 2016).
Qu.j Kej E Rd xd. are the key and query matrices of attention head (4,j), where d,. h
They are multiplied with the input X E-1 to obtain the query and key activations Qr, E-1
and K E.j X X both in the space IRd xt, We then perform the "scaled dot-product" of the query and key activations to obtain
Vdn (Q
1
T
K E.j R
(4)
which was introduced in Vaswani et al. (017)) and also used GPT2 (Radtord et al. 2019b).
The causal mask operator causal gext - gxxt allows the lower triangular portion of the input (including the diagonal entries) to pass through unchanged, and sets the upper triangular portion of the input -U, where U is very large positive number (some papers simply denote this UU as -cc). In other words, given any M E txt and (i, k) E [t] x [t).
[causal [M]]i, = [M] 1 if k;
(5)
[causal [M]]i, U, if k.
S R - Rtxt is the softmax operator, which computes the row-wise softmax output from the input square matrix. In particular, given square input matrix M R with its upper triangular portion set to -U (note that the causal mask operator indeed causes the input to S to have this property), we have
exp ([M)a.)
[S(M)) E exp([M]
if h;
(6)
[S(M)) = =0, <
To recap a bit, we have now explained how to compute the first major term in B namely T
S [causal Jah (Q.,
1] It reflects the attention pattern
(also called attention probabilities) of the attention head (4.j) illustrated in Figure 29 right half. Intuitively speaking, the (i, it) entry of this t by t matrix reflects how much the attention head moves the information from the previous layer l - at the source token the current layer at the target token position
position
Now what about X 2-1 V VeJ E Rdaxx, xd. is the value matrix of attention head (f,j).
is multiplied with X f-1 to obtain the value activation X ē.j E Bixd,
At this point, have shown how the whole term in equation 3 is computed.
Having computed the output of the all H attention heads in the attention block at layer E, we find the output of the attention block as follows:
X X 8-1 + Concat(Ae, (..
The operators are defined as follows:
Concat[-] is the concatenation operator, where Concat| Concat[Ar, (X (X R
An e
Wo, E xde the projection matrix (sometimes called output matrix) of layer f. In our implementation, we allow this layer to have trainable bias terms too.
Finally, having computed, layer by layer, the hidden outputs X for e E [Z.] we apply affine classifier (with softmax) to obtain the output of the model
f(x) = S(X WT class + beloss)
(8)
This output indicates the probability vector of the next word.
In this paper, set the dimension of the hidden embeddings d, 768.
52

---

D.2.2 Training details
In all of our experiments, we set the learning rate to 5 10-5, and weight decay to 10-4. We use batch size of 512, and train the model for 60k iterations. We use the AdamW optimizer in PyTorch. with 5k iterations of linear warmup, followed by cosine annealing to learning rate of O. Each model trained on a single V100 GPU; the full set of models take around days finish training.
D.3 High-level reasoning strategy of the 3-layer transformer
We complement the text description of the reasoning strategy of the 3-layer transformer in the main text with Figure 30 below. not only presents the main strategy the model, but also summarizes the core evidence for specific parts of the strategy.
Query type
Query for linear chain
Query for LogOp chain
Routing layer
Layer sends poote layer at QUERY position
Layer sends oute layer QUERY position
Leyer 3 @ ANSWER focuses -10% attention on QUERY position
Leyer @ ANSWER allocates -0% attention on
QUERY, instead focuses rules and facts
deiinngedd action layer
Layer 2 resid QUERY:
ANSWER: Linearly decodable "partial" answer
Layer
linearly decodaable answer
Answer
Layer 3 pre projection embd ANSWER: Linearly decodable relevant premise variables
Answer
Layer-3
post-prejection embo @ ANSWER: innerly decodable answer
Answer
Figure 30: High-level overview of how the 3-layer transformer solves the logic problem, particularly writing down the first answer token (the hardest place in the proof). As shown the grey blocks on the left, the model performs "routing" in layer 2 by sending routing signal hroate to layer (with its "sign" dependent on the query type), then the layer-3 attention block acts according to the "sign" of the routing signal sent to it. The middle (right) chain shows the strategy when the problem queries for linear (LogOp) chain.
D.4 Answer for the LogOp Chain
Evidence 3a: Distinct behaviors of affine predictors at different layers. We train two affine classifiers two positions inside the model (each with 10k samples): W resid,==2 at layer-2 residual stream, and W attn,t=3 at layer-3 attention block output, both at the position of ANSWER, with the target being the correct first token. In training, if there are two correct answers possible (e.g. OR gate, starting nodes are both TRUE or both FALSE), we randomly choose one the target; in testing, we deem the top-1 prediction "correct" if it coincides with one of answers. We observe following predictor behavior the test samples:
1. W Wattntt3 predicts the correct answer 100% of the time.
2. W resid,t=2 t=2 always predicts one of the variables assigned FALSE (in the fact section) if LogOp is the AND gate, and predicts one assigned TRUE if LogOp is the OR gate.
Evidence 3b: linearly decodable LogOp information from first two layers. We train an affine classifier at the layer-2 residual stream to predict the LogOp of the problem instance, over 5k samples (and tested on another 5k samples). The classifier achieves greater than 98% accuracy. We note that training this classifier at the layer-1 residual stream also yields above 95% accuracy.
Evidence 3c: identification of LogOp-chain starting nodes at layer 3. Attention heads (3.1) and (3.3), when concatenated, produce embeddings which can linearly decode the two starting nodes of
53

---

the LogOp chain with test accuracy greater than 98%. We also find that they focus their attention in the rule section of the context (as shown in FigureBIJ, Due to causal attention. this means that they determine the two starting nodes from the regOp-relevant rules. Remark. The above pieces cormmation--rffiemmen process To To further validate that the
of observations suggest the "partial embedding from the first two layers are indeed causally linked to the correct answer at the third layer, we perform an activation patching experiment.
Evidence 3d: linear non-decodability of linear chain's answer. To provide further contrasting evidence for the linear decodability of the LopOp chain's answer, we experimenaally show that is not possible to linearly decode the answer of the linear chain in the model. Due to the causal nature of the reasoning problem (it is only possible know the answer at or after the QUERY token position), and the causal nature of the decoder-only transformer, we train set of linear classifiers on all token positions at or after the QUERY token and up to the ANSWER token, and on all layers of the residual stream of the transformer. We follow the same procedure in Evidence 3c, except in this set of experiments, for contrasting evidence, QUERY is for the LopOp chain, while the classifier trained to predict the answer of the Linear chain. The maximum test accuracy of the linear classifiers across all aforementioned token positions and layer indices is only 32.7%. Therefore, the answer of the Linear chain is not linearly encoded in the model when QUERY is for the LopOp chain.
Evidence 3e: layer-2 residual stream at ANSWER important to correct prediction. We verify that layer-3 attention does rely on information in the layer-2 residual stream (at the ANSWER position):
Construct two sets of samples D1 and D22 each of size 10k: for every sample X E Di and X2,, 2.m E D2, the context of the two samples are exactly the same, except the LogOp is flipped, i.e. if Xx 1,r has disjunction, then x has the conjunction operator. If layer 3 of the model has no reliance on the Resid/=2 (layer-2 residual stream) for LogOp information at the ANSWER position, then when we run the model on any X 22 patching 2 n.2) with 2 (X.1) at ANSWER should not cause significant change to the model's accuracy of prediction. However, we observe the contrary: the accuracy of prediction degrades from 100% to 70.87%, with standard deviation 3,91% (repeated over 3 sets of experiments).
Observation: LogOp-relevant reasoning al the third layer. We show that the output from attention heads (3,1) and (3,3) (before the output/projection matrix of the layer-3 attention block), namely when concatenated, contain linearly decodable information about the two starting nodes of the LogOp chain. We frame this as multi-label classification problem, detailed as follows:
As,1 and
1. We generate 5k training samples and 5k test samples, each of whose QUERY is for the LogOp chain. For every sample, we record the target as a 80-dimension vector, with every entry set to 0 except for the two indices corresponding the two proposition variables which are the starting nodes of the LogOp chain.
2. Instead of placing softmax on the final classifier of the transformer, function. Moreover, instead of the Cross-Entropy loss, we use the Binary Cross-Entropy loss (namely the torch. nn. functional. binary_ srrryy_ross_eeeee settrooy_wtth__uee Torch, which directly includes the Sigmoid for numerical stability).
the Sigmoid
in Py-
3. We train ( A3,)) and with the targets and training loss defined above. We use constant learning rate of 0.5 x 10 -3 and weight decay of 10 10-2 The optimizer is AdamW in PyTorch.
affine classifier, with its input being the concatenated (a 512-dimensional vector) on every training sample,
4. We assign a "correct" evaluation of the model on a test sample only if it correctly outputs the two target proposition variable as the top-2 entries in its logits. We observe that the classifier achieves greater than 98% once it converges.
"In fact, the observations suggest that layer performs a certain "matching" operation. Take the OR gate as an example. Knowing which of the three starting nodes (for LogOp and linear chain) are TRUE, and which two nodes are the starting nodes for the LogOp chain are sufficient to determine the first token! This exact algorithm. however, not fully validated by evidence; we leave this as part of our future work.
54

---

0.40
Head (3.1)
Head (3.3)
0.35
0.30
0.15
0.10
0.00
G G Foo oos
PA.A raise
D
The
G
Phe
19
Figure 31: Attention statistics, averaged over 500 samples, all of which query for the LogOp chain. The x-axis is simply an example prompt that helps illustrate where the attention is really placed at. Observe that only attention head (3,2) pays significant attention to the fact section. The other two heads focus on the rule section. Note that none of them concentrate attention on the QUERY token. Reminder: due the the design of the problem, the rule, fact and query sections all have consistent length for every sample!
D.5 Extra remarks
Observation 3 supplement: linearly-decodable linear-chain answer at layer 2. We simply frame the learning problem a linear classification problem. The input vector of the classifier is the same as the input to the layer-3 self-attention block, equivalently the layer-2 residual-strean embedding. The output space is the set of proposition variables (80-dimensional vector). We train the classifier on 5k training samples (all whose QUERY is for the linear chain) using the AdamW optimizer, with learning rate set to 5 10-3 and weight decay of 10-2 We verify that the trained classifier obtains an accuracy greater than 97% on an independently sampled test set of size 5k (all whose QUERY is for the linear chain too).
Remarks on truth value determination. Evidence suggests that determining the truth value of the simple propositional logic problem is easy for the model, the truth value of the final answer is linearly decodable from layer-2 residual stream (with 100% test accuracy, trained on 10k samples) when give the model the context+chain of thought right before the final truth value token. This expected, as the main challenge of this logic problem is not about determining the query's 's truth value, but about the model spelling out the minimal proof with careful planning. When abundant CoT tokens are available, is natural that the model knows the answer even in its second layer.
55


Reviews:
[{"review_id": "6ZkgaQiOWl", "reviewer": "Reviewer_b3jQ", "summary": "This work presents an in-depth causal mediation analysis (CMA) of how pretrained LLMs (Mistral, Gemma; up to 27B parameters) solve formal logic problems. The authors uncover fine-grained functional roles of attention heads by identifying sparse, modular circuits responsible for reasoning. They decompose the identified circuits into sub-circuits exhibiting four distinct sets of attention heads and validate these roles through various techniques (intervention, patching, ablation).", "questions": "- Why do exactly four attention head types emerge? Could this modularization differ under other prompts or tasks?\n\n- Can the authors provide a minimal pseudo-code for discovering modular reasoning circuits applicable beyond the logic task?\n\n- How statistically robust is the attention focus in Figure 4b? For decision head (30,9), attention seems diffuse across multiple tokens, yet the interpretation singles out “U”.\n\n- In Table 1, how should we interpret negative logit differences (e.g., –0.97, –1.11)? Does the sign have causal meaning?\n\n- Can the authors expand on this claim: IE magnitude ≠ component necessity? How does the method distinguish compensatory effects from causal roles?\n\n- If decision heads track the correct fact variable when the model is correct, what happens when it is incorrect? How does attention shift, and what does it imply?\n\n- In line 219 (Sec. 3.1), where are the MLPs defined?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Pros:\n - this paper performs a fine-grained and relatively large-scale analysis of LLMs’ reasoning processes by examining symbolic logic tasks beyond single-hop inference. And it yields actionable insights into the internal modularity of large models.\n\n - It contributes to the growing body of mechanistic interpretability work, especially in understanding how attention heads are structured and composed for symbolic reasoning beyond one step.\n\n### Cons:\n\n- Causal mediation analysis methodology needs refinement:\n\n  - the use of text-only explanations in Section 2 makes the methodology hard to follow. Critical elements such as circuit construction, intervention steps, and component attribution should be formalized using math notation, diagrams, or pseudo-code. Without these, the method lacks generalizability and reproducibility.\n\n - several CMA assumptions may be violated or not addressed:\n\n   - the analysis tracks only the first answer token, assuming it encapsulates all reasoning. This may ignore important contributions to subsequent steps or generated proofs.\n\n   - the causal graph assumed (X → M → Y) does not reflect the dense interdependencies in a transformer; identifiability assumptions (e.g., no hidden confounders, functional invariance) are likely invalid.\n\n   - also, unadjusted upstream activations may confound the estimation of indirect effects, leading to overestimated effect sizes.\n\n   - prompt perturbations (e.g., flipping QUERY, shuffling rules) modify multiple features (position encodings, token distance), so changes in logits cannot cleanly be attributed to the modeled causal chain.\n\n   - this paper treats interventions as Monte Carlo over prompts, rather than stochastic model outputs, which may not be inconsistent with Pearl-style DE/IE definitions over probabilities.\n\n- Conceptual and methodological clarity:\n\n  - the definition of “causal chain” (e.g., “QUERY → Relevant Rule → Relevant fact(s) → Decision”) is introduced (lines 118–122) without formal backing. Why must the model explicitly follow this chain? How is it operationalized in model activations?\n\n  - the claim that “the model must execute the causal chain” to produce the first answer token is asserted without sufficient justification. Given that transformer-based reasoning is distributed and redundant, the claim may be overly rigid or oversimplified.\n\n  - the rationale behind exactly four types of attention heads remains ambiguous. Is this structure emergent or imposed by the design of the few-shot examples? Could alternative task formulations or data lead to different modular decompositions?\n\n- Insufficient rigor in key sections:\n\n  - in section 3.1, MLP references (e.g., MLP-0) are introduced abruptly without clear definitions. Terms like logit difference, patching, and attention head role should be better defined with notation as mentioned for clarity.\n\n  - interventions (e.g., altering QUERY tokens, flipping rules) are scattered across text. A structured summary table would improve clarity.\n\n   - figure 3 lacks explanation of the color encoding scheme; fig 4(b) lacks axis labels and a legend. The ordering of figures is also inconsistent with narrative flow.\n\n   - table 1 includes abbreviations that are undefined in the main text, making it unreadable without referring to the appendix. All terms describing sub-circuits need be introduced and explained concisely in the main paper.\n\n  - the method for identifying the four attention head types is underspecified. How are the roles inferred from weight visualization? How are components assigned to categories?\n\n- Concerns with experimental interpretations:\n\n  - section 3.2: Removing non-circuit heads may leave behavior unchanged, but that doesn’t necessarily prove their non-involvement—redundant heads may still contribute causally.\n\n  - section 3.3: Swapping rules affects token order and embeddings, introducing confounds that may account for changes in indirect effect, rather than proving lookup mechanisms.\n\n  - some analysis appears cherry-picked. For instance, attention head (23,6) is cited as moving QUERY to “:”, but the evidence for this movement is unclear. Further, figure 4c mentions both (23,6) and (23,3); clarification is needed.\n\n  - large indirect effect (IE) does not imply necessity. Fragile components may inflate IE, while robust but essential parts may show small IE due to downstream compensation. Similarly, large direct effect (DE) does not prove sufficiency due to non-additivity and redundant pathways.\n\n- presentation issues:\n\n   - the writing requires grammar and clarity checks in several places (e.g., lines 92, 148, 210).\n\n  - several critical technical definitions are scattered or missing, leading to cognitive overhead for readers unfamiliar with causal analysis in LLMs.", "was_revised": true, "final_justification": "My last comment to the authors is also my final justification, which I repeat here as below.\n\n> I thank the authors for their detailed response, which has addressed most of my concerns regarding the CMA setup, experiments, and the rationale behind the four types of attention heads. The commitment to open-source the code meaningfully improves the replicability and potential broader adoption of this method (though the write-up could be more streamlined). I also appreciate the additional experiments, which partially strengthen the empirical grounding. A discussion, planned by the authors, on the token-distance effect (a potential confounder) and on the generalizability of the proposed circuit surgery to distinct reasoning domains would further reinforce the claims made in this work. Given these points, I decide to raise my score to 4 to recommend acceptance. Look forward to seeing the open-sourced tool tested on new domains, which could substantially advance our understanding of LLM circuit mechanisms.", "rebuttal": "We sincerely thank you for the thoughtful questions and suggestions! We would be happy to continue the discussion on any of our responses below.\n\n# CMA experiments, methodology and interpretation.\n\nBefore delving into the technical details, we wish to emphasize that the use of _activation patching_ for localizing and interpreting hidden mechanisms in LLMs is popular in mechanistic interpretability, and has been explicitly cast as instances of Pearl’s Causal Mediation Analysis (CMA) in seminal works [1-3]. In particular, it is typically acknowledged that _activation patching offers imperfect (and potentially biased) estimates for the indirect and direct effects_, due to the LLM’s complexity and limits in computational resources. \n\nWith this overall consideration of CMA in interpretability research, we address your concerns below.\n\n> the causal graph assumed (X → M → Y) does not reflect the dense interdependencies in a transformer\n\nWe employ the simplified causal graph $X \\to M \\to Y$ as a _local abstraction_ of the LLM: in our circuit analysis, we analyze one hypothesized mediator at a time, treating the rest of the network as background, essentially approximating the indirect effects of the mediator. _This approach is common in mechanistic interpretability._\n\n> identifiability assumptions (e.g., no hidden confounders, functional invariance) are likely invalid. \n\nWhile the listed identifiability assumptions are largely satisfied by activation patching, some subtleties exist.\n\nFirst, please note that we sidestep many identifiability conditions in classical mediation analysis since we can observe the model’s “counterfactual” outcomes. By design of activation patching, no _exposure‑induced mediator‑outcome confounder_ exists. To perform activation patching, we run the model on the normal prompt, and hold the mediator’s output to its corresponding counterfactual activations, that is, we perform a hard _intervention_ $do(M = m_{counterfactual})$. For any potential $C$ in the LLM with $X \\to C \\to Y$ and $X \\to C \\to M$, the intervention cuts the link $C \\to M$, removing such confounders. _Exposure-outcome_ and _exposure-mediator confounding_ are also ruled out – the LLM’s forward pass is fully deterministic and “self-contained”, as both $Y$ and $M$ are only functions of $X$. Functional invariance can be shown similarly.\n\nThe _positivity_ assumption is subtler. Earlier approaches in mechanistic interpretability corrupt a mediator in a “messy” fashion, such as noise injection [4]. In comparison, our counterfactual activations are generated with much care to minimize “off-manifold” effects, as we use activation from altered prompts which differ _minimally_ from the normal one. We emphasize, however, that a balance has to be struck between efficiency of reaching reasonable conclusions and how fine-grained the differences are.\n\n> unadjusted upstream activations may confound the estimation of indirect effects…\n\nThe upstream activations of the mediator, including those in the lower layers and previous token positions, and parallel activations (e.g. from the attention heads in the same layer, if the mediator is an attention head) are identical with or without intervention. Only the mediator’s output is intervened. \n\n> “large indirect effect (IE) does not imply necessity…  large direct effect (DE) does not prove sufficiency…”, “...compensatory effects”\n\nTo demonstrate “necessity” of (a family of) attention heads in the circuit, we show\n1) large IE in the single-component patching sweeps, and \n2) when we ablate all non-circuit heads, further ablating any one of the attention families from the circuit _causes performance-recovery to drop from >90% to nearly trivial_. \n\nThis shows that the identified attention head families participate heavily in the LLM’s inference pathway. This is a (stronger) “necessity” result following [2]’s definition.\n\nWe believe that the rest of the comment is less about “sufficiency”, but more about the circuit’s “**faithfulness**” and “**completeness**” – more advanced criteria introduced in [5]. As shown in Section 3.2 and B.5, the discovered circuit is _nearly sufficient in recovering the model’s performance_ (94% for Gemma-2-9B, 98% for Mistral-7B) – **our circuit is faithful**. We note that **even the faithfulness criterion is not tested for or met in many existing mechanistic interpretability works** [6,7].\n\nNow, **our circuit is not necessarily _complete_**. Backup heads, albeit typically weak and noisy, have been observed in some LLMs, and is considered an open problem in mechanistic interpretability. Please note that **the completeness criterion is rarely tested for in many mechanistic interpretability works**, due to its _heavy computational demand_ of having to iteratively ablate the discovered causal mechanisms to track compensating pathways – an aspect which is particularly pronounced in our work due to the large model sizes. We will add a discussion on these more advanced criteria to our updated paper.\n\n> “Prompt perturbations… modify multiple features (position encodings, token distance)...”\n\nWe first clarify that the sets of patching experiments are independent of each other. For example, when we flip the QUERY token, we do not shuffle the rules: only the QUERY _value_ is changed, _no position encoding or other features vary in the prompt_. \n\nWe treat the rule-shuffling and fact-changing patching experiments as _cross-checks_ for our hypotheses of the sub-circuits in the LLMs. For example, the rule-locators, whose attention weights consistently track the queried rule, have high causal scores in both the QUERY-flipping and rule-shuffling experiments, but trivial scores in the fact-changing experiments. While the correlational and causal evidence do not fully eliminate nuanced factors such as _token distances_, the proposed circuit is still the more plausible explanation of existing evidence.\n\n> In Table 1, how should we interpret negative logit differences?\n\nThe scores in Table 1 are normalized. W.r.t. the logit-difference metric, a score of -1.0 shows that the model behaves as if it is solving the counterfactual problem, even though it is run on the normal prompt. It is attained with the _null circuit_, i.e. freezing all attention head activations to their counterfactual activations. A score of 1.0 is the other extreme: the model behaves as if it is naturally run on the normal prompts. 0.0 is the mid-point.\n\n# Writing and conceptual clarity.\n\n> Clarify “MLP references…”, “interventions…”, “figure 3…”, “fig 4(b)…”, “table 1…”\n\nThank you for these suggestions, we will address them in the updated version of our paper!\n\n> “definition of ‘causal chain’ …”, “…asserted without sufficient justification”\n\nThank you for raising this point about the QUERY → Decision chain. We intended it to be the (human-interpretable) _hypothesis_ of how the LLM implements its internal reasoning. We will adjust our phrasing in the updated version of our paper to ensure that this point is clear.\n\n> “...rationale behind exactly four types of attention heads remains ambiguous”, “ method for identifying the four attention head types is underspecified”\n\nWe primarily distinguish the attention-head families based on their attention patterns. While we do provide detailed explanations of their characteristics in Appendix B.4.2, B.8.2 and B.9.1, we will aim to surface succinct versions of these analyses into the main text for better clarity in our updated paper. \n\n> … four attention head types… modularization differ under other prompts or tasks?\n\nWe believe the exact types of attention heads for different tasks can change. However, functionality-wise, there have been observations about the usage of locator and mover heads in other mechanistic works, in GPT2 models [5]. An interesting future direction is to study what functional components persist across problems and models, and what emerges with model scale.\n\n> “How statistically robust … decision heads…”, \n\nWe show their (robust) attention statistics in Fig. 5(b)(iv). \n\n> “moving QUERY to ‘:’... (23,3)”\n\nDue to grouped query attention, head (23,6) has the value index (23,3). We primarily classify the mover heads based on their QUERY-invariant attention patterns, trivial query and key casual scores, and high value causal scores.\n\n**Pseudocode**.\nWe will add the pseudocode for our circuit analysis algorithm to our updated paper. The following is its backbone, in consideration of character limits.\n```\nA or B implies C. D implies E. A is true. B is false. D is true. What is the truth value of C?\n\nX  = prompt with Query = C\n\nX* = prompt with Query = E   (counterfactual)\n\n(1) run model(X)    → logits_norm\n\n(2) run model(X*)   → logits_ctfl\n\n(3) run model(X) with head h set to activation from X* → logits_interv\n\n(4) Estimate indirect effects (expression (1) in the Appendix)\n```\nWe currently manually set the threshold for top-k components to form the circuit (threshold setting is a major open problem in mechanistic interpretability). For more complex propositional problems, we can vary the QUERY token to choose which sub-part of the reasoning chain we wish to probe for.\n\n# References.\n\n1. J. Vig et al. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. NeurIPS 2020.\n2. F. Zhang et al. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR 2024.\n3. A. Geiger et al. Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability. JMLR 2025.\n4. K. Meng et al. Locating and Editing Factual Associations in GPT. NeurIPS 2022.\n5. K. Wang et al. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. ICLR 2023.\n6. A. Stolfo et al. A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis. EMNLP 2023.\n7. B. Wang et al. Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization. NeurIPS 2024."}, {"review_id": "8WceW39bCm", "reviewer": "Reviewer_9qMC", "summary": "This paper studies the mechanism of how pretrained LLMs perform propositional logical reasoning tasks. The authors constructed synthetic propositional logical reasoning tasks that contain facts, rules, and a query of the truth value of a Boolean variable as questions and contain the proof and the final answer as the response. By providing several in-context examples and investigating the first answer token logit by causal mediation analysis (CMA), the authors identified a sparse circuit (which contains 13 attention heads for Gemma-2-9B) that performs propositional logical reasoning. The attention heads in the circuit can be divided into four groups according to their roles, i.e., queried-rule locating heads, queried-rule mover heads, fact processing heads, and decision heads. Analysis on different models (Mistral-7B and Gemma-2-27B models) also revealed similar components, although the larger model’s (Gemma-2-27B) circuits have a greater degree of parallelism.", "questions": "What is the longest reasoning chain studied in this problem (it seems that the illustrative examples in the main text can be solved by two-hop reasoning)", "limitations": "See the Weaknesses section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The problem studied in this paper, i.e., how pretrained LLMs perform propositional logical reasoning tasks, is important, as propositional logical reasoning can be viewed as one of the most fundamental building blocks of more complex reasoning problems.\n\n2. The authors carefully designed a synthetic task and identified the circuits performing the reasoning across different models, and found similar functional units (such as attention heads serving as queried-rule locating heads).\n\n**Weaknesses:**\n\n1. The analysis for the logit difference by CMA only involves the first answer token. Although predicting the first token is non-trivial and is important for generating the whole output sequence, analyzing the first token only might not be sufficient to understand the underlying mechanism of how LLMs perform propositional logical reasoning.\n\n2. The input contains in-context examples. However, it is unclear how the in-context example will affect the underlying mechanism. For example, when there is no in-context example or when the number of in-context examples varies, will the circuit change?\n\n3. The problem studied in this paper is somewhat restricted, and the format is not that flexible. To understand how LLMs perform logical reasoning in a more general setting and which heads play the important role, it would be important to analyze the scenarios when the input format changes while keeping the set of facts and rules unaltered (e.g., exchanging the order of the rule area and the fact area, or rules and facts are interleaved) or slightly increasing the complexity of the problem (as the illustrative examples in the main text is relatively simple and might not be sufficient to discover the full mechanism of how LLMs perform propositional logical reasoning).\n\n4. It would be better to see some implications of the finding in this paper, such as how one can utilize the discovered circuits.", "was_revised": true, "final_justification": "The authors' responses addressed most of my concerns.\n\nFor the first point, I agree with the authors that studying all output tokens might be computationally prohibitive (and can be a future direction), and their further analysis of the first token for invoking the correct rule is nice.\n\nTherefore, I decided to raise my score to 4.", "rebuttal": "Thank you for your thoughtful and constructive feedback! Below are our responses to your questions and suggestions.\n\n> The analysis for the logit difference by CMA only involves the first answer token. Although predicting the first token is non-trivial and is important for generating the whole output sequence, analyzing the first token only might not be sufficient to understand the underlying mechanism of how LLMs perform propositional logical reasoning.\n\n\nThank you for this point. Indeed, it would be nice to study how the models write down all output tokens in the proof, but it is computationally infeasible across the large models we study given our resources. \n\nNevertheless, **we conduct a further analysis** on the _second most important place_ in the model’s generated proof, the **first token for invoking the correct rule** – recall that the model needs to invoke the correct fact, then the correct rule to infer the correct truth value of the queried proposition variable. We found that in both Gemma-2-9B and Mistral-7B, **the _queried-rule locators_ and _decision heads_ again exhibit strong indirect effects at this proof position, indicating _reuse_ of functional sub-circuits in their proof writing**. \n\nWe will add detailed versions of these results to our updated paper.\n\n> The problem studied in this paper is somewhat restricted, and the format is not that flexible. To understand how LLMs perform logical reasoning in a more general setting and which heads play the important role, it would be important to analyze the scenarios when the input format changes while keeping the set of facts and rules unaltered (e.g., exchanging the order of the rule area and the fact area, or rules and facts are interleaved)...\n\nThank you for this nice suggestion!\n\nWe conducted the following **new experiment for testing the robustness of the circuit across prompt formats**. As you suggest, we swapped the fact and rule sections in the prompts. Now all the logic problems are presented in the form “Facts: … Rules: … Question:...”. For example:\n\n_Facts_: O is false. Q is true. I is true. _Rules_: I implies C. Q or O implies H. _Question_: what is the truth value of H? _Answer_: Q is true. Q or O implies H; H is true.\n\nWe found evidence that **on Gemma-2-9B, the circuit for predicting the first answer token stays almost exactly the same**! In particular, we find that it is the same heads exhibiting high IE scores in the activation patching sweep. Due to the time limit, we have not yet performed the sufficiency test (as described in Section 3.2 and Appendix B.5, B.8.3) thoroughly, so we cannot report DE measurements here, but we think the _necessity-based evidence_ is already quite interesting to present here! We will add the more detailed and rigorous version of these results to our updated paper.\n\n> The input contains in-context examples. However, it is unclear how the in-context example will affect the underlying mechanism. For example, when there is no in-context example or when the number of in-context examples varies, will the circuit change?\n\nThank you for the comment – it is a good question as to if, and how much the circuit changes with respect to the number of in-context examples. It is, however, outside the scope of this paper. \n\nThe primary goal of this paper is _not_ about understanding circuit _formation_ in in-context learning, but rather _how transformers output tokens that describe simple but valid mathematical proofs_: this requires us to work in a regime where the models are sufficiently high in accuracy, in which case the logit difference metric becomes a valid metric for measuring causal effects of model components. If the model is low in accuracy, then it cannot separate the two most likely candidate answer tokens well (the two which we use as normal and counterfactual answers by construction, in our QUERY-flipping experiments). Basically, we need to eliminate possible confounders introduced by having overly limited in-context proof examples, which would make the model too uncertain about the proof structure."}, {"review_id": "5kdUl5ECC9", "reviewer": "Reviewer_rGZP", "summary": "This paper presents a circuit-based analysis of how Gemma-2-9b, Gemma-2-27b and Mistral-7b perform (a constrained form of) single-step logical deduction. Each task consists of two rules (implications, with a binary disjunction or conjunction as the precondition), three facts (truth values of atoms) and one query (question about the truth value of one atom, which only depends on one of the two rules). The findings indicate that all of these models contain circuits which decompose the task as locating the relevant rule, identifying the query token, processing the relevant facts and then making a decision. In contrast, the authors find that training small transformers from scratch on these tasks does not yield such circuits, indicating that this behaviour arises at scale (of the model or the training data).", "questions": "Suggestions:\n- Please include all **main** numerical results in the main text of the paper (in particular for section 3.2).\n\nQuestions:\n- Can you elaborate a bit more on the significance of the decision head? It appears that it is attending sharply to the atom for which a known fact can be used in one of the rules to derive the conclusion. Calling this a \"decision\" seems a bit far fetched - it almost seems more like \"fact locating\" to me?\n- What motivated your selection of models? In particular, did you consider studying a smaller model (e.g. 1b or 2b)? If it is feasible, comparing Gemma-2-2b to the 9b and 27b models would be particularly interesting (although it is very good that you also included a model belonging to a different family).\n\nCriteria under which my evaluation score would increase or decrease: My ability to judge the significance and novelty of this work is somewhat limited as this is not my primary area of research. As such I have predominantly focused on the clarity of the exposition of the material, which overall is already good, as well as the apparent scientific rigor of the study. As such I will only consider updating my score if the discussion with other reviewers clarifies the novelty and significance of the work, or if I feel that the answers to my questions above have further strengthened the paper.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is very clear and well written. Both the motivations and the findings are presented clearly.\n- The findings are interesting, especially due to the contrast between the LLMs and the small transformers trained from scratch.\n- Despite the laborious nature of this type of research, the authors do manage to find generalizable findings across three models (although the Gemma-2-27b study is somewhat limited)\n- While the problem is still very toy, it feels a bit more relevant to actual usages of LLMs than most prior work in this area (that I am aware of).\n\nWeaknesses:\n- Given the authors find some evidence that Gemma-2-27b carries out this task somewhat differently than Gemma-2-9b, it would have been nice to see Gemma-2-2b studied as well (to isolate the effect of scale vs. architecture).\n- Focusing uniquely on Gemma-2-9b in the main text makes the narrative clear but makes it harder to get a sense of the generalizability of these findings. Focusing on one model in the text but still including the main numerical results for all models (e.g. Table 2 for Mistral-7b) in the main text would have made for a clearer paper.\n- The influence of the decision head is not clear to me, which I believe is because of the focus on the model's state right before outputting the first token. However, I recognize that scaling up this analysis to arbitrary positions in the output sequence would be infeasible.\n- The results figures (in particular Figure 5 and similar figures in the appendix) could use some polish. Furthermore, the legends do not clarify what the red bars are (95% CI on the mean? +- one standard deviation?).", "was_revised": true, "final_justification": "I believe this paper to be contributing novel findings to the growing body of knowledge regarding the internal workings of LLMs. Like other mechanistic interpretability works it definitely has its flaws, in particular regarding the scope of the analysis, and the novelty of the method itself is unclear. However, I do believe the findings to nonetheless have significant scientific value, especially when taking the additional experiments carried out during the rebuttal period into account. Unlike some other reviewers I do not take issue with the writing or presentation of the methodology.\n\nOverall, I do not believe this paper to be groundbreaking in any way, but I do believe that it contributes new and meaningful knowledge to the literature and would therefore like to see it accepted.", "rebuttal": "Thank you for your valuable feedback! We are glad that you find the paper’s writing clear, and appreciate the mechanistic insights on the large and small models. We will address your comments below.\n\n> “Given the authors find some evidence that Gemma-2-27b carries out this task somewhat differently than Gemma-2-9b, it would have been nice to see Gemma-2-2b studied as well (to isolate the effect of scale vs. architecture).”, “...did you consider studying a smaller model (e.g. 1b or 2b)?”\n\nWe wish to emphasize that the main goal of this paper is to study _the latent processes behind how sufficiently capable LLMs write simple but valid (propositional logic) proofs_. We found that small LLMs (Gemma-2-2b, the GPT-2 models, etc.)’s proof accuracies are overly low even with a large number of in-context proof examples, showing that they do not appear to recognize the latent reasoning patterns. This is the primary reason for the study of larger LLMs in our paper, compared to the more standard sizes studied in many existing circuit analysis papers (mostly under 1B in size) [1-3].\n\n> Focusing uniquely on Gemma-2-9b in the main text makes the narrative clear but makes it harder to get a sense of the generalizability of these findings. Focusing on one model in the text but still including the main numerical results for all models (e.g. Table 2 for Mistral-7b) in the main text would have made for a clearer paper.\n\nThank you for this point, we will add Table 2 to section 3.2 in the main text, and highlight that we were also able to verify a functionally similar circuit in Mistral-7B.\n\n> Can you elaborate a bit more on the significance of the decision head? It appears that it is attending sharply to the atom for which a known fact can be used in one of the rules to derive the conclusion. Calling this a \"decision\" seems a bit far fetched - it almost seems more like \"fact locating\" to me?\n\nThere are two reasons why we group the decision heads instead of merging them into the family of fact-processing heads. \n\nFirst, we separate the two families based on their distinct attention patterns. We noted in Appendix B.4.2 and B.8.2 that the fact-processing heads’ attention weights usually do _not_ directly focus on the answer token itself, but are typically spread over the sentence containing the correct fact. In contrast, the decision heads always direct significant attention weight at the _precise answer token_, for all the LLMs we analyzed. Second, we found that the _decision_ heads tend to have _direct effects on the model logits_ in Gemma-2-9B (the main model analyzed in this paper), as we show in _Figure 25_ in the Appendix, while the fact-processing heads do not. \n\nWe will surface succinct versions of the distinction between the two attention-head families to the main text in the updated version of our paper.\n\n> The influence of the decision head is not clear to me, which I believe is because of the focus on the model's state right before outputting the first token. However, I recognize that scaling up this analysis to arbitrary positions in the output sequence would be infeasible. \n\nWe found that for **_rule invocation_ in the proof, the _decision heads_ again exhibit high importance in the Gemma-2-9B and Mistral-7B’s inference pathway**. \n\nMore specifically, we performed another set of patching experiments measuring the causal importance of the model components for writing down the _first token of the invoked rule_ – the second most important place in the proof, after the fact-invocation token. We found that the same _decision heads_ and _rule-locator heads_ exhibit strong causal scores (measured via logit difference) in both Gemma-2-9B and Mistral-7B, with similar attention patterns to fact invocation! This suggests that the LLMs reuse functional sub-circuits for writing down the proof. We will add detailed versions of these results to our updated paper.\n\n> The results figures (in particular Figure 5 and similar figures in the appendix) could use some polish. Furthermore, the legends do not clarify what the red bars are (95% CI on the mean? +- one standard deviation?).\n\nThank you for this point, we will make sure to add clarity to the figures in our updated paper.\n\n**References**.\n\n1. K. Wang et al. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. ICLR 2023.\n2. G. Kim et al. A Mechanistic Interpretation of Syllogistic Reasoning in Auto-Regressive Language Models. ArXiv Preprint, 2025.\n3. Y. Yao et al. Knowledge Circuits in Pretrained Transformers. NeurIPS 2024."}, {"review_id": "Vttu3bIwNg", "reviewer": "Reviewer_zZhs", "summary": "This paper utilizes a synthetic propositional logic problem set to explore the reasoning pathways and circuits of transformer-based LLMs with parameter sizes of up to 27 billion. By means of counterfactual variations, it identifies modular reasoning circuits within LLMs and the \"lazy reasoning\" phenomenon, in contrast to smaller transformers.", "questions": "1. The propositional logic problem studied is minimal (5 variables, 2 rules). How do the identified circuits scale to more complex reasoning tasks with longer logical chains or nested operators? Maybe you can test the models on extended problem variants with 3+ rules or nested logical operators (e.g., “A or (B and C) implies D”). Analyze whether the four attention head families (queried-rule locating, mover, fact-processing, decision) remain modular or require additional components. If the core circuit structure is preserved in more complex cases, this would strengthen claims about the universality of the identified mechanisms. A failure to generalize would indicate limitations in handling real-world multi-step reasoning.\n2. The analysis focuses almost exclusively on attention heads, but what is the role of MLPs and lower-layer embeddings in the reasoning process? For example, do MLPs contribute to logical operator processing or fact integration? Performing ablation experiments by freezing MLP activations or lower-layer embeddings (e.g., layers 1–8 in Mistral-7B) and measure the impact on reasoning accuracy is welcomed. If MLPs are shown to play a non-trivial role (e.g., ≥20% accuracy drop when frozen), this would require revising the claim that reasoning is primarily localized in attention heads. Conversely, if their impact is minimal, it validates the focus on attention-based circuits.\n3. The study is limited to Mistral and Gemma models. Do similar reasoning circuits exist in other LLM families like Qwen? Replicate the analysis on at least one additional model family (e.g., LLaMA or Qwen) to verify whether the four attention head families emerge universally. Compare attention patterns and circuit sparsity across architectures. If analogous circuits are found in diverse models, this supports the hypothesis that modular reasoning components are an emergent property of transformer pretraining. Discrepancies would highlight architecture-specific reasoning strategies.\n4. The contrast with 3-layer task-trained models mentions “intermingled reasoning” but lacks detailed mechanistic comparison. How exactly do the information flow and component specialization differ between small, task-specific models and large, general-purpose LLMs? Train additional small transformers (e.g., 4-layer, 6-head) on the same task and analyze their attention patterns using the same CMA framework. Compare metrics like circuit sparsity, layer-wise information flow, and reliance on early vs. late layers. If larger small models show emerging modularity (e.g., separate fact-processing and decision layers), it would challenge the claim that modularity is exclusive to LLMs. If not, it would reinforce the role of scale in enabling specialized components.\n5. Recommended citation: LogiCoT: Logical Chain-of-Thought Instruction-Tuning", "limitations": "The paper does not discuss potential societal implications of its findings, such as how insights into LLM reasoning circuits could be used to improve safety, bias mitigation, or adversarial robustness. While the focus is foundational, addressing these aspects would strengthen the work’s relevance to broader ML ethics discussions.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\nThe paper provides one of the first systematic mechanistic analyses of reasoning in large pretrained LLMs (up to 27B parameters).\nThe framework for circuit discovery (QUERY-based patching, CMA) offers a replicable methodology for analyzing reasoning in other LLMs. The emphasis on the first answer token as a critical latent reasoning point provides a focal point for future studies, as it captures the earliest stages of proof construction.\nCross-scale and cross-model comparisons reveal both conserved and scale-dependent mechanisms. \n\nWeaknesses:\nThe propositional logic problem studied is minimalistic (5 variables, 2 rules), which may not generalize to real-world multi-step reasoning tasks with greater syntactic/semantic complexity.\nThe analysis focuses almost exclusively on attention heads, sidelining the role of MLPs and lower-layer embeddings. For instance, MLP-0 is dismissed as a “nonlinear token embedding,” but its potential role in feature transformation is underexplored. Additionally, the 27B model analysis is “partial” due to computational constraints, leaving open questions about how its more parallelized circuit impacts reasoning efficiency or error modes.\nUnclear generalizability to other popular open models like Qwen.\nThe contrast with 3-layer models is valuable but underdeveloped. The small models are trained exclusively on the synthetic task, whereas LLMs are pretrained on diverse data, making direct mechanistic comparisons challenging. More analysis is needed to explain why task-specific training leads to intermingled reasoning, while general pretraining fosters modularity.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback, we are glad that you appreciate our methodology and mechanistic insights on the large and small models. Below are our responses to your comments and questions.\n\n> The propositional logic problem studied is minimal…\n\nWhile extending to longer problems is beyond the scope of this paper due to our constrained computational resources (longer context lengths coupled with the large models is too demanding for our computing resources), we did conduct two new experiments to understand how much the model reuses the identified subcircuits, under variations of our current setting.\n\n**_New experiment 1 - Prompt format variation_**. We conducted the following **new experiment for testing the robustness of the circuit across prompt formats**. We swapped the fact and rule sections in the prompt area. Now all the logic problems are presented in the form “Facts: … Rules: … Question:...”, for example:\n\n_Facts_: O is false. Q is true. I is true. _Rules_: I implies C. Q or O implies H. _Question_: what is the truth value of H? _Answer_: Q is true. Q or O implies H; H is true.\n\nWe found evidence that on **Gemma-2-9B, the circuit for predicting the first answer token stays almost exactly the same**! In particular, we find that it is the same heads exhibiting high IE scores in the activation patching sweep, with slightly different scores. Due to the time limit, we have not performed the sufficiency test (as described in Section 3.2 and Appendix B.5, B.8.3) thoroughly yet, so we cannot report DE measurements here, but we think even the _necessity-based evidence_ is quite interesting to present here! \n\n**_New experiment 2 - Rule-invocation subcircuit reuse_**. We conduct a further analysis on the second most important place in the model’s generated proof, the **first token for invoking the rule** – recall that the model needs to invoke the correct fact, then the correct rule to infer the correct truth value of the queried proposition variable. We found that in both Gemma-2-9B and Mistral-7B, **the _queried-rule locators_ and _decision heads_ again exhibit strong indirect effects at this proof position, indicating _reuse_ of functional sub-circuits in their proof writing**. \n\nWe will add the more detailed and rigorous version of these results to our updated paper.\n\n> “The contrast with 3-layer task-trained models mentions ‘intermingled reasoning’ but lacks detailed mechanistic comparison…”\n\nThank you for this excellent point. Due to the page limit we had to delay our mechanistic analysis of the 3-layer 3-head transformer to _Appendix D_ (with its backbone algorithm summarized in _Figure 29_) – we do see some major differences between how the small models and the LLMs solve the simple propositional problems. We will add further comparisons to our updated paper. For now, we highlight some interesting differences below.\n\nThe most pronounced difference is the _less interpretable and modular_ nature of the attention heads (especially their attention patterns) in the small transformer. Since the problem has many token positions which are “redundant” (e.g. “is”, the periods, spaces, etc.), the small transformer’s attention heads tend to make use of these positions in a much less predictable way than the large models. \n\nFor example, while we show, through a set of probing and ablation experiments, that when the logical-operator chain is queried, it is only at the _final_ layer of the attention heads that the model arrives at the right token to write down – meaning that they are effectively “_decision heads_” – their attention pattern statistics are fairly messy, scattered across token positions in the Rules section. This stands in great contrast to the _modularity_ of the attention head families and the _interpretability_ of their attention patterns found in the LLMs. A cautionary remark should be emphasized, however: such modularity is _not_ necessarily observable in _all_ LLMs. For instance, smaller and older LLMs such as Gemma-2-2B and the GPT2 models have very low proof accuracies, they might not even possess “reasoning circuits” as we see in the Gemma-2-($\\ge$9B) and Mistral-7B models.\n\n> Recommended citation: LogiCoT: Logical Chain-of-Thought Instruction-Tuning\n\nThank you for this reference, we will add it to our updated paper.\n\n**Discussion on future directions**. We sincerely appreciate your comments on the possible ways of extending the current analysis. Increasing the complexity of the propositional problems by increasing its length and involving nested operators is indeed a feasible direction. However, even without increasing the problems’ lengths (with heavier compute demands), we believe that there are meaningful directions to pursue. \n\nFirst, the _negation_ operator is an interesting one to study: its analysis might indicate whether certain “polarized” behaviors arise in the model, and whether the model reuses certain causal pathways when it needs to latently negate a (simple) clause. \n\nFirst-order logic problems are another possible extension direction of problem setting. We are especially interested in _how transformers latently bind quantifiers to variables, and how such binding affects the way they reason about sets_ – a foundational problem in how LLMs reason."}]

Author rebuttal:

Meta-review: The paper presents a mechanistic interpretability study of how large pretrained LLMs (Gemma-2-9B, Gemma-2-27B, and Mistral-7B) perform propositional logical reasoning. Using causal mediation analysis (CMA) and activation patching, the authors identify sparse modular circuits consisting of four functional families of attention heads (rule locators, rule movers, fact processors, decision heads). They show these circuits are faithful (recovering 94–98% of model performance), scale-dependent, and consistent across model families, contrasting them with small transformers trained from scratch that do not exhibit modular reasoning.

### Strengths of the paper:

- The reviewers (zZhs, rGZP, 9qMC) emphasize the novelty and significance of providing one of the first systematic mechanistic analyses of reasoning in large pretrained models, revealing both conserved and scale-dependent mechanisms.

- The work is technically sound and rigorous, with careful use of activation patching, sufficiency/necessity tests, and cross-model comparisons. The experiments are labor-intensive but yield meaningful and generalizable findings.

- The paper is clear and well written, situating itself within mechanistic interpretability and contributing a reusable framework for reasoning circuit analysis.

- The authors commit to open-sourcing the code, improving reproducibility and impact, and their rebuttal added further experiments confirming circuit robustness under prompt format variation and rule invocation.

### Weaknesses of the paper:

- The methodology relies primarily on attention heads, with limited exploration of the role of MLPs or embeddings.

- The reasoning tasks are restricted to short propositional logic problems, raising questions about generalization to more complex or real-world reasoning settings.

- The analysis of the 27B model is partial due to computational constraints.

- Some presentation aspects (e.g., pseudo-code, figure clarity, explanation of head categories) could be streamlined to reduce cognitive load for readers.

### Primary reasons for Accept (Spotlight)

The primary reasons for recommending Accept (Spotlight) are that the paper makes a principled, timely, and rigorous contribution to mechanistic interpretability. It advances our understanding of how LLMs implement symbolic reasoning by uncovering sparse, modular circuits with clear functional roles, verified through both necessity and sufficiency evidence. The cross-model comparisons and scale-dependent analyses provide robust insights of broad interest. While limited in scope to propositional logic, the findings establish a faithful methodology and framework that can be extended in future work. The paper’s novelty, methodological rigor, and clear empirical support make it a valuable addition to the NeurIPS community.

### Summary of the discussion and rebuttal

The authors provided detailed responses that successfully addressed key concerns. For R-b3jQ, they clarified methodological rigor by explaining how activation patching constitutes a valid causal intervention, addressed confounding concerns, and demonstrated circuit faithfulness (recovering 94–98% accuracy). They also agreed to streamline the presentation with pseudo-code and clearer definitions. For R-9qMC, they expanded experiments to show that functional sub-circuits (rule locators and decision heads) are reused when altering prompt formats and during rule invocation, alleviating concerns about overfitting to the first answer token. For R-rGZP, they provided additional causal importance experiments and clarified the functional role of decision heads, strengthening confidence in the interpretability of the circuits. Overall, the rebuttal and discussion period led to multiple upgraded scores, consensus among reviewers, and reinforced the contribution as a robust, reproducible, and high-impact study of reasoning circuits in LLMs.
Decision: Accept (spotlight)