Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models
Authors: Yukang Yang
Declan Iain Campbell
Kaixuan Huang
Mengdi Wang
Jonathan D. Cohen
Taylor Whittington Webb
Venue: ICML 2025 poster (2025)
Keywords: Mechanistic interpretability
reasoning
abstraction
large language models
Primary area: deep_learning->large_language_models
TL;DR: A three-stage architecture is identified that supports abstract reasoning in LLMs via a set of emergent symbol-processing mechanisms

Abstract: Many recent studies have found evidence for emergent reasoning capabilities in large language models (LLMs), but debate persists concerning the robustness of these capabilities, and the extent to which they depend on structured reasoning mechanisms. To shed light on these issues, we study the internal mechanisms that support abstract reasoning in LLMs. We identify an emergent symbolic architecture that implements abstract reasoning via a series of three computations. In early layers, *symbol abstraction heads* convert input tokens to abstract variables based on the relations between those tokens. In intermediate layers, *symbolic induction heads* perform sequence induction over these abstract variables. Finally, in later layers, *retrieval heads* predict the next token by retrieving the value associated with the predicted abstract variable. These results point toward a resolution of the longstanding debate between symbolic and neural network approaches, suggesting that emergent reasoning in neural networks depends on the emergence of symbolic mechanisms.

Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models
Yukang Yang'
Declan Campbell Kaixuan Huang'
Mengdi Wang'
Jonathan Cohen
Taylor Webb
Abstract
2022; Mirchandani et al., 2023), even rivaling human per- formance in some cases (Webb et al., 2023; Musker et al., 2024; Webb et al., 2025), but other studies have questioned these conclusions (Wu et al., 2024; McCoy et al., 2023; Lewis & Mitchell, 2024). In particular, LLMs appear to perform more poorly in some reasoning domains, such as mathematical reasoning (Dziri et al., 2024) or planning (Mo- mennejad et al., 2024); and, even in domains in which they have shown strong performance such as analogical reason- ing (Webb et al., 2023), some studies have questioned the robustness of these capabilities (Lewis & Mitchell, 2024).
Many recent studies have found evidence for emergent reasoning capabilities in large language models (LLMs), but debate persists concerning the robustness of these capabilities, and the ex- tent to which they depend on structured reason- ing mechanisms. To shed light on these issues, we study the internal mechanisms that support abstract reasoning in LLMs. We identify an emer- gent symbolic architecture that implements ab- stract reasoning via series of three computations. In early layers, symbol abstraction heads con- vert input tokens to abstract variables based on the relations between those tokens. In interme- diate layers, symbolic induction heads perform sequence induction over these abstract variables. Finally, in later layers. retrieval heads predict the next token by retrieving the value associated with the predieted abstract variable. These re- sults point toward resolution of the longstanding debate between symbolic and neural network ap- proaches, suggesting that emergent reasoning in neural networks depends on the emergence of symbolic mechanisms.
These conflicting findings raise the question: are LLMs gen- uinely capable of structured human-like reasoning, or are they merely mimicking this capacity by statistically approx- imating their training data? One way to answer this ques- tion is to look at the internal mechanisms that support this capacity. It has long been hypothesized that innate symbol- processing mechanisms are required to support human-like abstraction (Marcus, 2001; Dehaene et al., 2022; Wong et al., 2023). It has also been demonstrated that neural networks are capable, least in principle, of implementing some of the key properties of symbolic systems (Smolensky, 1990; Hummel & Holyoak, 2003; Kriete et al., 2013), and that the incorporation of these properties as architectural induc- tive biases can support data-efficient acquisition of abstract symbolic reasoning (Webb et al., 2021; Altabaa et al., 2024; Webb et al., 2024). It remains unclear, however, in the case of transformer-based LLMs that do not obviously possess such strong inductive biases, what mechanisms support their emergent capability for abstraction.
1. Introduction
Large language models (LLMs) have become the domi- nant paradigm in artificial intelligence, but there is ongoing debate concerning the limits and reliability of their capabili- ties. One major focus of this debate has been the question of whether they can reason systematically in an abstract or human-like manner. Many studies have documented im- pressive performance on various reasoning tasks (Wei et al.,
Here, we report evidence for a set of emergent sym- bolic mechanisms that support abstract reasoning in LLMSs. Specifically, we identify an emergent three-stage architec- ture consisting of the following mechanisms: 1) in early layers, symbol abstraction heads convert input tokens to abstract variables (i.c., symbols) based on the relations be- tween those tokens, 2) in intermediate layers, symbolic induction heads, perform sequence induction over these variables, and 3) in later layers, retrieval heads perform next-token prediction by retrieving the value associated with the predicted variable. These mechanisms capture two key properties of symbol processing. First, they operate over
'Department of Electrical and Computer Engineering, Prince- ton University, Princeton, NJ *Princeton Neuroscience Insti- tute, Princeton University, Princeton, NJ 'Microsoft Research, New York, NY. Correspondence to: Taylor Webb <tay- wor.w.ebbb @wwebb@gmail.co>....
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
ks
Output
ke
ixe
Retrieval heads
A
Symbolic
A
B
Induction
heads
A
A
Symbol
A
B
abstraction
ilege
iac
ptest
heads
iac
ilege
1ac
An
pteat
y1
pteat
\n
Input
Figure 1. Emergent Symbolic Architecture. Schematic depiction of the proposed three-stage architecture for abstract reasoning in language models. Example depicts architecture as applied to an algebraic rule induction task, involving either ABA or ABB rules instantiated with random tokens. Symbol abstraction heads identify relations between input tokens and, based on these relations, represent the tokens using consistent set of abstract variables aligned with their role in the relations. Symbolic induction heads perform sequence induction over abstract variables (i.e., they predict next variable based on the sequence observed the previous in-context examples). Retrieval heads predict the next token by retrieving the value associated with the predicted abstract variable.
representations of abstract variables that are invariant to the values that they are associated with. That is, the repre- sentation of abstract variable is the same regardless of which values instantiate that variable. Second, these mech- anisms utilize indirection, meaning that variables refer to content that is stored at different location than the variables themselves (i.e., they are pointers).
2. Approach
Figure/ depicts the proposed architecture as applied to an algebraic rule induction task involving sequences gov- erned by one of two identity rules, ABA or ABB. For each problem, two in-context examples were presented, followed by incomplete third example. The model was expected to generate the token that completes this third example. We instantiated rules using tokens randomly sampled from each model's vocabulary (ensuring that in-context examples within the same problem instance did not share tokens). We begin by presenting results from this rule induction task and the Llama-3.1 70B model, and then present results for additional tasks and models.
To identify and validate the presence of these mechanisms, perform experiments across three abstract reasoning tasks algebraic rule induction, letter string analogies, and verbal analogies and 13 open-source LLMs from four model families - GPT-2 (Radford et al., 2019), Gemma- 2 (Gemma Team, 2024), Qwen2.5 (Qwen Team, 2025), and Llama-3.1 (Dubey al., 2024) drawing on convergent evi- dence from a series of causal, representational,, and attention analyses. We find robust evidence for these mechanisms across all three tasks, and three out of four model families (Gemma-2, Qwen2.5, and Llama-3.1; with more equivocal results for GPT-2). Taken together, these results suggest that emergent reasoning in LLMS depends on structured, abstract reasoning mechanisms, rather than simple statisti- cal approximation. More broadly, these results suggest resolution to the longstanding debate between symbolic and neural network approaches, illustrating how neural networks can learn to perform abstract reasoning via the development of emergent symbol processing mechanisms.
We found that Llama-3.1 70B displayed 2-shot accuracy of 95% on the rule induction task. Although this task is relatively simple, especially when compared with some of the tasks that have been featured in recent debates over LLM reasoning (e.g., matrix reasoning (Webb et al., 2023)), it nev- ertheless offers paradigmatic case of relational abstraction. In particular, the use of completely arbitrary tokens ensures that the task cannot be solved by relying on statistical pat- terns specific to the tokens or associations among them, and for this reason it has previously been used to argue for the presence of pyrbol- mechanisms human cognition (Marcus et al., 1999), and to evaluate systematic generalization of abstract rules in neural networks (Webb et al., 2021). Accordingly, the ability to reliably solve this
2

---

Emergent Symbolic Mechanisms Support Reasoning in LLMS
task is already strongly suggestive of the presence of some form of symbol-processing. In the following sections, we describe specific mechanistic hypothesis for how symbol- processing might be carried out in this model.
first stage. This hypothesis is inspired by previous work induction heads, an emergent circuit that supports in- context learning in transformers (Elhage et al., 2021; Olsson et al., 2022). As originally formulated, this circuit performs a simple sequence induction mechanism: given a sequence that ends with a particular token, an induction head will look for previous instances of that token, and retrieve the token that succeeded it. Although this mechanism performs induction based only on in-context bigram statisties, sub- sequent work has identified heads that also compute more complex n-gram statistics (Akyürek et al., 2024). Here, we use the term 'induction' to refer to the more general process of predicting the next token based on in-context transition probabilities (i.e., beyond bigram statistics).
2.1. Symbol Abstraction Heads
Our hypothesis consists of three stages. In the first stage, input tokens are converted to symbolic representations. The inspiration for this hypothesis comes from the abstractor ar- chitecture (Altabaa et al., 2024), a variant of the transformer that implements a strong relational inductive bias (Webb et al., 2024). In that architecture, a modified form of at- tention (termed relational cross-attention is employed in which the values consist of a standalone set of learned em- beddings, rather than being conditioned on the input tokens as in standard self-attention. As result, the output of this attention operation is completely abstracted away from the identity of the input tokens, and instead only reflects the pattern of relations among those tokens (as encoded by the pattern of inner products between query and key embed- dings). These outputs can therefore be viewed as form of learned, distributed symbolic representations.
We hypothesize that symbolic variant of this mechanism is responsible for performing induction over sequences of symbols rather than literal tokens. We refer to the attention heads that carry out this mechanism as symbolic induction heads. Unlike standard induction heads, which operate over direct representations of the input tokens, symbolic induc- tion heads operate over representations of abstract variables (computed by symbol abstraction heads in previous layers). The output of symbolic induction heads is a prediction of the abstract variable associated with the next token. Em- pirically, we find that symbolic induction heads are distinct from standard induction heads (section 3.5).
Here, we hypothesize that an emergent form of this rela- tional attention operation is implemented by attention heads in early layers of the model. We refer to these heads as symbol abstraction heads. Concretely, the keys and query embeddings in these heads represent the input tokens, and the inner product between keys and queries represents the relations between these tokens. It is natural to interpret this operation as representing similarity relations (and this is the relevant type of relations in the rule induction task), but it is also possible for this operation to represent a broader class of relations (Altabaa & Lafferty, 2024) (as necessitated by some of the other tasks that we investigate). Importantly, hypothesize that the value embeddings in these heads do not carry information about the specific identity of the input tokens (i.e., they are invariant to the content of those tokens), but instead represent only their position. More precisely, we hypothesize that the value embeddings rep- resent the relative position of a token within an in co-contex example, this is precisely the information that's needed to compute the abstract variable associated with that token (e.g., the fact that the first token and the third token are the same in an ABA rule is precisely what determines that they share the same variable). Given that these conditions are met, the self-attention operation is equivalent to relational cross-attention (Altabaa et al., 2024), and the output of such an attention head will represent an abstract variable.
2.3. Retrieval Heads
Finally, in the third stage, we hypothesize that a separate mechanism is used to convert the abstract variables (sym- bols) to their associated tokens (values), by performing a simple form of retrieval. We refer to the attention heads that perform this retrieval operation as retrieval heads. The key and query embeddings in these heads represent abstract variables, and the value embeddings represent the corre- sponding input tokens. Retrieval heads perform the inverse of the relational attention operation performed by symbol abstraction heads. Given an input embedding representing an abstract variable (the prediction computed by symbolic induction heads in previous layers), this variable is matched with previous instances, and the associated token is retrieved. This be viewed able (i.e., a pointer to a particular location in memory) is used to retrieve the value associated with it (i.e., the data stored at that location).
form of indirection, wherein vari-
3. Results
3.1. Causal Mediation Analyses
2.2. Symbolic Induction Heads
We performed causal mediation analysis (Pearl, 2022; Meng et al., 2022; Wang et al., 2023; Todd et al., 2024) to isolate the hypothesized attention heads. In this analysis, embed-
In the second stage, we hypothesize that sequence induction is performed over the abstract variables computed in the
3

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
dings from one context are patched into another context. This approach can be used estimate the causal effect of embedding at particular layer, position, or attention head.
which was designed to test whether the activations repre- sent abstract variables, we expected that activation patching would convert the symbol for the token that completes the incomplete query, whereas for ctoken, we expected that activation patching would convert the literal token. See Section B.2 and Figures 5-7 for more details (including illustrated examples) on this analysis.
Our analysis had two conditions. In one condition, intended to isolate representations of abstract variables (i.e., sym- bols), we created two contexts in which the same token is associated with two different variables. Given a set of tokens A1, B,, AN By (where N 1 represents the number of in-context examples), we created one context cphstract that instantiated an ABA rule, and another context c2 oobetrect that instantiated an ABB rule (equivalent to BAA rule):
Figures 2(a) and 2(b) show the results of these causal medi- ation analyses when they were performed on both the aggre- gated attention head outputs and the MLP outputs (leaving the residual stream intact) at each sequence position and layer within the model. Consistent with our hypothesized architecture, the abstract causal mediation analysis (Figure 2(a)) revealed two distinct stages of processing, one in early layers of the model, with an effect that was largely concen- trated at the positions of the final item in each in-context example, and one in intermediate layers, with an effect that was concentrated at the final position in the sequence (i.e., the position at which the model must generate a completion to the query). These results are consistent with the hypoth- esized behavior (both in terms of specific positions within the sequence, and relative order across layers) of symbol abstraction heads and symbolic induction heads respectively. The token causal mediation analysis (Figure 2(b)) revealed later stage of processing that was also concentrated at the final position in the sequence, consistent with the hypothe- sized behavior of retrieval heads.
4 Aasstract AI, B1, AL, AN, BN abstroct B1, A1, A1, AN
Importantly, in this analysis, the final token in each in- context example, AN, is identical for both contexts, but it is associated with a different variable, based on its relations to the other tokens within the same in-context example.
We contrast this with another condition that is intended to isolate representations of literal tokens. Given the same set of tokens used for the previous condition, we created the following two contexts:
token A1, B1, A1 AN, BN
(3)
token A1, A1, ..... BN, AN
(4)
Next, we performed causal mediation analysis on the output of individual attention heads. To isolate symbol abstraction heads, we performed the abstract causal mediation analysis at the positions corresponding to the final item in each in- context example. To isolate symbolic induction heads, we performed the abstract causal mediation analysis at the final position in the sequence. To isolate retrieval heads, we performed the token causal mediation analysis at the final position in the sequence. The results (Figures 2(c)-2(e)) revealed a relatively sparse selection of attention heads, again conforming to the hypothesized three-stage structure.
In this condition, both contexts involve the same abstract rule, but the tokens used in the final example are swapped. We also perform version of both analyses using the ABB rule and average the results for the two rules.
Together, these analyses allow for a double dissociation be- tween representations of abstract variables (abstract causal mediation) vs. the tokens associated with those variables (token causal mediation). For each analysis, given two con- texts ei and c2, and pretrained lanuage model f() that outputs logits for all possible next tokens, we computed the causal mediation score developed by Wang et al. (2023):
3.2. Attention Analyses
We analyzed attention patterns to better understand the be- havior of the identified attention heads. Figure 3(a) shows the attention patterns for symbol abstraction heads. Our hypothesis predicts that attention should be directed from the third item in each in-context example to the first item for ABA rules (top), and should be directed from the third item to the second item for ABB rules (bottom). The results largely confirmed this hypothesis (the attention patterns for these positions are highlighted with red dashed lines). In- terestingly, the pattern became more focused for the second in-context example, suggesting that the symbol abstraction heads benefit from in-context learning.
= (f(c))leel1 f(ci) 1) - -(f(a)lueg= - -
(5)
where [() is the logit for answer y in context c1: [cci)[v] is the logit for answer y in the patched context cj, for which activations are patched from C2 to e₁ at specific layers, positions, and/or attention heads; Me, is the correct answer for context Cj; and yet is the expected answer for the patched context cj. Intuitively, this score reflects the ex- tent to which patching activations from context C2 to ej has the expected effects on the model's 's outputs. For cadstract
4

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
1.5
1.2
Caural Mediation Score
Layer
Causall Mediation
Layer
50
30
0.6
30
20
10
8 A
0x Ag
76
05
20
Vn
Ae 1n 20
M 4
A
02
Ag
A
0,
Ag
(a) Abstract Causal Mediation
(h) Token Causal Mediation
80
80
00
0.5
70
70
70
Causall Mediation Scare
Causal Mediation Score
Caucal Mediation Score
60
0.4
60
0.0
60
30
50
50
0.6
Layer
40
40
40
30
30
0.4
20
20
20
20
0.1
36
64
20
64
64
Head
Head
Head
(c) Symbol Abstraction Heads
(d) Symbolic Induction Heads
(c) Retricval Heads
Figure 2. Causal Mediation Results. (a) Abstract causal mediation effects at each layer and sequence position. X-axis shows the two contexts (eabstruct and c2 cabatruct) used for this analysis, aligned with each sequence position. (b) Token causal mediation effects each layer and sequence position, with corresponding ci Joken and c2 token contexts shown along X-axis. (c) Identification of symbol abstraction positions corresponding the last item in each of the two in-context examples. (d) Identification of symbolic induction hcads: abstract causal mediation effects for each attention head, for the final token the sequence. (e) Identification of retricval heads: token causal mediation effects for each attention head, for the final token in the sequence. Permutation testing performed to estimate the family-wise error rate across all scores in each plot, and scores were thresholded so that only scores significantly greater than zero (p 0.05) are shown.
heads: abstract causal mediation effects for each attention head, averaged
Figure 3(b) shows the results for symbolic induction heads. Our hypothesis predicts that attention should be directed from the final position the entire sequence (the separation token the end of the incomplete example) to positions of the final items in each in-context example, these the positions where the previous instances of the to-be- beeediccct abstract variable are located. The results confirmed this hypothesis. Again, the pattern was stronger for the second in-context example, consistent with a general effect of in- context learning.
Taken together, these results suggest that the model employs form of indirection, first computing the variable associated with the predicted token, and then using that variable as pointer to retrieve the token itself.
3.3. Representational Similarity Analyses
We also performed representational similarity analy- ses (Kriegeskorte al., 2008) to better understand the rep- resentations that were produced by each type of attention head. In this analysis, representations are modeled in terms of their similarity with one another. For each set of tokens A1, A Bx, we created four prompts, intended to dissociate representations of abstract variables (i.e., sym- bols) vs. literal tokens. The first two prompts were the the cabstract and c cabstract contexts used for causal mediation analysis, in which the same token plays differ- ent abstract roles. The other two prompts were based on abstract and e gbsttrct but the final instances of An and By were swapped. The resulting set of prompts predict one pattern pairwise similarity for abstract variables, and a
Figure 3(c)) shows the results for retrieval heads. Our hy- pothesis predicts that attention should be directed from the final position in the sequence to the positions correspond- ing to the tokens that will appear next (i.e., within the in- complete example). For ABA rules (top), we predict that attention should be directed to the first item in the exam- ple (corresponding the variable A), and for ABB rules (bottom) predict that attention should be directed to the second item in each example (corresponding to the vari- able B). This prediction too was confirmed by our analyses.
5

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
022
03
Ay (n
tn
Attention Score
Attention Score
Attemtion Score
Queries
Querles
Queries
A, 1n4, 8,
22
A, A,1,A
V.
A Antr Ai AA In Keys
As A AA A In Keys
Keys
8: &. A.A -,
0.2
d: la
0.3
Attention Score
Quertes
Queries
Querles
Attention
0.2
Attention
8, 8. Wn
6 -B84 161nA,
A, 8, th 1.
B, 1,, Keys
Keys
Keys
(a) Symbol Abstraction Heads
(b) Symbolic Induction Heads
(c) Retrieval Heads
Figure Attention Analysis. Analysis attention patterns (a) symbol abstraction heads, (b) symbolic induction heads, (c) retrieval heads. For each type of attention head, selected only heads with statistically significant CMA scores (shown in Figure2(c)- 2(e)), and a weighted average their attention patterns was computed using the causal mediation scores as weights. X-axis corresponds to keys (positions that attended to), Y-axis corresponds to queries (positions from which attention is directed). Top row depicts attention pattern for ABA problems, bottom row depicts attention pattern for ABB problems. Prompt templates are shown along each axis, with tokens aligned to their corresponding positions. Red dashed lines highlight positions of interest (discussed in text). Note that gegimming-of-sqqennee token is omitted.
different pattern of similarity for literal tokens.
Interestingly, although the output embeddings for symbol abstraction heads and symbolic induction heads primarily resemble the pattern predicted for abstract variables, they also show the diagonal bands predicted by the token simi- larity matrix, suggesting that they preserve some degree of token identity, and thus do not exclusively represent abstract variables. However, it is still possible for abstract variables to be represented in invariant manner in subspuce of these attention heads' outputs. To test this, we performed a decoding analysis which a decoder was trained to predict the abstract rule (ABA vs. ABB) for problems vnvolving one set of tokens, and then tested on problems involving a completely different set of tokens. This decoding analysis yielded nearly perfect generalization accuracy for both sym- bol abstraction and symbolic induction heads (>> 98% test accuracy, see Section C.7 for more details), indeed suggest- ing that abstract variables are invariantly represented in a subspace of the output embeddings for these heads.
Figure 4(a) shows the predicted pattern of pairwise similari- ties for representations abstract variables, with all pairs involving two instances of the variable A forming one block of high similarity, and all pairs involving two instances of B forming another block. By contrast, Figure 4(b) shows the predicted pattern of pairwise similarities for represen- tations of literal tokens. This pattern displays 3 diagonal bands, corresponding to pairs of the same token (regardless of whether this token assigned to the same variable).
Figure 4(c) shows the pattern of similarity observed for the output of symbol abstraction heads the third position in both in-context examples (averaged across the two exam- ples). The pattern closely resembles the abstract variable similarity matrix, indicating that the output of symbol ab- straction heads have representational structure similar to abstract variables. Figure 4(d) shows the pattern observed for symbolic induction heads at the final sequence position. This pattern also resembles the pattern predicted for abstract variables. Figure 4(c) shows the patten observed for retrieval heads at the final sequence position. This pattern closely resembles the token similarity matrix.
To gain a more precise understanding of the identified atten- tion heads, we also applied RSA the key, query, and value embeddings (Tables 2-3 and Figures 11-14). For symbol ab- straction heads, we found that queries primarily represented token identity, keys represented mixture of both tokens
6

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
Sinilarity
Similariy
Abstract Similarity Matrix
(b) Token Similarity Matrix
10
0.9
0.8
0.0
0.7
0.7
0.6
0.5
0.6
0.5
0.5
0.4
02
0.3
0.1
(c) Symbol Ahstraction Heads
(d) Symbolic Induction Heads
(c) Retrieval Heads
Figure 4, Representational Similarity Analysis. (a) Predicted pattern of pairwise similarity for representations of abstract variables. (b)
Predicted pattern for representations tokens. (c) Pairwise similarity for symbol abstraction head outputs, averaged across the third position in the two in-context examples. (d) Pairwise similarity for symbolic induction head outputs at the final sequence position. (e) Pairwise similarity for retrieval head outputs the final sequence position. For each attention head type, we computed weighted average of similarity matrices from all statistically significant heads, using the causal mediation scores weights.
and abstract variables, and values primarily represented the abstract variables. For symbolic induction heads, queries and keys primarily represented the relative position within each in-context example, while values primarily represented abstract variables. For retrieval heads, queries primarily represented abstract variables, keys represented mixture of both tokens and variables, and values primarily repre- sented the predicted token. These results further confirm the hypothesized mechanisms, namely that abstraction heads convert tokens to variables, symbolic induction heads make predictions over these variables, and retrieval heads convert variables back to tokens.
ablated. This performed for the full range of h 1...H heads in the entire model. We performed control experiment in which each head ablated in the previous experiment was replaced by the head in the same layer with the lowest causal mediation score. Addi- tionally, we implemented random baseline by ablating randomly selected attention heads. We measured the effects of these ablations in terms of the probability assigned the correct answer.
mediation
We found that these ablation experiments had dramatic effect for all three types of heads (Figures 15(a)-15(c)) In the ablation condition, the probability assigned to the cor- rect answer rapidly fell zero as more heads were ablated, whereas in the control conditions it was necessary ablate almost all attention heads have such an effect. These experiments confirmed that all three types of attention heads were both sufficient and necessary to perform the rule in- duction task.
3.4. A Ablation Analyses
The causal mediation analyses in section 3.1 demonstrate that the identified attention heads are causally sufficient, in sense that perturbing their outputs alters the model's responses in predictable manner. We also performed abla- tion analyses test whether these heads are necessary for the model perform the task.
3.5. Comparison with Induction Heads
For each type of attention head, we performed cumulative ablation analysis in which the heads with the top h causal
We investigated the relationship between symbolic induction heads and the standard induction heads identified in previous

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
work (Olsson et al., 2022). In that work, it was proposed that induction heads not only perform literal sequence induction, but may also perform fuzzy or abstract form of induction. This raises the question of whether symbolic induction heads are merely standard induction heads.
tors are first computed by symbol abstraction heads the level of individual in-context examples, and symbolic induc- tion heads are primarily responsible for aggregating them across in-context examples.
3.7. Evaluating Other Language Models
For each attention head, we computed the prefix matching score previously used to identify induction heads (Olsson et al., 2022), and compared this with the causal mediation score for symbolic induction heads. We found that these scores were very weakly correlated (r = 0.11, Figure 9a)). These results suggest that, despite the conceptual similarity between these two mechanisms, they are implemented by disjoint sets of attention heads.
To determine how widespread the identified mechanisms are, we applied our causal mediation analyses to 12 additional open-source language models of different sizes across four model families: GPT-2, Gemma-2, Qwen2.5, and Llama 3.1 (Figure 16). With the exception of the GPT-2 models, all tested LLMs displayed nearly perfect accuracy on the identity rules task with 10 in-context examples. Correspond- ingly, all of the LLMs that performed well on the identity rules task also displayed statistically significant scores for each type of attention head, and these heads were arranged in a three-stage hierarchy (Figure 16(b) and Figures 20- 22). By contrast, the GPT-2 variants did not show robust evidence for the presence of symbol abstraction heads (Fig- ure 16(b) and 18), consistent with their poor performance this task. These results further confirm the link between the identified mechanisms and abstract reasoning in LLMs, and suggest that symbolic mechanisms may only emerge at certain scales (in terms of model or training data size).
3.6. Comparison with Function Vectors
We also investigated the relationship between symbolic in- duction heads and function vectors (Todd et al., 2024), rep- resentations of an in-context task that are generated by a subset of attention heads. The symbolic induction heads identified in the present work have many similarities to the attention heads that generate function vectors, including: 1) they are found in intermediate layers of the model, 2) they primarily attend to the last item in each in-context example, and 3) they are causally implicated in in-context learning for relational tasks.
3.8. More Complex Reasoning Tasks
To address this, we computed the average indirect effect for each attention head, which represents measure of the extent to which its outputs constitute function vectors (Todd et al., 2024), and compared this with the causal mediation score for symbolic induction heads. This analysis revealed that these scores are indeed highly correlated (Figure 9(b); r = 0.86), suggesting that they are essentially the same set of attention heads. That is, the output of symbolic induction heads can be thought of as function vectors. This result provides novel perspective function vectors, suggesting that, where relevant, they can be used to implement sym- bolic forms of computation. This result also provides insight into the mechanism that computes function vectors, suggest- ing that it may be conceptualized as a form of sequence induction over abstract variables. Interestingly, this is con- sistent with recent findings that function vector heads often develop out of induction heads (Yin & Steinhardt, 2025).
Our results thus far have focused on a simple but paradig- matic case of abstract reasoning - algebraic rule induction. Do these results also extend to more complex reasoning tasks? To test this, we extended our causal mediation analy- ses to two other abstract reasoning tasks (Figure 23). First, we investigated letter string analogies involving either suc- cessor or predecessor relations (Hofstadter et al., 1995). Second, we investigated verbal analogies involving either antonym or synonym relations. We found that the same basic three-stage architecture was present for both of these tasks (Figures 24 and 25), suggesting that similar mecha- nisms are indeed involved in more complex abstract reason- ing tasks.
Interestingly, the causal mediation scores revealed com- plex pattern of within- and between-task correlations. For the identity rules task and letter string analogies, we found moderate to high correlations both at the within-task level (i.e., for different relations or rules in the same task; Table 4) and the between-task level (Table 5). However, for verbal analogies, the correlations (both within- and between-taske were lower or even negative. These results suggest that, although all three tasks employed the same high-level mech- anistic strategy, there was some specialization in the specific tasks and relations mediated by each attention head.
Additionally. we found that these heads are not the only mechanism involved in computing function vectors. Specif- ically, we found that symbol abstraction heads also play an important role. The previous analysis was based on function vector scores computed the the final position in the sequence. When function vector scores were instead computed based on the third item in each context-example (Figure 10), these scores were highly correlated with symbol abstraction heads (r 0.47), but not symbolic induction heads (r = 0.06). These results suggest that function vec-
8

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
4. Related Work
On the one hand, the emergent architecture identified here, that supports abstract reasoning via an intermediate layer of symbol processing, is strikingly at odds with characteriza- tions of language models as mere stochastic parrots (Bender et al., 2021) or 'approximate retrieval' engines (Wu et al., 2024). These results are also at odds with claims that neural networks will need innately configured symbol processing mechanisms in order to perform human-like abstract reason- ing (Marcus, 2001; Dehaene et al., 2022; Wong et al., 2023). On the other hand, these results can be viewed as a vindica- tion of longstanding claims that symbol-processing mecha- nisms of some form (whether they be innate or learned) are a necessary component supporting human cognitive abili- ties (Fodor& & Pylyshyn, 1988), insofar as they suggest that neural networks can acquire these abilities only by develop- ing their own form of symbol processing.
There is a rich history of work illustrating how various aspects of symbol processing might be implemented in neural networks. Work on the tensor product representa- tion (Smolensky, 1990) and manding-by-synchree mel & Holyoak, 2003) illustrated how dynamic variable- binding can be performed in neural networks. Kricte et al. (2013) demonstrated how indirection, the use of one vari- able to refer to another, can be implemented in a biologically plausible neural network. More recently, a series of studies illustrated how a relational bottleneck (Webb et al., 2024)-a strong inductive bias to perform relational processing can enable data-efficient learning of abstract reasoning capabili- ties in deep learning systems (Webb et al., 2021; Kerg et al., 2022: Altabaa et al., 2024). The primary contribution of our work, relative to these previous studies, is to demonstrate empirically that symbolic mechanisms can emerge in a large- scale neural network, and to illustrate how they operate to support abstract reasoning- Notably, the symbol abstraction heads identified in this work implement an emergent version of the abstractor architecture that was previously proposed to support relational learning (Altabaa et al., 2024)
(Hum-
It is interesting to consider the extent to which the identified mechanisms are truly emergent vs. dependent on innate as- pects of the model. The transformer architecture (Vaswani et al., 2017) does not obviously possess the strong relational inductive biases that characterize the abstractor (Altabaa et al., 2024) or other architectures designed to perform re- lational abstraction (Webb et al., 2024). However, trans- formers do have some inductive biases that seem relevant, including: 1) an innate mechanism for computing in-context similarity via the inner product between keys and queries, and 2) a form of indirection, in the sense that the keys and queries that are used to select information for retrieval are distinct from the values that are retrieved. In future work, it would be interesting to investigate the extent which these or other inductive biases contribute to the development of emergent symbol processing mechanisms.
There has also been much recent work investigating the in- ternal mechanisms that support various forms of abstract and structured task processing in large language models. This work has identified key primitives such as induc- tion heads (Olsson et al., 2022), function vectors (Todd et al., 2024), concept vectors (Opietka et al., 2025), binding IDs (Feng& & Steinhardt, 2024), and other mechanisms that play a role in relational processing (Merullo et al., 2023). Complementary work has also investigated how structured mechanisms emerge in smaller models trained on synthetic tasks (Wu et al., 2025; Grant et al., 2025; Al-Saeedi & Härmä, 2025; Tang al., 2025; Brinkmann al., 2024). We build on this previous work by identifying an integrated ar- chitecture that brings together multiple mechanisms. These include newly identified mechanisms - symbol abstraction and symbolic induction heads - that, respectively, carry out the processes of abstraction and rule induction needed to implement an emergent form of symbol processing that supports abstract reasoning in neural networks.
Finally, an important open question concerns the extent to which language models precisely implement symbolic pro- cesses, as opposed to merely approximating these processes. In our representational analyses, we found that the identified mechanisms do not exclusively represent abstract variables, but rather contain some information about the specific to- kens that are used in each problem. On the other hand, using decoding analyses, we found that these outputs contain a subspace in which variables are represented more abstractly. A related question concerns the extent to which human rea- soners employ perfectly abstract vs. approximate symbolic representations. Psychological studies have extensively doc- umented 'content effects", in which reasoning performance is not entirely abstract, but depends on the specific content over which reasoning is performed (Wason, 1968), and re- cent work has shown that language models display similar effects (Lampinen et al., 2024). In future work, it would be interesting to explore whether such effects are due to the use of approximate symbolic mechanisms, and whether similar mechanisms employed by the human brain.
5. Discussion
In this work, we have identified an emergent architecture consisting of several newly identified mechanistic primi- tives, and illustrated how these mechanisms work together to implement form of symbol processing. These results have major implications both for the debate over whether language models are capable of genuine reasoning, and for the broader debate between traditional symbolic and neural network approaches in artificial intelligence and cognitive science.
9

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
Impact Statement
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph,
N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly,
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
T., DasSarma, N., Drain, D., Ganguli, D., Hatfield- Dodds, Z., Hernandez, D., Jones, A., Kermion, J., Lovitt, I.,, Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A math- ematical framework for transformer circuits. Trans- former Circuits Thread, 2021. https:/transsfrmmer Set.iissppb/200///taaeeeeo///nae
References
Akyürek, E., Wang, B., Kim, Y., and Andreas, J. In-context language learning: architectures and algorithms. In Pro- ceedings of the st International Conference on Machine Learning, pp. 787-812, 2024.
Feng, J. and Steinhardt, J. How do language models bind en- tities in context? In The Twelfth International Conference on Learning Representations, 2024.
Al-Saeedi, A. and Härmä, A. Emergence of symbolic ab- straction heads for in-context learning in large language models. In Proceedings of Bridging Neurons and Sym- bols for Natural Language Processing and Knowledge Graphs Reasoning@ COLING 2025, pp. 86-96. ELRA and ICCL,, 2025.
Fodor, J. A. and Pylyshyn, Z.. W. Connectionism and cogni- tive architecture: A critical analysis. Cognition, 28(1-2): 3-71, 1988.
Gemma Team. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.001 2024.
Altabaa, A. and Lafferty, J. Approximation of relation functions and attention mechanisms. arXiv preprint 20Xiv:2402.088566 2024.
Grant, S., Goodman, N. D., and McClelland, L. Emergent symbol-like number variables in artificial neural networks. arXiv preprint 0rXiv:2501.06141 2025.
Altabaa, A., Webb, T. W., Cohen, J. D., and Lafferty. J. Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in transformers. In The Twelfth International Conference on Learning Repre- sentations, 2024.
Hofstadter, D. R., Mitchell, M., et al. The copycat project: A model of mental fluidity and maaloy--making Advances in connectionist and neural computation theory, 2(205- 267):2-3, 1995.
Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and trans- parency. pp. 610-623, 2021.
Hummel, and Holyoak, K. J. A symbolic-conneetionist theory of relational inference and generalization. Psycho- logical review, 110(2):220, 2003.
Kerg, G., Mittal, S., Rolnick, D., Bengio, Y., Richards, B., and Lajoie, G. On neural architecture inductive biases for relational tasks. arXiv preprint wrXiv:2266.05056, 2022.
Brinkmann, J., Sheshadri,
A., Levoso, and Bartelt, C. A mechanistic analysis of transformer trained on symbolic sulti-step reasoning task. arXiv preprint arXiv:2402. 1917, 2024.
V. Swoboda, P.,
Kriegeskorte, N., Mur, M., and Bandettini, P. A. Repre- sentational similarity connecin the branches of systems neuroscience. Frontiers in systems neumscience, 2:249, 2008
Dehaene, S., Al Roumi, F., Lakretz, Y., Planton, S., and Sablé-Meyer, M. Symbols and mental programs: hy- pothesis about human singularity. Trends in Cognitive Sciences, 26(9):751-766, 2022.
Kriete, T., Noelle, D. Cohen, J. D., and O' O'Reilly, R. C. Indirection and symbol-like processing in the prefrontal cortex and basal ganglia. Proceedings of the National Academy of Sciences, 110(41):16390-16395 2013.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama herd of models. arXiv preprint 2407: 2024.
Lampinen, A. K., Dasgupta, I., Chan, S. C., Sheahan, H. R., Creswell, A., Kumaran, D., McClelland, J. L.,, and Hill, F. Language models, like humans, show content effects on reasoning tasks. PNAS nexus, 3(7):pgae 2024.
Dziri, N., Lu, X., Sclar, M., Li, L.,, Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024.
Lewis, M. and Mitchell, M. Evaluating the robustness of analogical reasoning in large language models. arXiv preprint arXiv:2411.14211 2024.
10

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
Mareus, G. F. The algebraic mind: Integrating connection- ism and cognitive science. MIT press, 2001.
Smolensky, Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46( 46(1-2):159-2111 1990.
Marcus, G. F., Vijayan, S., Bandi Rao, S., and Vishton, P. M. Rule learning by seven- woven-month-old infants. Science, 283 (5398):77-80, 1999.
Tang, C., Lake, B., and Jazayeri, M. An explainable trans- former circuit for compositional generalization. arXiv preprint arXiv:2502. 2025.
McCoy, R. T., Yao, S., Friedman, D., Hardy, M., and Grif- fiths, T. L. Embers of autoregression: Understanding large language models through the problem they are trained solve. arXiv preprini arXiv:2309. 13638, 2023.
Todd, E., Li, M., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function vectors in large language models. In The Twelfth International Conference Learning Representations, 2024.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neu- ral Information Processing Systems, 35:1 35:17359-17372, 2022.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, Jones, L., Gomez, A. N., et al. Attention all you need. Ad- vances in neural information processing systems, 30(1): 261-272, 2017.
Merullo, J., Eickhoff, C., and Pavlick, E. A mechanism for solving relational tasks in transformer language models. 2023.
Wang. K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: circuit for in- direct object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations, 2023.
Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess,
D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng. A. Large language models as general pattern machines. In Conference on Robot Learning, pp. 2498-2511 PMLR, 2023.
Wason, P. C. Reasoning about rule. Quarterly journal of experimental psychology, 20(3):273-281, 1968.
Momennejad, I., Hasanbeig. H., Vieira Frujeri, F., Sharma, H., Jojie, N., Palangi, H., Ness, R., and Larson, Eval- uating cognitive maps and planning in large language models with cogeval. Advances in Neural Information Processing Systems, 36, 2024.
Webb, T., Holyoak, K. J., and Lu, H. Emergent analogi- cal reasoning in large language models. Nature Human Behaviour,
2023.
Webb, W., Sinha, I., and Cohen, J. Emergent symbols through binding in external memory. In The Ninth Inter- national Conference on Learning Representations, 2021.
Musker, S., Duchnowski, A., Millière, R., and Pavlick, E. Semantic structure-mapping in Ilm and human analogical reasoning. arXiv preprint arXiv:2406.13:3 13803, 2024.
Webb, T. W., Frankland, S. M., Altabaa, A., Segert, S., Kr- ishnamurthy, K., Campbell, D., Russin, J., Giallanza, T., O' O'Reilly, R.. Lafferty, J., et al. The relational bottleneck as an inductive bias for efficient abstraction. Trends in Cognitive Sciences, 2024.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., al. In-context learning and induction heads. arXiv preprint arXiv:2209.11880 2022.
Webb, W., Holyoak, K. J., and Lu, Evidence from counterfactual tasks supports emergent analogical reason- ing in large language models. PNAS Nexus, 4(g) 05 2025. ISSN 2752-6542 135
Opietka, G., Rosenbusch, H., and Stevenson, C. E. Ana- logical reasoning inside large language models: Con- cept vectors and the limits of abstraction. arXiv preprint arXiv:2503.cm 036666 2025.
doi: 10. 10.1093/pnasnexus/
Pearl, J. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pp. 373-392. 2022.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D.,, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2025.
Wong, I.,, Grand, G., Lew, A. K., Goodman, N. D., Mans- inghka, V. K., Andreas, J., and Tenenbaum, J. B. From word models to world models: Translating from natural language to the probabilistic language of thought. arXiv preprint arXiv:23066 2672, 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
Wu, Y., Geiger, A., and Millière, R. How do transformers learn variable binding in symbolie programs? arXiv preprint arXiv:2505.2089, 2025.
Wu, Z., Qiu, L.., Ross, A., Akyürek, E., Chen, B., Wang. B., Kim, N., Andreas, J., and Kim, Y. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 1819-1862, 2024.
Yin, K. and Steinhardt, J. Which attention heads matter for in-context learning? arXiv preprint arXiv:2502. 14010, 2025.
12

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
A. Code and Hardware
All code was written in Python using the TransformerLens and HuggingFace libraries. Experiments on Llama-3.1 70B and Qwen2.5 72B were conducted on two NVIDIA 80G H100 GPUs while experiments on other models of smaller sizes were conducted on single H100 GPU. All model weights are loaded in the bíloat format.
All code and data will be released at https://gidhub..cmm/uuuaaa M23LLLMS
B. Implementation Details
B.1. Rule Induction Task: Evaluating Task Performance
To evaluate performance on the rule induction task, we randomly selected English tokens from the LLama-3 vocabulary to form 2,000 prompts. We used the following 2-shot prompt format:
A3 B3"
B.2. Causal Mediation Analysis
Algorithm Causal Mediation Analysis
Input: context pair (c1,02); language model f(c) E RA, outputs the logits for all possible next tokens at the last position of prompt c; vocabulary size A; the i-th value of vector f(e): f() [i]; the correct answer for c1 ye, the expected answer for the patched context c: Mef.
1: For Cj. measure the difference between the output logit for yle and you i.e. Afe. = f(ci)luel f(c))loo)
2: Cache the internal activations after feeding 22 into the model.
Replace activations for selected model component (e.g., output of the entire attention block, or individual attention head, at particular layer and token position) in c1 with corresponding activations in c2. yielding patched context fe;-
4: Compute logit difference for patched context:
Af;; = f(ci)lue)) - (c1)
(6)
5: Compute causal mediation score:
=
Afe Afe, = - -ff f(e))
(7)
Output: Causal Mediation Score
s
Algorithm 1 explains the causal mediation procedure for given context pair (c1, We performed three separate versions of this analysis, each targeting one of the hypothesized attention head mechanisms, as described below.
Targeting Symbol Abstraction Heads To target the hypothesized symbol abstraction heads, we used the context pair (cghstract, o cabstract), ). In one version of this analysis, cohstract involved an ABA rule, and cg cahstraet involved an ABB rule, Importantly, the same tokens were used for both contexts, but with swapped variable assignments, such that the correct answer to complete both contexts was the same. We also performed version of this analysis in which c cafstract was an ABB rule and cohstract was an ABA rule. We used 100 prompts for each version of the analysis, yielding 200 prompts in total. Patching was performed at the position of the final token in each of the two in-context examples. See Figure 5 for an illustration and further explanation of this analysis.
Targeting Symbolic Induction Heads. To target the hypothesized symbolic induction heads, we performed the same analysis described above (for symbol abstraction heads), except that patching was performed at the final position in the sequence, i.e. the position at which the model expected to generate the missing token. See Figure 6 for an illustration of this analysis.
Targeting Retrieval Heads. To target retrieval heads, we used the context pair (ctoken, c2 dtoken). ). In one version of this analysis, zooenn involved an ABA rule, and cjoken involved the same ABA prompt, but with the position of the final two tokens swapped. We also performed a version of this analysis in which both token and of dooknn involved an ABB rule. Patching was performed the final position in the sequence. See Figure 7 for an illustration of this analysis.
13

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
Activations. We performed CMA on types of activations: 1) the output of the entire attention block (including both the attention head and MLP outputs) specific layer and token position, and 2) the output of specific attention head (hefore the additional projection matrix over concatenated head outputs) at specific layer and token position. The first analysis was intended to provide course -ganneed,, but comprehensive, account of where the causally relevant computations were located (i.e., which layers and token positions). The second analysis was intended to provide more fine-grained analysis (focusing on individual attention heads) of the causally relevant model components at those locations.
Permutation Testing. We performed statistical tests to identify significant heads for each head type. Specifically, we used permutation testing to model the null distribution for these analyses. For each attention head h. the average causal score over multiple context pairs measures the difference between the means of two groups, i.e., the logit difference for the original context Aʃe, and the logit difference for the patched context Afc- To model the null distribution, we randomly permuted these two quantities, such that there was a 50% chance they would remain the same. and a 50% chance Afe, would be swapped with fc; and vice versa. This was repeated 5,000 times. We then identified the threshold c for which there was a fame error rate of p < 0.05. That is, we identified the threshold for which there was a less than 5% chance that even a single attention head would have causal mediation score exceeding this threshold in the null distribution. We considered only heads with scores that passed this threshold to be statistically significant.
14

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
Context c1 (ABA rule)
ks
Oulput of retrieval heads
Output symbolic induction heads
Output of symbol abstraction heads
B
A
A
B
A
A
4
+
iac
ilege
iac
ptest
yi
ptest
Input
Patched context c1*
ixe
Output of retrivaal heads
(patching from c2 to c1)
Output symbolic induction heads
Output of symbol abstraction heads
B
1
iac
ilege
iac
ptest
yi
ptest
Input
Context c2
ks
Output of retrieval heads
(ABB rule)
Output symbolic Induction heads
Output of symbol abstraction heads
B
B
B
A
4
4
ilege
iac
iac
yi
ptest ptest
ixe
ks
Input
Figure Illustration of Causal Mediation Analysis targeting Symbol Abstraction Heads. In this example, context ei involves an ABA rule, and context cu involves ABB rule. Importantly, both contexts are formed from the same token sets, but with the tokens assigned to different variables, such that correct prediction e1 should therefore have effect model components that encode literal tokens (i.e., the retrieval heads), since the output of these components should be the same in both contexts. This intervention should, however, have an effect on model components that encode abstract variables, since these differ between the contexts. target symbol abstraction heads, we patched from cz cı the position of the final tokens in each in-context example, thus intervening on the abstract variable representation for these tokens (and therefore converting the representation of an ABA rule into representation ABB rule). In this example, the correct answer Yez for context C₁ is 'ks' (i.c., Av) ) and the expected answer Veę for the patched context cj is 'ixe' (i.c., Bx).
next token the same both contexts ("ks') ()ss" Patching from context
15

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
Context c1 (ABA rule)
ks
Output of retrieval heads
Output symbolic induction heads
Output of symbol abstraction heads
B
A
B
A
A
4
+
iac
ilege
iac
ptest
yi
ptest
Input
Patched context c1* (patching from c2 to c1)
ixe
Output of retrivaal heads
Oulput symbolic induction heads
Output of symbol abstraction heads
B
A
A
B
A
A
1
iac
ilege
iac
ptest
yi
ptest
Input
Context c2
ks
Output of retrieval heads
(ABB rule)
Output symbolic induction heads
A
B
B
B
A
Output of symbol abstraction heads
4
4
ilege
iac
iac
yi
ptest ptest
ixe
ks
Input
Figure Illustration of Causal Mediation Analysis targeting Symbolic Induction Heads. intervention targeting symbolic induction heads was the same the intervention targeting symbol abstraction heads, except that patching performed at the final position in the sequence, at which the model was expected to generate the token. This intervention should have an effect on process of predicting the abstract variable associated with the final token, but not the process of retrieving the final token itself.
16

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
Context c1 (ABA rule)
ks
Oulput of retrieval heads
Output symbolic induction heads
Output of symbol abstraction heads
B
A
A
B
A
A
4
4
+
iac
ilege
iac
ptest
yi
ptest
Input
Patched context c1*
ixe
Output of retrivval heads
(patching from c2 to c1)
A
Output of symbolic induction heads
Output of symbol abstraction heads
B
A
A
B
A
A
1
iac
ilege
iac
ptest
yi
ptest
Input
Context c2
ixe
Output of retrieval heads
(ABA rule with tokens swapped in last example)
Output symbolic Induction heads
Output of symbol abstraction heads
B
A
B
B
A
4
iac
ilege
iac
ptest
yi
ptest
ixe
ks
Input
Figure Illustration of Causal Mediation Analysis targeting Retrical Heads, example, both contexts ei and da involve an rule, but the order of the tokens in the final in-context examples swapped. This means that the correct answer differs between these two contexts, and therefore model components that encode literal tokens such as retrieval heads should affected by this intervention. Model components that encode abstract variables, however, should not be affected by this intervention. since the abstract rule in both contexts the same. To target Retrieval Heads, we patched from d2 to C₁ at the final position in the sequence. In this example, the correct Ve, for context is 'ks' (i.e., Ax), and the expected answer Vet for the patched context oj is 'ixe' (i.e., Bx).
17

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
B.3. Attention Analyses
For each individual attention head, the attention map was averaged over 1,378 prompts each for the ABA and ABB tasks, again limiting the analysis to prompts that the model answered correctly.
B.4. Representational Similarity Analyses
For one set of tokens [(An, B=) we built four different contexts, i.c.
A1, B1, A1. AN, BN
(8)
A1, B1, A1, ...... Bv, An
B1, A1, A1. BN,AN
B1, Ai..., AN, Bn
We randomly selected 40 token sets formed into the above contexts to measure the similarity of attention head activations at certain token positions. We then compared these similarity matrices with hypothesized similarity matrices based on either abstract variables or token identities. We applied this analysis to four separate attention head components: keys, values, queries, and outputs. Figure 4 shows the RSA on the output of significant heads for each head type.
For symbol abstraction heads, we measured query similarity and output similarity at the third item in the first two in-context examples, and keys and values at the first two items. For symbolic induction heads, we compared the values at the third item in the first two in-context examples and the outputs at the final position of the prompt. For retrieval heads, we used the queries and outputs at the final sequence position and keys and values at the first two items in the last incomplete query. These positions were chosen because they most directly targeted the hypothesized computations in the proposed three-stage architecture.
For the keys and queries of symbolic induction heads, tested two hypotheses. Inspired by the original induction head hypothesis (Olsson et al., 2022), the first hypothesis is that these embeddings represent the abstract variable at the previous position (just as the key and query embeddings in induction heads represent the token at the previous position). However, this mechanism is limited to performing induction via bigram statistics. An alternative hypotehsis is that these embeddings represent the relative position within an in-context example. This mechanism can support induction based on more complex n-gram statistics. Our original task, involving either ABA or ABB rules, does not enable us to distinguish between these two hypotheses, both hypotheses predict the same similarity matrix. To address this, we tested variant of this task in which the two hypotheses dissociate. Specifically, we used a task involving the following length-4 identity rules: AABA, ABCB, and ABCC. We performed RSA based the similarity of the embeddings at the third and fourth items in each in-context example.
B.5. Ablation Analyses
We randomly selected set of 40 prompts that do not overlap with the prompts used in causal mediation analyses. Starting from the heads with highest causal mediation scores, we gradually increased the number of ablated heads, and evaluated the effect on the probability of the correct answer. As a control, we performed the same analysis, but replaced each ablated head with the head in the same layer with the lowest causal mediation score. We also included a random baseline by randomly selecting heads to ablate. The results in Figure 15 are averaged over both ABA and ABB tasks.
B.6. Induction Heads
Following (Olsson et al., 2022), we used the prefix matching score as a measure for induction heads. We created a prompt involving a repeated sequence 50 random tokens. The prefix matching score is defined as the average attention score from each token to the tokens that directly follow the previous instance of the same token. We averaged results over 4 random seeds.
18

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
B.7. Function Vectors
As described in (Todd al., 2024), function vectors are aggregated over heads with high causal mediation score. The answers for each in-context example shuffled to form corrupted prompt and the causal indirect effect (CTB) is defined the recovery of the probability for correct answers after replacing the attention head output at the final position in the corrupted run with the average embedding across multiple prompts from the clean run. The average indirect effect (AIE) was taken over 50 prompts each for ABA and ABB tasks.
C. Additional Experimental Results
C.1. Comparison of Symbol Abstraction Heads, Symbolic Induction Heads, and Retrieval Heads
0.5
Symbol Abstractian Head Score
Symbolic Induction Head Score
0.8
Symbol Abstraction Head
0.2
02
0.4
0.1
0.1
0.2
0.0-
.
-0.2
-0.4 -02
0.0
0.2
0.4
0.6
0.8
1.0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
Symbolic
(a)
(b)
(C)
Figure 8. (a) Comparison of symbol abstraction head scores and symbolic induction head scores. (b) Comparison of symbol abstraction head scores and retrieval head scores. (c) Comparison of symholic induction head scores and retrieval head scores. Each dot represents single attention head, with all heads across all layers displayed. The scores for each type of attention head are largely orthogonal to each other, indicating that they form disjoint of attention heads.
19

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
C.2. Comparison with Induction heads and Function Vectors
We compared the symbolic induction head scores with prefix matching scores, used measure for standard induction heads (Olsson et al., 2022), and average indirect effects, used measure for function vectors (Todd et al., 2024), across all attention heads (Figure 9).
0.008
Prefix Matching Score
Average Indirect Effect
0.006
0.6
0.004
0.4
0.002
0.2
Symbolic Induction Head Score
Symbolic Induction Head Score
(a)
(b)
Figure (a) Comparison of symbolic induction heads standard induction heads. (b) Comparison of symbolic induction heads and function vector attention heads.
Apart from the original implementation in (Todd et al., 2024) on the embeddings at the final sequence position (Section B.7), we also computed the function vector scores by patching the activations at the position of the third item in each in-context example. Table and Figure 10 show the result of comparing function vector scores based on either of these two positions (last position in sequence vs. position of third item in each in-context example) to the scores for symbol abstraction heads and symbolic induction heads. The results indicate that symbol abstraction head scores are correlated with funetion vector scores the third item in each in-context example, while symbolic induction head scores are correlated with function vector scores the final position in the sequence.
Symbol Abstraction Head Score Symbolic Induction Head Score
Third-item shird-item--osition Function Vector Score Final-position Function Vector Score
0.47
0,06
0.06
0.86
Table Comparison between Function Vectors and Symbol Abstraction Heads/Symbolic Induction Heads. Function vector scores (average indirect effect) computed based embeddings either at the third item in each in-context example, the final position in the sequence (original implementation in (Todd et al., 2024)). Results reflect the correlation coefficient for comparisons between function vector scores and symbol abstraction head and symbolic induction head scores. Corresponding scatter plots shown in Fig. 10.
20

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
mrd-ite-ppssioon Function Vector Score
Third-tem-posttorn Function Vector Score
0.0015
0.0015
0.0010
0.0005
0.0005
-0.0005
-0.0005
001
0.0
0.1
0.2
0.3
0.4
0.3
-025
0.00 0.25
0.30
0.75 1.00
Symbol Abstraction Head Score (a)
Symbolic Induction Head Score (h)
Final pool-posioon Function Vector Score
Final-position Function Vector Score
0.006
0.006
0.802
0.002
0.000
0.000
Symbol Abstraction Head Score (c)
Symbolic Induction Head Score
(d)
Figure 10. Scatter Plots for Comparisons between Function Vectors and Symbol Abstraction Heads/Symbolic Induction Heads. Each dot represents single attention head, with all heads across all layers displayed. (a) Symbol abstraction head scores were correlated with function vector scores evaluated at the third items the in-context examples, but (c) not at the final position. head scores were highly correlated with function vector scores evaluated the final position, but (b) not at the third items in the in-context examples.
(d) Symbolic induction
21

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
C.3. Representational Similarity Analyses for Different Attention Head Components
Figures 4-14 show the hypothesized similarity matrices and the actual similarity matrices computed from the key, query, value, and output embeddings. For illustration purposes, used only 10 different token sets to compute the similarity in Figures 11-14.
10
10
98
Simil
09
(a) Abstract Similarity Matrix (Queries/Oupuuts
(b) Token Similarity Matrix (Queries/Outputs)
(c) Query Similarity Matrix
(d) Output Similarity Matrix
09
Simbsity
Sintark
07
07
06
(e) Abstract Similarity Matrix (f) Token Similarity Matrix (Keys/Values)
(g) Key Similarity Matrix
(h) Value Similarity Matrix
(Keys/Values)
Figure Representational Similarity Analysis for Symbol Abstraction Heads. (a), (b), (e). (f) are the predicted abstract or token
similarity matrices; (c), (d), (g), and (h) are the actual embedding similarity matrices.
08
92
Siitarry
02
(a) Abstract Similarity Matrix (Queries/Ouputt))
(b) Token Similarity Matrix (Queries/Outputs)
(c) Query Similarity Matrix
(d) Output Similarity Matrix
09
02
siitarty
05
00
(e) Abstract Similarity Matrix (Keys/Values)
(I) Token Similarity Matrix (Keys/Values)
(g) Key Similarity Matrix
Value Similarity Matrix
Figure 12. Representational Similarity Analysis for Retrieval Heads. (a), (b), (e), and the predicted abstract token similarity
matrices; (c), (d), (g), and (h) are the actual embedding similarity matrices.
22

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
1.
Similarty
Similarity
05
03
(a) Abstract Similarity Matrix (Outputs)
(b) Token Similarity Matrix (Outputs)
(c) Output Similarity Matrix
88
48
Similantyy
Similantty
0.6
A3
(d) Abstract Similarity Matrix
(c) Token Similarity Matrix Matrix(Valucs))
(f) Value Similarity Matrix
(Values)
Figure 13. Representational Similarity Analysis for Symbolic Induction Heads (Outputs and Values). (a), (b), (d), and (e) are the predicted abstract or token similarity matrices; (c) and (f) are actual embedding similarity matrices.
39
Sinilrtty
0.6
(a) Within-instance Position Similarity Matrix (Queries)
Previous Abstract Variable Similarity Matrix (Queries)
Query Similarity Matrix
Similariy
Similarity
(d) Within-instance Position Similarity Matrix (Keys)
(c) Previous Abstract Variable Similarity Matrix (Keys)
(f) Key Similarity Matrix
Figure 14. Representational Similarity Analysis for Symbolic Induction Heads (Queries and Keys). (a), (b), (d), and (e) are the predicted abstract or token similarity matrices; (c) and (f) are the actual embedding similarity matrices.
23

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
The Pearson correlation coefficients for all representational similarity analyses (representing the correlation between each hypothesized similarity matrix and the similarity matrix for specific attention head component) shown in Tables 2 and 3. These coefficients were computed based on the lower triangular parts of each similarity matrix (excluding the diagonal).
Symbol Abstraction Heads
Symbolic Induction Heads
Retrieval Heads
Embedding
Abstract RSA
Token RSA
Abstract RSA
Token RSA
Abstract RSA Token RSA
Queries
0.30
0.41
0.57
0.06
Keys
0.62
0.22
0.50
0.2.
Values
0.67
0.17
0.69
0.19
0.07
0.55
Outputs 0.56
0.24
0.69
0.03
0.11
0.35
Table RSA Summary Results. Pearson correlation hetween hypothesized similarity matrices and similarity matrices for specific model components. Each column represents the pearson correlation between hypothesized similarity matrix (abstract token RSA) and the similarity matrix for a particular component (query, key, value, or output embeddings) of a particular set attention heads. Output RSA results correspond to those shown in Figure 4, Similarity matrices for query, key, and value embeddings shown in Figures 11-13. For the key and query embeddings symbolic induction heads, we tested two additional hypotheses requiring a modified task (see Table 3).
Embedding Previous Abstract Variable RSA Within instance Position RSA
Queries
0.33
0.63
Keys
0.29
0.73
Table Symbolic Induction Head RSA Results. We tested two hypotheses for the key and query embeddings the symbolic induction heads. The first hypothesis (Previous Abstract Variable RSA) was that these embeddings represent the abstract variable the previous position. This hypothesis is based on the induction head mechanism, where keys and queries represent the previous token. The second hypothesis (Within-instance Position RSA) that these embeddings represent the relative position of token within the current in-context example. The results indicated that the key and query embeddings more closely matched the second hypothesis. Similarity matrices shown in Figure 14.
C.4. Ablation Results
1.0
1.0
1.0
Ablation Control Random
Ablation
Ablation
Control
Control
Random
Random
P(Corrett Answeri
P(Corret Answer)
P(Correct Answer)
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
3000 Number of heads ablated
2000
4000
1000
3000
Number of heads ablated
Number of heads ablated
(a) Symbol Abstraction Heads
(b) Symbolic Induction Heads
(c) Retrieval Heads
Figure Ablation Analyses. (a) Cumulative ablation symbol abstraction heads in decreasing order of causal mediation scores. Control condition involves ablation of heads in the same layers but with the lowest causal mediation scores. Random refers to randomly selecting heads ablation. (b) Cumulative ablation symholic induction heads. (c) Cumulative ablation retrieval heads, Error bars for random baseline reflect standard deviation over 10 runs.
24

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
C.5. Evaluating Other Language Models
Figure 16 summarizes the accuracy of 13 models identity rule tasks averaged over both ABA and ABB rules and the number of significant heads found in each model. Figure 17 shows the model performance for problems involving either ABA ABB rules separately. Figures 18-22 show the causal mediation results for this task in all models. Note that GPT-2 does not show robust evidence for the presence of symbol abstraction heads (Figure 19).
1.0
G772 Small (32AM)
Owen2.5 75
Grli MoJaa 15:5-1
Owenz2 148
Number Significant Heads
G77.2 Large
Owen2.5 328
0.8
Ulmm--1.
Germa 225
Accuracy
0.6
20
0.4
15
6212 (7719)
0L12XL (1.50
9nun25
Owee25
10
Gomms-2 23
Gamma 2
Jame 708
370
0.0
5
6
7
8
10
Number In-context Examples
Symbsit Infietion leed Retroot Hext
(a) Generation Performance
(b) Number of Significant Heads
Figure Results for different language models the identity rules task. Accuracy for each model function of the number of in-context examples. Error bars reflect 95% binomial confidence intervals. (b) Number of statistically significant symbol abstraction heads, symbolic induction heads, retrieval heads each model. We applied the same causal mediation analyses as in Section 3.1 using 10-shot prompts. Permutation testing performed to estimate the family-wise enror rate across all heads, and statistical significance was determined based on a threshold of < 0.05.
1.0
Accuracy
Accuracy
0.4
GPT Small (3248)
Owen2.5
GPT Small (124M)
Owen2.578
GPT-2 Nedium (1351)
Qwen2.5 148
08752 Medium (555M))
Owenz 145
GPT-2 Large (774M)
Qwen2.5 228
GPT-2 Large (774M)
Owen2.5
0.2
GPT-2 (1.581
Owen2.5 728
0.2
GP75 (3.58)
Owen2.5 228
Lsma
Gemma
Usma 58
0.0
0.0
1
3
5
6
8
10
1
3
6
8
10
Number of In-context Examples
Number of In-context Examples
(a) Rule ABA
(b) Rule ABB
Figure 17. Model Performance for Identity Rules Tasks, We tested model performance on 500 prompts cach for the ABA and ABB rules, across different numbers of in-context examples. Error bars reflect 95% binomial confidence intervals.
25

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
Symbol Abstraction Head
Symbolic Induction Head
Retrieval Head
10 11 12
11 12
12
2.4
Causa Mediation Score
10
Causal Mediation Score
1.8
Causa Mediation
Layer
1.2
0.6
2
10
11
12
10
6
11
12
Head
Head
(a) GPT-2 Small (124M)
Symbol
Symbolic
24
24
24
0.15
21
21
Causal Mediation Score
18
15
Mediation
15
Mediation
0.09
Layer
Layer
32
12
0.03
3
12
Head
Head
Head
(b) GPT-2 Medium (335M)
Symbol Abstraction Head
Symbalic Induction Head
Retrieval Head
36
32
0.04
32
3.6
32
Med lation Score
Causal Mediation Score
28
28
24
0.03
24
Mediation
1.8
20
Layer
Layer
16
16
1.2
12
12
12
10
12
14
16
18
20
=0
12
10
12
14
16
18
20
12
4
8
10
12
14
16
18
20
=0
Head
Head
Head
(c) GPT-2 Large (774M)
Symbal Abstraction Head
Symbolic Induction Head
Retrieval Head
48
4,8
2.4
42
42
42
Causal Mediation Score
Causal Mediation Score
36
36
1.8
30
Mediation
Layer
24
24
2.4
1.2
18
18
18
12
18
18
21
15
18
21
Head
Head
(d) GPT-2 XL. (1,5B)
Figure 18. Causal Mediation Results for GPT-2 Models. From left right, the heatmaps display significant symbol abstraction heads,
symbolic induction hcads, and retrieval heads. Permutation testing was performed to estimate the family- wise error rate, and statistical significance determined based threshold of 0.05.
26

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
8 10 11
Causal Mediation Score
Causal Mediation Score
8
Layer
Layer
Head
Head
(a) GPT-2 Small (124M)
Rule ABA
Rule ABB
21
0.24
Causal Mediation Score
21
Causal Mediation Score
18
18
15
Layer
Layer
12
12
0.12
8
10
12
14
1
6
10
12
14
16
Head
Head
(b) GPT-2 Medium (335M)
Rule ABA
Rule ABB
36
0.10
Causal Mediation Score
Causal Mediation Score
0.08
24
Layer
15 20
Layer
20
16
12
10
Head
Head
GPT-2 Large (774M)
Causal Mediation Score
42
Causal Mediation Score
36
0.08
36
30
Layer
Layer
24
24
18
0.04
18
12
12
0.02
15
18
25
Head
Head
(d) GPT-2 XL (1,5B)
Figure 19. Symbol Abstraction Heads in GPT-2 Models for ABA vs. ABB Rules. Permutation tests revealed (Left) very few significant abstraction heads for problems involving ABA rules, (Right) no significant abstraction for problems involving ABB rules.
27

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
Symbol Abstraction Head
Symbolic Induction Head
Retrieval Head
24.26
24
8.4
24
21
1.6
21
21
Causal Mediation Score
6.3
18
18
18
1.2
Causa Mediation
Mediation
15
15
Layer
15
12
12
0.8
2.1
1.6
Head
Head
Head
(a) Gemma-22 2 2B
Symbol Abstraction Head
Symbolic Induction Head
Retrieval Head
3942
3942
3942
6.8
5.6
35
35
Causal Mediats Mediation Score
1.2
30
30
25
Mediation
25
Layer
Causal Med
20
20
15
0.6
15
15
10
10
10
1.7
12
8
10
14
=0
8
12
14
16
=0
10
12
16
50
Head
Head
(b) Gemma-2 9B
Symbol Abstraction
Symbolic Induction Head
Retrieval Head
46
0.8
40
40
Causal Mediation Score
15
7.5
50
Mediation
30
Mediation
80
25
Layer
25
Layer
20
20
15
10
10
2.5
10
24
Head
Head
Head
(c) Gemma-2 27B
Figure 20. Causal Mediation Results for Gemma-2 Models. From right, heatmaps display significant abstraction heads,
symbolic induction heads. and retrieval heads. Permutation testing performed to estimate the family-wise rate, and statistical significance was determined based on a threshold of < 0.05.
28

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
Symbol Abstraction Head
Symbolic Induction Head
Retrieval Head
28
5.2
28
3.2
24
24
Cousal Mediation Score
21
2.4
10
Causal Mediation
18
Causal Media Mediation
13
Layer
15
Layer
15
2.6
1.6
12
12
9
12
15
18
21
28
50
12
15
18
21
24
28
12
15
18
21
24
28
85
Head
Head
(a) Qwen2.5 7B
Symbol
Symbolic
3.6
42
1.6
Causal Mediation Score
Causal Mediation Score
36
36
2.7
30
0.3
Causa Mediation
30
Layer
Layer
24
24
1.8
0.8
18
18
18
12
0.1
12
12
0.9
Head
Head
Head
(b) Qwen2.5 14B
Symbol Abstraction Head
Symbolic Induction Head
Retrieval Head
64
64
56
56
Causa Mediation Score
Causal Mediation Score
48
1.8
0.6
40
40
Layer
Layer
32
0.2
52
0.4
1.2
24
16
0.1
16
10
15
20
30
35
=0
10
15
20
25
30
35
40
10
15
20
25
30
35
40
Head
Head
Head
(c) Qwen2.5 32B
Symbal Abstraction Head
Symbolic Induction Head
Retrieval Head
80
0.4
70
70
0.8
Causal Mediation Score
60
60
50
Causal Mediation
39
Causal Mediation
0.6
Layer
Layer
40
0.2
40
20
50
50
0.4
20
20
20
0.2
10
16
24
48
56
16
24
48
16
32 Head
40
Head
(d) Qwen2.5 72B
Figure 21. Causal Mediation Results for Qwen2.5 Models. From left to right, the heatmaps display significant abstraction heads. symbolic induction heads, and retrieval heads. Permutation testing was performed to estimate the family-wise error rate, statistical significance determined based threshold of 0.05.
29

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
Symboo Abstraction Head
Symbolic Induction Head
Retrieval Head
32
7.2
28
Causa Mediation Score
28
Mediatio Score
Causa Mediation Score
24
24
24
5.7
0.9
20
20
20
Layer
Layer
10
3.6
0.6
12
12
12
16
24
28
5.0
12
20
24
28
50
12
16
20
24
28
32
50
Head
Head
Head
(a) Llama-3.1 8B
Symbol Abstraction Head
Symbolic Induction Head
Retrieval Head
00
118
70
70
1.6
Causal Mediation atiio Score
60
0.6
60
1.4
50
Causal Mediation
50
0.8
Causa Mediation
1.2
Layer
Layer
40
0.4
40
0.6
30
30
30
0.4
0.6
20
0.2
20
20
0.4
10
0.1
0.2
24
Head
Head
Head
(b) Llama-3.1 70B
Figure 22. Causal Mediation Results for Llama-3.1 Models. From left to right, the heatmaps display significant abstraction heads,
symbolic induction heads, and retrieval heads. Permutation testing was performed to estimate the family- wise error rate, statistical significance was determined based on threshold of < 0.05.
30

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
C.6. More Complex Abstract Reasoning Tasks
Letter string analogies
Successor
Predecessor
[ iip [ijk] [ [ijil
[ikillike
[ b c I I b
[cde [bde]
[wxy y ]
[xy
Verbal analogies
Synonym
Antonym
impoverished
poor
rich
poor
courageous
brave
fearful
brave
lazy
energetic
Figure 23. Letter String Analogy and Verbal Analogy Tasks. Letter string analogy problems involved either successor or predecessor relations. Verbal analogy problems involved either antonym or synonym relations. Similar to identity rules task, problems were constructed so enable the crcation of pairs of problems that involved the same completion tokens governed by different abstract rules. For instance, in the example verbal analogics, the answer to both problems is 'idle'. Holding the completion tokens constant, while varying the abstract rule, enabled the causal mediation analysis to dissociate attention heads involved in coding abstract variables vs. literal tokens.
Figure 23 depicts the letter string analogy and verbal analogy tasks. Similar to the design of the identity rules task, these tasks were designed in a manner that allowed a dissociation between representations of literal tokens vs. abstract variables.
C.6.1. LETTER STRING ANALOGY TASK
Task Setup. Letter string analogies were originally developed by (Hofstadter al., 1995). Here, adopted an imple- mentation similar to (Webb et al., 2023), who developed letter string analogy dataset for testing LLMs. Our dataset involved 2-shot problems, governed either by successor predecessor relation. For the successor relation, a set of three letters (e.g., jij k)) was transformed such that the final letter was converted into its alphabetic successor (e.g.. jij IJ). For the predecessor relation, a set of three letters (e.g.. [j IJ) was transformed such that the first letter was converted into its alphabetic predecessor (e.g., [ k IJ). This design made it possible to construct pairs of problems using the same set of tokens, where the first and third letters of the transformed set were the same in both the successor and predecessor problems. This was important for our causal mediation analyses, as described below. We created 500 prompts each for the predecessor and successor tasks, randomly sampling token sets for each pair of problems. We evaluated Llama-3.1 70B this task, counting responses correct only if all three letters in the completion set were correct. The model achieved 99.2% and 82.0% 2-shot accuracy for the successor and predecessor tasks respectively.
Causal Mediation Analyses. We performed causal mediation analysis to identify attention heads involved representing either abstract variables or literal tokens. By using the same set of tokens to create successor vs. predecessor problems, we were able to dissociate attention heads involved in these two processes. Specifically, the successor and predecessor problems for each pair had completion sets that shared the first and third letter. For instance, given the following instance of k] ji I]. and the following instance of a predecessor relation: [j k 1] [i I], the first and third letters
successor relation: in the completion sets of both problems are 'i' and 'I'. But despite involving the same tokens, these letters shared different relations with the corresponding letters in the first set. Specifically, in the successor relation, the first letter stays the same, and the third letter is transformed into its successor, whereas in the predecessor relation, the first letter is transformed into its predecessor, and the third letter stays the same. To target mechanisms involved in representing abstract variables, we therefore focused our causal mediation analysis on the first and third letters in the completion set.
31

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
To target symbol abstraction heads, we patched attention head outputs from corresponding successor predecessor problems (or vice versa) the position of the first and third letters in the completion sets for the in-context examples. For instance, in the example shown in Figure 23, patching was performed on the letters 'i' and 'I' in the second set of brackets the first row, and the letters "b' and "e" in the second set of brackets on the second row.
To target symbolic induction heads, we first appended the correct answer to the prompt. We then targeted the tokens that immediately preceded the first and third letters the completion set for the final (incomplete) query example. These are the token positions at which the model required to generate the first and third letters for the completion set. For instance, in the example shown Figure 23, the correct answer '[w x z]' was appended the successor problem, and the correct answer "[w y z]' appended to the predecessor problem. Patching brackets in the third row, and the second letter in the appended correct answer completion sets (e.g., 'x' in the successor problem, and "y" in the predecessor problem).
then performed from the open bracket in the second set of
The expected answer for both of these causal mediation analyses (i.e., the expected output given that the patching was effective) was obtained by applying the alternative rule to each problem. For instance, in the successor problem shown in Figure 23, the expected completion [[w yJ' was obtained by applying the predecessor relation, yiclding '[v x yJ' We used the sum of the causal mediation scores defined in Algorithm the positions preceding the first and third letters the completion set.
Finally, to target retrieval heads, we created variants of the problems described above in which the query example used completely different tokens. For instance, given the successor problem shown in Figure 23, with the following complete query example: [w x y] [w x z], we created an alternative problem with the following query example: [q s] [q r t]. The correct answer was appended both contexts, and patching was performed between two instances with the same rule (either successor or predecessor), at the positions preceding the three letters in the completion bracket, and including the first and second letters). The expected answer for this analysis is obtained simply by appending the tokens from the alternative context pair. For instance, when patching from 'lq r s] [q r t]' to '[w y] [w z]', the expected completion set is '[q t]'. We measured the causal mediation scores at the positions preceding all three letters in the completion set.
(i.e., beginning with the open
Figure 24 shows the results of this analysis in Llama-3.1 70B. The analysis was performed on a set of 100 prompts for each rule. Only prompts that the model answered correctly were used. Permutation testing was performed to identify attention heads with scores that were significantly greater than (p < 0.05). The results were averaged over both patching directions and predecessor sxieeeessor->uccccsoooo The results qualitatively similar to the pattern observed for the identity rules tasks. That is, symbol abstraction heads appeared primarily in early layers, symbolic induction heads appeared primarily in intermediate layers, and retrieval heads appeared primarily in later layers.
(i.e., prccessor-xpredeceeet
80
70
0.6
70
70
Causal Mediation Score
Causal Mediation Score
60
60
1.2
60
2.1
50
50
Causal Mediation
50
Layer
0.4
Layer
Layer
40
00
40
1.4
30
30
0.6
30
20
20
20
0.7
10
50
10
32
64
56
Head
Head
Head
(a) Symbol Abstraction Heads
(b) Symbolic Induction Heads
(c) Retrieval Heads
Figure 24. Letter String Analogy Results. Causal mediation for the letter string analogy task in Llama-3.1 70B. Permutation
testing was performed to estimate the family-wise error rate, and statistical significance was determined based on threshold of p < 0.05.
32

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
C.6.2. VERBAL ANALOGY TASK
Task Setup. We developed a dataset of four-term verbal analogies. Verbal analogies a common analogy format, involving two pairs of words that share the same relation. We created problems that involved either antonym synonym relation. The format used for presenting problems is shown in Figure 23. We created pairs of problems, one involving antonym relation, and the other involving a synonym relation, in which the correct answer was the same word. For instance, the pair "lazy: idle' involves a synonym relation, while the pair 'energetic : idle' involves an antonym relation, despite both pairs ending with the word 'idle'. To create this dataset, we prompted GPT-4 to generate sets of words, in which the first pair of words were synonyms, the second pair words were also synonyms, but the first and second pairs of words were antonyms with eachother (e.g., lazy, idle, energetic, and active). We then manually inspected each of words ensure that they involved the correct relation), and filtered any sets that were redundant. This yielded 46 unique word sets.
We then created 500 pairs of synonym and antonym problems by sampling from these word sets. Each problem involved two in-context examples and an incomplete query example, each created from one of the sampled word sets. We then evaluated 1lama-3.1 70B on this task. Accuracy was computed by comparing the logit assigned to the correct answer with the logit assigned an incorrect answer generated by applying the incorrect relation to the query word. For instance, an antonym problem with the query word 'energetic", the antonym relation), and the incorrect answer was 'active' (generated by applying a synonym relation). This procedure ensured that the model valid synonyms and antonyms for any given word). Llama-3.1 70B displayed 2-shot accuracy of 77. 77.0% and 88.4% for the synonym and antonym tasks respectively.
correct answer was 'idle' (generated by correctly applying
not penalized for generating unanticipated but correct answer (since there typically many
Causal Mediation Analyses. To identify attention heads involved in representing abstract variables, we patched from synonym to antonym problems in same pair, or from antonym to synonym problems. Each pair of problems had the
80
70
70
0.14
Causal Mediation Score
Causa Mediation Score:
50
0.10
Layer
40
0.04
Layer
30
0.05
20
0.02
0.02
Head
Head
Symbol Abstraction Heads (Antonym)
(b) Symbol Abstraction Heads (Synonym)
80
80
12
70
1.6
70
Causal Med asinn Score
Causal Mediation Score
60
0.4
40 50
50
50
Causal abon
Layer
Layer
Layer
40
40
0.6
30
30
0.2
20
20
0.1
10
10
16
24
32
x0
32
40
48
56
64
=0
48
56
64
Head
Head
Head
(c) Symbolic Induction Heads (Antonym) (d) Symbolic Induction Heads (Synonym)
(e) Retrieval Heads (Both)
Figure 25. Verbal Analogy Results. Causal mediation scores for the verbal analogy task in Llama-3.1 70B. Permutation testing was performed estimate the wase rate, and statistical significance determined based threshold p < 0.05. Because the scores for symbol abstraction heads and symbolic induction heads have low or even nogative within-task task correlations, we display separate figures for these heads for antonym vs. synonym relations.
33

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
same correct answer, despite involving different relations. To identify symbol abstraction heads, patching was performed at the position of the second word in each of the in-context examples (e.g., 'poor' and 'brave' in the example shown in Figure 23). To identify symbolic induction heads, patching was performed at the position of the final colon, i.e., the token at which the model was required to generate the completion to the query example. The expected output for both of these analyses was obtained by applying the incorrect relation to the prompt. For instance, when patching from the synonym example to the antonym example in Figure 23, the expected answer (given that patching was effective) was "active" (generated by applying the synonym relation to 'energetic"). To target retrieval heads, we patched between two problems involving the same relation, but using different sets of words. Patching was again performed the position of the final colon. We measure the causal mediation scores following Algorithm 1. For answers that involve multiple tokens (i.e., when single word was tokenized into multiple tokens), instead of computing the causal mediation scores based logits for individual tokens, we computed these scores based on joint log probabilities for each answer (i.e., summing the log probabilities over all tokens for each answer).
Figure 25 shows the results of this analysis in Llama-3.1 70B. The analysis was performed on a set of 100 prompts for each rule, and only prompts that the model answered correctly were used. The same permutation testing procedure as used the other tasks was applied, For symbol abstraction and symbolic induction heads, we found that causal mediation scores for the synonym task (identified by patching from synonym to antonym problems) and the antonym task (identified by patching from antonym to synonym problems) were weakly or even negatively correlated (see Table 4). We therefore present these results separately for antonym vs. synonym relations. For retrieval heads, we found that the scores for these relations were correlated, so results were averaged over both patching directions. The results were again consistent with the hypothesized three-stage architecture, with symbol abstraction, symbolic induction, and retrieval heads appearing in early, intermediate, and later layers respectively.
Attention Head Type
IR
LSA|
VA
Symbol Abstraction Heads
0.49 0.33
0.19
Symbolic Induction Heads
0.54
0.57
-0.63
Retrieval Heads
0.83 0.94 0.46
Table 4. Within-task Correlation Results. Results reflect Pearson correlation coefficient between causal mediation scores for two relation types within each task. Each columa represents a task. IR: identity rules (ABA vs. ABB); LSA: letter string analogy (successor vs. predecessor); VA: verbal analogy (antonym synonym). All correlation coefficients pass two-sided permutation tests 0.05).
Head Score Type
IR vs LSA| VA vs LSA
VA vs IR
Symbol Abstraction Heads
0.20
0.08
0.08
Symbolic Induction Heads
0.42
0.34
0.36
Retrieval Heads
0.43
-0.05
-0.10
Table Between-task Correlation Results. Results reflect Pearson correlation coefficient between causal mediation scores for two separate tasks. Each column represents correlation between two tasks. IR: identity rule; LSA: letter string analogy: VA: verbal analogy. All correlation coefficients pass two-sided permutation tests (p < 0.05).
34

---

Emergent Symbolic Mechanisms Support Reasoning in LLMs
1.0
MLP Causal Soore
Retrieval Heads
35
Signfiicnt Retrieval Heads (curnulaa atiuulllivv)
Signifiiant Retrieval
MLP Gausal Score
Sign sgnificamtt Retrivval H Heads (cumuulaai coumulaatiio attiv
30
0.8
MLP Causa Score
MLP Caural Score
1.25
20
20
-15
0.75
0.50
10
0.2
0.25
0.0
0.00
10
20
30
40
50
60
70
80
0
10
20
30
60
50
60
70
80
Layer
Layer
(a) Predecessor
(b) Successor
Figure 26. MLP Involvement in Computing More Complex Relations. For tasks that do not involve same/different relations, relational inference cannot be performed by retrieval alone. We tested the role that the MLP layer played in computing relational transformations for more complex relations. For the letter string analogy task, we found that causal mediation effects in MLP layers were primarily concentrated after the appearance retrieval heads. This result suggests that the model first retrieves the corresponding letter from the incomplete query example, and then uses subsequent MLP layers to apply the relational transformation (i.e., successor or predecessor).
C.7. Decoding Abstract Variables
We trained a one-layer linear probe to decode abstract variables (A B) based the outputs of symbol abstraction and symbolic induction heads. We used the heads identified in Llama-3.1 70B. For symbol abstraction heads, we used outputs corresponding to the position of the final item in each in-context example. For symbolic induction heads, we used outputs corresponding to the final position in the sequence. We created 500 prompts for each rule (ABA and ABB), each involving completely disjoint token sets. We split these into a training set of 200 prompts, a validation set of 100 prompts, and a test set of 200 prompts. Importantly, the prompts in the training and test sets involved completely disjoint sets of tokens. This ensured that any ability to generalize from the training to test set would be based on an invariant representation of the abstract variable, rather than the specific tokens used to create the problems. Table 6 shows the results of this analysis.
Symhol Abstraction Head] Symbolic Induction Head
Decoding Test Accuracy
68.6
98.10%
Table 6. Abstract Symbol Decoding Results. Results for linear decoder trained to predict abstract variables based on output of symbol abstraction and symbolic induction heads Llama-3.1 70B. Results show test accuracy. Training and test sets involved completely disjoint token sets.
C.8. Error Analyses
We compared RSA results for symbol abstraction and symbolic induction heads in correct vs. error trials. Specifically, we computed the Pearson correlation between the similarity matrices for these attention head outputs and the hypothesized similarity matrix based on abstract variables. The results (Table 7) indicated that abstract variables were better represented the correct trials than in the error trials.
Symhol Ahstraction Head Symbolic Induction Head
Correct Trials
0.52
0.63
Error Trials
0.47
0.49
Table 7. Comparisons of Abstract Variable RSA between Correct Trials and Error Trials. Scores reflect the correlation between pairwise similarity matrix of head outputs and expected similarity pattern defined by abstract variables. These scores are significantly higher in correct error trials for both symbol abstraction heads and symbolic induction heads (permutation test, p 0.05). RSA correlation scores are averaged across all the statistically significant heads for each head type.
35


Reviews:
[{"review_id": "9ABXEzhSnE", "reviewer": "Reviewer_ekki", "summary": "This paper investigates the internal mechanisms that support abstract reasoning in LLMs, focusing on the open-source model Llama3-70B. The paper makes a contribution to the ongoing debate about the reasoning capabilities of LLMs by proposing a novel three-stage symbolic architecture and providing empirical evidence to support its existence. \nThe authors test this hypothesis using a simple but paradigmatic algebraic rule induction task (ABA/ABB rules), where the model must predict the next token in a sequence based on in-context examples. Llama3-70B achieves 95% accuracy on this task, suggesting robust performance. Through a series of mechanistic interpretability techniques—including causal mediation analysis, attention pattern analysis, representational similarity analysis, and ablation studies—the authors identify and validate the roles of the proposed attention heads.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the thoughtful and detailed feedback. We present detailed responses below to address each of the issues raised. Throughout these responses, we refer to new results that can be viewed here: \n\nhttps://anonymous.4open.science/r/RB-F30A/13386.pdf\n\n## Additional models and tasks\nWe have tested **12 additional models** (Figures 1, 6, 8-10), including GPT-2 (small, medium, large, and extra large), Gemma-2 (2B, 9B, and 27B), QWEN-2.5 (7B, 14B, 32B, and 72B), and Llama-3.1 8B (along with our original tests on Llama-3.1 70B), and **two additional tasks** (Figures 2-4) including a letter string analogy task and a verbal analogy task. With the exception of GPT-2 (see ‘Testing smaller models’ below), we find that our results are qualitatively replicated across all of these models and tasks. These results strongly suggest that the identified mechanisms are a ubiquitous feature of abstract reasoning in sufficiently large language models.\n\n## Defining ‘symbol processing’\nWe define ‘symbol processing’ in terms of two key properties: \n- Symbolic representations are *invariant* to the content of the values that they are bound to. That is, the representation of the abstract variable ‘A’ should be the same regardless of which values this variable is assigned to. Although the abstraction and symbolic induction head outputs preserve some information about specific tokens, we find that they still contain a subspace that represents abstract variables in an invariant manner (see ‘Additional evidence for invariant symbolic representations’ below). \n- Symbol processing mechanisms employ *indirection*, meaning that variables refer to content that is stored at a different location than the variables themselves (i.e., they are pointers). In the identified architecture, the retrieval heads use the inferred symbols to retrieve the associated tokens from earlier positions in the sequence. That is, the symbol representations function as pointers that identify the address of the to-be-retrieved tokens.\n\nWe will add an explicit statement to the revised paper that clearly defines symbol processing in these terms, and add more discussion explicitly relating this definition to the results.\n\n## Additional evidence for invariant symbolic representations\nThe RSA results showed that the outputs of abstraction and symbolic induction heads preserve token identity to some extent, which may seem to suggest that they do not represent abstract variables in an invariant manner. However, it is possible for these heads to invariantly represent abstract variables within a specific subspace, while also representing token identity. To test for this possibility, we performed an experiment in which a linear decoder was trained to predict the abstract variable (A or B) based on the outputs of these heads, and tested on its ability to *generalize out-of-distribution to problems involving completely novel tokens*. The decoder achieved nearly perfect (>98%) accuracy for both types of heads (Table 3). These results demonstrate that a subspace exists in which abstract variables are represented in an invariant manner, despite the fact that concrete tokens are also represented in other regions of the embedding space (note also that tokens are represented much more weakly, as shown in Table 1). \n\n## Testing smaller models\nUnlike the larger models that we tested, none of the GPT-2 variants were able to reliably perform the task (Figure 1), and they did not show robust evidence for the presence of abstraction heads (Figure 1 and 6). These results were also not consistent between the two rule types (ABA vs. ABB, see Figures 5 and 7), again suggesting a lack of robustness. These results suggest that symbolic mechanisms may only emerge at certain scales (whether in terms of model or training data size). These results also strengthen our argument that abstract reasoning in language models depends on the presence of emergent symbolic mechanisms – language models that do not develop these mechanisms are not able to reliably solve abstract reasoning tasks.\n\n## Responses to other comments\n- Implications for designing more robust AI systems: One implication of these results is that the identified mechanisms could potentially be built directly into the architecture of language models. This has been explored to some extent in architectures such as the Abstractor (which implements a mechanism similar to abstraction heads), but could be taken further by incorporating the other mechanisms that we identify.\n- Impact of training regime and architecture: We agree that it would be very interesting to investigate how aspects of the training regime and model architecture contribute to the emergence of symbolic mechanisms. These experiments would necessitate very extensive resources, as they would involve training language models from scratch, but we consider this an important direction for future work.\nWe will add further discussion of these issues to the revised paper.", "claims_and_evidence": "1. The paper claims that the identified mechanisms perform symbolic processing, but the evidence is not entirely convincing. The representations produced by abstraction and symbolic induction heads are not perfectly abstract (they retain some token-specific information), suggesting that the mechanisms may not be fully symbolic.\n\n2. The paper does not provide a clear definition of what constitutes \"symbolic\" processing in the context of neural networks, which weakens the claim.\n\n3. The study is limited to a single task (ABA/ABB rule induction) and a single model (Llama3-70B). While the task is paradigmatic for studying relational abstraction, it is unclear whether the identified mechanisms generalize to more complex reasoning tasks or other models.\n\n4. The paper does not explore whether these mechanisms emerge in smaller models or models with different architectures, which would strengthen the claim that they are a general feature of LLMs.\n\n5 The claim that the findings resolve the symbolic vs. neural network debate is overstated. The paper does not provide sufficient evidence to conclude that the identified mechanisms are truly symbolic or that they generalize to other tasks and models.", "methods_and_evaluation_criteria": "1. The use of an algebraic rule induction task (ABA/ABB rules) is well-justified for studying abstract reasoning in LLMs. This task requires the model to identify and apply abstract rules (e.g., repetition or alternation) based on in-context examples, making it a paradigmatic case of relational abstraction. The use of arbitrary tokens ensures that the task cannot be solved by relying on statistical patterns, which is critical for isolating abstract reasoning capabilities.\n\nThis method has merits: The task is simple yet effective for probing the model’s ability to perform abstract reasoning.\nIt has been used in prior work to study systematic generalization in neural networks and symbol-processing in human cognition, providing a strong foundation for comparison.\n\nHowever, The task is relatively simple compared to more complex reasoning tasks, e.g., mathematical reasoning, planning, or analogical reasoning, which might limit the generalizability of the findings.\n\n2. The methods and evaluation criteria used in the paper are generally appropriate for the problem and application at hand.", "theoretical_claims": "No theoretical claims proposed in this paper.", "experimental_designs_or_analyses": "The causal mediation analysis is a rigorous and appropriate method for identifying the causal role of specific components (e.g., attention heads) in the model’s behavior. By patching activations from one context to another, the authors isolate the contributions of different heads to the model’s predictions. However, the analysis is somewhat limited to the specific task and model studied, and it does not rule out alternative explanations for the observed behavior (e.g., other emergent mechanisms).\nThe interpretation of the causal mediation scores depends on the assumptions of the analysis, which are not fully discussed in the paper. For example, it is unclear whether the patching procedure introduces any artifacts or biases.\n\nThe experimental designs and analyses in the paper are generally sound and appropriate for the problem and application at hand. The algebraic rule induction task is well-suited for studying abstract reasoning, and the mechanistic interpretability techniques (causal mediation analysis, attention pattern analysis, representational similarity analysis, ablation studies) provide multiple lines of evidence to support the proposed architecture.", "relation_to_broader_scientific_literature": "The contributions of this paper are related to 1. address the debate about the robustness and nature of these capabilities, with some studies questioning whether LLMs rely on structured reasoning or merely approximate it through statistical patterns; 2. mechanistic interpretability of Transformers", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n\n1 The submission identifies and characterizes new types of attention heads (abstraction heads, symbolic induction heads, retrieval heads) that support abstract reasoning in LLMs. It also provides a detailed, mechanistic account of how these heads work together to perform a form of symbol processing.\n\n2. It contributes to the symbolic vs. neural network debate by demonstrating how neural networks can develop symbol-like processing capabilities without explicit architectural biases. It uses a combination of causal mediation analysis, attention pattern analysis, representational similarity analysis, and ablation studies to validate the proposed mechanisms.\n\n3. It builds on prior work in emergent reasoning, systematic generalization, and mechanistic interpretability, while providing new insights into the role of function vectors and induction heads.\n\nWeakness:\n\n1. The study focuses on a single, relatively simple task (ABA/ABB rule induction) and a single model (Llama3-70B), raising questions about whether the findings generalize to more complex reasoning tasks or other models.\n\n2. The paper claims that the identified mechanisms perform symbolic processing, but the evidence is not entirely conclusive. The representations produced by abstraction and symbolic induction heads are not perfectly abstract (they retain some token-specific information), which weakens the claim.\n\n3 The paper does not thoroughly rule out alternative explanations for the observed behavior, such as statistical approximations.", "other_comments_or_suggestions": "1. Could the authors clarify the definition of \"symbolic\"? The paper frequently uses the term \"symbolic\" to describe the identified mechanisms, but it is not always clear what this means in the context of neural networks. Providing a clearer definition (e.g., distinguishing between discrete, rule-based symbols and distributed, approximate symbols) would strengthen the paper’s claims and help readers understand the nature of the proposed mechanisms.\n\n2. The paper could expand its discussion of the broader implications of the findings for AI and cognitive science. For example, how might these insights inform the design of more robust and interpretable AI systems? How do these mechanisms compare to human cognitive processes?\n\n3. To strengthen the generalizability of the findings, consider testing the identified mechanisms on a wider range of tasks (e.g., mathematical reasoning, planning, or analogical reasoning) and models (e.g., smaller or larger LLMs, models with different architectures).\nExplore whether these mechanisms emerge under different training regimes or datasets, which would provide insights into their dependence on specific training conditions.\n\n4. The paper does not thoroughly rule out alternative explanations for the observed behavior, such as statistical approximations. Including additional experiments or analyses to address these possibilities would make the findings more robust.\n\n5. The paper can also include a more detailed discussion of its limitations, such as the simplicity of the task, the focus on a single model, and the potential dependence on specific architectural features of the transformer. Acknowledging these limitations would provide a more balanced perspective on the findings.", "questions_for_authors": "See the above questions and comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pZ6WKgLEy3", "reviewer": "Reviewer_8kkE", "summary": "The paper studies the internal mechanisms of a Llama3-70B on an in-context learning task. Specifically, they study an abstract reasoning task in which the model is given multiple demonstrations of the form ABA or ABB, where A and B correspond to randomly selected tokens, and on the final example the model has to predict either A or B. \n\nThe authors identify a combination of multiple attention heads that are causally responsible for solving the task: (1) *Abstraction heads* extract relational information about the input tokens in each demonstration, (2) *Symbolic induction heads* perform induction over the relational information, (3) *Retrieval heads* predict the subsequent token by retrieving the value associated with the extracted relational variable. \n\nThey causally verify their observations using activation patching experiments. Specifically, they verify the existence of token-independent, abstract variables containing relational information. Additionally, they analyse attention patterns and compare the similarity of attention head outputs to evaluate whether they match their expectation. \n\nFinally, they contrast induction heads as identified in Olsson et al. (2022) with the symbolic induction heads identified in their setting. Interestingly, they find that for symbolic attention heads the prefix matching score does not correlate with the causal mediation score, suggesting that these are different from induction heads. Instead, those heads appear to be involved in the creation of function vectors (Todd et al., 2023).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the thoughtful and detailed feedback. We present detailed responses below to address each of the issues raised. Throughout these responses, we refer to new results that can be viewed here: \n\nhttps://anonymous.4open.science/r/RB-F30A/13386.pdf\n\n## Additional related work\nThank you for bringing these additional studies to our attention. We agree that they are highly relevant, and will cite them in the revised paper. We include below a discussion of the relationship to our results:\n- Al-Saeedi and Harma identify a type of attention head that they refer to as ‘symbolic abstraction heads’ in smaller transformer models trained from scratch on identity rule tasks. Based on the attention patterns that they present, these heads seem to correspond to what we refer to as ‘symbolic induction heads’. Relative to that work, our study provides many complementary lines of evidence (RSA, attention analyses, ablations, causal mediation), identifies two additional types of heads, and demonstrates the involvement of these mechanisms in large pre-trained language models. However, the setting studied by Al-Saeedi and Harma provides a complementary line of evidence that may be especially useful in future work investigating the contribution of training regime and architecture (which requires training models from scratch).\n- The other studies investigate different types of reasoning, namely logical reasoning (Dutta et al.) and planning (Brinkmann et al. and Saparov et al.). One difference with our study is that language models have generally not been found to be very good at performing these types of reasoning reliably, whereas the analogical reasoning and rule induction setting that we investigate in our work is one where language models have shown stronger performance. This provides an opportunity to understand how language models are solving the types of problems that they seem to be able to solve in a relatively robust manner.\n\nWe will add further discussion of these issues to the revised paper.\n\n## Analysis of failure cases\nWe agree that it is important to understand how the identified mechanisms contribute to the model’s failures as well as successes. To address this, we looked at how the RSA results differed between correct and error trials. We found that the outputs of abstraction heads and symbolic induction heads represented abstract variables more precisely (had a higher correlation with the abstract RSA matrix) in correct vs. error trials (Table 5). However, the effect size of this difference was very small for abstraction heads, suggesting that the effect was driven primarily by differences in the symbolic induction heads. One interpretation of these results is that abstraction heads correctly encode symbols on error trials, but symbolic induction heads do not successfully aggregate these results across in-context examples, perhaps due to interference from other heads. We will include these results in the revised paper.\n\n## Additional results\nWe have also included a number of additional results in the rebuttal file, including results for **12 additional models** (GPT-2 small, medium, large, and extra large; Gemma-2 2B, 9B, and 27B; QWEN-2.5 7B, 14B, 32B, and 72B; and Llama-3.1 8B), **2 additional tasks** (letter string analogies and verbal analogies), more detailed analyses of the identified attention heads (looking at the representations formed by keys, queries and values), and further tests of symbolic invariance.", "claims_and_evidence": "The experiments are generally supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The evaluation criteria, e.g. the causal mediation score for the activation patching experiments, make sense for the problem at hand.", "theoretical_claims": "The paper does not make any theoretical claims.", "experimental_designs_or_analyses": "I did check the soundness and validity of the experimental design and analyses; most importantly, the activation patching experiments, representational similarity analyses as well as the comparison with induction heads and function vectors.", "relation_to_broader_scientific_literature": "The paper fits well into the research direction aimed at (mechanistically) understanding reasoning in language models. In contrast to some prior work, this paper studies an in-context reasoning task in which function vectors (Todd et al., 2024) seem to emerge. While Todd et al. (2024) introduce the notion of abstract computations being represented in simple vectors in language models, they do not mechanistically study how any of those emerge. Thus, this paper is an in-depth study of how one specific function vector emerges in a large language model.", "essential_references_not_discussed": "There have been a number of papers that mechanistically study some form of reasoning in language models. This includes works such as: \n- A. Al-Saeedi and A. Harma, ‘Emergence of symbolic abstraction heads for in-context learning in large language models’, in Proceedings of Bridging Neurons and Symbols for Natural Language Processing and Knowledge Graphs Reasoning @ COLING 2025, 2025, pp. 86–96.\n- J. Brinkmann, A. Sheshadri, V. Levoso, P. Swoboda, and C. Bartelt, ‘A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task’, in Findings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 4082–4102.\n- S. Dutta, J. Singh, S. Chakrabarti, and T. Chakraborty, ‘How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning’, Transactions on Machine Learning Research, 2024.\n- A. Saparov et al., ‘Transformers Struggle to Learn to Search’, in The Thirteenth International Conference on Learning Representations, 2025.\n\nGiven the relevance of these works, it would be valuable to contextualize this study within their findings and highlight how it builds upon or differs from their approaches.", "other_strengths_and_weaknesses": "Overall, I think this is solid work aiming at mechanistically understanding in-context reasoning in language models. It builds upon existing work in the field and studies reasoning in an original setting with interesting results and findings. Notably, the authors study this task in Llama3-70B, which presents additional challenges due to its size. Despite this, the paper remains clear and easy to follow.\n\nThe main weaknesses are the limited engagement with prior work studying similar settings, and at times a lack of depth in the analyses. For example, it would have been interesting to study failure cases of the model.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XpaTi9UsiT", "reviewer": "Reviewer_yD8H", "summary": "This paper investigates the mechanisms behind how Large Language Models (LLMs) perform two simple abstract reasoning tasks related to algebraic identity rules (left and right). They identify three types of attention heads: abstraction heads, symbolic induction heads, and retrieval heads, which are implicated in performing the task. The attention patterns match the predicted values very closely, and show evidence that LLMs do indeed possess the capability to do symbolic processing to some extent.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the thoughtful and detailed feedback. We present detailed responses below to address each of the issues raised. Throughout these responses, we refer to new results that can be viewed here: \n\nhttps://anonymous.4open.science/r/RB-F30A/13386.pdf\n\n## Additional models and tasks\nWe have tested **12 additional models** (Figures 1, 6, 8-10), including GPT-2 (small, medium, large, and extra large), Gemma-2 (2B, 9B, and 27B), QWEN-2.5 (7B, 14B, 32B, and 72B), and Llama-3.1 8B (along with our original tests on Llama-3.1 70B), and **two additional tasks** (Figures 2-4) including a letter string analogy task and a verbal analogy task. With the exception of GPT-2, we find that our results are qualitatively replicated across all of these models and tasks. These results strongly suggest that the identified mechanisms are a ubiquitous feature of abstract reasoning in sufficiently large language models.\n\n## Direct analysis of key, query, and value embeddings\nTo gain a more precise understanding of the identified attention heads, we have now performed additional analyses applying RSA to the key, query, and value embeddings (Tables 1-2 and Figures 11-14). For abstraction heads, we found that queries primarily represented token identity, keys represented a mixture of both tokens and abstract variables, and values primarily represented the abstract variables. For symbolic induction heads, we found that queries and keys primarily represented the relative position within each in-context example, while values primarily represented abstract variables. For retrieval heads, we found that queries primarily represented abstract variables, keys represented a mixture of both tokens and variables, and values primarily represented the predicted token. These results further confirm the hypothesized mechanisms, namely that abstraction heads convert tokens to variables, symbolic induction heads make predictions over these variables, and retrieval heads convert symbols back to tokens.\n\n## Additional analyses on relationship to function vectors\nIn the original submission, we showed that function vector scores and symbolic induction head scores are very highly correlated, indicating that symbolic induction heads are responsible for producing function vectors. We have now performed additional analyses that suggest a more complex interpretation (Table 4). Specifically, when function vector scores are computed based on the final position in the sequence, these scores are highly correlated with symbolic induction heads, but not with abstraction heads. In contrast, when function vector scores are computed based on the final item in each context-example, these scores are highly correlated with abstraction heads, but not symbolic induction heads. These results suggest that function vectors are first computed by abstraction heads at the level of individual in-context examples, and symbolic induction heads are primarily responsible for aggregating them across in-context examples.\n\n## Related work on relationship between induction heads and function vectors\nThank you for bringing the paper from Yin et al. to our attention. We agree it is highly relevant and will make sure to cite it in our revised paper. It is especially interesting that Yin et al. find evidence that function vector heads evolve from induction heads. Our results suggest an explanation for this finding: function vector heads (i.e. symbolic induction heads) can be viewed as performing induction over symbolic (rather than literal) inputs.\n\n## Additional related work\nThank you for pointing out the additional related work from Liu et al. and Song et al. We agree that these studies are also relevant, and will make sure to cite them in the revised paper.\n\n## Relationship to previous-token heads\nIt is unlikely that the abstraction heads are performing a function similar to previous-token heads. Previous-token heads copy the token from the position t-1 into the residual stream at position t. By contrast, abstraction heads are responsible for computing *which* token (t-1 or t-2) is related to the token at position t, and what that relationship is. That is, abstraction heads do not copy the token from positions t-1 or t-2. They compute whether the token at position t is related to the token at position t-1 or t-2, and what that relationship is.", "claims_and_evidence": "While the setup and experiments are simple, they are thorough and well thought out. I found the claims well-supported by the evidence.", "methods_and_evaluation_criteria": "There are no benchmarks used here, just a simple dataset. But the proof-of-concept of symbolic processing is sufficient for the goal of the paper.", "theoretical_claims": "There are no theoretical claims in the paper.", "experimental_designs_or_analyses": "The experiments are well-designed and nice counterfactuals are used to elicit different responses from the network to study information flow/processing.", "relation_to_broader_scientific_literature": "This work attempts to show that LLMs trained on natural text corpora are capable of developing symbolic processing mechanisms that reason over abstract concepts. I am not aware of previous work that has studied this specifically, though the original induction heads thread from Anthropic did suggest that induction heads may be able to process both literal and abstract relationships. It is nice to see this in a concrete, though simple form. \n\nI will also point out (though do not expect the authors to have been aware of/cited it due to its release date) a recent paper that attempts to clarify the difference between induction heads and function vector heads [1], showing that literal copying and few-shot ICL performance are mediated by different sets of heads.\n\n___\n\n[1] Yin, et al. Which Attention Heads Matter for In-Context Learning? 2025. (https://arxiv.org/abs/2502.14010)\n\n[2] Liu, et al. Transformers Learn Shortcuts to Automata. ICLR 2022. (https://openreview.net/forum?id=De4FYqjFueZ) \n\n[3] Song, et al. Out-of-distribution generalization via composition: a lens through induction heads in Transformers. (https://arxiv.org/pdf/2408.09503)", "essential_references_not_discussed": "The literature awareness of this paper is great! Here are a couple works that might help contextualize the contributions of the paper such as [2] who study what algorithms transformers trained on symbolic data settings such as algebraic groups learn, or [3] which study induction heads for \"symbolized\" language reasoning, though I think their may still fit the standard token-induction and not symbolic induction studied here. However, I think the work already cited is pretty well-covered.", "other_strengths_and_weaknesses": "Overall, I think this is a solid paper and gained some new insights while reading it.\n\nI think the experiments section is a little light and I'd like to see additional evidence or more concrete analysis on the connection of the three kinds of attention heads - whether they are directly connected or there are other mediators between them.\n\n- There are only two tasks for which symbolic induction is tested (ABA, and ABB). One thing that could strengthen the claims of this paper is to examine other setups where symbolic induction may be helpful/used. A simple extension that I imagine wouldn't change the results you find are things like ABCABC..., or ABBCABBC...?", "other_comments_or_suggestions": "- In Line 155-157, can you clarify why you say the final token is the final token in each in context example, is the same for both contexts? Did you mean that A_N is the target token for both examples?\n\nTypos:\nLine 320: patten -> pattern", "questions_for_authors": "1. Have you studied these classes of attention heads on a broader class of relations?\n2. Are the function vector heads only symbolic induction heads or did they relate to the abstraction heads or retrieval heads you find?\n3. Are the abstraction heads you identify a more general form of \"previous token heads\" identified in previous circuit work [4,5]?\n\n\n___\n[4] Elhage, et al. In-context Learning and Induction Heads. 2022. (https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html)\n\n[5] Wang, et al. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. (https://arxiv.org/abs/2211.00593)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BSJeXeiH41", "reviewer": "Reviewer_Pe4s", "summary": "This paper studies an important topic of whether symbolic representations and behaviors emerge in LLMs when performing abstract reasoning tasks. The paper argues that symbolic abstraction and operations appears at different levels of attention (low-level abstraction, mid-level prediction, high-level retrieval), and verified these hypothesis via causal mediation analysis, attention plots and ablation studies. All analysis was conducted on a simple reasoning tasks (generating strings following the pattern ABA or ABB from two-shot examples), and demonstrated that the attention analysis aligns well with the expected behavior under these controlled experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the thoughtful and detailed feedback. We present detailed responses below to address each of the issues raised. Throughout these responses, we refer to new results that can be viewed here: \n\nhttps://anonymous.4open.science/r/RB-F30A/13386.pdf\n\n## Additional models and tasks\nWe have tested **12 additional models** (Figures 1, 6, 8-10), including GPT-2 (small, medium, large, and extra large), Gemma-2 (2B, 9B, and 27B), QWEN-2.5 (7B, 14B, 32B, and 72B), and Llama-3.1 8B (along with our original tests on Llama-3.1 70B), and **two additional tasks** (Figures 2-4) including a letter string analogy task and a verbal analogy task. With the exception of GPT-2, we find that our results are qualitatively replicated across all of these models and tasks. These results strongly suggest that the identified mechanisms are a ubiquitous feature of abstract reasoning in sufficiently large language models.\n\n## Direct analysis of key, query, and value embeddings\nTo gain a more precise understanding of the identified attention heads, we have now performed additional analyses applying RSA to the key, query, and value embeddings (Tables 1-2 and Figures 11-14). For abstraction heads, we found that queries primarily represented token identity, keys represented a mixture of both tokens and abstract variables, and values primarily represented the abstract variables. For symbolic induction heads, we found that queries and keys primarily represented the relative position within each in-context example, while values primarily represented abstract variables. For retrieval heads, we found that queries primarily represented abstract variables, keys represented a mixture of both tokens and variables, and values primarily represented the predicted token. These results further confirm the hypothesized mechanisms, namely that abstraction heads convert tokens to variables, symbolic induction heads make predictions over these variables, and retrieval heads convert symbols back to tokens.\n\n## Clarification about retrieval heads\nThe hypothesis about retrieval heads is not that they retrieve tokens that are most *similar* to the abstract variable. The hypothesis is that they retrieve the tokens that are *bound* to the abstract variable. For example, for an ABA rule, with the following incomplete query example:\n\nla (A), li (B), ?\n\nThe final variable is A, and in this example A is bound to the token ‘la’. Retrieval heads take the predicted variable (A) as input, and retrieve the token that’s bound to it (‘la’) to predict the token that will come next. To make this clearer in the revised paper, we will clearly describe this in terms of variable-binding, rather than using the ambiguous phrase ‘associated with’.\n\n## Intuitions on the origins of emergent symbolic mechanisms\nWe agree that it is interesting to consider the factors leading to the emergence of symbolic mechanisms. In the discussion section, we have already included some discussion of aspects of the transformer architecture that may contribute (innate similarity mechanisms, indirection), but aspects of the training data may also contribute, including: 1) the inherently relational nature of language, 2) factors such as ‘burstiness’ that have been related to in-context learning [1], and 3) the massive scale and generality of LLM training data, which may force the development of more general-purpose mechanisms. Directly analyzing the relative contribution of these factors will necessitate training language models from scratch under various conditions, which would require extensive resources, but we consider this an important avenue for future work. We will add further discussion of these issues to the revised paper.\n\n[1] Chan, S., Santoro, A., Lampinen, A., Wang, J., Singh, A., Richemond, P., ... & Hill, F. (2022). Data distributional properties drive emergent in-context learning in transformers. Advances in neural information processing systems, 35, 18878-18891.\n\n## Additional related work\nThank you for bringing the paper from Grant et al. to our attention. We agree that it is highly relevant. Relative to our work, their study investigates smaller neural networks trained from scratch on specific tasks rather than large pre-trained models, and also investigates a different task setting, but the high-level emphasis on emergent symbolic representations is closely related to our work, and we will make sure to cite it in our revised paper.\n\n## Clarification on description of 2\nWe will correct the description of equation 2 to describe it as a ‘BAA’ rule, while also clarifying that ABB and BAA rules are equivalent in this task (they involve the same pattern of relations).", "claims_and_evidence": "In a narrow sense, the evidence provided by the paper, as given by the experimental section, supports well the paper's claim that LLMs demonstrated remarkable symbolic behavior for the \"ABA, ABB\" reasoning task.\nHowever, to claim that LLMs developed intrinsic symbolic mechanism for general abstract reasoning tasks (as the title and introduction implies), more evidences and analysis need to be provided beyond the simple \"ABA, ABB\" task.", "methods_and_evaluation_criteria": "The methods (causal mediation analysis, attention ablation etc.) makes a lot of sense for the purpose of identifying symbolic behaviors, under the controlled setup of performing \"ABA, ABB\" tasks.", "theoretical_claims": "The paper could have provided a deeper theoretical foundation on the connection between connectionism and symbolism, as well as elaborated on why symbolism has been historically essential in the development of AI.", "experimental_designs_or_analyses": "The experiments are well-designed and analyzed from the perspective of causal intervention.", "relation_to_broader_scientific_literature": "This paper is well-contextualized connected to the established literature on the philosophical symbolism of artificial intelligence, as well as the historical development of neurosymbolic integration in AI model design.", "essential_references_not_discussed": "The paper could have cited a highly-related recent paper:\nEmergent Symbol-like Number Variables in Artificial Neural Networks, Satchel Grant, Noah D. Goodman, James L. McClelland, arXiv 2025", "other_strengths_and_weaknesses": "Strength:\nThe paper studies a really important and intriguing topic of the emergence and role of symbolism in neural networks. If it is successfully proven that symbolic and formal logic abstractions do exist in LLMs internal representations as an emergent phenomenon, this would significantly help researchers understand the inner-working of LLMs, and even how intelligence works in general.\n\nWeakness:\nThe cases studied by the paper is purely synthetic and a little over-simplistic. They are not strong enough to support the general claim that symbolic abstraction emerges from LLM reasoning process.", "other_comments_or_suggestions": "In Sec 3.1, right above Eq. (1), it should be \"that instantiated an BAA rule\" instead of \"...that instantiated an ABB rule\"?", "questions_for_authors": "1. The specific examples (\"ABA\", \"ABB\") studied by the paper are highly structured and position-driven. While it is plausible that the model has developed an abstraction of the symbolic patterns from the in-context examples, these specific tasks are perhaps too simple for the paper to draw reliable conclusion that symbolic mechanisms are emergent from LLM reasoning process.\n\nTo systematically study the symbolic behaviors of LLM, it would be more convincing to conduct experiments on general reasoning and logical inference tasks, for example maths and arithmetics, first-order logic etc.\n\n2. The study of the paper is based on few-shot learning setup. It remains unclear whether the symbolic abstraction behavior still holds in broader LM use cases like instruction following. For example, instead of giving concrete examples for ABA or ABB, just instruct the model with \"Generate 10 random triplets, each following the pattern ABB\". Will we still be able to observe similar abstraction mechanism as the paper analyzed?\n\n3, Given that modern LLMs have already demonstrated impressive capability in formal reasoning tasks, it is not hard to believe that LLMs already developed intrinsic symbolic representations and inference rules for these tasks. However, one missing piece is to understand how LLMs developed such a capability from training data. It would be great if the paper provides some analysis or just intuition on how the LLMs acquired such capability, and what types of training data is critically responsible for the symbolic emergence.\n\n4. The paper argues that \"we hypothesize that the value embeddings in these heads do not represent the identity of the input tokens, but instead represent only their position.\" (line 132). If this is the case, then the value embeddings is decoupled from semantic representation, and simply became a position indicator. Wouldn't it be easier to verify this hypothesis just by clustering value embeddings from different examples? If the embedding contains little semantic info, then these vector clusters should be clearly position-dependent.\n\n5. It was not very clear how the retrieval heads work: If like what the paper argues (the retrieval head retrieves actual tokens closest to the latent symbolic variable), then shouldn't Fig. 4 (e) be a plot between variables (A, B) and the token vocabulary?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the internal mechanisms that support abstract reasoning in LLMs, focusing on the open-source model Llama3-70B. It is argued that emergent reasoning in neural networks depends on the emergence of symbolic mechanisms. Evidence for this claim is presented in the form of a discovery of an emergent symbolic architecture in LLama3-70b that implements abstract reasoning via a series of three computations: abstraction, induction and retrieval. Experiments are carried out on algebraic rule induction tasks.

This paper received mostly positives reviews and after the rebuttal all reviewers are in favor of accepting. Reviewers agree the interpretability techniques deployed are appropriate, and the results are sound. There is also broad agreement that the results presented are of broader interest, especially because of the new heads/circuits that are identified. A general weakness noted among the reviewers was a lack of results for other models and tasks, which the authors addressed in the rebuttal. Other remaining questions, such as about clarifying what is meant with "symbolic processing" were mostly addressed in the author response and will hopefully make their way into the paper. The AC concurs with the recommendation to accept.
Decision: Accept (poster)