How Do Transformers Learn Variable Binding in Symbolic Programs?
Authors: Yiwei Wu
Atticus Geiger
Raphaël Millière
Venue: ICML 2025 poster (2025)
Keywords: variable binding
mechanistic interpretability
causal interventions
transformers
language models
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: Developmental interpretability study of a small transformer trained to perform variable binding on programs

Abstract: Variable binding---the ability to associate variables with values---is fundamental to symbolic computation and cognition. Although classical architectures typically implement variable binding via addressable memory, it is not well understood how modern neural networks lacking built-in binding operations may acquire this capacity. We investigate this by training a Transformer to dereference queried variables in symbolic programs where variables are assigned either numerical constants or other variables. Each program requires following chains of variable assignments up to four steps deep to find the queried value, and also contains irrelevant chains of assignments acting as distractors. Our analysis reveals a developmental trajectory with three distinct phases during training: (1) random prediction of numerical constants, (2) a shallow heuristic prioritizing early variable assignments, and (3) the emergence of a systematic mechanism for dereferencing assignment chains.
Using causal interventions, we find that the model learns to exploit the residual stream as an addressable memory space, with specialized attention heads routing information across token positions. 
This mechanism allows the model to dynamically track variable bindings across layers, resulting in accurate dereferencing. 
Our results show how Transformer models can learn to implement systematic variable binding without explicit architectural support, bridging connectionist and symbolic approaches.

How Do Transformers Learn Variable Binding in Symbolic Programs?
Yiwei Wu Atticus Geiger'
Raphaël Millière
Abstract
Example 5-Hop Program
Directed Graph
Variable binding ability to associate vari- ables with values-is is fundamental to symbolic computation and cognition. Although classical architectures typically implement variable bind- ing via addressable memory, it is not well under- stood how modern neural networks lacking built- in binding operations may acquire this capacity. We investigate this by training a Transformer to dereference queried variables in symbolic pro- grams where variables are assigned either numer- ical constants other variables. Each program requires following chains of variable assignments up four steps deep to find the queried value, and also contains irrelevant chains of assignments acting as distractors. Our analysis reveals a de- velopmental trajectory with three distinct phases during training: (I) random prediction of numeri- cal constants, (2) shallow heuristic prioritizing early variable assignments, and (3) the emergence of systematic mechanism for dereferencing as- signment chains. Using causal interventions, we find that the model learns to exploit the resid- ual stream as an addressable memory space, with specialized attention heads routing information across token positions. This mechanism allows the model to dynamically track variable bindings across layers, resulting in accurate dereferencing. Our results show how Transformer models can learn to implement systematic variable binding without explicit architectural support, bridging connectionist and symbolic approaches.
referential depth
referentivll depth?
j-k
k=2
Distractor Chain
Variable
t=8
Variable Chain
distractor ohaln b=j
lnne
Inn
Quary
(Answvrr)
refurentivl depth
W=
line
line
query
Other Numerical Constant
Figure I. Example 3- H-Hop Program. While our programs have 17 lines, this example only has 7 lines for illustration. The variable chain of the query variable (w) includes 3 variable assignments "hops." This program also includes 3 irrelevant variable assign- ments that act distractors. An interactive version of this plot for any program can be viewed on & var variablescope.org
tation and cognition. It enables systems to represent and manipulate structured information by maintaining relation- ships between abstract roles and their concrete instantiations. In classical computer architectures, variable binding im- plemented through addressable read/write memory, where variables serve addresses pointing to memory locations containing their bound values. This mechanism separates computational machinery from specific values, allowing general general-purpose -pupose algorithms to operate on arbitrary inputs without requiring dedicated computations for each possible input-output pairing.
The question of how neural networks might implement vari- able binding, if at all, has been central point of contention in debates between classicist and connectionist theories of cognitive architecture (Smolensky, 1990; Gallistel & King, 2011; Alhama & Zuidema, 2019). Classicists argue that connectionist models face significant challenges in account- ing for the systematic and compositional processing that variable binding enables symbolic systems. Although some theorists acknowledge that neural networks could in principle exhibit systematic behavior, they suggest that this would merely result from implementing classical archi- tecture with explicit variable binding operations (Marcus 1998; Marcus et al. 1999; Marcus 2001; cf. Elman 1999; Seidenberg & Elman 1999a;b; Geiger et al. 2023).
1. Introduction
Variable binding, the ability to associate abstract variables with specific values, is fundamental operation in compu-
'Pr(Ai"RR Group 'Maqqurrie University. Correspondence to: Yiwei Wu <redroomsd@gmm @gmail.com> Raphaël Millière matianel.iiliieremmm.dee
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
The advent of the Transformer architecture provides a new

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
(4) Test Set Accuracy
[b] Accuracy by Correct Answer Line Position
100%
102%
Fros
CoroetAesse
80%
80%
Anturnoy
80%
Acsurazy
40%
40%
20%
12
13
14
0%
15
10
10
16
(c) Accuracy by Number of Hops in Query Variable Chain
(d) Distribution of Numerical Predictions
100%
Prodiction
80%
Number
Assurayy
80%
Proportion
40%
0.4
20%
0.2
0.0
10
16"
10"
Training Steps
Training Stops
Figure 2. Behavioral results showing the learning dynamics of our model across training steps. (a) Overall test set accuracy demonstrates three distinct learning phases, with rapid improvements at and steps. (b) Accuracy breakdown by correct answer line position (1-16) (c) Average accuracy of model checkpoints depending the number of "hops" of the query variable chain in test set programs. (d) Distribution of model predictions across numerical constants. Vertical dashed lines (at step 800 and 14000) identify the beginning of each major nonlinear transition phase in the developmental trajectory.
avenue for exploring these questions about variable binding in neural networks. Transformers be seen imple- menting form of read/write memory through their residual stream trream-thhhhe high-dimensional vector space at each token po- sition that acts as a communication channel between layers. This space is addressable through learned linear projections, allowing attention heads to read from and write to specific subspaces (Vaswani et al., 2017). This raises the possibility that Transformers might learn to use their residual stream to implement variable binding operations without explicit architectural support for symbol manipulation.
In this paper, we investigate the open question of how the capacity for variable binding emerges in neural networks during training. We focus on a variable dereferencing task using synthetic programs with a Python- like syntax. These programs consist of 17 lines, where each line (except the last) contains a variable assignment statement of the form var = const or var var. The final line contains a query instruction #var: that asks the network to output the value bound to the specified variable. Fig. 1 illustrates this task with a simplified 5-line example.
Our analysis uncovers a distinct three-phase developmen- tal trajectory in the model's learning process. The model begins with (1) random predictions of numerical constants, then progresses to (2) shallow heuristics that prioritize early variable assignments, before finally developing (3) a system- atic mechanism for variable dereferencing. Through careful causal interventions, we demonstrate that the model Iearns to use its residual stream as an addressable memory space, with specialized attention heads routing information across token positions to track variable assignments. Notably, we find that the model's final solution builds upon, rather than replaces, its earlier heuristics. This finding challenges tradi- tional narratives suggesting that neural networks abandon
Recent mechanistic interpretability research has begun to in- vestigate the internal mechanisms underlying variable bind- ing in Transformers. Davies et al. (2023) and Prakash et al. (2024) uncover variable binding circuitry by learning binary masks over model components that mediate variables and values. Additional studies have shown the presence of "bind- ing ID vectors" that associate entities with their attributes, with these vectors occupying a specific "binding subspace" within the activation space (Feng & Steinhardt, 2024b;a; Dai et al., 2024). These findings suggest that variable binding may emerge as a necessary capability for language models to satisfy their training objectives.
2

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
early strategies when they discover systematic solution to a task. The observed developmental trajectory provides new perspective how neural networks can acquire structured reasoning capabilities without explicit symbolic machin- ery. Our results show that Transformer models can imple- ment fundamental aspects of symbolic computation through learned mechanisms, contributing valuable evidence to on- going debates about connectionist and symbolic approaches to computation and cognition.
sequence of variable assignments that does not contribute to determining the final value of the queried variable but might complicate the task by introducing irrelevant variable relationships. These distractors prevent the network from solving the task through simple pattern matching, instead requiring it to accurately track the relevant variable bind- ings. For example, in the program shown in Fig. 1, the chain + j - k + represents a distractor that branches from the queried variable's ' referential chain at variable j.
To facilitate transparent and reproducible interpretability research, we developed Variable Scope, an interactive web platform that allows researchers to explore and verify our experimental findings. The platform includes interactive visualizations of the program structure, training checkpoint evaluation, model developmental trajectory, causal interven- tion experiments, and subspace experiments. This platform builds on previous efforts to present experimental results interactively, such as the Distill Circuits Thread, while pro- viding more granular tools to visualize and analyze the evolution of a neural network over the course of training (Cammarata et al., 2020). Through Variable Scope, we aim establish a new standard for open and collaborative mech- anistic interpretability research: á var wrrriaalecoppe.or
Program Sampling Procedure Wc generate dataset of 500,000 programs. Our splits are: training (450,000 pro- grams, 90%), validation (1,000 programs, 0.2%)., and testing (49,000 programs, 9.8%). Programs use 26 lowercase letters (a-z) as variables and single-digit numbers (0-9) as values. Every line follows a four-token structure: (1) a variable on the left-hand side, (2) an equality "=", (3) either variable or a value on the right-hand side, and (4) a newline token.
Each line has 30% chance of having a numerical constant on the right-hand side of the equality sign, and a 70% chance of having a variable reference, provided that previously defined variables exist. When a variable appears on the right-hand side, multiple variable chains could potentially be extended. We deliberately favor longer chains by sampling with probability proportional to the cube of the chain length. The query variable is selected using the same weighting scheme. Finally, we use rejection sampling to balance the dataset across the four possible referential depths (1-4 hops).
2. Training a Transformer to Run Programs
2. The Task
Our task requires a model to process programs with variable assignments and determine the final value of a queried vari- able. Fig. shows an abbreviated 7-line example program. The final line queries variable w, which has a value of 2. To determine this value, the model must trace through chain of variable assignments: = j. k, and finally A = 2. This dereferencing process requires the model to track and retrieve variable bindings throughout the program.
This sampling procedure ensures the presence of sufficiently long distractor chains in the data, preventing the model from achieving high performance simply by selecting values associated with the longest chains. Instead, the model must learn to follow the referential paths of each query.
2.2. Training a Transformer
We train a Transformer model from scratch our task. Our model is architecturally similar to GPT-2 (Radford et al., 2019), with 37.8M parameters. The model has 12 layers, each with attention heads with a head dimension of 64 and residual stream dimension of 512. We implement LayerNorm (Ba et al., 2016) before each attention and MLP block, use rotary positional embeddings (RoPE) (Su et al., 2024), apply GELU activations between layers (Hendrycks & Gimpel, 2016), and a dropout rate of 0.1. We do not tie the input and output embedding weights. After training, our model achieves near-perfect performance on the held-out test set (over 99.9% accuracy), showing it has learned a robust mechanism for variable binding and dereferencing.
We define referential depth as the num- ber of assignment steps needed to reach a numerical value from a queried variable v, where for a chain of assignments V = U1, 111 = U2...., Un-1 = c (with c being a numerical constant), the referential depth is n. Our programs contain variable chains with depths ranging from I to 4. These chains form directed graph structure where variables are nodes and assignments are edges. To correctly dereference a queried variable, the model must traverse this graph along the relevant path while ignoring irrelevant branches. For instance, the program in Fig. 1 has a referential depth of 3, as three steps or "hops" are required to trace from the queried variable to its final value (w + j k + 2).
Referential Depth
2.3. Phase 1: Predicting Numbers Instead of Letters
Distractors Our programs include distractor variable chains that are either wholly independent from the queried variable chain or branch out from it. A distractor chain is any
In the first phase (training steps 0 to 800), the model per- forms at approximately chance level, achieving only around
3

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
12% accuracy (Fig. 2a). During this phase, the model only learns that the answer should be numerical value rather than a variable name or special token. This is reflected in the model's prediction distribution, where numerical tokens receive a higher probability mass, though in a notably imbal- anced and unstable pattern (Fig. 2d). Despite recognizing that it should output numbers, the model has not yet devel- oped any systematic strategy to identify which particular numerical constant is the correct answer.
3. Causal A Analysis
To understand how the model learns to solve our variable binding task, we use mechanistic interpretability techniques (Cammarata et al., 2020; Saphra & Wiegreffe, 2024; Sharkey et al., 2025) grounded in causal analysis (Cao et al., 2020; 2022; Csordás et al., 2021; Geiger et al., 2021; 2024; Chan et al., 2022; Mueller et al., 2024). These methods allow us to reverse engineer the model's internal mechanisms and pro- vide insights into how neural networks implement symbolic operations. An important feature of our experiment is its developmental perspective (Lovering et al., 2021; Liu et al., 2022; Nanda et al., 2023; Merrill et al., 2023): we track how the network's ' internal mechanisms evolve throughout training instead of analyzing only the fully trained model.
2.4. Phase 2: Learning Early Line Heuristies
The second phase (training steps 1200 to 14000) begins with sharp performance jump from 12% to 56% accuracy. During this phase, the model develops two primary predic- tion strategies or heuristics seuristies-iimplifeed decision rules that approximate a solution without fully solving the task. These include: (1) a "line-l heuristic" that selects the numerical constant from the first program line, and (2) a "line-2 heuris- tic" that selects the numerical constant from the second line (when the right-hand side of that line contains a number). Fig. 2c illustrates this development by showing the model's accuracy over time for programs where the correct answer appears on specific lines. Accuracy for programs with an- swers on line increases dramatically to over 90%, while accuracy for programs with answers on line 2 reaches ap- proximately 65%. The line-1 heuristic proves particularly effective because our program generation process tends to place the root values of longer reference chains in earlier lines. This is evident in the model's higher Phase 2 accuracy on programs with longer query variable chains (Fig. 2c). For multi-hop programs, the root value-the numerical constant at the end of the chain- frrquently appears in the first or second line, making these early-line heuristies surprisingly successful. Conversely, 1-hop programs show slower con- vergence because their answer (a numerical constant directly assigned to the queried variable) could appear on any line, making these position-bast based heuristics less reliable.
At the core of our approach is the interchange intervention method (Vig et al., 2020; Geiger et al., 2020). This tech- nique involves taking specific components from the model's computation graph and replacing them with values they would have if the model were processing a different, care- fully constructed counterfactual inpur-a modified version of the original input that differs in a controlled way. By observing how these interventions affect the model's out- put, we can identify causal relationships between internal components and model behavior.
Constructing Counterfactual Inputs We use inter- change interventions to conduct a causal tracing experiment (Meng al., 2022), allowing to track how information about specific tokens flows through the model from input to output. While Meng et al. injected noise into activa- tions, our approach instead replaces activations with values from specially constructed counterfactual inputs. For each program in our dataset, we create a counterfactual by first identifying the "root" of the variable chain (the original numerical value that determines the final value of the query variable), and then modifying this root value to a different number. When we intervene by replacing components with their counterfactual values, a successful intervention causes the model to output this new number instead of the original one. This provides clear signal indicating which model components are responsible for propagating the root value through the network to the final prediction.
2.5. Phase 3: Systematic Variable Binding
In Phase 3 (training steps 34000 to 105400), we observe another sharp performance transition, with accuracy jump- ing from 56% to 99%. During this phase, accuracy rapidly improves across all reference depths and distractor config- urations, eventually exceeding 99.9% on the test set. The model demonstrates robust performance regardless of query variable chain length (Fig. 2c) the position of the cor- rect answer within the program (Fig. 2b). This dramatic improvement suggests that the model has acquired a gen- eral solution capable of tracing variable chains to any depth while appropriately ignoring distractor assignments.
3.1. Final Checkpoint Analysis
We intervene at two strategic locations in the model archi- tecture. First, we intervene on the residual stream of the Transformer the block of hidden vectors for each token that serves as input to each layer. Second, we intervene on the outputs of individual attention heads to trace how infor- mation moves across token positions. These complementary interventions allow us to pinpoint where and how variable binding information flows through the network.
4

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
Loght Difference
Layer
10
9 1
2.
Token Position
First Information
Sedond Information movommnt
Third information
movement
Movlnn movement
Moving Myyer
Moving root
(a): We patch the residual stream incoming Transformer blocks across all layers and tokens, using counterfactual input where the root value 8 replaced with stream each position replaced by its counterfactual value. Heatmap colors show normalized logit differences: the change in pre-softmax activation (logit) of the new number between patched and original runs, divided by the maximum difference. Blue indicates where interventions increase the probability of the new number.
number. Logits computed by running forward pass with the residual
Phoglame
Programm
Programs
Programs
Programs
Programs
Programs
CorrectAnrwnn On
100
Gorrect Answer On Line
Success Rato (4)
Averge Logh Orhroros Ineestol
Layee
20
Correct Anewer On Anotherr Line
1
Laper1t
Rei Croth pon Charr Jecablo
Finnl Coler
Rei Davin 2H13
Daey Vepatio
Rei Caeen Fand and for Drom Five
Coles
Ghors preal Croter
Davin 19v9 for Cree snd
Bavin FIH Rel brem Fiim Chars vensable Final Grion
Bachheet Cvty vesaat
for fur Dweet pou
thech yova cwelh shol
our cover puette
(b): Aggregated patching results targeting only query vari- able. the following colon, and the right-hand-sd hind-side (RHS) to- kens at different depths of the query assignment chain (e.g.. measures
Cvmry
(c): Patching results targeting individual attention heads. For each (head, token) position, replace only that head's contri- bution the residual stream with counterfactual value and compute logits. We analyze only programs where the correct answer appears after line 2. Layers without significant signals are excluded but shown in Appendix K.
"8". "b", and "q" in Fig. 3(a)) Patching how often the intervention causes the model to predict the new number as its top choice, averaged across test programs. A high success rate indicates that the intervened component contains causally relevant information about the output.
Figure 3. Patching analysis results on the final model checkpoint. In all experiments, logits are computed by passing the patched hidden states through the unembedding matrix measure how interventions affect token predictions.
5

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
Tracing a Single Example Consider the example pro- gram and its pre-residual stream patching results shown in Fig. 3(a). This program contains query referential chain (z ( + b - 8) formed by three assignment statements: b b= q = i,, and = = q. Our patching analysis reveals that token locations with high logit difference values across lay- ers correspond precisely to the right-hand side positions of these assignment statements, along with the query variable and colon token in the final query line. This clear pattern of information flow, visible in Fig. 3(b), motivates us to per- form systematic pre-residual stream patching experiments on the inputs to each layer.
grams where the early-line heuristic (predicting values from initial program lines) fails. For clarity, we exclude layers where all heads show signal below significance threshold. We use Y.X to denote the Xth head in layer Y.
The attention head patching results reveal a pattern consis- tent with the information movement observed in the residual stream analysis (Fig. 3(b), third row). For the first hop in the variable reference chain, heads 6.5 and, to a lesser ex- tent, 7.7 play crucial roles. The second and third hops mediated primarily by heads 7.2, 8.3, and 9.4. In the final processing stage, heads 11.2, 11.3, and 11.7 transfer this information to the output token position.
Intervening on the Residual Stream The attention mech- anisms in each layer can move information in the residual stream between different tokens. This path of information flow depends on the specific input provided to the model. When performing interchange interventions on the residual stream, we dynamically select token locations rather than intervening at fixed token indices. Our Transformer model uses causal attention, meaning that information can only flow from left to right; consequently, later tokens contain information about earlier tokens. Based on this property, we conduct intervention experiments on tokens that appear on the right-hand side (RHS) of the equality token "=".
Interestingly, head 8.3 participates in both the second and third hops of the reference chain. While the first hop pro- cessing spans layers 6-7 and the second hop spans layers 7-8, head 8.3 handles both transitions rather than having separate specialized heads for each hop. This is evidence that similar representations are used across hops and token positions for the systematic solution.
3.2. Circuit Development Throughout Training
To understand how the systematic mechanism for variable binding emerges over time, we conducted residual stream in- tervention experiments across different training checkpoints. We track intervention success rates at key (layer, token posi- tion) combinations and observe both the persistent influence of develodee heuristics and the gradual emergence of systematic variable-tracing mechanism.
Specifically, we intervene along the variable chain that leads to the correct answer, labeling each intervention point ac- cording the referential depth of its line in the chain. For example, in the program shown in Fig. 3(b), we refer to the token "8" as the Ref. Depth 1 RHS token, the second occurrence of "b" as the Ref. Depth 2 RHS token, and so on. We also target the query variable and the colon token following it for intervention. The results in Fig. 3 show two rough patterns. In the first pattern, the number immedi- ately moved to the colon token layer 1, which happens for 2-, 3-, and 4-hop programs when the correct answer is on line 1. In the second pattern, the number travels along the query variable chain, which happens in all other graphs to some extent. Notable exceptions are 1-hop programs where the answer is on line I, which seem split between the two patterns, and 4-hop programs, where the signal lost entirely after the Ref. Depth 2 RHS token.
We focus on three critical interventions in the network at po- sitions that serve distinct computational roles: (1) Variable value information transferred to the final colon token imme- diately after layer indicates the operation of line-specifer heuristics that bypass multi-step variable tracking. (2) Sue- cessful interventions at RHS tokens in layers 6-9 reveal how the model traces variable values by propagating information along reference chains step by step. (3) At the query variable token in layer 10, the model performs the final matching between the query variable and its dereferenced value.
Fig. illustrates the developmental trajectory of our model's variable binding mechanism across programs of different complexity. For I-hop programs (Fig. 4(a)), we first ob- serve activity at the layer 6 RHS token position (tracing to referential depth 1), followed by the layer 10 query variable position, with patching success rates reaching high levels by training step 60000. For 2-hop programs (Fig. 4(b)), a simi- lar but delayed developmental pattern emerges, with layer and 8 RHS token positions (corresponding to referential depth 2) becoming important.
These results show that the model develops its general mech- anism top of existing line-specific heuristics from Phase 2, overriding them only when needed.
Intervening on Attention Heads We apply the same root value replacement intervention to individual attention heads and present the results in Fig. 3(c). Specifically, we inter- vene on each head's output as it contributes to the residual stream at specific token positions, allowing us to trace infor- mation flow through the network. To isolate the systematic variable binding mechanism, we focus our analysis on pro-
The developmental pattern becomes more complex for pro- grams with higher referential depths. In 3- and 4-hop pro- grams (Fig. 4(c) and (d)), deeper RHS token positions in
6

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
(+) Programs Where Correct Anewer is on Line > (Neither Nor 2)
(b) Programs Where Correct Answer on Line (Neither Nor 2)
1-Hop Programs
24Hop Programs
100%
Pasching Stuzzes Ratt
80%
Paeching Stroeees Rate
80%
40%
40%
20%
20%
Training Steps
Training
(c) Programs Where Correct Answer is on Line > (Neither Nor 2)
(4) Programs Where Correct Answer is on Line (Neither Nor 2)
3-Hop Programe
4-top Programs
100%
Pasthing Secooess Ratt
80%
80%
80%
60%
Palthinn
20%
20%
0%
20000
40800
30000
120000
0%
20000
D0000
acoco
Training Stups
Training Steps
Programs Where Correct Answer Line
(1) Programs Where Correct Answer on Line (But Not
Aggregated Across fo-4-Hop Programs
Aggregated Across 14o-44Hop Programs
100%
Patshing Success Rote
10%
Parching Surrees
20%
60%
40%
40%
0%
20000
40800
60003
a00co
100000
0%
20000
A0000
6000
a2000
100000
Training Steps
Training Steps
Layer 2 Final colon token
Layer 8 Dept leri token
Layer Depth RHS token
Layer Depti Ri-S token
Layer Depth
Layer Depit takon
Layer 8 Depth RHS tokon
Layor 10 Query var tokon
Figure 4. Evolution of patching success rates across training steps, revealing the development of line-specific heuristics and general variable binding mcchanism. Vertical dashed lines at steps 800 and 14,000 mark the transitions between major phases identified in our behavioral analysis in Fig. 2. (a)-(d) show patching success rates for programs where the correct answer is not line I separated by the number of hops in the program. We selectively intervene on key (layer, token position) combinations: the query variable token (layer 10), final token (layer 2), and depth-specific RHS tokens (layers 6-9) that analysis identified as important causal mediation locations for tracking variable bindings where most of the information movement occurs. This highlights the emergence of the general circuit. (c) shows patching results aggregated across all programs where the answer is on line 1. The heuristic consistently used throughout training. (f) shows patching results aggregated across all programs where the answer is on line 2. The hcuristic is used only in the initial phase.
7

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
layers 8 and 9 show both delayed emergence and dimin- ished activation strength. These positions, corresponding to referential depths 3 and 4, exhibit attenuated activation patterns reflecting the increased difficulty of tracking longer reference chains. This systematic correspondence between network layer depth and reference chain length suggests the model develops specialized circuit that propagates variable binding information through successive layers.
line (targeting the numerical constant subspace). The other predicted the variable name typically found the LHS (targeting the variable name subspace, probed via the RHS token's 's residual stream). The L1 penalty encourages spar- sity, effectively selecting small subset of components most predictive for each task. This process yielded 10 principal components for numerical constants and 26 for variable names.
The contrast between Fig. 4(e) and Fig. 4(f) provides valu- able insight. Fig. 4(e), depicting programs where the answer appears on line 1. shows that carly-learned line-specific heuristics persist throughout training, as evidenced by con- sistently high success rates (over 80%) when patching the layer 2 final colon token position. contrast, Fig. 4(f), which focuses on programs where the answer does not ap- pear on line 1, reveals different pattern: while the heuristic (measured at the layer 2 final colon token) shows initial suc- cess, the systematic mechanism (measured at depth 2 RHS locations) develops more slowly compared to Fig. (db)
Using these components, we then extracted activations from the residual stream feeding into layer 6 across a subset of programs. To visualize the structure within these subspaces, we projected the vectors in 2D using UMAP (McInnes et al., 2018). We generated visualizations at three distinct training checkpoints (steps 1200, 17600, and 105400) to observe their evolution, shown in Fig. 5. The UMAP projections demonstrate increasing separation over training. By the final checkpoint, distinct clusters emerge for different nu- merical values (Fig. 5(a)) and variable identities (Fig. 5(b)), revealing the numerical and variable subspaces.
This contrast reveals that the model builds its variable binding mechanism on top of simpler line-specific heuris- ties rather than replacing them. The line-specific heuris- tie (tracked at layer 2's final colon token) remains active throughout training, with the general mechanism (tracked through the RHS token positions) only overriding it when necessary to predict the correct answer. This composite approach enables the model to process simple cases using the heuristic while using more sophisticated algorithms for programs that require them.
To validate the causal role of these subspaces, we performed interchange interventions on each one. Unlike earlier inter- ventions, these swapped only the residual stream subspaces spanned by the selected component sets (10 for numerical constants, 26 for variable names) between original and coun- terfactual programs targeting either the numerical constant or the variable name. These interventions had high suc- cess rates: 92.17% for numerical constants and 87.08% for variable names. This provides strong causal evidence that the identified subspaces encode their respective information types, confirming our hypothesis.
3.3. Numerical and Variable Subspaces
4. Discussion
To investigate how the final model represents and tracks nu- merical constants and variable names, we hypothesized that these are encoded in separate subspaces within the residual stream. Given that our model uses causal attention, we fo- cus our analysis on the residual streams at token positions corresponding to the right-hand side of each program line in the query chain.
Our study shows how neural networks can acquire capa- bilities traditionally associated with symbolic computation, particularly variable binding and derefereneing, Analysis of the model's learning trajectory reveals that the path to sys- tematic variable binding proceeds through distinct phases with qualitatively different strategies.
To test this hypothesis, we first aimed to identify these po- tential subspaces. We applied principal component analysis (PCA) to reduce the 51 512-dimensional residual stream ac- tivations to 256 principal components at the target RHS locations. We performed this analysis on a subset of test samples selected to exclude cases solvable by a line-1 heuris- tic (where the correct answer is on the first line). This fil- tering ensures that the observed internal states reflect the systematic mechanism rather than a shallow heuristic.
Notably, we find that the model's final solution builds upon, rather than replaces, the heuristics learned in ear- lier phases. This adds nuance to the traditional narrative about "grokking", where models are thought to discard su- perficial heuristics in favor of more systematic solutions. Instead, our model maintains its carly-line heuristics while developing additional mechanisms to handle cases where these heuristics fail, suggesting cumulative learning where sophisticated capabilities emerge by augmenting simpler strategies.
To isolate the components most relevant to each hypothe- sized subspace, we trained two linear classifiers with L1 regularization on these 256 principal components. One clas- sifier predicted the numerical constant on the RHS of the
Causal interventions show the model uses its residual stream as addressable memory, with specialized attention heads routing information to track variable assignments. This
8

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
Checkpaint
Chackpohtt 17603
Checkpoint Step 105400
Number
(a): 2D UMAP projection of the 10 principal components identified predictive of numerical constants. PCA was first applied to RHS token residual streams (from non-heuristic test samples) reducing dimensionality from to 256. An Ll-regularized linear classifier then selected these 10 components. The projection visualizes these residual stream activations (from the input to layer 6) across three training checkpoints: steps 1200, 17600, and 105400.
Checkpoint 1200
Checkpoint 17600
Checkpoint 105400
Variable
(b): 2D UMAP projection of the 26 principal components identified as predictive of variable names. PCA was first applied to RHS token residual streams (from non- nou-heuristic test samples) reducing dimensionality from 512 to 256. An LI-regularized linear classifier then selected these 26 components. The projection visualizes these residual stream activations (from the input to layer across three training checkpoints: steps 1200, 17600, and 105400.
Figure 5. Evolution of 2D UMAP for numerical-constant and variable-name residual stream subspaces (input layer 6) across training.
provides concrete evidence for how symbolic computation can emerge from continuous vector operations. Importantly, probing experiments suggest that the model develops an efficient strategy focused on tracking only the informa- tion necessary for dereferencing, rather than maintaining complete program state.
5. Conclusion
Our study reveals how Transformers can learn to per- form variable binding and dereferencing capability tra- ditionally associated with symbolic computaatonn-wtheee built-in symbolic operations. Through mechanistic anal- ysis, demonstrate that the model implements variable binding by repurposing its residual stream address- able memory space. These findings contribute to ongo- ing debates about connectionist and symbolic approaches to computation. We present an interactive web platform & variablescope.org org to support reproducible research.
These findings have implications for cognitive science and machine learning, showing how symbolic capabilities can emerge from neural architectures without built-in symbolic operations. This also suggests that the development of phisticated reasoning capabilities might be better supported by training regimes that allow progressive refinement and composition of simpler strategies.

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
Acknowledgments
through differentiable weight masks. In International Conference on Learning Representations, 2021. URL
This work was in part supported by a grant from Open Philanthropy.
Dai, Q. Heinzerling, B., and Inui, K. Representational Analysis of Binding Language Models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17468-17493, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
Impact Statement
This research was conducted with the goal of understand- ing how language models develop and combine different circuits. By contributing reproducible mechanistic inter- pretability research and open scientific methodology, this work supports broader efforts make AI systems more transparent and understandable.
Davies, X., Nadeau, M., Prakash, N., Shaham, T. R., and Bau, D. Discovering variable binding circuitry with desiderata, 2023. URL https::/arxxw.orr//aa 03637.
References
Alhama, R. G. and Zuidema, W. A review of computational models of basic rule learning: The neural-symbolic de- bate and beyond. Psychonomic bulletin & review, 26(4):
Elman, J. L. Generalization, rules, and neural networks: A simulation of Marcus et. al. HTML document, 1999.
1174-1194, 2019.
Feng. J. and Steinhardt, J. How do language models bind entities in context? In The Twelfth International Confer- ence on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024a. URL nttps://openreviee eww.ew http:/wwwwwweeetnnneeeeeeee ntt.://orruum//d2/3366o0000
J. J. Kiros, J. R., and Hinton, G. E. Layer normalization, 2016. URL h
/
06450.
Cammarata, N., Carter, S., Goh, G., Olah, C., Petrov, M.,
Schubert, L., Voss, C., Egan, B., and Lim, S. K. Thread: Circuits. Distill, 2020. doi: 10.23915AA /002i4 URL https://distiie /ttpllpp///////uuuuuii
Feng. J. and Steinhardt, J. How do language models bind entities in context? In The Twelfth International Confer- ence on Learning Representations, ICLR 2024, Vienna, Austria, May 7- 2024. OpenReview.net, 2024b. URL https://openrevieer wwtp://www....eeooommmmd/////aaa
Cao, N. D., Schlichtkrull, M. S.. Aziz, W., and Titov, How do decisions emerge across layers in neural models? Interpretation with differentiable masking. In Webber, B., Cohn. T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pp. 3243-3255, Online, November 2020. Association for Computational Linguistics. doi: 10. 118865//1120200 http:/0000..mmmmmaaa.a.a0.0.. URL https:://claatttol//a//// 262/.
Feng, J., Russell, S., and Steinhardt, J. Monitoring La- tent World States in Language Models with Propositional Probes, June 2024.
Gallistel, C. and King, A. Memory and the Compu- lational Brain: Why Cognitive Science will Trans- form Neuroscience. Blackwell/Marlladd Lectures in Language and Cognition. Wiley, 2011. ISBN 9781 9781444359763. URL https://booke goookk..oogle.com (aokss?id=o@@pHgwweeee
org 2000/ emnlp-main.
Cao, N. D., Schmid, L., Hupkes, D., and Titov, Sparse in- terventions in language models with differentiable mask- ing. In Proceedings of the Fifth BlackboxNLP Work- shop on Analyzing and Interpreting Neural Networks for NLP, 2022. URL httph: http:///oo.oogg///// 111100.8153/v1 18653/v1/ 2022. blackboxnlp-1.2.
Geiger, A., Richardson, K., and Potts, C. Neural nat- ural language inference models partially embed theo- ries of lexical entailment and negation. In Alishahi, A., Belinkov, Y., Chrupala, G., Hupkes, D.,. Pinter, Y., and Sajjad, H. (eds.). Proceedings of the Third Black- boxNLP Workshop on Analyzing and Interpreting Neu- ral Networks for NLP, pp. 163-173, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 12005///10000000aaaaooooo/ / //aclanthology http:///w.....//./..oaooooooo/// blackboxnlp-1.16.
Chan, L., Garriga-Alonss A., Goldwosky-Dill, N., Greenblatt, R., Nitishinskaya, J., Radhakrishnan, A., Shlegeris, B., and Thomas, N. Causal scrubbing, a method for rigorously testing interpretability hy- potheses. Al Alignment Forum, 2022. https://www. al alignmentforum. org/ orgg/posts/Jwwhhyytttzz/ s
URL https:
Geiger, A., Lu, H., Icard, T., and Potts, C. Causal abstractions of neural networks. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan,
Csordás, R., van Steenkiste, S., and Schmidhuber, Are neural nets modular? inspecting functional modularity
10

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 9574-9586, 2021. URL https: opproeedingss.eurrrs...c//pppp/p /hasshh /hash/ (f562227449955500002222222200100 html.
Meng, K., Bau, D., Andonian, A. J., and Belinkov, Y. Locat- ing and editing factual associations in GPT. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL wtrpeerreiieennee/oorummm2m
Merrill, W., Tsilivis, N., and Shukla, A. A tale of two circuits: Grokking as competition of sparse and dense subnetworks. In ICLR 2023 Workshop on Mathemati- cal and Empirical Understanding of Foundation Models, 2023.
Geiger, A., Carstensen, A., Frank, M. C., and Potts, C. Re- lational reasoning and generalization using nonsymbolic neural networks. Psychological Review, 130(2):308-) 333, 2023. ISSN 1939-1471. doi: 10. 10.10377/e00000371.
Mueller, A., Brinkmann, J., Li, M., Marks, S., Pal, K., Prakash, N., Rager, C., Sankaranarayanan, A., Sharma, A. S., Sun, J., Todd, E., Bau, D., and Belinkov, Y. The quest for the right mediator: A history, survey, and theo- retical grounding of causal interpretability, 2024. URL https://arxiv http//axxvv.r.ggaaaa hts//bbs/24400 01416.
Geiger, A., Ibeling, D., Zur, A., Chaudhary, M., Chauhan, S., Huang, J., Arora, A., Wu, Z., Goodman, N., Potts, C., and leard, T. Causal abstraction: A theoretical foundation for mechanistic interpretabillity, 2024. URL https:// arxiv.org/abs//238 04709.
Nanda, N., Chan, L., Lieberum, T., Smith, J., and Stein- hardt, J. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL htttss//ooeennevveeeeeeffforoorr wwt.://orruomidddffsrooooss
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.084155 2016.
Liu, Z., Kitouni, O., Nolte, N. S., Michaud, E., Tegmark, M., and Williams, M. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651- 34663, 2022.
Prakash, N., Shaham, T. R., Haklay, T., Belinkov, Y., and Bau, D. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In The Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net, 2024. URL https://openrewe forum?1d=8sK<ANOF22.
Loshchilov, and Hutter, F. Fixing weight decay regular- ization in adam. CORR, 0txx/1711.051011 2017. URI. http://arxiv.. httpp://arxiv.org/bbbsssso wttp:/r//////.//..////// 05101.
Lovering, C., Jha, R., Linzen, T., and Pavlick, E. Predicting inductive biases of pre-trained models. In International Conference Learning Representations, 2021. URL wttpr//wwww...eeoooooooiinnnnnna
Radford, A.,
Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99, 2019.
Magic. Hashhop: Long context evaluation. https:// ..t.ub.commmaaaaee. commm//mgiiiprrduut///sshhhhoooo 2024.
Saphra, N. and Wiegreffe, S. Mechanistic? In The 7th Black- boxNLP Workshop, 2024. URL net met/foorum?iidsschAAABPPP
Marcus, G. F. Rethinking eliminative connectionism. Cog-
nitive psychology, 37(3):243-28) 1998.
Seidenberg, M. S. and Elman, J. L. Networks are not 'hid- den rules'. Trends in Cognitive Sciences, 3(8):288-22) 289, 1999a
Marcus, G. F. The Algebraic Mind: Integrating Connec- tionism and Cognitive Science. The MIT Press, 04 2001.
ISBN 9780262279086. doi: 10.7551/mitpress/|| 187.001. 0001. URL
Seidenberg, M. S. and Elman, J. L. Do infants learn gram- mar with algebra or statistics? Science, 284(5413):433- 433, 1999b.
1187. 001 10000.00.....
Marcus, G. F., Vijayan, S., Rao, S. B., and Vishton, P. M. Rule learning by seven-month-old infants. Science, 283 (5398):77-80, 1999.
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq. L., Goldowsky- -Dill, N., Heimersheim, S., Ortega, A., Bloom, J., Biderman, S., Garriga-Alonso,
A., Conmy, A., Nanda, N., Rumbelow, J., Wattenberg,
McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint
M., Schoots, N., Miller, J., Michaud, E. J., Casper, S.,
Tegmark, M., Saunders, W., Bau, D., Todd, E., Geiger, A., Geva, M., Hoogland, J., Murfet, D., and McGrath,
2018.

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
T. Open problems in mechanistic interpretability, 2025. URL https://arxiv.rr httpss///arrtt.orr/ab/////// 16496.
A. Expanded Related Works
Synthetic Tasks. Our work expands on previous research using synthetic tasks investigate how Transformers handle highly structured data, including reference chains. Zhang et al. (2023) showed through their LEGO task that Trans- formers can develop specialized attention heads for match- ing identical tokens and processing local operations, sug- gesting mechanisms for tracking variable relationships. the hashhop benchmark (Magic, 2024) assesses whether mod- els can learn to resolve chains of references in long con- texts while avoiding patten-maaching shortcuts. while these works focus length generalization and path-finding in graphs, our task emphasizes sequential execution order and memory operations, providing focused test bed for in- vestigating variable binding mechanisms in Transformer architectures.
Smolensky, P. Tensor product variable binding and the rep-
resentation of symbolic structures in connectionist sys- tems. Artificial Intelligence, 46(1):159-216,, 1990. ISSN
0004-3702. doi:
90007-M. URL https::/ww..ciiieeeeeee sciemce/articcle////////////2/22
com/
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 568:127063, 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems, pp. 5998 6008, 2017.
Vig, J.. Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. Investigating gender bias in language models using causal mediation analysis. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. I 12388-12401. Curran Associates, Inc., 2020. URL neurips. http://....cppppp...eee////a/ppe 92558022227775511126622222222222
Variable Binding in Pretrained Transformers. Recent work has also made progress in understanding how pre- trained Transformer-base language models may implement variable binding mechanisms. Our work builds on and com- plements these findings by providing a detailed mechanistic account of how variable binding emerges during training.
Davies et al. (2023) developed an automated approach to identify shared variable binding circuitry in LLaMA-13B that retrieves variable values for multiple arithmetic tasks. Using causal mediation experiments with carefully designed desiderata, they localized variable binding to 9 attention heads and one MLP in the final token's 's residual stream.
pdf.
Zhang, Y.. Backurs, A., Bubeck, S., Eldan, R., Gu- nasekar, S., and Wagner, T. Unveiling transformers with LEGO: A synthetic reasoning task, 2023. URL https://openreview. w
This suggests that variable binding capabilities can be im- plemented by sparse subset of model components working together.
Feng & Steinhardt (2024a) analyzed LM representations and identified general binding ID mechanism present in every sufficiently large model from the Pythia and LLaMA families. They showed that LMs' internal activations rep- resent binding information by attaching binding ID vectors to corresponding entities and attributes, with these vectors forming continuous subspace. Through careful causal in- tervention experiments, they found that binding IDs are used consistently across different tasks and can be transplanted between tasks, suggesting the emergence of general bind- ing mechanism.
Building on this work, Dai et al. (2024) discovered that LMs encode ordering information in a low-rank subspace that causally determines binding behavior. By using dimension- ality reduction techniques, they identified an "Ordering ID" subspace distinct from pure positional information, provid- ing novel geometric view of how binding is implemented. Their causal intervention experiments showed that editing activations in this subspace could systematically alter which attributes are bound which entities.
12

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
Feng al. (2024) introduced "propositional probes" to mon- itor how language models internally represent relationships between entities and their attributes. Using Hessian-basee algorithm, they identified binding subspace where tokens that should be bound together (like a person and their oc- cupation) have high similarity, while unbound tokens do not. This binding subspace was found to be fairly robust, maintaining accurate representations even when the model's outputs became unfaithful due to biases adversarial at- tacks.
The design of our synthetie programs intentionally uses minimal symbolic vocabulary, with only numerical con- stants (0-9) and single-character variables (a-z). This con- trolled environment enables precise causal analysis of how variable binding mechanisms emerge during training.
Despite its apparent simplicity, the task demands sophis- ticated computational capabilities: the model must track multi-step variable dependencies through chains of refer- ences while ignoring distractor assignments.
Prakash et al. (2024) studied how fine-tuning affects en- tity tracking mechanisms, finding that fine-tuning enhances existing mechanisms rather than creating new ones. They showed that the same circuit implementing entity tracking in the base model persists in fine-tuned versions with im- proved performance, suggesting that binding capabilities can be strengthened through targeted training.
C. Tokenization Details
We employ a simple character- tokenizer where each individual character treated as a single token, including all variable names, numerical constants, and special characters like \n and =,
Our work complements these findings by providing a de- tailed analysis of how variable binding emerges during train- ing. While prior work has focused on identifying binding mechanisms in pretrained models, we examine the devel- opmental trajectory through which these mechanisms are learned. Our results show that the model progresses through distinct phases- from random prediction to shallow heuris- ties to systematic binding-wii wwith rapid nonlinear transitions between phases. This developmental perspective provides new insights into how neural networks acquire structured reasoning capabilities.
D. Choice of Sampling Strategy
We carefully selected the program sampling strategy to en- sure it would effectively test the model's capacity to track variable assignments. Early experiments with simpler dis- tributions showed that models could solve the task without developing genuine variable binding mechanisms, instead relying surface-level heuristics.
We found that several distribution characteristics made the task too simple. Programs with fewer distractor chains, uniform sampling of variables, or shorter reference chains allowed models develop shortcuts. For instance, without a weighted sampling approach, models could simply learn to associate the correct answer with the variable appearing in the longest chain, bypassing the need to track actual variable bindings.
B. Program Structure
A sample synthetic program (abbreviated from the full 17- line version) looks like this:
To address this, we implemented a sampling strategy where chains are selected for extension with probability propor- tional the chain length cubed. This cubic weighting cre- ates programs with multiple lengthy chains, preventing the model from relying solely chain length as a heuristic. Additionally, we used rejection sampling to balance our dataset across the four referential depths while maintain- ing sufficient distractor chains that branch from the main query chain. The resulting distribution effectively forces the model to track specific variable bindings through the pro- gram's 's execution rather than using pattern matching based surface features.
a
2 b = a
d = 2
= d
b
e
# C
The lines highlighted in green form the reference chain for the queried variable c, while the lines in orange show an independent distractor chain.
The synthetic program structure can be formally described by the following grammar:
program ++ 26 16 query
E. Training Setup
stmt + var (const var)
We train our model from scratch using standard causal lan- guage modeling on complete sequences, where each se- quence consists of the 17-line program string (16 lines of variable assignments and final query line) and its expected
query + #var:
var + [a-z]
const + [0-9]
13

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
result. Importantly, the model has no prior exposure to natural language or programming data, and must learn mechanism for variable binding and dereferencing purely from our synthetic data. The model is trained for 15 epochs using the Adam W optimizer with 31 = 0.95, 32 = 0.999, and a base learning rate of 1 x 10 (Loshchilov & Hutter, 2017) with a batch size 64 programs.
sampled and excluded from the training set but added the test set. As illustrated in Fig. 7, the accuracy progression and final performance of this model closely mirror those of the model trained on the complete dataset. This indicates that the model effectively generalizes to novel combinations it was not exposed to during training, confirming its ability for compositional reasoning rather than mere memorization.
The learning rate follows linear decay schedule with warmup. Starting from zero, the learning rate linearly in- creases to 1 x 10 over 750 warmup steps, then linearly decays to zero over the remaining training period. For reg- ularization, we applied dropout with a rate of 0.1 and a weight decay coefficient of x 10-4.
Original vs. Composiiional Model Accuracy
120%
Model
Compestional geneniiziion
40%
40%
F. Training Accuracy Across Multiple Random Seeds
0%
We examined the stability of the observed learning phenom- ena by performing multiple training runs, each initialized with a different random seed (specifically, seeds 42, 256, 416, 512, 1024, and 3407). Seed 42 corresponds to the original training run analyzed earlier in the main text. The evolution of test accuracy during training for each of these runs is shown in Fig. 6. All runs exhibit a learning dy- namic characterized by three distinct phases with similar transition points, consistent with the pattern identified in Fig. 2. This reproducibility strongly suggests that the ob- served three-phase learning dynamic is robust property of the training process rather than artifact of specific random initialization.
Traitin Staps
Figure 7. Training accuracy comparison for evaluating composi- tional generalization. The model trained with 10% variable/num- ber combinations held out achieves comparable test accuracy throughout training to the model trained on the full dataset.
H. Generalization to Longer Programs
To assess generalization beyond the training distribution, we evaluated the model on programs with lengths varying from
2 to 25 lines, contrasting with the fixed training length of 16 lines. Fig. presents the model accuracy stratified by the line number containing the correct answer. The results indicate robust generalization when the correct answer is located on line 2 or beyond, where high accuracy is main- tained across diverse program lengths. However, when the correct answer is on line 1, the model appears to rely on a shallow line-1 heuristic. Consequently, the model exhibits poor generalization for programs substantially different in length from the training regime, with accuracy dropping sharply for lengths below 14 and above 17 lines, since the learned heuristic itself does not generalize well beyond the specific 16-line structure seen during training.
Teel Accurasy Acroes Model Seeds
Model
Sant (wxgg nal mode))
Bood 1234
Bood 256
Bood 3407
Ausurey
60%
Bood $12
Sent 416
40%
20%
05
10"
Training Stops
Figure 6. Comparison of test set accuracy curves across multiple training runs initiated with different random seeds. All runs exhibit similar three- phase learning dynamics and transition points with slightly different convergence speeds.
I. Generalization to Programs with More Hops
We also evaluated the model's ability to generalize pro- grams with significantly more hops than encountered during training. While trained on programs with maximum of 4 hops, the model was tested on programs ranging from to 13 hops. The results are shown in fig. 9. For pro- grams where the correct answer lies beyond the first line, the model demonstrates high accuracy across all hop counts tested. This suggests the acquisition of a systematic mech-
G. Generalization to Unseen Combinations
We explicitly checked for potential memorization by train- ing a model on a subset of the original training data. Specifi- cally, 10% of variable/number combinations were randomly
14

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
Model Accuracy Program Length
by line. To investigate this hypothesis, applied linear probes the output of each Transformer layer, specifically at newline token positions (An) where program state updates occur. These probes attempt to extract information about the current program state after each line of code.
100%
80%
Apsurecy
60%
20%
For our analysis, we represented program states as (26, 11) tensors-one one dimension for each possible variable name (26 lowercase letters) and another for each possible value (digits 0-9 plus nil for unassigned variables). We trained separate linear probes for each layer using the final model checkpoint.
0%
6
10
12
14
16
18
20
24
Program Le nigh (ines)
Programs Where Anreei en Line
rams Where ComeetÄneeer Une (But Not
Programe Aorert Une 2 (Netter
Figure 8. Generalization performance across varying program lengths (2-2) 25 lines). Performance is broken down by the line number containing the correct answer. While the model (trained on length 16) generalizes well when the answer is on line or later, accuracy drops significantly for first-line answers programs shorter or longer than the training length, the model leans line-1 hcuristic that failed to generalize on program lengths.
The results presented in Table 1 do not support our hy- pothesis of explicit program state representation: even the best-performing layer (layer 6) achieves only 30.87% accu- racy when predicting variable values (excluding unassigned variables) and merely 8.90% accuracy when predicting com- plete program states. These poor results suggest the model does not maintain complete program state in a linearly decodable format at any single vector location.
anism rather than reliance memorized patterns that is specific to up to programs with up 4 hops in the training set.
This finding is particularly significant when contrasted with our successful causal intervention experiments. While linear probing attempted to extract complete program state from individual vectors, dynamic flow of specific information through the network. The success of these causal interventions indicates that the model implements variable binding not as static state repre- sentations but as dynamic process of information routing, The model appears have learned a more efficient strat- egy that tracks only the relevant variable bindings through specialized attention patterns, rather than representing the entire program state-a hypothesis further explored in our main analysis.
On the other hand, when the correct answer is located on the first line, the accuracy degrades faster as the number of hops increases. This performance drop aligns with the limitations of shallow heuristic, similar the results on program length generalization.
patching experiments revealed the
Accuracy Number Hops
100%
80%
Accurayy
60%
Layer
State Acc (%)
Var. Acc (Excl. Nil) (%)
20%
1
7.71
21.28
0%
11
13
2
8.42
25.36
Number of Hops
3
8.78
28.56
Programs Where Arower on Line
4
8.87
28.52
Programs Where CompstAnswer Une (But Not
5
8.73
29.80
Programs' CoreeiArsweer Line 2 (Nalther
6
8.90
30.87
7
8.88
30.72
8
8.83
30.03
Figure Generalization performance across varying program hop counts. The model maintains high accuracy on programs up to 13 hops (far exceeding the maximum 4 seen during training) when the correct answer is not on lines 1 or 2, indicating systematic solution. In contrast, accuracy degrades for programs where the
9
8.85
29.61
10
8.73
28.90
11
8.72
28.66
12
8.77
28.51
answer resides line or line 2 as hop count increases.
Table 1. Linear probing accuracy at program state and variable (excluding nil values) levels by layer.
J. Linear Probing Results
K. Single Head Activation Patching Results for All Layers
The model's strong performance suggests it might develop structured internal representations of program states, poten- tially simulating the execution of variable assignments line
15

---

How Do Transformers Learn Variable Binding in Symbolic Programs?
Ref Depth Programs
Ref Depth 2 Programs
Ref Depth 3 Programs
Depth Programs
Ref Depth Programs
Ref Depth 2 Programs
Ref Depth 3 Programs
Ref Depth Programs
Head
Hoad
Head
Head
Head
Layer
Layer
Heed
Head
Layera
Layer 8
Hwad
Head
Heard
Head
Head
Hoad
Head
Head
Head 2-
Head
Head 9
Head
Layers
Head
Layer
Hoad4 4
Head
Head
Head
Head
Head
Hoad
Awerage Dffrrnncee
Layer
Layer 10
Head
Head
Head
Heed
Head
Head
Hoad
Layars
Head
Layer 11
Head44
Head 5-
Head
Head
Head
Head
Head?
Head
Head
Head
Head
Layer 12
Layer
Deeth Rvg
Onery Vanable
Calon
Dandi Rie
RelDre me Quey vanuble
calor
Rel Dopth Rio Rel mo
Re Oupn Rig
Qrery Varatta Fins color
Rel Crepn F810 Rei Crpth Rivd Rei Chem Rivg Ref Drenh Fivg Cuarr Mariale
Calon
Duysh mo
Orey tamasee
Caal
Rei ReiDmem mi Dapan mit
Orey semsale
Pinel colon
Rei Chesh Rio Ref Copet Rid
Rei Ownn Rin
Coery Hemase Pinel Color
Cepal Rvo Ref Dennn Rue Re Coeth rive
Rel Death Vansale Oowy
Colon
Figure 10. Patching results targeting individual attention heads across all 12 layers of the model. For each (head, token) position, replace only that head's contribution to the residual stream with its counterfactual value and compute logits.
16


Reviews:
[{"review_id": "loVFjg4Y3o", "reviewer": "Reviewer_stBZ", "summary": "This paper performs a mechanistic interpretability study on a transformer trained on a synthetic task that requires tracking values assigned to variables in a mock programming language. Programs consist of 16 variable assignments, and the transformer has 12 layers. The authors use interchange interventions to identify the path through the transformer that the original value of the variable is transmitted through. Given a variable whose value is queried at the end of the sequence, the transformer solves the task by attending to the last assignment to that variable, doing so across multiple layers if necessary to figure out the chain of assignments and identify the constant used for the first assignment. This is intermixed with naive solutions that are specific to the first and second lines. The authors analyze the evolution of this solution over training and identify three main phases.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thorough review and positive feedback that our paper is \"straightforward and generally well-written\" with experiments that \"validate the authors' claims convincingly.\" \n\n## Responses to your questions:\n\n### Q1&Q2: Why exactly 17 lines? Did you try testing on longer/shorter programs? What happens with chains longer than 12?\nSee the response to Reviewer ZyPs for the results of new generalization experiments.\n\n### Q3: Did you try reproducing this over multiple training runs?\nYes, we trained additional models with different random seeds. All runs exhibit the same three distinct learning phases with similar transition points, demonstrating the robustness of our findings. See [the test set accuracy plot across different training run seeds](https://imgur.com/a/comparison-of-test-set-accuracy-across-different-seeds-OXfXTSF) for more details (direct link to new plot: https://i.imgur.com/cvf0vzO.png).\n\n**Action:** We will add this new experiment and plot.\n\n### Q4: Did you try other distributions?\nYes, we explored different sampling strategies during task design. Our final distribution was deliberately constructed to be as challenging as possible while remaining learnable. \n\nSimpler distributions (e.g., with fewer distractor chains, uniform sampling of variables, or shorter chains) made the task trivially solvable using surface-level heuristics. For instance, without our weighted sampling approach (where we choose chains to extend with probability proportional to chain length cubed), the model could simply learn to associate the answer with the longest variable chain.\n\nOur rejection sampling procedure ensures balance across referential depths while maintaining sufficient distractor chains that branch from the main query chain. This forces the model to genuinely track variable bindings rather than rely on pattern matching. \n\nThis challenging distribution was essential for studying true variable binding mechanisms rather than shortcut learning.\n\n**Action:** We will make this more explicit in the appendix.\n\n### Minor points\nThe logit differences were normalized to a [0,1] scale for visualization clarity. We agree this could be clearer and will explicitly state the normalization in the figure caption.\n\nWe acknowledge your concern about figure readability. In the camera-ready version, we will:\n\n- Increase font sizes throughout all figures\n- Use more visually distinct colors and line styles in Fig. 2b and Fig. 4\n- Add clearer legends and improve visual mapping between colors and values\n\n**Thanks again for your helpful feedback! Do these clarifications and planned changes address your concerns?**", "claims_and_evidence": "Yes, for the most part.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "This paper does not have theoretical claims.", "experimental_designs_or_analyses": "Yes, although I did not check the appendix.", "relation_to_broader_scientific_literature": "This is a mechanistic interpretability paper on a new synthetic variable assignment task.", "essential_references_not_discussed": "Not that I'm aware of.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The paper is straightforward and generally well-written.\n2. The experiments essentially validate the authors' claims convincingly in terms of how information flows through the transformer.\n3. The phases discovered during training are interesting.\n4. They made a snazzy website to view the results.\n\n**Weaknesses**\n\n1. The scope of the paper is limited to a single transformer trained on a simple synthetic task, so its real-world impact is limited.\n2. The synthetic task is simple and could have been made more complicated. See Questions.\n3. Some of the figures are hard to read.", "other_comments_or_suggestions": "1. 155 right: this is redundant\n2. Fib 2b is too hard to read. The colors and numbers should be related visually, perhaps with different shades of gray.\n3. Fig 4: It's hard to keep track of the line colors.", "questions_for_authors": "1. Why exactly 17 lines for all examples? Did you try testing on longer or shorter programs to check if the construction it learns is sensitive to the length of the training data?\n2. It seems like the learned solution is limited by the number of layers in the transformer. What happens when you test on any examples with chains longer than 12?\n3. Did you try reproducing this over multiple training runs?\n4. This analysis is likely sensitive to the distribution of the training data. Did you try other distributions?\n5. Fig 3a: Why is logit difference on a scale from 0 to 1? Even if it's probabilities, shouldn't it be on a scale from -1 to 1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ojObORJXaI", "reviewer": "Reviewer_kEn5", "summary": "The authors investigate how variable binding is learned by decoder-only transformer models. In particular, they use causal interventions to understand how transformers propagate binding in tasks where they are asked for the value of a variable, e.g., \"c\" where (\"c=b, p=f, b=a, f=2, a=1\") is given. Their key findings are: 1) A general mechanism for dereferencing is acquired in later stages of training, and gradually subsumes heuristics the model learned early on, 2) the full program state is not stored in the residual stream, and 3) certain attention heads' role in routing to dereference can be understood. Finding #1 is particularly interesting, as previous narratives around the acquisition of capabilities in transformers hypothesized that these replace early heuristics and are learned alongside, rather than on top of, them.\n\nThe paper is well-written and clear, and a very nice interactive webpage is provided alongside the paper where results are clearly explained and can be explored.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment that our paper is \"well-written and clear\" with claims that are \"well-supported.\" We're especially pleased you highlighted our most interesting finding: that relevant circuits form with \"behavior itself being fairly uninteresting.\"\n\n## Response to your question: Linear probing vs. patching\n> Why do you think linear probing works so poorly yet you are able to see clean effects from patching?\n\nOur linear probing experiment attempts to extract a complete program state from a single vector, while our patching experiments reveal the causal path of specific information through the network. The latter succeeds because the model implements variable binding as a dynamic process of information routing rather than as static state representations.\n\nThe poor linear probing results suggest the model doesn't maintain a complete program state in a linearly decodable format. Instead, our patching results reveal that the model learns a more efficient strategy that dynamically tracks only the relevant variable bindings through specialized attention patterns.\n\n**Action:** We will make this more explicit.\n\n## Addressing your other feedback:\n- **Grammar issues:** We will correct all grammatical errors, including:\n    - Line 95: \"open question of how does\" → \"open question of how\"\n    - Line 136: \"hand side of the quality\" → \"equality\"\n    - Line 291: We'll rephrase the statement about uniform representations\n\n- **Figure density:** We will redesign Figure 4 with:\n    - A larger, clearer legend at the top\n    - Thicker, more visually distinct lines\n    - Consistent labeling schemes\n\n**Action:** We will implement all these improvements in the camera-ready version.\n\n**Thanks again for your helpful feedback! Do these clarifications and planned changes address your concerns?**", "claims_and_evidence": "Claims are well-supported.", "methods_and_evaluation_criteria": "They used the existing method of interchange interventions which is a sufficient patching method to isolate heads relevant to the mechanism they are studying.", "theoretical_claims": "No theoretical claims are made in the body of the paper.", "experimental_designs_or_analyses": "Experiments are suitable.", "relation_to_broader_scientific_literature": "The paper presents various results around variable binding in transformers. The most interesting of these by far is the way in which the relevant circuits form, with the behaviour itself being fairly uninteresting. As such, the paper forms a reasonable contribution to the literature.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "* Figure 4 is very dense; would prefer large legend at the top, and perhaps thicker lines. Some of the color differences are not that substantial.\n* Figure 3 - text very small again. Maybe there is a way you can label ticks with an example as reference to make it easier to parse; e.g., say for example \"x=3:\" and then label x, =, 3, :.\n* Line 95 column 2 - \"open question of how does...\" incorrect grammar.\n* Line 136 column 2 - \"hand side of the quality\" → \"equality\".\n* Line 271 - there is a \"TODO\".\n* Line 291 column 2 - \"this is evidence that a uniform...\" incorrect grammar, and not sure this is a completely valid claim. Representations may be similar (not uniform), but the attention head being implicated on both is a function of the representation after q,k projections, not residual stream? Though I concede likely to be similar.\n* Both in the intro and conclusion you make reference to the ongoing symbolic vs. connectionist debate and claim to contribute by showing that models can solve such tasks, but models have long been shown to complete vastly more complex tasks that require handling complex inter-dependencies, so this may be overblown.", "questions_for_authors": "Why do you think linear probing works so poorly yet you are able to see clean effects from patching?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jMgD41hwD1", "reviewer": "Reviewer_eScW", "summary": "In this paper, the \"variable binding\" ability of Transformer model is studied, which is to autonomously assign correct values to symbolic variables. The paper focuses on controlled experimental design. A symbolic program dataset is constructed, which consists of programs that involve value passing among variables. Based on the learning results on this dataset, it is concluded that the Transformer model has strong variable binding ability without specific training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback that our paper \"proposes an interesting paradigm in analyzing the symbolic learning ability of Transformer.\" We appreciate your critical assessment and have conducted new experiments to address your concerns.\n\n## Addressing key concerns:\n### 1. \"The model does not distinguish between symbols and values\"\n\nWe conducted new subspace analysis experiments specifically to address this concern:\n\n1. We applied PCA to the Transformer's residual stream vectors for the right-hand-side tokens of program lines across many different inputs.\n2. We trained linear classifiers (with L1 regularization) identify principal components of the residual stream correlated with numerical constants (10 components) and variable names (26 components), respectively.\n3. We performed interchange interventions on these selected components, using counterfactual examples that target the numerical constant or the variable name, achieving high success rates for both numerical constants (92.17%) and variable names (87.08%). \n\n**New findings:** The 2D UMAP projections of these subspaces across three training checkpoints (steps 1200, 17600, 105400) reveal a clear evolution toward separated clusters for these token types. By the final checkpoint, the model distinctly represents symbols and values in different subspaces. See the new [the variable-name subspace scatter plot](https://imgur.com/a/2d-umap-projection-of-variable-name-residual-stream-subspace-projection-26-selected-pca-components-KBEft1t) and [the numerical-constant subspace scatter plot](https://imgur.com/a/2d-umap-projection-of-numerical-constant-residual-stream-subspace-projection-26-selected-pca-components-SCyS18B) for more details (direct links to new plots: https://i.imgur.com/ugP0baG.jpeg, https://i.imgur.com/479j8Ch.jpeg).\n\n**Action:** We will add these new experiments and plots.\n\n### 2. Questions about training\n> How are the three phases decomposed?\n\nThe three phases emerge naturally during training without external parameter changes. They represent qualitative shifts in strategy that correspond to sharp transitions in performance (from ~12% to ~56% to >99% accuracy). We use a linear decay learning rate schedule with 750 warm-up steps, but the phase transitions do not align with learning rate schedule changes.\n\n> What objective is used for training the Transformer?\n\nWe use standard autoregressive next-token prediction (causal language modeling) as stated in the Appendix section \"Training Setup.\"\n\n**Action:** We will add this information more explicitly in the main text.\n\n> How to explain Fig. 2c? Why 1-hop accuracy converged slower than those with more hops?\n\nThis results from our task structure. In 1-hop programs, the answer is always a numerical constant directly assigned to the queried variable. The model's early \"line-1/line-2 heuristic\" works well for multi-hop programs where the first line often contains the root value, but performs poorly on 1-hop programs where the answer could be on any line.\n\n**Action:** We will explain this in the figure caption.\n\n### 3. Task design\nWhile we acknowledge the controlled nature of our task, its simplicity is intentional. By using a minimal design, we can perform detailed causal analysis that would be challenging with more complex symbolic tasks. Our new generalization experiments show the model learns a truly general mechanism rather than simply memorizing patterns.\n\nWe would like to clarify an important aspect of our experimental design: our model is trained entirely from scratch without any pre-training on natural language or programming tasks. When the model first encounters our variable assignment syntax (e.g., \"a=1\"), it has no prior knowledge of what these symbols mean or how they relate to each other. From this perspective, using alternative symbolic relationships like \"a is left of b\" would be functionally equivalent - the model would still need to learn these relationships from scratch.\n\nOur focus was on creating a minimal yet challenging test bed where we can precisely analyze how variable binding mechanisms emerge during training. The apparent simplicity of our task masks its fundamental complexity: the model must still develop sophisticated capabilities to track multi-step variable dependencies while ignoring distractors, regardless of the specific symbolic relationship used.\n\nThe controlled design was crucial for our mechanistic interpretability goals, allowing us to isolate and analyze the precise causal pathways through which variable binding is implemented in the network.\n\n## Additional improvements:\n- We will fix all typos, including the dataset split percentages\n- We will improve the clarity of Fig. 2d\n- We will add clearer explanations of the training objective and methodology\n\n**Action:** We will implement all these improvements in the camera-ready version.\n\n**Thanks again for your helpful feedback! Do these clarifications and planned changes address your concerns?**", "claims_and_evidence": "In my view, the claims are not well-supported by the evidence: 1) The constructed dataset is too limited. 2) The experimental results lack deep insights. More details are discussed in the experimental design part below.", "methods_and_evaluation_criteria": "Even though the constructed programs are interesting, I think that the dataset design is still limited to justify the results. More details are discussed in the experimental design part below.", "theoretical_claims": "No theoretical results are included in the paper.", "experimental_designs_or_analyses": "1. The construction of the datasets is limited. The programs involves only number assignments among variables, which is a very special type of symbols. It would be better to introduce broader type of symbols (e.g. a is left to b, b is left to c, then a is left to c) and more complicated program structures. In my view, this is not difficult since variable assignment is common in programming tasks.\n\n2. The experimental results are somehow superficial. In the current experiments, the values and symbols are treated equally as tokens. From the results, it is hard to tell whether the model distinguishes between them and actually learns what is a symbol or a number. In my view, we can also explain the experimental results from simply the view of attention weights. After training, the Transformer model just associate the symbols based on attention weights according to their common appearance in the dataset. From this view point, the results do not reflect insight on symbolic perspective. I think the current experiments lack strong evidence to eliminate this explanation.", "relation_to_broader_scientific_literature": "The paper is related to the researches on understanding the principles of Transformer models. Even though the purpose of the paper is to study symbol bindings, in my view, the results are similar to analyzing the attention weights among tokens in previous researches.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\n- I think the paper proposes an interesting paradigm in analyzing the symbolic learning ability of Transformer. \n\nWeaknesses:\n\n- The experimental design is not sufficient to support the major arguments as discussed above.", "other_comments_or_suggestions": "- The writing can be significantly improved. The paper is hard to read due to the lacking of clarity. Furthermore, the paper needs further proofreading. There are a number of typos in the paper, such as:\n\n  - Line 129: 90% training, 0.2% validation, and 0.98% testing.\n\n  - Line 270: Appendix TODO\n\n- The visual effect in Fig. 2d needs refinement.", "questions_for_authors": "- How three phases are decomposed? Are they just three stages of decreasing the learning rate of SGD optimization, or naturally appeared in the experiments without external parameter changing?\n\n- What objective is used for training the Transformer? Is it auto-regressive next token prediction? \n\n- How to explain Fig. 2c? I wonder why 1-hop accuracy converged slower than those with more hops.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Io4VSxgQrD", "reviewer": "Reviewer_ZyPs", "summary": "This paper studies how transformers can learn to implement variable binding through training. In particular, the authors focus on a specific task where the transformer is given a synthetic program where variables are assigned either numerical values or other variables, and the target is to correctly return the numerical value corresponding to a queried variable. By closely inspecting and probing the training trajectories, the authors identify three distinct phases on how transformers learn to implement variable binding. Initially, it simply recognizes the desired output should be a numerical value, then, it leans to use heuristics and shortcuts to generate correct output occasionally, and finally, it leans a systematic mechanism and produces nearly perfect performance over the test set. The study offers some interesting observations on how transformers can be trained to acquire capabilities involving manipulating and tracing symbols.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your assessment that we \"carefully designed the experiments to empirically validate [our] claims\" and for your thoughtful suggestions. Your questions about generalization inspired several new experiments that strengthen our findings.\n\n## Does the model actually learn a systematic mechanism?\n\nWe appreciate the critical perspective on whether the model indeed learns a systematic mechanism in addition to a shallow heuristic, and conduct generalization experiments to further investigate the matter. Since our model is trained from scratch rather than pre-trained, testing on completely unseen tokens would mean testing on untrained embeddings, which wouldn't be meaningful. Instead, we conducted three rigorous generalization experiments:\n\n- **Held-out variable/number combinations**: We trained a model where specific variable/number combinations, randomly sampled and constituting ~10% of the original training data, were excluded from training but included in testing. The model successfully generalized to these unseen combinations, demonstrating compositional generalization rather than memorization. See [the compositional generalization plot](https://imgur.com/a/comparison-of-original-model-vs-compositional-generalization-model-qq9Z8wF) for further details. (direct link to new plot: https://i.imgur.com/esnjS3O.png) \n\n- **Program length generalization**: Testing on programs of 2-25 lines of assignment (vs. 16 in training), we found excellent generalization on programs where the answer is not on the first line, however when the answer is on the first or second line and the first line heuristic is used we see poor performance on shorter and longer programs. See [the program length generalization plot](https://imgur.com/a/model-accuracy-by-program-length-PPHwEyT) for further details (direct link to new plot: https://i.imgur.com/dIaCWx8.png)\n\n- **Chain depth generalization**: Testing on programs with 1-13 hops (vs. max 4 in training), we again observed near-perfect generalization, even for 13-hop programs despite our model having only 12 layers. We again see that when the answer is on the first or second line and the first line heuristic the model is worse at higher depth programs. See [the chain depth generalization plot](https://imgur.com/a/model-accuracy-by-number-of-hops-KZytngj) for further details. (direct link to new plot: https://i.imgur.com/C2XEVwc.png)\n\nIn the paper, we claim that the model uses a shallow heuristic for programs where the answer is in the first or second line, and uses a systematic solution for other programs. \n\nWhen the answer isn’t on the first or second line, the model generalizes far beyond the structures seen during training – to unseen variable/constant combinations, deeper chains, and longer programs. These results further support our claim that the model learns a \"systematic mechanism for dereferencing\" rather than memorizing patterns. \n\nMoreover, when the answer is on the first or second line, we see seriously degraded generalization performance. These results further support our claim that these are exactly the inputs that a shallow heuristic is used to solve the task.\n\n**Action:** We will add these new experiments and plots.\n\n## Addressing clarity issues:\n- We will provide clearer definitions of technical terminology\n- We will add details on how logits are computed at each layer in Figure 3\n- We will fix the missing reference in Figure 3(c)\n- We will ensure consistent naming and referencing throughout (e.g., \"fig. 2a\" vs \"Fig. 2d\")\n\nWe will improve the readability of all figures by:\n- Increasing font sizes\n- Adding clearer labels and legends\n- Ensuring consistent formatting\n\n**Action:** We will implement all these improvements in the camera-ready version.\n\n**Thank you for your valuable suggestions that led to these additional experiments! Do these results and planned improvements address your concerns about whether we've demonstrated a truly systematic mechanism?**", "claims_and_evidence": "Yes, the authors have carefully designed the experiments to empirically validate their claims and hypotheses.", "methods_and_evaluation_criteria": "Although the task is simple and synthetic, it allows to verify the claims of the paper in a controlled way. However, in addition to the current empirical setting, I think that the authors should really consider a test set where the program structure remains the same as the training set, but the variables and the numerical values differ from the training set. Since the underlying logic to implement variable binding does not restrict to a fixed set of variables names or numerical values, in order to claim that the transformer \"leaned a systematic mechanism for variable binding\", it is necessary to demonstrate that the model is able to achieve nearly perfect performance on a test set where neither variable names nor numerical values are seen during training. For example, the authors may consider splitting the 26 letters into 13 for training and 13 for testing, and additionally choose integers from 11 to 20 for the numerical values in the test set.\n\n## update after rebuttal\n\nDuring the rebuttal, the authors provided a suite of new empirical results to further verify their claims in the paper. The new experiments cover 3 distinct out-of-distribution settings and, in my opinion, substantially improve the quality of the paper. I updated my initial rating based on the new results. Overall, I think this is a good work and I'd like to recommend acceptance.", "theoretical_claims": "NA (There is no theoretical claim that requires a proof in the paper)", "experimental_designs_or_analyses": "Overall, the experimental design is good. My major concern is on the choice of the test set. Since the authors claim that at the end of training, there is \"emergence of a systematic mechanism for dereferencing\", it is important to have a thorough test on whether the nearly perfect performance is indeed due to learning a systematic mechanism, rather than being just memorization (since the test set and the training set use the same set of variable names and numerical values, and the program is highly structured).", "relation_to_broader_scientific_literature": "Understanding how transformers can learn to manipulate symbols and perform basic reasoning tasks is very important. Although the distinct learning phases are not very surprising, given that in some other cases it has also been shown that the learning curve of transformers demonstrates multiple jumps followed by long plateaus, the detailed causal analysis in this paper is interesting.", "essential_references_not_discussed": "I am not aware of important references that are not discussed.", "other_strengths_and_weaknesses": "The clarity of the paper can be greatly improved. Definitions and technical terminology should be properly stated. The authors should provide details on how logits are computed at each layer in Figure 3. In the caption of Figure 3(c), there is a missing reference to Appendix which is now read as \"shown in Appendix TODO\".", "other_comments_or_suggestions": "I see occurrence of \"fig. 2a\", \"Fig. 2d\", \"Figure 3(a)\". Please be consistent with naming and referencing.", "questions_for_authors": "I would be very interested to know what happens if the variables names and numerical values in the test set differ from those in the training set.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies variable binding in Transformer neural network on a synthethic task that requires tracking values assigned to variables in a mock programming language. Leveraging tools from mechanistic interpretability, it is shown how the capacity for variable binding forms in three key stages, and what role various parts of the Transformer play for implementing variable binding.

This paper initially received mixed reviews. While reviewers are largely positive about the presentation, the quality of the experiments, the validity of the claims and the overall contribution, there are a few limitations worth highlighting. In particular, only a single configuration of the Transformer is studied, and only a single task is considered, which affects the generality of the claims made. Reviewers pointed out how "the values and symbols are treated equally as tokens", which leaves some doubt whether the model really treats them differently, and how it is necessary to compare on an out-of-distribution test-test. Both of these concerns were reasonably well addressed during the rebuttal.

Following the reviewer discussion, all reviewers are in agreement that this work should be accepted, and two reviewers increased their scores. The AC agrees that this paper is of sufficient quality and that there is enough of a contribution to accept. That said, I would ask the authors to adjust their title in light of the feedback received to better reflect the contribution of the paper, including its limitations.
Decision: Accept (poster)