Emergent Symbol-like Number Variables in Artificial Neural Networks
Authors: Satchel Grant
Noah Goodman
James Lloyd McClelland
Venue: Accepted by TMLR ()

Abstract: What types of numeric representations emerge in neural systems, and what would a satisfying answer to this question look like? In this work, we interpret Neural Network (NN) solutions to sequence based number tasks through a variety of methods to understand how well we can interpret them through the lens of interpretable Symbolic Algorithms (SAs)—precise algorithms describable by rules operating on typed, mutable variables. We use GRUs, LSTMs, and Transformers trained using Next Token Prediction (NTP) on tasks where the correct tokens depend on numeric information only latent in the task structure. We show through multiple causal and theoretical methods that we can interpret raw NN activity through the lens of simplified SAs when we frame the neural activity in terms of neural subspaces rather than individual neurons. Using Distributed Alignment Search (DAS), we find that, depending on network architecture, dimensionality, and task specifications, alignments with SA’s can be very high, while other times they can be only approximate, or fail altogether.  We extend our analytic toolkit to address the failure cases by expanding the DAS framework to a broader class of alignment functions that more flexibly capture NN activity in terms of interpretable variables from SAs, and we provide theoretic and empirical explorations of Linear Alignment Functions (LAFs) in contrast to the preexisting Orthogonal Alignment Functions (OAFs). Through analyses of specific cases we confirm the usefulness of causal interventions on neural subspaces for NN interpretability, and we show that recurrent models can develop graded, symbol-like number variables within their neural activity. We further show that shallow Transformers learn very different solutions than recurrent networks, and we prove that such models must use anti-Markovian solutions—solutions that do not rely on cumulative, Markovian hidden states—in the absence of sufficient attention layers.

Published in Transactions on Machine Learning Research (07/2025)
Emergent Symbol-like Number Variables in Artificial Neural Networks
Satchel Grant
grantarb
Deportments of Psychology and Computer Science
Stanford University
Noah D. Goodman
Crpoddaansstaanaaoo edu
Departments Psychology and Computer Science Stanford University
James L. McClelland
jumceSstanfor edu
Departments of Psychology and Computer Science
Stanford University
Reviewed on OpenReview: https: oppenreveennn net/ forum? YPnYpáru5W
Abstract
What types of numeric representations emerge in neural systems, and what would satisfying answer to this question look like? this work, we interpret Neural Network (NN) solutions to sequence based number tasks through a variety of methods to understand how well we can interpret them through the lens of interpretable Symbolic Algorithms (SAs) - precise algorithms describable by rules operating on typed, mutable variables. We use GRUs, LSTMs, and Transformers trained using Next Token Prediction (NTP) tasks where the correct lokens depend numeric information only latent in the lask structure. We show through multiple causal and theoretical methods that we can interpret raw NN activity through the lens of simplified SAs when we frame the neural activity in terms of neural subspaces rather than individual neurons. Using Distributed Alignment Search (DAS), we find that, depending on network architecture, dimensionality, and task specifications, alignments with SA's can be very high, while other times they can be only approximate, or fail altogether. We extend our analytic toolkit to address the failure cases by expanding the DAS framework to broader class of alignment functions that more flexibly capture NN activity in terms of interpretable variables from SAs, and we provide thcoretic and empirical explorations of Linear Alignment Functions (LAFs) in contrast to the preexisting Orthogonal Alignment Functions (OAFs). Through analyses of specific cases we confirm the usefulness of causal interventions on neural subspaces for NN interpretability, and we show that recurrent models can develop graded, symbol-like number variables within their neural activity. We further show that shallow Transformers learn very different solutions than recurrent networks, and prove that such models must use anti- mariMMarovvin solutions solutions that do not rely cumulative, Markovian hidden states-in the absence of sufficient attention layers.
ooutions-sslutiions
1 Introduction
We can see examples of the modeling power of Neural Networks (NNs) in both biological NNs (BNNs) from the impressive capabiliuies of human cognition, and in artificial NNs (ANNs) where recent advances have had such great success that ANNs have been crowned the "gold standard" many machine learning communities Alzubaidi al.] 2021). The inner workings of NNs, however, are still often opaque. This is, in part, due their representations being highly distributed. Individual neurons can play multiple roles within a network in what's called population encoding. In these cases, human-interpretable information is encoded across

---

Published in Transactions on Machine Learning Research (07 (07/2023) /025)
1+1
1+1
1+1-1
1-1-11
BOS D
ODTO
D
T
R
R
EOS BOS
D
D
T
R
EOS
GRU/LSTM
Transformer
Figure 1: Different architecture's solutions achicving the same accuracy on a numeric equivalence task. The rectangles represent tokens for lask in which the model must produce the same number of R lokens ending with the EOS token as it observed D tokens. The T token indicates the end of the D tokens (see Methods 3.1 p. The thought bubbles represent the values of causally discovered neural variables encoded within the models' representaliinss The recurrent models encode a single count variable that increments up before the T token and down after the T token, with 0 indicating the end of the task. Transformers learn solution in which they recompute the task relevant information from the input tokens at each step in the sequence. All NoPE transformers align with the displayed Transformer solution. RoPE transformers can partially rely on positional information unless they are trained on a variant of the task that breaks number-position correlations.
populations of neurons rather than within any individual unit (Hinton et al. 1986 Smolensky
1988
Olah
et al.
2017 2020 Elhage
al.] 2022 Scherlis et al. 2023 Olah 2023
Symbolic Algorithms (SAs), in contrast, defined as processes that manipulate distinct, typed entities according to explicit rules and relations, can have the benefit of consistency, transparency, and generalization when compared to their neural counterparts. A concrete example of an SA is a computer program, where the variables are distinct, typed, mutable entities, able to represent many different values, processed by well defined functions. There are many existing thcories that posit the necessity of algorithmic, symbolic, processing for
higher level cognition (Do & Hasselmo 2021 Fodor Pylyshyn 1988 Fodor, 1975 1987 Newell 1980 1982
Pylyshym 1980: Marcus 2018 Lake et al. (017].. Human designed symbolic cognitive systems, however, lack the expressivity and performance of NNs. This is apparent in the field of natural language processing
where neural architectures trained on vast amounts of data (Vaswani et al. 2017 Brown et al.] 2020 Kaplan et al. 2020) have swept the field, surpassing the pre-existing symbolic approaches. Despite the differences between NNs and SAs, it might be argued that NNs actually implement simplified SAs; or, they may approximate them well enough that seeking neural analogies to these simplified SAs would be a powerful step toward an interpretable, unified understanding of complex neural behavior. In one sense, this pursuit is trivial for ANNs, in that ANNs are, by dcfinition, aligned to the computer program that defines them. The algorithms they acquire through training, however, not trivial to understand, and their complexity can be so great. that simplified SAs become: useful for explaining, predicting, and controlling their behavior. This approach of seeking to characterize NNs (either biological or artificial) in terms of simplified, interpretable SAs is one way of construing the goal of many research programs in cognitive science, neuroscience, and mechanistic interpretability.
In this work, we narrow our focus to numeric cognition and ask, how can we understand neural implementations of numeric values at the level of SAs? Numeric reasoning has the advantage of being well studied in humans of different ages and experience levels, which provides a powerful domain for comparisons between BNNs and ANNs (Di Nuovo & Jay] 2019). And numeric domains provide the benefit of tasks built upon well defined variables. We focus on a numeric equivalence task that was used to test the numeric abilities of humans whose language lacks explicit number words (Gordon] 2004). The task is formulated as a sequence tokens, requiring the subject to produce the same number of response tokens as quantity of demonstration tokens initially observed at the beginning of the task. This task is interesting for computational settings because the training labels vary in both identity and sequence length, and quantities are never explicitly labeled. Similar
2

---

Published in Transactions on Machine Learning Research (07/2025)
versions of this task have also been used in previous theoretical and computational work EElNNagarr et al.
2023
Weiss et al. 2018 Behrens et al.j 2024), providing platform to expand our understanding of seemingly disparate modeling systems in unified ways.
What sorts of representations do ANNs use to solve such a task and how do they arrive at these representations? Do the networks represent numbers in a shared system, or do they use different systems for different situations? Is it propitious to think about their representations as though they are discrete variables in an SA, or would it be better to think of their neural activity change over the course of training, and do the answers vary based on task and architectural details? How can we unify the way we understand NN solutions in satisfying ways for cognitive scientists, ncuroscientists, and computer scientists alike? We set out to understand NN neural activity through the lens of simplified, interpretable SAs using causal interventions to support our interpretations.
graded continuum? Do the answers to these questions
In this work, we pursue these questions by training Gated Recurrent Units (GRUs) (Cho et al. 2014). Long Short-Term Memory cells (LSTMs) (Hochreiter & Schmidhuber [197], and Transformers on numeric equivalence tasks using Next Token Prediction (NTP). We then provide causal, correlational, and theoretical analyses such activation patching, Principal Component Analysis (PCA), attention visualizations, and Distributed Alignment Search (DAS) (Geiger et al. 2021 2023) to understand the networks' representations and solutions, and we introduce the notion of an Aligmment Function to the DAS framework which we use to frame neural activity in terms of functions of interpretable variables. We summarize our contributions as follows:
1. We show through causal interventions the emergence of graded symbol-like neural variables in RNNs, where neural variables are defined representational subspaccs that causally align with variable from an SA, and they are "graded' or "symbol-like" when they exhibit signatures of a continuum rather than being fully discrete.
2. We show that soemingly insignificant task variations can drastically affect the NN's alignment to the SAs, molivating us to extend the DAS framework to a broader class of alignment functions, including linear alignment functions (LAFs). We show that LAFs can be used to control NN behavior through their neural activity aligned to interpretable SAs, and we provide thcoretical analysis of LAFs and Orthogonal Alignment Functions (OAFs) to better understand their properties.
3. We show empirically that shallow Transformers use an auti-Markovian solution to the numeric tasks, and we show theoretically that Transformers must use anti-Markovian solutions in all tasks in the absence of sufficient attention layers.
4. Through our specific analyses, we confirm the utility of interpreting NNs the level of neural subspaces and using causal interventions to make claims about NN solutions.
We use results to encourage the use of multiple causal interpretability tools for any representational analysis, to highlight functional differences that might emerge from architectural/tast constraints, and to demonstrate the nuances of interpreting neural activity at the level of SAs.
2 Related Work
Many prior works have attempted to describe ANNs through SAs. Some relevant examples include Lindner et al. 2023) who built a system to compile human written code into transformer models and Michaud et al. (2024) who perform lincar and symbolic regression on simplified NN represcntations to generate programs that perform like the NNs. Both of these cases focus on generating sufficient SAs for network behavior rather than relating the internal mechanisms of an existing NN to the resulting SAs. Another prominent example is the work of Nanda et al. (2023) who showed that small transformers trained on modular addition use discrete Fourier transforms combined with trig identities to solve the task. Nanda et al. used a combination of examinations and ablations to defend their claims.
We wish to highlight the importance of using causal manipulations for interpreting NN solutions rather than relying on correlational analyses. Causal inference broadly refers to processes that isolate causal effects of individual components within larger system (Pearl 2010). An abundance of causal interpretability
3

---

Published in Transactions on Machine Learning Research (07/)025)
variants have been used to determine what functions are being performed by NN models' activations and circuits (Olah et al. 2018 2020 Wang et al. 2022 Geva et al.] 2023 Merrill et al. 2023 Bhaskar et al. 2024 Wu et al. 2024). Vig et al. (2020) provides an integrative review of the rationale for and utility of causal mediation in NN analyses. We rely and build upon DAS for many of our analyses. DAS can be thought of as a specific lype of activation patching (also referred to as causal tracing) (Meng et al. 2023 Vig
ct al. 2020: Heimersheim & Nanda 2024). In scction 3.4, we introduce the notion of generalized alignment functions which can be thought of as a specific class of mapping models from Ivanova et al. (2022) where the
mapping model in our case defines an invertible, causal relationship between neural subspaces from NN and
interpretable variables from an SA: the success of the relationship is measured through NN behavior. Other
works such as Williams et al. (2021) have explored differences between orthogonal and linear relationships between distributed representations where they largely focused their analyses on formally defining metrics on representational dissimilarity.
Many publications explore ANNs' abilitics to perform counting tasks (Di Nuovo & McClclland
2019 Fang
et al. Sabathiel et al. 2020 Kondapaneni & Perona 2020 Nasr et al., 2019: Zhang al. 2018 Troil
2018
et al.
2018
and closely related tasks (Csordás et al. 2024) Our tasks and modeling paradigms differ from
many of these publications in that numbers are only latent in the structure of our tasks without explicit
teaching of distinet symbols for distinct numeric values. El-Naggar et al. (2023) provided a theoretical
treatment of Recurrent Neural Network (RNN) solutions to a parentheses closing task, and Weiss et al. (018)) explored Long Short-Term Memory RNNs (LSTMs) (Hochreiter & Schmidhuber 1997p and Gated Recurrent
Units (GRUs) (Cho et al. 2014) in a similar numeric equivalence task looking at the activations. These works showed correlates of magnitude scaling solution in both theoretical and practically trained ANNs. Our work builds on their findings by using causal methods for our analyscs, expanding the models considered, and introducing new theoretical and empirical analyses. [Berrnns et al. (2024) explored transformer counting solutions in a task similar to ours. Our work extends beyond theirs by exploring positional encodings, avoiding explicit labels of the numeric concepts, using causal analysos, and providing a theoretical focus on solutions.
3 Methods
3.1
Numeric Equivalence Tasks
Each task we consider is defined by Next-Token Prodiction (NTP) task over sequences, in the example shown in Figure 0 The goal of the task is to reproduce the same number of response tokens as demonstration tokens observed before the Trigger (T) token. Each sequence starts with a Beginning of Sequence (BOS) token and ends with an End of Sequence (EOS) token. Each sequence is generated by first uniformly sampling an object quantity from the inclusive range of to 20 where 20 was chosen to match the human experiments of Pitt et al. (2022). The sequence is then constructed as the combination of two phases. The first phase, called the demonstration phase (demo phase), starts with the BOS token and continues with a series of demo tokens equal in quantity to the sampled object quantity. The end of the demo phase is indicated by the trigger token after the demo tokens. This also marks the beginning of the response phase (resp phase). The resp phase consists of a series of resp tokens equal in number to the demo tokens. After the resp tokens, the end of the sequence is denoted by the EOS token.
During the teacher forced, NTP model training, we include all tokens the NTP loss. During model evaluation and DAS trainings, we only consider tokens in the resp phase-which are fully determined by the demo phase. During model trainings, we hold out the object quantities 4, 9, 14, and 17 as way to examine generalization. We chose 4, 9, 14, and 17 to semi-unifrmlly cover the space of possible numbers while including even, odd, and prime numbers and ensuring that training covered 3 examples at both ends of the training range. A trial is considered correct when all resp tokens and the EOS token are correctly predicted by the model after the trigger. We include three variants of this task differing only in their demo and resp token instances.
Multi-Objes Task: there are 3 demo token instances (D1, D2, D2} with a single response token instance, R. The demo tokens are uniformly sampled from the 3 possible instances. An example input sequence with
4

---

Published in Transactions on Machine Learning Research (07/2025)
object quantity of 2 could be: 'BOS D3 D1 T,, with ground truth response of 'R R EOS'. All possible tokens are contained in the set (BOS, Dj: D2, D3: T, R. FOS).
Single-Object Task: there is single demo token instance, D, and a single response token instance, R. An example of the imput sequence with an object quantity of 2 is: 'BOS D D T', with a ground truth response of 'R FOS'. All possible tokens are contained in the set {BOS, D, T, R, ROS).
Same- Object Task: there is a single token instance, C, used for both the demo and resp tokens. An example of the imput sequence with an object quantity of 2 is: 'BOS C T,, with ground truth response of 'C EOS'. All possible tokens are contained in the set (BOS, C, T, EOS}.
For some transformer trainings, we include Variable-Length (VL) variants of each task to break count- position correlations. In these variants, each token in the demo phase has a 0.2 probability of being sampled as unique 'void" token type, V, that should be ignored when determining the object quantity of the sequence. The number of demo tokens will still be equal to the object quantity when the trigger token is presented. As an example, consider the possible sequence with an object quantity of 2: "BOS V D V T R R EOS'.
3.2 Model Architectures
The recurrent models in this paper consist of Gated Recurrent Units (GRUs) CCho et al., 2014), , and Long Short-Term Memory networks (LSTMs) (Hochreiter & Schmidhuber 1997). These architectures both have
Markovian, hidden state vector that bottlenecks all predictive computations following the structure:
h1+1 (11,,0))
(1)
f g((h+t)
(2)
Where he the hidden state vector at step t, z, is the imput token at step t, f is the recurrent function (cither a GRU or LSTM cell), and g is a multi-layer perceptron (MLP) used to make a prediction, denoted 8141, of the token at step ++1.
We contrast the recurrent architectures against transformer architectures (Vaswani et al. 2017 Touvron
et al. 2023 Su et al.] 2023) in that the transformers use a history of input tokens, X. each time step, f, to make prediction:
21, .. Fz), at
4++1 f(x.)
33)
Where f now represents the transformer architecture. We show results from layer, single attention head transformers that use No Positional Encodings (NoPE) (Haviv et al. 2022) and Rotary Positional Encodings (RoPE) (Su et al. 2023). Refer to Supplemental Figure 5 for more model and architectural details. We also consider one-layer transformers with No Positional Encodings (NoPE) in Results section 4.2.2 For all of our analyses except the training curves in Figure 3 we first train the models to > 99% accuracy on their respective tasks before performing analyses. The models are evalnated on 15 sampled sequences of each of the 16 trained and 4 held out object quantities. We train model seeds for each training condition. One seed from the transformer models in both the Variable-Length Multi-Object and Variable-Length. Same-Object tasks were dropped without replacement due their low accuracy.
3.3 Symbolic Algorithms (SAs)
In this work, we examine the alignment of 3 different SAs to the models' distributed representations.
Up-Down Program: uses single numeric variable, called the Count, to track the difference between the number of demo tokens and resp tokens at each step in the sequence. It also contains Phase variable to determine whether it is in the demo or resp phase. The program ends when the Count is equal to 0 during the resp phase.
Up-Up Program: uses two numeric variables the Demo Count and Resp Count to track quantities at cach step in the sequence. It uses a Phase variable to track which phase it is in. This program increments the Demo Count during the demo phase and increments the Resp Count during the resp phase. It ends when the Demo Count equal to the Resp Count during the resp phase.
5

---

Published in Transactions on Machine Learning Research (07/2025)
Context Distributed (Ctx-Distr) Program: queries history of inputs at each step in the sequence, assigns a numeric value to each, and sums their values to determine when to stop (contrasted against encoding a cumulative, Markovian quantity variable). More specifically, this program uses an Input Value variable for each input token, and assigns the Input Value a value of for demo tokens and -1 for resp tokens and computes the sum of the Input Values at each step in the sequence to determine the count. This program outputs the EOS token when the sum 0 and the sequence contains the T token.
We include Algorithms 2 and B in the supplement which show the pseudocode used to implement the Up-Down, Up-Up, and Ctx-Distr programs in simulations. Refer to Figure ! for an illustration of the Up-Down strategy and the Ctx-Distr strategy that is observed in some transformers.
It is important to note that there are an infinite number of causally equivalent implementations of these SAs. For example, the Up-Down program could immediately add and subtract from the Count at every step of the task in addition to carrying out the rest of the program as previously described. We do not discriminate between programs that are causally indistinct from one another in this work.
3.4 Distributed Alignment Search (DAS)
DAS measures the degree of alignment between a representational subspace from an NN and symbolic variable from a symbolic algorithm (SA) by testing the assumption that the model hidden state h E Rdm can be written as orthogonal rotation = Qh, where Q E Rdmxdm is orthonormal, E Rd consists of contiguous subspaces encoding high-level variables from SAs, and dm is the size of the hidden state. The benefit of this alignment is that it allows us to understand the NN's activity through interpretable variables and it allows us to manipulate the value of these variables without affecting other information.
Concretely, DAS performed on the Up-Down program tests the hypothesis that z is composed of subspaces Ecount encoding the Count, xphase encoding the Phase, and Zextra encoding extraneous, irrelevant activity.
eeount
2 =
Ephase
(4)
Textra
Each Zmer E Rewsa is a column vector of potentially different lengths salisfying the relation doaunt + dphase + dextra = dm. Under this assumption, the value of a high-level variable encoded in h can be freely exchanged through causal interventions using:
=
-var) + Dvar Qhar))
(5)
Where Dvar E Rdmxxdn is a manually chosen, diagonal, binary matrix with dyar non-zero elements used to isolate the dimensions that make up ēvar, here is the source vector from which the subspace activity is harvested, hir the target vector into which activity is substituted, and h² the resulting intervention vector that we then use to replace hrr in the model's processing, allowing the model to make predictions based a replaced value of variable var following the intervention.
DAS relies on the notion of counterfactual behavior to create intervention data to train and evaluate Q. For a given SA, we know what the SA's behavior will be after performing a causal intervention on one of its variables. The resulting behavior from the SA after intervening on a specific variable and keeping everything else in the algorithm and task constant is the counterfactual behavior. This counterfactual behavior can be used as a training signal for Q using next-token prodiction. Q can cquivalently learn any row permutation of the subspaces in 2, thus we can restrict our searches to values of D that have contiguous non-zero entries. We can then brute-force search over independent trainings with different values of dyar, selecting the var) pair with the best results. Unless otherwise stated, we try values of dear equal to either 16 or half of dm and take the better performing of the two. See Supplemental Figure S for a closer examination of how d affects results.
We perform our causal interventions on individual time steps in the sequence. We run the model up to an independently sampled timestep in the target sequence, taking its latent representation at that point as the target vector, h We do the same for the source vector, here at timestep from a separate source
6

---

Published in Transactions on Machine Learning Research (07/2025)
sequence. We then construct hy using Equation 5 and continue the model's predictions starting from time t, using bi in place of n.
For the LSTM architecture, we perform DAS on a concatenation of the h and recurrent state vectors [Hochreteer & Schmidlhuber 1997]. In the GRUs, we operate on the recurrent hidden state. In the transformers, we operate on the residual stream following the first transformer layer (referred to as the Layer Hidden States in Supplementary Figure 5 or the input embedding layer. We use 10,000 intervention samples for training and 1,000 samples for validation and testing. For all data, uniformly sample trial object quantities, and unless otherwise stated, we uniformly sample intervention time points, t and u, from sequence positions containing demo tokens or response tokens (excluding BOS, trigger, and EOS tokens). We orthogonalize the rotation matrix using PyTorchi ' orthogonal parameterization with default settings. We train Q with a batch size of 512 until convergence, selecting the checkpoint with the best validation performance for analysis. We use a learning rate of 0.001 and an Adam optimizer. See more detail in Supplement A.44
DAS Evaluation: Once our rolation matrix has converged, we can evaluate the quality of the alignment using the accuracy of the model's predictions on the counterfactual outputs held out intervention data. We consider trial correct when all deterministic tokens are predicted correctly using the argmax over logits. We report the proportion of trials correct as the Interchange Intervention Accuracy (IIA) (as used in previous work /eigeer et al. 2023
DAS Alignment Functions: an effort to understand the solutions employed by the Same-Object RNNs, we introduce relaxations of the orthogonal rotation matrix used in DAS. We do this by substituting the orthogonal matrix Q with a general invertible function ((h). We name this class of invertible DAS functions alignment functions due to their potential to 'align' or encode the relationship between the neural activity and the specified interpretable variables. Formally, we can write the model's aligued representation, 2, in terms of an invertible function, f, where == In this work, we only extend DAS to linear cases of f of the form f(h) = X(h + b) where X E RpdmXda is an invertible symmetric matrix and b E Rd is a bias vector. Using & E {trg, sre} to denote that the same aligument function is applied to both the target and source vectors before the intervention, we reformulate Equation terms of f:
= = 1(10= =0 X(1 + hy = x-1((1 10--Dvac) D + Dvar=
(6) (7)
With this formulation, we are able to train X and b using the same counterfactual sequenees used to train Q in Equation 5) We refer to the original DAS analyses as using an Orthogonal Alignment Function (OAF) and the linear formulation from Equations and 7 as the Linear Alignment Function (LAF). In our experiments, we construct X (MMT + eT) where M E Rdm is a matrix of learned parameters initially sampled from a centered gaussian distribution with a standard deviation of 1, I Rd Xdm xdm is the d,,, xdm is a diagonal matrix to learn
identity matrix, e 0.1 to prevent singular values equal to 0, and S E Rdmxddm a sign for each column of X using diagonal values 8;, = Tanh(a;) + ((sig((Tanh(a.)) where cach a, is a learned parameter and = 0.1 to prevent 0 values.
3.4.1 Activation Substitutions
RNN Individual Activation Substitutions: We explore direct substitutions of individual neurons in the Multi-Object trained RNN models to demonstrate the relative ineffectiveness of individual neuron patching compared to the rotated subspace patches used in DAS. Tn these experiments, we attempt to transfer the Count in accordance with the Up-Down program by directly replacing the activation value of a single neuron (A.K.A. dimension) in the RNN's hidden state vector at time step with the activation value of the same neuron at time step from a different sequence. This is equivalent to Equation 5 using an identity rotation with a single, non-zero value in D corresponding to the index of the specified neuron. We perform these interventions individually for every model neuron and evaluate the model's IIA using the expected behavior from the Count interventions.
Transformer Hidden State Substitutions: A sufficient experiment to determine whether a Transformer is using Markovian states is to examine its behavior after replacing all activations in its most recent hidden state vector from time & from a target sequence with hidden state vector from time from different.
7

---

Published in Transactions on Machine Learning Research (07/2025)
source sequence. If the post-intervention behavior matches that of the source sequence after time u, then the state has encoded all behaviorally relevant information in its activation vector and we can conclude that the intervened transformer state is Markovian. If the post-intervention behavior ignores the substitution and matches the target sequence after time t, then we can conclude that the states are anti-Markovian. In the two-layer transformers used in this work, we only need to perform this intervention on the output (A.K.A residual stream or hidden states) of the first transformer layer, as the output of the second layer can no longer transmit information between token positions (see Supplement A.5 for more details). See Supplement A.4.6 for specific intervention data examples.
GRU
LSTM
GRU
LSTM
1.0
1.0
Count
Resp Count
5 0.5
0.5
0.0
Multi-Object Same-Otyect
Mult--Otject Orthogonal Aligrment
Same-Object
0.0
Mutli-Otject Same-Objact
Mulli-Obyect
Same-Obyect
Orthogonal Alignment
Linear Alignment
Linear Alignment
The Interchange intervention accuracy (IIA) for variables from different. SAs for different tasks and architectures. The displayed IIA for the Count and Phase variables comes from the Up- program. The IIAs for the Demo Count and Resp Count variables come from the Up-Up program. IIA measurements show the proportion of trials where the model correctly predicts all counterfactual R. and EOS tokens following a causal intervention. The DAS alignment function is displayed below each panel.
Figure 2:
4 Results
4.1
Recurrent Neural Networks
4.1.1
Numeric Neural Variables
The left side of Figure 2 shows the DAS aliguments using the orthogonal aligument function for RNNs trained on the Aulti-Object and Samee-Oject tasks. In the Multi- Object recurrent models, we see that the most aligned SA is the Up-Down program from the higher IIA in the Count and Phase variables compared to the Demo Count and Resp Count variables from the Up-Up program. We use this as supporting evidence for the claim that the Multi-Object GRUs and LSTMss possess neural variables for the Count and Phase of the task and use solution that is causally consistent with the Up-Down program. A visualization of the first two principal components of the solution found by a Multi-Object GRU in Supplemental Figure 16 further supports this interpretation. During the demonstration phase the projected states take steps in one direction along a line; at the trigger token, the trajectory jumps in direction perpendicular to this line; and during the response phase the states travel parallel to the demonstration phase trajectory, albeit in the reverse direction.
The existence of these numeric neural variables stands as a proof of principle that neural systems do not require explicit exposure to discrete numeric symbols, nor do they need built-in counting principles to develop symbol-like representations of numbers. Such variables can emerge in these architectures simply from learning to perform exact quantity matching task.
We now focus on the Same- Object task where the demo tokens and resp tokens share single token id within the task. We see from Figure that each of the Same-Object GRU OAF aliguments have relatively low IIA. To aid in understanding how the model solves this task variant, we show the first two principal components of the hidden states of a Samo-Object GRU in Supplemental Figure [7 As before, the states progress along line in one direction during the demonstration phase, and they jump to a line paralleling the first when transitioning to the response phase. In this case, however, the jump returns back to the start of the parallel
8

---

Published in Transactions on Machine Learning Research (07/2025)
line and then proceeds along this line in the same direction as in the demonstration phase; furthermore, the rate of progress and the stopping point in the response phase both depend on the object count of the trial. Thus, the main PCs of this model cannot be simply described in terms of either the Up-Down or Up-Up SAs. We include an additional DAS analysis in Supplement A.3 in which we introduce a now SA that more closely
describes our observations from the PCA in Figure 17 These additional aliguments, however, also have a relatively low IIA of 58.1% using an OAF.
We considered whether the state spaces of the Same-Object models could be aligned to an SA using an LAF instead of the OAF. We see in Figure 2 that the LAF ILA for the Same-Object GRUs reaches very high accuracy (>95%) for the Count and Phase variables of the Up-Down program. In accordance with the OAF IIA and what we observed in Figure 17, we do not conclude that the model has distinct Count and Phase variables in the same way as the Mujti-Object models. It does mean, however, that such variables are recoverable from its state space via an invertible linear mapping, and can be causally intervened upon after such recovery, before being mapped back into the state space of the model to produce desired, prodictable behavior.
We also see in Figure 2 that the GRU LAF alignments reach very high IIA (>95%) for the Demo Count and Resp Count variables of the Up-Up program. It appears that the flexibility of the LAFs allows the states of networks to be aligned with conceptually distinct programs. In this case, it is possible to compose the Count. variable of the Up-Down program using the difference between the Demo Count and Resp Count variables from the Up-Up program. This perhaps explains why both LAF alignments result in high IIA for both the Multi- and Same- Same-Object tasks as shown on the right side of Figure 2 We show in Supplemental Section A.3 that LAFs do not achieve high ITA for all possible SAs, where the LAF ITA only reaches 72.2% and 89.4% for the Multi-Object and Saje-O GRUs respectively.
In sum, LAFs can allow for causal interventions of high accuracy under a wider range of circumstances than OAFs, though, this measure alone is perhaps less diagnostic of the specific SA that best aligns with the NN's learned solution. We return to LAF' in Sections 4.1.5 and 4.1.6 to better explore their theoretic and empirical properties.
4.1.2 Individual Activation Substitutions
We performed direct substitutions of individual activation values in recurrent models' hidden state vectors to demonstrate the importance of operating on a subspace of the neural population rather than on individual neurons. We turn our attention to the raw activation traces in the topmost panel (b) of Figure | and note that neurons 12 and 18 (shown in blue and black) have high correlation with the Count of the sequence. These traces came from an LSTM with dm 20. In this model, we attempted interchange interventions that transferred the raw activity from both neurons 12 and 18 in an attempt to transfer the value of the Count. These interventions achieved an IIA of 0.399 on the behavior generated from the Up-Down program. Furthermore, we observed no consistent pattern of behavior (i.e. off by one errors) following the interventions. We include this result as a cautionary demonstration that interpreting correlations without interventions can be problematic; and the interventions appear to suggest that interpreting raw NN activations can be misleading.
4.1.3 Graded Neural Variables
By increasing the gramularity of our analyses, we uncovered a graded effect of the content of the values involved in the interchange interventions. We can see this in Figure 3 (c) and (d). We see a gradience in the IIA, where the interventions have a relatively smooth decrease in IIA when the quantities involved in the
intervention are large and when the intervention quantities have a greater absolute difference. This indicates that the neural variables have some level of graded continuity despite the task using fully discrote numeric values. We refer to such neural variables as graded or symbol-like. This notion of graded, symbol-like variables can be contrasted against cases where the errors are uniformly distributed, independently of the values used in the interventions. It notable that human behavior many numeric tasks exhibits similar graded effects, called the size and the distance effect (Moyer & Landauer 1967). Some rescarchers note that differential experience may account. for these effects (and showed this in neural network simulations) (Verguts et al.
9

---

Published in Transactions on Machine Learning Research (07 2025)
1.0
Accuracy
0.8
0.8
- 0.5
0.5
126
0.6
Count
236
0.6
0.0
lesk Parformnce Symbollc Aignment
0.0
256
0.4
16
0.4
Reap Demo
Derro
Swep Tasp
(a)
0
Training Epoch
500
(b)
Hidden Size
(c)
Target Count
10
15
20
(d) Intervention Distance
10 15 20
In all panels, the IIA comes from DAS using an Orthogonal Alignment trained on the Count variable in the Up-Down program. The models are all Multi-Objeet GRUs. (a) Shows task accuracy and ITA over the course of training for architectures with different sizes of the recurrent state h. We note the correlation between IIA and accuracy, with relatively little change as training continues. (b) Shows the final IIA for the GRUs as a function of increasing hidden state sizes. (c) Shows the IIA from the 128d GRU as a function of the Count value from the source hare (denoted by color) and the Count before the intervention in the target h (shown on the x-axis). The cyan, dashed line represents the mean IIA over all interventions for given larget count highlighting the unequal distribution over target source pairs. (d) DAS IIA from the 128d GRU
Figure 3:
function of the absolute difference between the target and source counts. The colors indicate the Phase of hare the left and 4 the right. Panels (c) and (d) show that the value of the variable during the interventions somewhat smoothly affect the resulting IIA. See Supplemental Section A.4.7
for detail on the data used in panels (c) and (d).
2005). In our experiments, the task training data provides more experience with smaller numbers, as the models necessarily interact with smaller quantities every time they interact with larger quantities. This is perhaps a causal factor for the greater intervention error at larger numbers, but we do not explore this further. The DAS training data suffers from a similar issue due to the fact that we use a uniform sampling procedure for the object quantities that define the training sequences and we uniformly sample the intervention indices from appropriate tokens in these sequences. This results in a disproportionately large nunber of training interventions containing smaller values.
The graded neural variables raise the question of how best to interpret neural norssrks. We remind ourselves that the ANN is built on a symbolic computer program, and thus, this program will always align perfectly with the ANN by definition. The non-trivial goal of our work is to find SAs that characterize the computations such programs learn such that we can predict and control the NNs' activity unified, interpretable ways. The symbolic gradience that we observe in our models serves as partial motivation for the LAFss examined in Section 4.5
4.1.4 Model Width and Developmental Trajectories
We see in Figure 3(a) that although many model widths can solve the Multi-Object task, increasing the number of dimensions in the hidden states of the GRUs seems to improve the IIA of the Up-Down alignment. We can also see from Figure 3 (b) that the larger models tend to have better ITA. Although our results are for RNNs on linear tasks, an interesting related phenomenon in the LLM literature is the effect of model scale on performance (Brown et al. 2020 Kaplan et al. 2020). We do not concretely explore why increasing dimensionality improves ITA, but we speculate that with greater dimensionality comes a greater likelihood that any two variable subspaces will be orthogonal to one another.
Model Width:
Developmental Trajectories: Turning our attention to the learning trajectories in Figure 3 that the models' task accuracy and IIA begin to transition away from 0% at similar epochs and plateau at similar epochs. This finding can be contrasted with an alternative result in which the alignment curves significantly lag behind the task performance of the models. Alternatively, there could have been a stronger upward slope of the ILA following the initial performance jump and platean. In these hypothetical cases, a possible interpretation could have been that the network first develops more complex solutions, or it could have developed unique solutions for many different input-output pairs and subsequently unified them with further training. The pattern we observe instead is consistent with the idea that the worksrks are biased
we see
10

---

Published in Transactions on Machine Learning Research (07/2025)
towards simple, unified strategies early in training. Perhaps our result is expected from works like Saxe et al. (2019) and [axx et al. (2022) which show an inherent tendency for NNs trained via gradient descent to find solutions that share network pathways. This would provide driving force towards the demo and resp phases sharing the same representation of a Count variable.
4.1.5 Understanding Alignment Functions
In this section, we ask why Linear Alignment Functions (LAFs) improve ITA? To answer this, we reformulate the model's neural activity, h, in terms of activity component vectors E Rdm; h x Uz E 2,44 - b, where X and make up the LAF, U = X for notational ease, 2 is a vector composed of interpretable subspaces from Equation 4 and z; refers to the value of the 7th dimension of z. The interchange intervention in Equation 7 is equivalent to exchanging weighted activity components 2,Ug;
dm
h = varr are + (1 Dvar)= - - -
(iSdvar
are
dvar } 4Tg
(8)
(=1
h"
-b b Arruy
2r -b + Uvar,
Irg 2i
Uyart
(9)
Where uvar, indicates that the activity corresponds to the intervened variable subspace and Uyer' is all other activity. If orthogonal, then by definition, each inner product (us, uz) when i + j. Thus (zyu4, 2z01) z4zy (ut, uy) 0 too. If U is orthogonal, then is its inverse, and thus (U-1 2,,, U0 2-) 0 when * j due to orthogonal matrices preserving inner products. Thus when using function, the intervened subspaces are also orthogonal in the original neural space. This is not necessarily the case, however, for LAFs, where U is linear invertible matrix because (U-12;u; 2, U U U01 25Uz) need not be equal
orthogonal alignment
Neuron Index
Token Type
BOS
Object Count
Activalions
DO D2
Activalion
10
0
-10
T
R
Inverse Aligned Phase Activation
1
Autivetiion
10
Aligned Phase Activation
0
-10
Token Type
DO
D1
Aclivetion
10
Inverse Aligned Count Activation
0
R
-10
EOS
BOS
EOS
2
Aligned Count Activation
Sequence Step
(a)
(b)
Figure 4: (a) The top pancl shows values of h for different trials and input token types projected onto the aligned dimension a 1D linear aligument, equal to DX + b), for the Phase variable. The bottom panel shows the same for the Count variable. The h vectors are collected from 15 trials for each object count ranging from 1-20 from single Multi-Object LSTM of size 20 (size chosen for relative simplicity of raw activity). The IIA for the Phase 84.7% and the Count from individual trials with object counts of 2, 8, and 16. Dot colors redundantly encode token type. (b) These pancls show neural variables in the models' original neural space and show the importance of causal analysis. Each panel shows the activation for individual neurons averaged over 15 trials each with an object count 15. The x-axis shows steps in the trials. The topmost panel shows the raw values. We label two specific neurons (index 12 and 18 within the R20 vector) that have a visibly high correlation with the Count of the sequence. We show Section 4.1.2) that these two neurons are insufficient to causally transfer the Count between representations. The middle and bottom panels show the Phase and Count neural variables projected back into the original neural space by inverting the aligned activity using X x-1 (h 6)) b.
82.6%. The connecting lines trace the states
11

---

Published in Transactions on Machine Learning Research (07/2025)
to O. This means that the LAF can allow the intervened subspaces to be non-orthogonal in the original neural space, thus allowing the LAF to formulate each variable from the SA in linear terms of the other variables.
We demonstrate this by first writing h as a combination of alignment vectors corresponding to each variable. Using the Count and Phase variables as an example, we decompose h as follows:
h = Ucount Zcount + Uphase Ephase + VextraRextre -
(10)
where Ever E Rever and Uvar E Rdm Xduas, We can now solve for Zcount by rearranging Equation 10
Ueount = - Uphose - Uextrazezte +
(11)
In the case where U is orthogonal, then Urount = Ueount and Ueoane Uphase = UcouneUeztre = 0, meaning that Ecount must be defined independently of Ephase. This is not the case, however, for a non-orthogonal U, which means that each Eaar can be written in linear terms of the other variables in the SA when using LAFs.
4.1.6 Visualization of symbolic algorithm variables with alignment functions
While IIA alone may not be fully diagnostic of the SA that best characterizes the computations performed by an NN, further analysis based on a successful aligument can potentially shed further light. on the algorithm a network is using, and it can aid in understanding how neural variables are encoded in the raw neural activity. We provide Figure 4 (a) and (b) to illustrate these points. The Figures were obtained using an LAF with dvar- = 1 aligned to a Multi-Object LSTM with hidden dimensionality of 20 (dm = 20). We chose dvar and dm = 20 for visualization purposes, and we use an LAF for its improved IIA when using dvar = 1 relative to an OAF on the smaller model size.
Figure 4 a) shows a set of h vectors projected into the 1-dimensional, aligned Phase and Count subspaces taken from the LSTM at many different time steps and trials; we illustrate trajectories in cach subspace for individual trials with object counts of 2, 8, and 16. The vertical axis in each panel is used to facilitate the visualization of individual trajectories and their relation to specific token types. One can see in the top panel of (a) how the Phase variable has a value of -1 with slight jitter independent of the demonstration token instance. The Phase then transitions to a positive value at the T token, and stays positive throughout the response phase, though, with some contamination by the Count. Similarly, one can see in the bottom panel of 4a) how the Count variable progresses from right to left in integer-like steps during the demonstration phase, takes one step further left at the T token, then jumps back 2 steps to the right at the first R token to correctly represent the decremented value of the Count after the first response token. Lastly, it steps rightward until the Count reaches 0, after which the EOS is produced.
Figure (b) shows the raw neural activity in the top panel for all 20 of the neurons in the model averaged over 15 sampled trials with object counts of 15, and the bottom two panels show the inverse of only the aligned activity for each individual variable, equal to 2phase - b in the middle panel and Zcount Mccuntt - in the bottom. We provide this figure as a demonstration of how to view the causally relevant neural activity in the NN's original neural space through the lens of the aligned variables. We can see that many of the raw neurons can play a causal role in both the Phase and Count neural coding, confirming notions of distributed coding from prior work (Rumelhart et al. 1986. Smolensky
1988 Elhage et al. 2022 Olah 2023
4.2 Transformers
In this section, we demonstrate through empirical and theoretical means that shallow transformers solve the task by recomputing the solution to the task at each step in the sequence. We refer to this class of solutions as mari-Markovian, named for their inductive bias against cumulative, Markovian states. We begin by demonstrating that using the previous transformer layer's outputs as the inputs to the attention mechanism in the next layer restricts transformers from using Markovian solutions that use more sleps than attention layers. We then demonstrate a theoretical solution for simplified versions of the Single-Object and Multi-Object numeric equivalence tasks in one layer NoPE transformer architectures, and we causally verify that such a solution emerges empirically. Lastly, we show through causal interventions that similar solutions can emerge in two layer RoPE transformers.
12

---

Published in Transactions on Machine Learning Research (07/ 2205)
4.2.1 Anti-Markoviar States
In this section, we demonstrate why Transformer solutions that use Markovian states in the residual stream require a new attention layer for every new step in the sequence. To show this, we focus on a simplified transformer architecture that only includes an embedding layer and the self-attention mechanism within cach layer. To justify this simplification, we note that the attention mechanism is the only mechanism in the transformer that provides opportunity to transmit state information between token positions in the sequence. With this simplification, we can write the output of a single transformer layer as:
-. "]" + ( [h M ... MJ] = [1--1 14-1
-i--... "D
(12)
where Mf E Ra are column vectors from the transformer residual stream, l denotes the attention layer where = O is the output of the embedding layer, refers to the positional index in the sequence, and attne(z) refers to the attention mechanism. We denote a cumulative state at step m in a Markov chain as 8m, and we denote the encoded state in residual stream vector as n,((=)) We assume that the attn function can only produce and encode sm+1 at time if sm is already encoded at time < t, and we assume that is produced in the embedding layer. Then the cumulative state gets updated with each transformer layer as follows:
hg
hg ... hg = Embedding(220 21, 22, .... xt)
1 (1((0))
hb ... h! = Ino ((e)
h
h2
... ng]
+ ngn
attnr ( [N0 (10(0)))
hy
hy
(00)
122(0r))
122(00)
h??
[bb (20)
(())
h22
h! +
attna( (mp 1,100
mt)) n1
[ng (n((
h2 nb/600]]
[nb (,00)
11t 11-1] +
li2
h2
attnee [1o (x1,
h (01(()))
h2 (02)
hi n)
Where 2e denotes the input token id at time We can see that in the best case scenario, the cumulative state can only be transmitted and updated one layer at time (we provide more formal proof in Supplement A.7 Thus the two layer transformers in our work are architecturally insufficient for using solution that involves Markovian states.
We experimentally verify that the two layer RoPE transformers used in this work use anti-Markovian states by performing Transformer Hidden State Substitutions as outlined in Methods Section 3.4.1. Indeod, these substitutions leave the NNs' behavior largely unaffected with an IIA of 0.964 on the original behavior in the Multi-Objer RoPE transformers and 0.949 for the Variable-Length Multi- Objeit RoPE transformers.
We note that generative techniques like scratch pad Nye et al. 2021 and Chain-of-Thouie Thought (CoT) (Wei et al. 2023) allow for transformers to track a cunulative state in the form of self-generated input embeddings. We might expect recurent models to benefit less from CoT in this respect.
4.2.2 Simplified NoPE Transformers
To better understand how a transformer could impement anti-Markovian solution to the Multi-Objer task, we include theoretical treatment of a single-layer NoPE Transformer that is trained on simplified version of the Multi-Object task: it excludes the BOS and T tokens from the sequences and only has one possible demo token instance D that is different from the resp token instance R (i.c. an object count of 2 would result in the sequence 'D R R E'). The self-autention calculation for a single query 4. E Rdd from a response token, denoted by the subscript r, is as follows:
Attention(g. K) = (softmaxx
KT gr
e Va
Ui
s U (13)
ソソ=
8
En
e Va
1=1
13

---

Published in Transactions on Machine Learning Research (07/2025)
Where d is the dimensionality of the model, is the sequence length, K Rd is a matrix of column vector << k;
keys, V E Rdxn is a matrix of column vectors Ui, and s = e va using i to denote the positional index of the key and the superscript r to denote that the came from a response token. We refer to sf4 as the strength-value of the 1th token for the query 9r-
In the first layer following the embeddings a NoPE transformer, each of the queries for the response tokens will produce equal strength-values for a given key-value pair regardless of the position from which the response token and demo tokens originated. This is because NoPE does not add positional information to the embeddings. Thus, assuming that the attention mechanism is performing sum of the count contributions from each token in the sequence, we should be able to use the sfu, to increment and decrement the model's decision to produce the EOS token from any given response loken in the following way:
Incremeented/Ateeniinnn... K, K2V)
87 + プド= (s, + Σ 8:0s)
1
(14)
Where the subscript r in the strength 8, and value v, denotes that the originating token for the key-value pair is a response token. We can decrement the count using a key-value pair from a demonstration token. To verify our thcoretical treatment, we performed a simulation using a single-layer NoPE transformer trainod on the simplified Single-Object lask. Using the streng(h-value additions outlined in Equation [4 we were able to change the position at which the transformer produced the EOS token with 100% accuracy. We include results for other transformer architecture variants in Supplemental Figure (c).
4.2.3 RoPE Transformers
To determine how the RoPE transformers perform the tasks, we first looked at the attention weights for both of its two layers (sce Supplemental Figure 11 h. The R and EOS queries give surprisingly little attention to the R tokens. In Supplemental Figure G we show DAS results on the Imput Value variable from the Ctx-Distr SA where a numeric value is assigned to each token and the values of all previous tokens are summed al each step in the sequence. The Multi-Object transformers achieved an 11A of 0,800 for this alignment. We took this to mean that the Multi Objeet transformers were at least partially using a positional readout to solve the task. To understand transformer solutions that rely less on a positional readout, we also examined set of transformers trained the Variable-Length variant of the Multi-Objeet task that disrupts count-position correlations. These transformers achieved a higher IIA of 0.935 for the same DAS aligmment. The lower IIA of the original Multi-Object transformers is consistent with the notion that they rely, in part, on a positional readout, rather than a summing operation, to solve the task.
5 Conclusion
In this work we used both causal and correlational methods to interpret emergent representations of numbers in several types of NNs. We used these methods to discover and characterize graded, symbol-like like nunber variables within the representations of two types of RNNs; we introduced an extension of DAS that adds flexibility our ability characterive neural activity in terms of functions of interpretable symbolic variables; we explored theoretical and empirical transformer solutions to the tasks; and we showed that transformers must use anti-Markovian solutions in the absence of sufficient layers.
As we have noted before this article, ANNs are by definition symbolic programs, but the algorithms they learn are latent in their connection weights and are not directly observable by inspection of the ANN code. Our goal in aligning NNs and SAs is to find predietive, interpretable, and controllable ways of understanding the learned neural activity. Because NNs are example-driven, gradient- gradient-based learners, their learned solutions do not necessarily need to match interpretable SAS exactly. Under idcalized conditions, they can come to conform more and more precisely to such algorithms, but they generally do so gradually through training. We believe these points characterize many aspects of the algorithms humans and animals learn through experience, making NNs good models of many aspects of human coguition and development.
When networks learn to conform to simple SAs, they tend to do so in narrow task settings, disconnected from the richness and complexity of real experience ServaanSShreiber et al. (1991). This situation applies to
14

---

Published in Transactions on Machine Learning Research (07/2025)
our models, whose experience is extremely narrow, limited to correct instances of sequence- numeric equivalence task. In more complex cases, such as natural language, debate has raged for decades about whether NNs are adequate for extracting the symbolic rules that many argue underlie human language abilities. In our view McClelland 2013)), such rules are often useful characterizations for the goals of interpretability, but should not be embraced as the complete story without skeplicism. Nevertheless, we strongly encourage work that socks to understand the computations of neural systems through their alignment with SAs.
6 Acknowledgments
Thank you to the PDP Lab for thoughtful discussion and the Stanford Psychology department for funding. Thank you to the ICLR and TMLR reviewers for greatly improving the paper. And huge thank you to Zen Wu for great discussions and assisting with numerous questions about DAS early in the project's development.
References
Laith Alzubaidi, Jinglan Zhang, Amjad J Humaidi, Ayad Al-Dujaili, Ye Duan, Omran Al-Shamma,, J Santa- maría, Mohammed A Fadhel, Muthana Al-Amidie, and Laith Farhan. Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. Journal of Big Data, :(1), 2021. ISSN 2196-1115. doi: 10.1186/440337-02-0004444
URL https://doi. /tt/:///w..g/00....///0000000000
Jimmy Lei Ba, Jamic Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
URL https: :
//axxiv.or//aat 0t////6600000655
Freya Behrens, Luca Biggio, and Lenka Zdeborová. Counting in small transformers: The delicate interplay between attention and foed-forward layers, 2024. URL httL_tttsss//arr..or/////////// 1154..09000/
Adithya Bhaskar, Dan Friedman, and Danqi Chen. The houristic core: Understanding subnetwork generaliza- tion in pretrained language models, 2024. URL http://pe//aaxtt.oor///////0.../
Tom B. Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-dccoder for statistical machine translation. CoRR, abs/1406. 2014. URL http://arrd..org/aaa///aae
Róbert Csordás, Christopher Potts, Christopher D. Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-lincar representations, 2024. URL https://arxiv.org
abs/2408. 10920
A Alessandro Di Nuovo and Tim Jay. Development of numerical cognition in children and artificial systems: a review of the current. knowledge and proposals for multi-disciplinary research. Cognitive Computation and Systems, 1(1):2 11, 2019. doi: https: httpps:/dooi.org/ 10...00099/cc.000 2018..00..00000 URL https://ietresearrs onlinelibrary. wwtprrarr.wllll....... com www./////a///.../..o.o
ccs. 2018. 2008.0004
Alessandro Di Nuovo and James L. McClelland. Developing the knowledge of number digits in child-like robot. Nature Machine Intelligence, 111)):594-655, 2019. ISSN 2522-5839, doi: 10000///42266019-00000
URL
doi htt.://.....rr0..00000//22220000
Quan Do and Michael E. Hasselmo. Neural Circuits and Symbolic Processing. Neurobiology of learning
and memory, 186:10755,, December 2021. ISSN 1074-7427. doi: 10.1006//nlm.22211.077... 1075.5 URL
https://www..r hcwp://wwwwnncb..... ncbi. nlm. mih, http://///aatt..le/////11222111/
15

---

Published in Transactions on Machine Learning Research (07/2025)
Nadine El-Naggar, Andrew Ryzhikov, Laure Daviaud, Pranava Madhyastha, and Tilmann Weyde. Formal and empirical studies of counting behaviour in relu rnns. In Frangois Coste, Faissal Ouardi, and Guillaume Rabusseau (eds.), Proceedings of 16th edition of the International Conference on Grammatical Inference, volume 217 of Proceedings of Machine Learning Research, pp. 129.222. PMLR, 10-13 Jul 2023. URL https: http::///roceddnss.. wwm.mlrrppeeee arress/v217/el-naaar223 .tmm
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hadfield-oddds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuils Thread, 2022.
2022/022//// /toy model /tt///iddxxitttt html.
M. Fang, Z. Zhou, S. Chen, and J. L. McClelland. Can a recurrent neural network learn to count things? Proceedings of the 40th Annual Conference of the Cognitive Science Society, pp. 360-365, 2018.
Jerry A. Fodor. The Language of Thought. Harvard University Press, 1975. ISBN 978-0-674-51030-2. Google-Books-ID Books-ID: XZwGLBYLbg4C.
Jerry A. Fodor.
Psychosemantics: The Problem of Meaning in the Phyllosophy of Mind. MIT Press, 1987.
Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis.
Cogmition, 28(1):3-71, March 1988. ISSN 0010-0277. doi: 10100//0000-222278800000113 URL https:
//www.sciencedire....c.. http://ddeeettcccooccccceeeeeeee
com htt.a///cc............oooooooo/.
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks.
CoRR, abs/2106.029977 02097, 2021. URL
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. Finding alignments between interpretable causal variables and distributed neural represcntations, 2023.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in aulo-regressive language models, 2023. URL https://arxiv..rrtt argg/bbs/3304 14767
Peter Gordon. Numerical cognition without words: Evidence from Amazonia. Science, 306(5655):496 2004. ISSN 00368075. doi: 10. 1...18//scieccc.044400
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information, 2022. URL httpps://axxxv.or////////222 16634
Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching, 2024. URL https: //arxiv. org htwggabb/244..155555
G. F. Hinton, J. T.. McClelland, and D. F. Rumelhart. Distributed representations. In D. F. Rumelhart and J. L. McClelland (eds.), Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, pp. 77-109. MIT Press, 1986.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Compul., 988:17335 1780, nov 1997. ISSN 0899-7667. doi: (0.1122/ne00.1997.9882 URL https://aai..rrg// w1w../ecc...c.o.m www.ee............ 8. 8.1735
Anna A Ivanova, Martin Schrimpf, Stefano Anzellotti, Noga Zaslavsky, Evelina Fedorenko, and Leyla Isik. Beyond linear regression: mapping models in cognitive neuroscience should align with research goals. arXiv preprint Xrx:::2008. 10668, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Nechar Kondapaneni and Pietro Perona. A Number Sense as an Emergent Property of the Manipulating Brain. Xiv, pp. 1-,,, 2020. URL http:://aaaii...orga////220....0 04132
16

---

Published in Transactions on Machine Learning Research (07/2025)
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Build-
ing machines that learn and think like people. Behanioral and Brain Sciences, 40:e253,
Jamuary 2017. ISSN 0140-525X, 1469-1825. doi: 10. 10.0017/90149533X601133377
URL
https://www cwwbridgee....ee htt.a//ww............a...a///aar
David Lindner, János Kramár, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik. Tracr: Compiled transformers as laboratory for interpretability, 2023. URL
/bs//
2301. 05062
Gary Marcus. Deep learning: A critical appraisal, 2018. URL http://aaxxi..orr//////0..000000
James McClelland. Capturing gradience, continuous change, and quasi-regularity in sound, word, phrase, and meaning. In The handbook of language emergence, pp. 53 80. Wiley Online Library, 2015.
Kevin Meng, David Bau, Alex Andonian, and Youatan Belinkov. Locating and editing factual associations in gpt, 2023. URL https:///arstt: http:///axvvvvv/a wwtp://wr//...a....a.//// 2202.05262
abs
William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits: Grokking competition of sparse and dense subnetworks, 2023. URL
Eric J. Michaud, Isaac Liao, Vedang Lad, Ziming Liu, Anish Mudide, Chloe Loughridge, Zifan Carl Guo, Tara Rezaei Kheirkhah, Mateja Vukelić, and Max Tegmark. Opening the ai black box: program synthesis via mechanistic interpretability, 2024. URL http::://arrxiv.org//b//////////
Robert S. Moyer and Thomas K. Landauer. Time required for judgements of numerical incquality. Nature, 215:1519-1523 1967. URL https://api htt.appi.semmannccchhclle..s http://ttt.ctotooooooooooooooo/a 4298073
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interprotability, 2023. URL
Khaled Nasr, Pooja Viswanathan, and Andreas Nieder. Number detectors spontaneously emerge in a deep neural network designed for visual object recognition. Science Advances, 5(5):1-11, 2019. ISSN 23752548.
doi: 10.1126/ veciavv.av7990:
Allen Newell. Physical symbol systems. Cognétive Science, 4(2):135 183, April 1980. ISSN 0364-0213.
doi: 12011000000000011110000000003110
URL sttps::/www..sceeceddiine http:///ttt.ttto////c//eeeee//aa
50364021380800152
Allen Newell. The knowledge level. Artificial Intelligence, 18(1):87 127, Jamuary 1982 ISSN 0004-3702.
doi:
URL http:://www.eideeeddeeett
000437028220012211
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021. URL https 2
//arxiv. stxxiv..rr//abs2112.00011
Chris Olah. Distributed reprosentations: Composition & superposition. httpss//rranstfomerr-caaaal
pbb/D0///mpprrpsttiiorrooopppi.p 2023.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.23915/d1sttil..0000.
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexan- der Mordvintsev. The building blocks of interpretability. Distill, 2018. doi: 10.23915 2...23//iititll....m //t//dissil.00001111
hupps//distiill.pbb/200 hbugll/ophbbo///uddddnnhbbhess
17

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 1a0991//disill.0022224
http://iistlllpu//22200 /t//2/000000//t// 22220//rrcuits/zoommii
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performacce deep learning library. CoRR, abss1992.01173, 2019. URL http://arxii http://aaxxi..ooaabb//aa/a/ /
Judea Pearl. An Introduction Causal Inference. The International Journal of Biostatisties, 6(2):7, February 2010. ISSN 1557-4679. doi: 00--222211557-400 1203. URL https://www.nee. /cwp::////www.biii....mmm wwm.ii.nmm.ii....mmmmmmmmm nov.
puc/artiiclee/PMcc233313
Benjamin Pitt, Edward Gibson, and Steven T. Piantadosi. Exact number concepts are limited to the
verbal count range. Psychological Science, 33(3):371-38) 2022. doi: 1011770666677211444552 URL
PMID: 35132893.
Zenon W. Pylyshyn. Computation and cognition: Issues in the foundations of cognitive science. Behavioral and Brain Sciences, 3(1):111-18) 169, 1980. ISSN 1469-1825. doi: 101007/50140555X0002233 Place: United
Kingdom Publisher: Cambridge University Press.
D. E. Rumelhart, J. L. McClelland, and PDP Research Group (eds.). Parallel Distributed Processing. Volume 1: Foundations. MIT Press, Cambridge, MA, 1986.
Silvester Sabathiel, James L. McClelland, and Trygve Solstad. Emerging Representations for Counting in a Neural Network Agent Interacting with Multimodal Environment. Artificial Life Conference Proceedings, ALIFE 2020: The 2020 Conference on Artificial Life:736-745 07 2020. doi: 10.1162/isal 00333. URL https://doi... org/ 10. 11t...1122iiaa..
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. A mathematical theory semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 1157733:1 May 2019. URL http:///x>: http//xxxx.ii.oo.o org/ 10. www.//pa........... 1aaia2611122211
ISSN 1091-6490. doi: (00073/paas.120220000
Andrew M. Saxe, Shagun Sodhani, and Sam Lewallen. The neural race reduction: Dynamics of abstraction in gated networks. 2022.
Adam Scherlis, Kshitij Sachan, Adam S. Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks, 2023. URL https: http://aww....r//a///22000.../1
David Servan-Schreiber, Axel Cleeremans, and James L McClelland, Graded state machines: The representa-
tion of temporal contingencies in simple recurrent networks. Machine Learning, 7:161 -99,, 1991.
Paul Smolensky. On the proper treatment of connectionism. 1988.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Mario-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Alexander Trott, Caiming Xiong, and Richard Socher. Interpretable counting for visual question answering. 6th International Conference Learning Representations, ICLR 2018 Conference Track Proceedings, pp. 1-18, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. CoRR, abs/1706.03762, 2017. URL http:
//arxiv.org// www. httg/abs/1706..o 03762
18

---

Published in Transactions on Machine Learning Research (07 (025))2
Tom Verguts, Wim Fias, and Michaël Stevens. A model of exact small-number representation. Psychonomic Bulletin & Review, 12(1):66 80, February 2005. ISSN 1069-9384. doi: 10.3758/bl0319960
Jesse Vig, Sobastian Gchrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreling neural ulp: The case of gender bias, 2020. URL https: httts://arxiiv. org. httpp://aarri..ooraaa///0000 12265
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in gpt-2 small, 2022. URL htttp:///arx: org/abs/ 2211.00595
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thouggt prompting clicits reasoning in large language models, 2023. URL https://arxiv. 109.://22..vv0rrr//////2..0191
Gail Weiss, Yoav Goldberg, and Eran Yahav. On the practical computational power of finite precision rmns for language recognition, 2018. URL https://arxxv.orrg wwwp:///wwrraaba/////e//// 04908
Alex Williams, Erin Kunz, Simon Kornblith, and Scott Linderman. Generalized shape metrics on neural representations. Advances in Neural Information Processing Systems, 34:4738 4750, 2021.
Zhengxuan Wu, Atticus Geiger, Thomas Ieard, Christopher Potts, and Noah D. Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca, 2024. URL htttp:::/aaxvv..rr//bb//////
Yan Zhang, Jonathon Hare, and Adam Bennett. Learning to count objects in natural images for visual question answering. 6th International Conference on Learning Representations, ICLR 2018 Conference Track Proceedings, pp. 1-17, 2018.
19

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
A
Appendix / supplemental material
A.1 Additional Figures
Logits
Linear
Layer 2
Hidden States
Add
Outputs
Residual Stream
Add
Outputs
Residual Stream
Norm
RoPE
Head Self
idettity
Layer 1
Hidden States
Add
Ouiputs
Fead Natwork
Residual Stream
Add
Attention Outputs
Residual Stream
Single
Identity
Embeddings
Figure 5: Diagram of the main transformer architecture used in this work. The white rectangles represent activation vectors. The arrows represent model operations. All normalizations are Layer Norms (Ba et al.) 2016). The majority of the DAS interchange interventions are performed on Hidden State activation voctors from Layer 1 individual time-steps. We offer further granularity in the Input Value interventions by performing DAS the embeddings that are projected into the key and value vectors for the Layer self-attention.
20

---

Published in Transactions on Machine Learning Research (07 (025)
The DAS results for each model and task variant. Each alignment function was trained on single causal variable with a dvar (subspace size) of the better performing out of 16 or 64 dimensions. Performance values are reported as IIA under the task name.
Table 1:
Model
Alignment
Algorithm
Variable
Multi-Object
Same-Object
Single-Object
GRU
Orthogonal
Up, Down
Count
0.9464
0.3698
0.908
GRU
Orthogonal
Up, Down
Phase
0.9368
0.373
0.889
GRU
Orthogonal
Up, Up
Demo Count
0.4807
0.2503
0.606
GRU
Orthogonal
Up, Up
Resp Count
0.4173
0.4024
0.477
GRU
Linear
Up, Down
Count
0.9906
0.991
0.995
GRU
Lincar
Up, Down
Phase
0.9884
0.9922
0.991
GRU
Linear
Up, Up
Demo Count
0.9174
0.9744
0.984
GRU
Linear
Up, Up
Resp Count
0.9223
0.9808
0.989
LSTM
Orthogonal
Up, Down
Count
0.993
0.958
0.989
LSTM
Orthogonal
Up, Down
Phase
0.991
0.95
0.991
LSTM
Orthogonal
Up, Up
Demo Count
0.5416
0.86
0.409
LSTM
Orthogonal
Up, Up
Resp Count
0.5374
0.8007
0.439
LSTM
Linear
Up, Down
Count
0.9928
0.9922
0.992
LSTM
Lincar
Up, Down
Phase
0.9914
0.9912
0.99
LSTM
Linear
Up, Up
Demo Count 0.9846
0.9846
0.990
LSTM
Linear
Up, Up
Resp Count 0.9862
0.9697
0.991
Table 2: The DAS results for each model and task variant using linear alignment function with dvar (subspace size). Each alignment function was trained on a single causal variable. Performance values are reported as IIA under the task name.
Model
Alignment
Algorithm
Variable
dvar
Multi-Object
Same-Object
GRU
Lincar
Count Up, Count Down
Count
1
0.9436
0.68
GRU
Linear
Count Up, Count Down
Phase
I
0.8568
0.870
GRU
Lincar
Count Up, Count Up
Demo Count
1
0.758
0.507
GRU
Linear
Count Up. Count Up
Resp Count
1
0.8339
0.890
LSTM
Linear
Count Up, Count Down
Count
1
0,9236
0.864
LSTM
Lincar
Count Up, Count Down
Phase
1
0.955
0.86
LSTM
Linear
Count Up, Count Up
Demo Count
1
0.912
0.775
LSTM
Linear
Count Up, Count Up Resp Count
0.9494
0.900
Table The DAS results for the transformers. Fach DAS training was performed on a single causal variable with a dvar (subspace size) of the better performing out of 24 or 64 dimensions. Performance values are reported as IIA under the task name.
Model
Alignment
Algorithm
Variable
Multi-Object
Same-Object
NoPE Transformer
Orthogonal
Context Distributed
Input Value
0.882
0.982
NoPE Transformer
Orthogonal
Count Up, Count Down
Count
0.1112
0.110
RoPE Transformer
Orthogonal
Context Distributed
Input Value
0.8004
0.935
RoPE Transformer
Orthogonal
Count Up, Count Down
Count
0.1274
0.124
21

---

Published in Transactions on Machine Learning Research (07 (07/202) (025)
Layer NoPE Transtormer
Layer RoPE Transtormer
Strength Value Inverventions
1.0
1.0
$ 0.5
0.5
Input Value
Simple
Layer NoPC
Laaye NoPE
2layer RoPE
0.0 Multi-Object
VL Multi-Object
Multi-Object
Multi-Object
Multi-Object (c)
(a)
(b)
Figure 6: (a) and (b) show the interchange intervention accuracy (IIA) on the Count from the Up-Down program and the Input Value from the Ctx-Distr program aligned to the Transformer architectures using DAS with an Orthogonal alignment function. VL denotes models trained on the Sariable-Length version of the task. The Input Value encodes an assigned value (+1, -1, or 0) to each incoming token that used recalculate the count each step in the sequence. The DAS analysis is applied the model embeddings for the Input Value, and the residual stream after the first transformer layer for the Count. We can see that the Criiall--eength transformers have stronger alignment to the Input Value interpretation in which the Multi-Object transformers can rcly, to some degree, on positional information. (c) IIA for streugth-value interventions described in Section 4.2.2 These interventions add and subtract from the count using the strength-value within an attention computation. Strength-value are computed from the last response query, key, and value in the sequence from the layer which interventions are performed. The displayed IIA is taken from the better performing of the possible altention layers.
with an
1.0
1.0
1.0
0.8
0.8
0.8
Trial Accuracy
Trial Áccuracy
Trial Accuracy
0.6
0.6
0.6
0.4
0.4
0.4
Single-Osject L5TM Slgppe CRU
1 Laysi NoPE Laye
0.2
0.2
0.2
Layer
SamreCejjct
Layar
Layer
Mulli Ob LSTM
VL 2 Laver NoPE Layer NoPE
VL 2 Layer NoPE
0.0
Multi Ob act ORU
0.0
0.0
Layor NofE
10
20
25
30
10
20
25
30
10 Target Count
20
25
30
(a)
Target Count
(b)
Target Count
(c)
Figure 7: (a) RNN task performance measured as the proportion of trials correct. Object quantity refers to the number of demo Lokens in a sequence preceding the trigger loken. The evaluation data consists of 15 sampled sequences (even when only one configuration exists for that object quantity).
(b) Transformer performance the Multi-Object task. VL indicates the Variable-Length version of the task. One model seod was droppod from each the NoPE and RoPE models trained on the Variable-Length Multi-Object task due lower than 99% accuracy. (c) Transformer performance the Same- Objeet task. VL. indicates the Variable-Tennth) version of the task. All NoPE model seeds performed below 99% accuracy on the Same- Object task.
22

---

Published in Transactions on Machine Learning Research (07 (225))
IIA by Size of Interchange Subspace
1.0
Alignment Accuracy (IIA)
Count
0.5
0.0
20
40
60
80
100
120
Subspace Size
Figure 8: An exploration of the performance of the orthogonal DAS aligument as a function the size the interchange subspace for a randomly selected Multi-Object LSTM model seed on the Count variable. The x axis shows dcount while the y axis shows IIA. This is the number of dimensions substituted in the intervention.
GRU (MultiObj)
1.0
IIA
0.5
0.0
1
2
16 Subspace Size
3
4
32
64
Figure 9: An exploration of the performance of the orthogonal DAS alignment as a function of the size of the interchange subspace for randomly selected Multi-Object GRU model seed the Count variable. The axis shows deount while the axis shows IIA. This is the number of dimensions substituted in the intervention.
23

---

Published in Transactions on Machine Learning Research (07/202) 2025)
1.00
1.00
11A
0.75
d 0.75
Multi-Object LSTM Count
Same- hame-Object GRU Count
Obje-Object Count
Multi-Object GRU Count
0.50
Multi-Object LSTM Demo Count
0.50
Same-Obir Object GRU Demo Count
Same- Same-Object LSTM Demo Count
Multi-Object GRU Demo Count
1
2
3
1
2
3
(a)
(b)
Subspace Size
Subspace Size
Figure 10: (a) An exploration of the DAS IIA the y-axis using the Linear Alignment function with varying sizes of dvar (the size of the intervention subspace) on the x-axis for the LSTM models. (b) An exploration of the DAS IIA on the y-axis using the Linear Alignment function with varying sizes of d,ar (the size of the intervention subspace) on the x-axis for the GRU models.
Layer O
Layer 1
BOS D2
BOS
D2
D1
D1
D3
D3
D1
D1 D2
D2
T
T
RAR
R
R
R R
R
R R
R
EOS
EOS
BOS
D2
D1
D3
D1
D2
T
R
R
R
R
R
EOS
BOS
D2
D1
D3
D1
D2
R
R
R
R
R
EOS
Figure 11: Attention weights for single transformer with two layers using rotary positional encodings trained
on the Multi- Object Task. Queries are displayed on the vertical axis order of their appearance starting the top. Keys displayed the horizontal axis starting from the left. Queries only able to attend to themselves and preceding keys.
24

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Layer 0
Layer 1
BOS D2 D1 D3 D1
BOS D2
D1 P3 D1 D2 T R
D2
T
R
R R R
R
R
R
R
EOS
EOS
BOS
D2
D1
D3
D1
D2
T
R
R
R
R
R
EOS
BOS
D2
D1
D3
D1
D2
T
R
R
R
R
R
EOS
Figure 12: Attention weights for single transformer with two layers using rotary positional encodings trained
the Variable-Length variant of the Multi-Object Task. Queries are displayed on the vertical axis in order of their appearance starting at the top. Keys are displayed on the horizontal axis starting from the left. Queries are only able to attend to themselves and preceding keys.
Layer 0
Layer 1
BOS D2
BOS D2
D1
D1
D3
D3
D1
D1 D2 T
D2
T
R
R
R
R
R
R
R
R
EOS
EOS
BOS
D2
D1
D3
D1
D2
T
R
R
R
R
R
EOS
BOS
D2
D1
D3
D1
D2
T
R
R
R
R
R
EOS
Figure 13: Attention weights for single transformer model seed with two layers and no positional encodings
(NoPE) trained on the Multi-Objec Task. Queries are displayed on the vertical axis order of their appearance starting at the top. Keys displayed the horizontal axis starting from the left. Queries only able to attend to themselves and preceding keys.
25

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Layer 0
Layer 1
BOS D2
BOS D2
D1
D3 D1
D3 D1
D2
D2 T
T
R
R R R
R
R
R
R
EOS
EOS
BOS
D2
D1
D3
D1
D2
T
R
R
R
R
R
EOS
BOS
D2
D1
D3
D1
D2
T
R
R
R
R
R
EOS
Figure 14: Attention weights for single transformer with two layers using positional encodings (NoPE) trained on the Variable-Lengthh variant of the Multi-Object Task. Queries are displayed on the vertical axis in order of their appearance starting at the top. Keys are displayed on the horizontal axis starting from the left. Queries are only able to attend to themselves and preceding keys.
26

---

Published in Transactions on Machine Learning Research (07 (025))
A.2 Model Details
All artificial neural network models were implemented and trained using PyTorch (Paszke et al 2019
Nvidia Titan X GPUs. Unless otherwise stated, all models used an embedding and hidden state size of 128 dimensions. To make the token predictions, each model used two layer multi-layer perceptron (MLP) with GELU nonlinearities, with & hidden layer size of 4 times the hidden state dimensionality with 50% dropout on the hidden layer. The GRU and LSTM model variants each consisted of a single recurrent cell followed by the output MTP. Unless otherwise stated, the transformer architecture consisted of two layers using Rotary positional encodings (Su ct al., 2023). Each model variant used the same learning rate scheduler, which consisted of the original transformer (Vaswani el al., 2017) scheduling of warmup followed by decay. We used 100 warmup steps, maximum learning rate of 0.0001, minimum of 1e-7, and decay rate of 0.5. We used batch size of 128, which caused cach epoch to consist of 8 gradient update steps.
A.3 Increment-Up Symbolic Algorithm
In this section we explore an additional SA that we name the Increment-UD Up Program. This SA was in part inspired by the PCA shown in Figure [7] This SA has variables: the Phase, Progress, and the Increment. The algorithm uses the Progress variable to track progress along fixed interval from 0 to the maximun count which is preset constant. We refer to the maximum count as the Interval. To solve the numeric equivalence tasks, the algorithm increments the Progress according to an increment size which is defined by the Increment variable. During the demonstration phase, the algorithm initializes the Increment to Interval In our case the Interval constant is 20, but we refer to it as the Interval variable for generality. The algorithm progressively adds this increment multiplied by the Interval to the Progress with each successive demonstration token: Progress Progress Increment Interval. Upon reaching the trigger token the end of the demonstration phase, the Increment is set Progress and Progress is subsequently reset to 0. The algorithm then increments the Progress by the new value of Increment (still multiplied by the Interval) until the Progress reaches value greater than or equal to the Interval. At this point, the SA returns the EOS token. See Algorithm for more details.
We compare the Increment-Up and Up-Down alignments to Multi-Object and Same-Object GRUs in Figure 15 This figure was made from the same model seed used to produce Figure 17 We see that IIA for the Increment- Up SA is relatively low for all considered models in the OAF case. ITA is better in the LAF case, but still does not achieve the strong accuracy of the UpwDown SA. We use this result to demonstrate that LAFs cannot find good alignments with all SAs.
1.0
1.0-
Up-Down (Count)
Incr, Progress
f 0.5
LA
0.5
0.0
0.0
Multiobject
SameObject Orthogonal Alignment
MultiObject Linear Alignment
SameObject
Figure 15: The Interchange intervention accuracy (IIA) for variables from the Up-Down and Increment- SAs using different alignment functions for single model seed for each of the Multi-Object and Same-Obir Object GRUs The model seed is the samo as that used for Figure [7 The displayed IIA is for the Count variable from the UowDown SA (Up-Down Count) and the Progress variable from the Increment-Up SA (Iner. Progress). We see that even though the LAF raises ITA for both SAs, does not achieve comparable accuracies. We use the Increment-Us Up SA a negative baseline experiment to demonstrate the non-triviality of high LAF IIA.
27

---

Published in Transactions on Machine Learning Research (07/2025)
A.4 DAS Training Details
A.4.1 Rotation Matrix Training
To train the DAS rotation matrices, we applied PyTorch's default orthogonal paramelrization to a square matrix of the same size the model's state dimensionality. PyTorch creates the orthogonal matrix as the exponential of a skew symmetric matrix. In all experiments, we selected the mumber of dimensions to intervene upon half of the dimensionality of the state. We chose this value after an imitial hyperparameter search that showed the number of dlmensions had little impact on performance (see Figure B)_ We sample 10,000 sequence pairs for the intervention training dataset. See Supplement A..433 for more details on intervention data construction and examples. We use learning rate of 0.001 and batch size of 512. We removed models with performance below 99% to limit our DAS results to perfectly performing models thus simplifying our interpretations of the results. We chose 99% accuracy instead of 100% due to slight numerical underflow in accuracy caleulations and due the fact that half of the Mariable-Length Same- Object models would have been dropped due to low performance.
A.4.2 Symbolic Program Algorithms
Algorithm One sequence step of the Up-Down Program
q f- Count
+- Phase
t input token
if = BOS then
b BOS beginning of sequence token
< 0, p 0 return sample(D)
sample a demo token
else if y D then
D set of demo tokens
g - + return sample(D)
else if = T then --1 1
T is trigger token
else if == R then
R is response token
q + 1
end if
if (g = & (p -== 1) then
return EOS
EOS is end of sequence token
end if
return R
28

---

Published in Transactions on Machine Learning Research (07/2025)
Algorithm One sequence step of the Up-Up Program
d < Demo Count
- Resp Count
P +- Phase
y < input token
if BOS then
BOS beginning of sequence token
d <- 0, + 0, p return sample(D)
sample demo token
else if E D then d - d+1 return (Dmple(D)
D D is set of demo tokens
else if - T then p <-
is trigger token
else if y == R then
R is response token
f- 1
end if
if (d === r) & (p == 1) then
return EOS
EOS is end of sequence token
end if
return R
Algorithm One sequence step of the specific Cix-Distr Program
list of previous values excluding the most recent step
{ <- Input Value
The value of the most recent token 0 indicates the demo phase, is the response phase
+ Phase
y < input token
v.append(e)
8 < SUM(()
if BOS then L e 0, +
BOS is beginning of sequence token
return sample(D)
> sample a demo token D Sun is 0 or less in the response phase EOS is end of sequence token D. T trigger token, R is response token
else if < 0 and = then
return EOS
clsc if y/ = T or y = R then
p
l -1
return R
else if E D then (-1 end if
b D is set of demo tokens
if p = then
return R
else
return sample(D)
end if
29

---

Published in Transactions on Machine Learning Research (07/2025)
Algorithm 4 One sequence step of the Increment-Up Program
m + Interval
f- Progress
P + Phase
f Increment
y imput token
if BOS then
BOS is beginning of sequence token
- 0, p - 0, i + return sample(D)
sample demo token
else if (y E D or = R) and < m then
イー + i m
else if = T then
is trigger token
p p-1
= 1
<- 0 end if
if (q 2 m) and (p == 1) then
return EOS
EOS is end of sequence token
else if 2 m and == then
return T
else if 0 then
return sample(D)
else
return R
end if
A.4.3 DAS Intervention Data
Here expand upon the intervention data used to train and test the DAS rotation matrices. We organize this section into programs, variables, and tasks. For each DAS training, we train a single orthonormal matrix and only create interventions that depend on a single variable from the corresponding program. To construct an intervention sample, we first sample a target sequence and source squence and a positional index from each sequence. We limit positional indices to the demo and resp tokens. We then compute the values of each of the variables using the symbolic algorithm up to the positional index for both the target and source. The value of the variable of focus is then transferred from the source into the the target variable. We then continue the target sequence based on the new value. When the target sequence's counterfactual sequence begins in the demo phase, uniformly sample the number of demo sequence steps before placing the trigger token such that the Count (or Demo Count does not exceed the maximum count used in the task. We note that this makes the samples not strictly counterfactual literature, but the desired effect is the same as the true counterfactual comes from the same distribution.
the definition used in the causal inference
A.4.4 Up-Down Program Examples
Count Variable: Interventions attempt to transfer the representation corresponding the difference between the number of resp tokens and demo tokens. Interventions are only performed at positional indices corresponding to demo resp tokens.
30

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
Multi- Object Examples
1 BOS D1 BOS D3 D2
9 BOS D2 D1 TR BOS D1 D2 D1 T R R EOS
4 BOS D1 D3 T.R. BOS D2 D2 T R R EOS D2 T R EOS
Source Sequence
BOS D2 D1 D1 BOS D2 T R EOS
Target Sequence
Original Labels
D2 D3 E BOS
Counterfactual
D2 D3 TR EOS
R R R EOS
R EOS
Single-Objed Examples
1 BOS D
3 BOS D BOS D
4 D
Source Sequence
BOS D
Target Sequence
BOS
BOS T EOS R R EOS
BOS D D R EOS DTR EOS
Original Labels
DDTR R R R EOS DDTRRR EOS
R EOS
Counterfactual
R EOS
Sume-Objec Examples
1
3 BOS CCTC BOS CCCTC
4 CCSC
Source Sequence
BOS C BOS CC
BOS CCC BOS C C EOS C EOS
Target Sequence
BOS C CTCC EOS CTC EOS
Original Labels
CCTCCCCEOS EOS CCTCCC EOS
C EOS
Counterfactual
C EOS
Phase Variable: Interventions transfer the representation corresponding to the Phase of the sequence (whether it counting up or counting down). Interventions are only performed at positional indices corresponding to demo resp tokens.
Multi- Objet--beetttt Examples
1 BOS D1 BOS D2 D1 D3 D1 TRRRE EOS D3 D2 TRRRR EOS
2 BOS D3 D₃ D2 BOS Da T EOS D2 TR EOS
3 BOS D2 D1 TR BOS D1 D3 D1 TR
4 BOS D2 D3 TRR BOS D2 D1 TRR R EOS R EOS
Source Sequence
Target Sequence
Original Labels
R EOS
Countorfactual
R EOS
Single-Object Ezamples
3 BOS DDDD BOS D DDDR.
4 BOS D TRRR BOS D
Source Sequence
BOS D BOS D D DDTRRRR, EOS
BOS D D BOS D DTT R. FOS D T R EOS
Target Sequence
Original Labels
R R EOS
D R EOS R EOS
Counterfactual
DDTRRRE EOS
R R EOS
Same-Object Examples
3 BOS CCCC
4 BOS BOSC
Source Sequence
BOS C
BOS CCC
Target Sequence
BOS CC
BOS CTC
BOS CCCTC
BOS C CTCO CTCCEOS
Original Labels
CCTCCCCEOS ECTCCCGEOS
EOS
EOS
Counterfactual
C T C EOS
C C EOS
C EOS
31

---

Published in Transactions on Machine Learning Research (07 (025)
A.4.5 Up-Up Program Examples
Demo Count Variable: Interventions attempt to transfer the representation corresponding to the number of demo tokens in the sequence. Interventions are only performed at positional indices corresponding to demo or resp tokens. We remove training and evaluation samples in which the Demo Count is less than the Resp Count
Multi- Object Ezamples
1
2
Source Sequence
BOS D1 BOS D3 D2 T R R BOS T R EOS
BOS D2 D3 D3
BOS D2 D1 T.R. BOS D1 D2 D1 T.
BOS D1 D3 R R BOS D2 D2 T R R EOS D2 T R R R ROS
Target Sequence
BOS D2 D2 D3 TRR R EOS
Original Labels
R R EOS
Counterfactual
R EOS
R FOS
Single-Obeell Ezamples
2 BOS D D D BOS D D DTRR R EOS
3 BOS DDTRR
Source Sequence
BOS T) BOS D D T R R EOS T R EOS
BOS DDTRE BOS D DTRE R EOS DTRRREOS
Target Sequence
BOS DDDTR
Original Labels
R EOS
Counterfactual
R EOS
EOS
Hume-Objedt Examples
Source Sequence
BOS C BOS C C T C C EOS T C EOS
BOS CCC BOS C CCCTCC
BOS CCTCC
BOS CCTCC BOS C CTCCEOS CTCCCEG
Target Sequence
BOS C CCCTC
Original Labels
C EOS C EOS
Counterfactual
C EOS
Resp Count Variable: Interventions attempt to transfer the representation corresponding to the mumber of response tokens in the sequence. Interventions are only performed al positional indices corresponding to demo or resp tokens. We remove samples from the training and evaluation sets that transfer Resp Count greater than the Demo Count into the response phase.
Multi- Object Examples
9
4 BOS D1 D, D, TRRR BOS D2 D2 TR R EOS D2 FOS
Source Sequence
BOS Dj D3 D3 BOS D3 Do T R R EOS T RR R EOS
BOS D2 BOS D2 D2 D3 TRR R EOS
BOS D2 D1 TRR BOS D1 D2 D1 TR
Target Sequence
Original Labels
R R EOS
Counterfactual
R R FOS
FOS
Single-Objer Ezamples
3 BOS DDTRR. BOS DDDTR
4 BOS D DDDTRRE BOS D
Source Sequence
BOS D BOS DD
BOS D BOS D DDDTRR
Target Sequence
Original Labels
TR R EOS TR R FOS
EOS R FOS
R EOS
D T R EOS
Counterfactual
R FOS
D T FOS
Hame-Objeee Examples
1
3 BOS CCTCO
4 BOS CCCTCCC BOS C CTCC EOS CT EOS
Source Sequence
BOS C C BOS C C T EOS C EOS
BOS C BOS CCCTCC C EOS CC C EOS
Target Sequence
BOS CCCTC
Original Labels
C BOS
Counterfactual
EOS
32

---

Published in Transactions on Machine Learning Research (07/2025)
A.4.6 Ctx-Distr Program Examples
We perform these interventions directly by subslituting the source hidden state into the target hidden state without using DAS. Each intervention examines whether the state encodes sufficient information to transfer the NN's behavior from the source sequence into the target sequence. If the NN uses a Markovian hidden state, then transferring the hidden state from one position to another should result in corresponding transfer of behavior. In the case that the NN uses anti-Markovian states, then we would expect the model's behavior be unchanged at token positions that did not receive interventions. Higher accuracies correspond to behavioral transfer. Interventions only performed at positional indices corresponding to non con-terminal response tokens.
Anti-Markovian States:
Multi- Object Examples
3
Source Sequence
BOS D1 D3 TRR. BOS D3 D22 R R R EOS R R EOS
BOS D2 T BOS D2 D2 D3 T R R EOS RRD R EOS
BOS D2 D1 TR BOS D1 D2 TRR EOS EOS
BOS D1 D3 D3 TRR BOS D2 D1 D2 TR R R EOS
Target Sequence
Original Label
Counterfactual
R R EOS
Single-Object Examples
1
2 BOS DT R BOS D D DRR R.R. R EOS R R EOS
3
4 BOS D BOS DDDTR
Source Sequence
BOS D BOS D
BOS DDDR BOS D
Target Sequence
Original Label
R EOS R EOS
EOS
R R EOS R EOS
Counterfactual
EOS
Same- Object Examples
1
2 BOS CTC BOS CCCTC
3
Source Sequence
BOS CCTCC BOS CCC
BOS CCCTC
BOS CCCTCC BOS CCCTC
Target Sequence
BOS CCTCC
Original Label
CC EOS
EOS
C EOS
Counterfactual
C C EOS
CCC EOS
EOS
C C EOS
Input Value Variable: These interventions attempt to transfer the representation corresponding to the value with which the tokens contribute to the cumulative difference between the demo and resp tokens. A value of +1 is assigned to demo tokens, a value of is assigned to resp lokens, and the algorithm stops when the sum of the values is equal to in the resp phase. Interventions are only performed at positional indices corresponding to demo or resp tokens, and we restrict the number of demo tokens to be at least 2 when intervening on the demo phase. This latter restriction to avoid cases where the cumulative value is negative.
33

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Multi- Object- Examples
1 BOS D1 BOS D3 D2 TR R EOS TR R EOS
2 BOS D2 BOS D2 D2 D3 T
Source Sequence
BOS D2 D1 TRR BOS D1 D2 D1 TR
BOS D1 D3 D, TRRR BOS D2 D1 D2 T R R R EOS D2 T R EOS
Target Sequence
Original Labels
R EOS
R R EOS
Counterfactual
R R R EOS
R R EOS
Single-Objer Examples
1 BOS D BOS D T R R EOS T R R EOS
2
Source Sequence
BOS
BOS DDTRR
BOS DDDTRRE BOS D D R R R EOS D R EOS
Target Sequence
BOS D DDDTRR
BOS DDDTR
Original Labels
R EOS
R EOS
Counterfactual
R R.R. EOS
R EOS
Game-Objed Examples
1
2 BOS C BOS C CCTCC C EOS
3 BOS CCTCC
Source Sequence
BOS C BOS C C TC C EOS T C EOS
BOS CCCCCCCC BOS CC CTCCC EOS CTC EOS
Target Sequence
BOS CCCTC
Original Labels
CC EOS
Counterfactiual
C EOS
C EOS
A.4.7 DAS Gradience Evaluation Data
The data used for Figures 3 (c) and (d) was constructed by sampling a single target sequence for every object count ranging from 1-20, and source sequences with object counts incrementing by for cach target sequence. Interventions were then constructed for each target count, source count pair within each sequence pair. This procedure repeated for three times, each with different number of steps in the demo phase before providing the trigger token. The number of continued demo steps was 1, 4, and 12 respectively.
A.5 Context Distributed Interventions
We detail in this section why our Ctx-Distr interchange interventions sufficient to demonstrate that the transformers use solution that secefference//ecoommpe
the relevant information to solve the lasks al each step in the sequence. The hidden states in Layer 1 are a bottleneck at which a cumulative counting variable must cxist were to use a strategy like the Up-Down or Up-Up programs. This is because the Attention Outputs of Layer 1 are the first activations that have had an opportunity communicate across loken positions. This means that the representations between the Residual Stream of Layer up to the Residual Stream of Layer cannot have read cumulative state from the previous token position other than reading off the positional information from the previous positional encodings. The 2-layer architecture is then limited in that it has only one more opportunity to transfer information between positions the attention mechanism in Layer 2. Thus, if a hidden state at time were to have encoded a cumulative representation of the count that will be used by the model at time + 1, that cumulative representation must exist in the activation vectors between the Residual Stream 1 in Layer 1 and the Residual Stream 0 Layer 2. is using such a cumulative representation, then when we perform a full activation swap in the Layer hidden states then the resulting predictions should be influenced by the swap.
A.6 Variable-Length Task Variants
Here we include additional tasks to prevent the transformers with positional encodings from learning a solution that relies reading out positional information. We introduce Variable-Length variants of each of the Multi-Object, Single-Object, and Same-Objed tasks. In the Variable-Length versions, each token in the demo phase has 0.2 probability of being sampled as unique "void" token type, V, that should be ignored when determining the object quantity of the sequence. The number of demo tokens will still be equal to the object quantity when the trigger token is presented. We include these void tokens as a way to vary the length of the demo phase for a given object quantity, thus breaking correlations between positional information and object quantities. As an example, consider the possible sequence with a object quantity of 2: 'BOS D V D T R R FOS'
34

---

Published in Transactions on Machine Learning Research (07 (025)
A.7 Anti-Markovian Proof
Notation.
We use the following symbols throughout the theorem and proof:
zy the input token ID position i in the sequence.
Ex,od( the embodding of token zz this taken to be the "residual stream" output of layer t= 0.
& the layer index. We number layers so that l = 0 is the embedding layer, and & = 1,2... are the successive self-attention layers.
t: the final position in the sequence whose cumulative state we wish to encode.
hg Rd the residual-stream vector layer l and position i. In particular,
s0+ Embed (x:), 2
= 0,
and for 82 1,
hf = h + attn ,..., . t)
attn;()): the causal self-attention update at layer e, which may attend only to token-wisc lincar functions of positions S when computing K4
sy the "Markovian" cumulative state after sceing tokens up position i. By hypothesis,
8; g($i-1, v:), su I(,, zo),
and we require hy exactly when 2 i.
gi the state-update function g: (previous state, current token) new state.
#layers: the total number of self-attention layers in the Transformer (excluding the embedding layer).
Theorem: In a Transformer with causal self-attention, suppose that after & layers, position in the residual stream carries the full Markovian state
8; 1
if and only if 8 126 Then to encode s at position one must have
#layers 2 t.
We proceed by induction on the number of layers l.
Base case (L 0). Layer 0 is just the embedding layer:
h so 200) h. Embed(z.)
(i 2 1).
Thus only position 0 carries the cumulative state, and for any 2 1, L = 0 < t is insufficient to encode
Inductive step. Assume that after layers,
h 8i, i56 l
h
+ Str i> l.
Consider layer & For each position i,
h E+1
nf attnet1 (16.
35

---

Published in Transactions on Machine Learning Research (07 (07/)025)
if then by causality the attention may attend only to positions ,....,E+1.. By the inductive hypothesis, for j S & we have ht 8j, and h = ((*e+1) where f is some function that has not. seen or produced 8-+1- Hence the attention head can compute
se+1 g(st,
and add it its residual stream, yielding M 88+1-
If i >> 2+1, then no information can traverse more than one new position per layer, position i still does not have
Therefore after + layers exactly positions o,...., + 1 carry the states so...., 8241- This completes the induction.
Hence to encode the state at position i, the Transformer must have at least t layers.
A.8 Principle Components Analysis
MultiObject GRU
Token Type
BOS
DO
10
D1
D2
T
PC 1
Start
R
EOS
0
Trajectory Object Count
16
- 5
0
5 PC 2
10
Figure 16: Top 2 Principal Components (PCs) of the hidden states of a single Multi-Object GRU model seed where representations are taken from 15 sampled trials for each object quantity from 1 to 20 in the Multi-Object task variant. Fach dot represents a hidden state representation projected into the first 2 PCs. We plot select trial trajectories exemplify how the states change for given trial. Green points indicate the start of plotted trajectory, black points indicate an intermediate step, and red points indicate the end of plotted trajectory. The blue line plots single trajectory from start finish with object quantity of 2. Similarly, the pink and green lines follow single trajectorics of 8 and 16 respectively.
36

---

Published in Transactions on Machine Learning Research (07/2025)
SameObject GRU
Token Type
BOS
10
Trajectory Object Count
C (Phase 0)
Start
T
C (Phase 1)
PC 2
EOS
End
0008
-10
-5
0
5
10
PC 1
Figure 17: Top 2 Principal Components (PCs) of the hidden states of a single Same-Object GRU model seed where representations are taken from 15 sampled trials for each object quantity from 1 to 20 in the Same-Object task variant. Each dot represents a hidden state representation projected into the first 2 PCs. We plot select trial trajectories to exemplify how the states change for given trial. Green points indicate the start of plotted trajectory, black points indicate an intermediate step, and red points indicate the end of plotted trajectory. The blue line plots single trajectory from start. to finish with object quantity of 2. Similarly, the pink and green lines follow single trajectorics of 8 and 16 respectivcly.
37


Reviews:
[{"review_id": "qPfEcZZxsh", "reviewer": "Reviewer_co4F", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n\n1.\tThe paper focuses on an important problem: whether and how symbol-like representations emerge in neural networks when trained on tasks that require symbolic reasoning.\n\n2.\tMost parts of the paper are well structured and clearly written.\n\n3.\tThe paper provides detailed experimental settings, which enhances reproducibility.\n\nWeaknesses:\n\n1.\tIn Section 3.1, the description of the numerical equivalence task can be made clearer. A step-by-step example may help. For instance, “We consider the Single-object task and choose the object quantity to be 3. Then, the input sequence to the model is [BOS, D, D, D, T]. Given this input, the model is expected to output the same number of response tokens R as the number of demo tokens D presented in the input. Therefore, the expected completion is [R, R, R, EOS].” Furthermore, I’m still not sure whether the demo tokens remain the same across different input sequences. For example, in Single-object tasks, will there be two input sequences [BOS, D, D, D, T] and [BOS, E, E, E, T], where D and E are both considered demo tokens?\n\n2.\tThe justification for choosing certain hyperparameters (e.g., the number of replaced dimensions in DAS) could be stronger. Currently there is no ablation study on how different numbers of replaced dimensions in DAS would affect the result.\n\n3.\tIn Figure 4, the paper finds that recurrent neural networks usually have a low alignment accuracy on *Same-object* tasks, but the reason is not well explored.\n\n4.\tThe DAS method requires one to manually curate a set of symbolic algorithms. Only with those manually curated algorithms at hand, one can then measure the alignment between the neural representation and the symbolic algorithm. This approach is potentially difficult to scale up, as the symbolic algorithms are crafted by humans and usually takes some effort. It would be better if this method could be extended to directly identify whether the neural network implements a specific symbolic algorithm.\n\n5.\tThe DAS method only roughly measures the alignment between the neural representation and a specific symbolic algorithm, but it does not identify the circuit that implements the algorithm. In other words, taking transformers as an example, the method does not tell us which model component (An attention head? Or an MLP?) is responsible for computing the addition and subtraction operation, and which part of the hidden states is responsible for storing intermediate results. This limits the significance of the discovery presented in this paper.\n\n6.\tThe study does not explicitly quantify how much of the network’s performance is explained by these symbolic representations. It is good that the authors analyze the correlation between the alignment to the symbolic algorithm and network’s performance, but it should be noted that this analysis is correlational rather than causal.\n\n7.  The task and the models are relatively simple. The paper might be more impactful by considering more complex numerical tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates whether neural networks encode symbol-like numeric representations when trained on numeric equivalence tasks using next token prediction. Through causal interventions (DAS), the paper shows that recurrent neural networks use a cumulative counting strategy, while transformers recompute numeric values at each step. The paper also discusses graded neural representations and architectural differences, suggesting that neural networks approximate but do not fully internalize symbolic computation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tModify the description of the numerical equivalence task to make it clearer.\n\n2.\tProvide more justification or ablation study on how certain hyperparameters are chosen.\n\n3.\tExplain why recurrent neural networks have a low alignment accuracy on *Same-object* tasks.\n\n4.\tPlease give any comments (or revisions, if possible) to Weaknesses 4-6.\n\n5.\tThe overall layout of the figures can be improved. Figure 2 and Figure 3 appear early in the main text, but are not referenced until the last two pages.\n\n6.\tI’m still confused about what Figure 3 (a) and (b) try to express. It is better that the authors go through them step by step, with detailed explanations on the legends.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ajq2JAYf0r", "reviewer": "Reviewer_GPsF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- This work clearly outlines potential hypothesized mechanisms responsible for performing the task and conducts multiple causal experiments to accept or reject these hypotheses.\n\n- It rigorously examines both RNN and transformer-based models, identifying differences in their underlying mechanisms for performing the same task. This analysis provides novel insights into how neural architecture influences the learned mechanisms.\n\n- Unlike most mechanistic interpretability studies, this work not only investigates the mechanisms after training but also explores how they develop during the training process.\n\n**Weakness**:\n- My primary concern with this work is the lack of clarity in its experiments, which appear cluttered and do not clearly articulate their purpose. Specifically, it does not explicitly describe which variable(s) of which symbolic algorithm are being aligned. The counterfactuals and targets of each experiment are unclear, making it difficult to understand their rationale and significance.\n  - In Section 3.5, the text states that Figure 4 presents results for patching the residual stream after layer 1 to verify the absence of the cumulative count variable—meaning that patching these vectors from different time steps should not affect the final output. However, the first line of Section 4 describes Figure 4 as representing DAS results, creating confusion.\n  - Additionally, in Section 3.5, the authors discuss a DAS experiment aimed at aligning the Last Value variable in the Ctx-Distr program but do not specify the counterfactuals and targets. Similarly, patching at the attention subblock is mentioned without clarifying which variable this experiment seeks to align or in which layer(s) it is conducted.\n  - Moreover, in Figure 4, the Count variable is aligned for the up-down and up-up SAs, but the corresponding experiment, along with its counterfactuals and target, is not explicitly detailed.\n\n- Finally, this work does not cite several existing studies that have attempted to mechanistically understand how LLMs represent numerical values in their internal activations. Some of these relevant works include [1–4, inter alia].\n\n[1] Kantamneni et al, “Language Models Use Trigonometry to Do Addition”, 2025.\n\n[2] Liu et al, “Towards Understanding Grokking: An Effective Theory of Representation Learning”, 2022.\n\n[3] Nanda et al, “Progress measures for grokking via mechanistic interpretability”, 2023.\n\n[4] Zhong et al, “The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks”, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work examines how RNN and transformer-based language models trained on next-token prediction represent numeric concepts in their internal representations by conducting multiple causal experiments. It provides insights into the differences in the underlying mechanisms that enable RNNs and transformer-based models to perform the same task. Additionally, the study highlights a correlation between the emergence of task performance and the alignment of mechanisms during the training process among other insights.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical recommendations**: \n- For each hypothesized mechanism, clearly enumerate every causal experiment separately, specifying their corresponding counterfactuals and targets—ideally, one for each variable in the mechanism. Explicitly describe which variable in the algorithm is being aligned and explain why the proposed causal experiment is meaningful. Additionally, provide details on the granularity of the patching experiment, clarifying whether it is conducted at the residual stream level, attention subblock level, or individual neuron level.  \n\n- Define what is meant by the term “theoretical neural solution,” as mentioned in the second paragraph of Section 4.2 and Figure 3. Furthermore, provide additional context to clarify the reasoning presented in the second paragraph of Section 4.2.  \n\n- Lastly, define the term “graded alignment” as introduced in Section 4.3.\n\n**Additional recommendations**:\n- Please clarify the rationale for using DAS. While DAS is designed to identify the subspace encoding a specific causal variable, this work does not appear to focus on specific subspaces.\n\n- Additionally, cite [5] and provide an explanation of how its findings could help interpret the results of this study.\n\n[5] Haviv et al, “Transformer Language Models without Positional Encodings Still Learn Positional Information”, 2022.", "broader_impact_concerns": "I do not have any concerns regarding the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kZhXRIy7LG", "reviewer": "Reviewer_JpkP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The paper presents an interesting study of how different neural network architectures solve a simple mathematical/symbolic problem. \n- The study considers different variations of the problem and different hyperparameters, supporting the robustness of their findings. \n- The causal experiments are convincing and support the claims. \n- The paper is very well written and easy to follow. \n\nWeaknesses: \n- The studied task is relatively simple. \n- The fact that neural networks can approximate symbolic computation is not necessarily surprising, given prior work in the field of mechanistic interpretability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the emergence of symbol-like number representations in neural networks trained on next-token prediction tasks. Specifically, they study GRUs, LSTMs and transformer models, trained on different variations of numeric equivalence tasks. Then, they use distributed alignment search (DAS) to investigate the alignment between the representations and different hypothesised symbolic algorithms. \n\nThe key finding is that these models learn symbol-like representations that are used to solve the task. Specifically, they find that recurrent models learn a \"count up, count down\" algorithm, whereas transformer models adopt a \"context-distributed\" strategy that queries a history of inputs at each step of the sequence. They also perform additional causal interventions to validate the results of DAS, and study the development of those variables during training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Optional: \n- I believe the discussion and conclusion section, which touches on a number of important issues or additional experiments, could benefit from paragraph titles.", "broader_impact_concerns": "I do not have any concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)