Mechanistic evaluation of Transformers and state space models

## Abstract

Abstract State space models (SSMs) for language modelling promise an efficient and performant alternative to quadratic-attention Transformers, yet show variable performance on recalling basic information from the context. While performance on synthetic tasks like Associative Recall (AR) can point to this deficiency, behavioural metrics provide little information as to why —on a mechanistic level—certain architectures fail and others succeed. To address this, we conduct experiments on AR and find that only Transformers and Based SSM models fully succeed at AR, with Mamba a close third, whereas the other SSMs (H3, Hyena) fail. We then use causal interventions to explain why. We find that Transformers and Based learn to store key–value associations in-context using induction heads. By contrast, the SSMs compute these associations only at the last state, with only Mamba succeeding because of its short convolution component. To extend and deepen these findings, we introduce Associative Treecall (ATR), a synthetic task similar to AR based on PCFG induction. ATR introduces language-like hierarchical structure into the AR setting.
We find that all architectures learn the same mechanism as they did for AR, and the same three models succeed at the task. These results reveal that architectures with similar accuracy may still have substantive differences, motivating the adoption of mechanistic evaluations. github.com/aryamanarora/tinylang

## 

## Mechanistic evaluation of Transformers and state space models

## Mechanistic evaluation of Transformers and state space models

### 1 Introduction

Transformers with quadratic attention remain the dominant architecture in language modelling despite numerous proposed efficient alternatives. Most notably, state space models (SSMs) achieve impressive perplexities and benchmark scores [e.g. Gu and Dao, 2024]. Yet, SSMs exhibit deficiencies that benchmarks often fail to capture; for example, they struggle to perform retrieval, i.e. copying from the context Jelassi et al. [2024], Wen et al. [2024], Waleffe et al. [2024], Bick et al. [2025].

Controlled synthetic tasks can make these limitations clear by isolating specific capabilities and enabling expressive experimentation at small scales across architectures. Particularly, much work has used the associative recall (AR) task as a testbed for studying in-context retrieval across architectures. In turn, AR has informed the design of novel LM architectures [e.g. Based; Arora et al., 2024b].

Yet performance on synthetic tasks is measured solely via behavioural metrics like task accuracy. This is a missed opportunity: an advantage of these synthetic tasks is that they are designed to isolate a specific behaviour that implicates a mechanistic solution. For example, language models should solve AR by storing key–value associations in-context at the value, a mechanism termed the induction head in Transformers Olsson et al. [2022], Fu et al. [2023].
We should therefore directly check whether each architecture learns induction as part of performance evaluation on AR.

Here, we propose using tools from mechanistic interpretability to directly analyse the mechanisms used to solve synthetic tasks. We use causal interventions Geiger et al. [2024] on model internals to understand how these tasks are learned and implemented across a variety of architectures ([section 4](https://arxiv.org/html/2505.15105v2#S4)). This allows us to track the emergence (or lack thereof) of the correct association and retrieval mechanisms inside the model, beyond just observed task accuracy. Through comprehensive experiments on AR, we find that all SSMs except Based learn an inefficient direct-retrieval solution to AR, and that Mamba strongly relies on its short convolution component to perform AR.

To deepen our findings, we introduce Associative Treecall (ATR), a novel synthetic retrieval task more similar to real-world natural language retrieval than AR ([section 3](https://arxiv.org/html/2505.15105v2#S3)). ATR uses a probabilistic context-free grammar (PCFG) to generate hierarchical data, on which we ask AR-like queries. Since keys and values need not be adjacent to each other, ATR requires a true non-positional retrieval mechanism, which may challenge architectures that are designed for AR. Interestingly, we observe the same mechanisms are implicated across architectures on ATR as on AR, indicating that association mechanisms are not task-dependent.

Our results offer a framework for better understanding and evaluating synthetic task performance in terms of mechanistic interpretability. Mechanistic evaluations reveal fundamental differences between architectures beyond what we learn from behavioural performance, thus serving as a new tool for architecture analysis and design.

### 2 Related work

##### Associative Recall.

Associative Recall (AR)(^1^11Also known as associative retrieval, associative memory, or induction.) is a synthetic task that evaluates in-context retrieval for language model architectures, from early work on recurrent neural networks [Graves et al., 2014, Ba et al., 2016, Danihelka et al., 2016, Zhang and Zhou, 2017] to modern SSMs [Fu et al., 2023, Poli et al., 2023, Lutati et al., 2023, Jelassi et al., 2024, Arora et al., 2024a, b, Gu and Dao, 2024, Dao and Gu, 2024, Trockman et al., 2024, Liu et al., 2024a, Okpekpe and Orvieto, 2025, Li et al., 2025b, Wang et al., 2025]. An AR task consists of a sequence of key–value pairs followed by a single query key; the goal is to produce the corresponding value. For example,

- A 2 C 3 F 9 D 1 C $\to$ 3

Here, the correct next token is 3, since it is the value associated with the key C in context.
Despite being synthetic, AR has a direct analogue in natural language: induction, referring to in-context copying of sequences [Elhage et al., 2021, Olsson et al., 2022]. Arora et al. [2024a, b] show that architecture-level improvements on AR translate directly to natural-language induction.

##### Mechanistic interpretability.

In order to measure the contribution of individual model components (neurons, layers, etc.) to output behaviour, we can apply causal interventions on neural network internals [Geiger et al., 2021, 2024]. Informally, the core idea is to overwrite an activation at a specific component using a counterfactual input. If this changes model behaviour, then that component is causally relevant to the mechanism underlying that behaviour.

Some prior work in mechanistic interpretability has studied how some language models solve in-context retrieval tasks like induction and multiple choice question answering [Olsson et al., 2022, Lieberum et al., 2023, Brinkmann et al., 2024, Wiegreffe et al., 2025, Bick et al., 2025], as well as the training dynamics of Transformers on toy tasks using mechanistic metrics [Nanda et al., 2023, Reddy, 2024, Singh et al., 2024, Edelman et al., 2024, Tigges et al., 2024, Yin and Steinhardt, 2025]. Yet thus far, architectural comparisons on synthetic tasks have not made use of causal interventions.

### 3 Synthetic retrieval tasks

Induction, wherein key–value associations are stored in-context, is the memory-efficient mechanism implicated for retrieval tasks like AR in quadratic attention Transformers.
Yet AR can also be solved through naïve positional association, and indeed SSMs theoretically learn a less efficient solution [Jelassi et al., 2024].
To elucidate this, we apply our mechanistic evaluation framework to compare architectures on two synthetic retrieval tasks: Associative Recall and Associative Treecall (ATR). Compared to AR, ATR is a novel language-like task with tree structure and more parameters for controlling task difficulty ([section 3.2](https://arxiv.org/html/2505.15105v2#S3.SS2)). Critically, ATR cannot be solved with naïve positional association, enabling us to explicitly test if models learn different mechanisms for association in the hierarchical setting. We build upon prior work on formal-language synthetic tasks [White and Cotterell, 2021, Valvoda et al., 2022, Hahn and Goyal, 2023, Strobl et al., 2024, Allen-Zhu and Li, 2024, Akyürek et al., 2024, Pandey, 2024, Lubana et al., 2024, inter alia].

Figure: Figure 1: PCFG: An illustrative example of a PCFG and its components, with an example derivation (with final string) below. AR vs. ATR: Comparing AR and ATR using example documents; both tasks provide a document with key–value associations in-context and ask a query about one such association. However, associations in ATR need not involve adjacent tokens and are tree-structured.
Refer to caption: x1.png

#### 3.1 Associative Treecall

Since a standard AR document ([section 2](https://arxiv.org/html/2505.15105v2#S2.SS0.SSS0.Px1)) consists of adjacent key–value pairs, one can associate each key with its corresponding value solely using relative position. Yet many natural language retrieval tasks require association over latent hierarchical structure. For example:

- John had chicken and Mary had pork. The chicken was eaten by $\to$ John

Answering this query requires associating John with chicken and Mary with pork, and then retrieving the appropriate association for John. A solution employing relative positional association would not robust to the possible range of variation (John had some chicken, John decided to have chicken, etc.).

This type of retrieval is widely studied in cognitive science as binding. The mechanisms underlying natural-language binding in neural networks/LLMs have been examined by Greff et al. [2020], Kim and Schuster [2023], Feng and Steinhardt [2024], Prakash et al. [2024], Li et al. [2025a], Prakash et al. [2025]. Yet no synthetic analogue of this task exists to isolate this mechanism and enable direct comparison to AR. ATR thus allows us to study how different architectures implement binding, and ask if these solutions generalize from simple AR.

An ATR corpus is drawn from a synthetic probabilistic context-free grammar (PCFG) whose parameters we set. Each document consists of a string sampled from the PCFG, with latent structure made up of parent–child relations between symbols, followed by a divider token (EOS) and a query about one such relation. The PCFG has one special property which establishes the parent–child relationships: for the right-hand side of each production rule, the rightmost symbol is always a terminal, and is the parent of the symbols created by this production. We sample strings by selecting an iid nonterminal and recursively applying production rules according to the PCFG distribution. We show an example in [Figure 1](https://arxiv.org/html/2505.15105v2#S3.F1) and formalise definitions in [appendix A](https://arxiv.org/html/2505.15105v2#A1).
Since the number of tokens separating parents and their children may vary, ATR cannot be solved by a positional associative mechanism.

#### 3.2 Parameters

**Table 1: Parameters used for constructing a PCFG. We define PCFGs in Greibach Normal Form (GNF); see Appendix [A](https://arxiv.org/html/2505.15105v2#A1) for more details.**
| Param. | Description |
| --- | --- |
| $H$ | Is the head terminal at the left or the right of each production? |
| $d_{\textrm{max}}$ | Maximum depth permitted for the PCFG to generate. |
| $L_{\textrm{max}}$ | Maximum number of symbols of the right-hand side of a production rule. |
| $R_{\textrm{max}}$ | Maximum number of production rules for each nonterminal. |
| $\lvert\mathcal{N}\rvert$ | Number of nonterminal symbols in the PCFG vocabulary. |
| $\lvert\Sigma\rvert$ | Number of terminal symbols in the PCFG vocabulary. |
| $r_{\Sigma}$ | Relative weightage on choosing a terminal when sampling production rules. |

##### PCFG setup.

For each experiment, we generate a single PCFG to use across all models to ensure fair comparisons, with parameters in Table [1](https://arxiv.org/html/2505.15105v2#S3.T1). We also reject any samples that have more than $1024$ symbols, which only affects the sampling distribution for the most complex PCFGs we use.

##### Queries.

Each PCFG sample of length $n$ provides us with a set of $n-1$ eligible parent–child queries (i.e. a tree with $n-1$ edges). However, terminals may occur multiple times, so a query about a specific symbol may present ambiguity; thus, when presenting a query we consider it to only refer to the rightmost instance of that symbol.(^2^22This is the same setup as AR with rewrites Rodkin et al. [2025].) Therefore, the maximum number of eligible queries over all samples is $\min(n-1,\lvert\Sigma\rvert)$. To minimise the ability to heuristically guess, we inversely weight parent–child pairs by the parent’s child count when sampling queries.

#### 3.3 Methodology

##### Datasets.

We generate synthetic pretraining and evaluation datasets for both tasks. For each setting, the trainset has $100,032$ examples and the eval/dev sets have $320$ examples. In AR, we use disjoint key and value vocabularies; in ATR, keys and values are both sampled from the set of terminals. In each document, we separate the document from the query with a divider token, and provide only a single query.
Example AR/ATR documents are in [Figure 1](https://arxiv.org/html/2505.15105v2#S3.F1); further details in [appendix C](https://arxiv.org/html/2505.15105v2#A3).

##### Models.

We pretrain models from scratch on a variety of synthetic tasks. We use the exact architecture implementations from the zoology(^3^33[https://github.com/HazyResearch/zoology](https://github.com/HazyResearch/zoology)) library [Arora et al., 2024b], except for behaviour-preserving modification of the LM backbone to enable interventions with pyvene(^4^44[https://github.com/stanfordnlp/pyvene](https://github.com/stanfordnlp/pyvene)) [Wu et al., 2024] on the sequence mixers, MLPs, and layer blocks. The LM backbone for all architectures is the same, with pre-norm blocks of alternating sequence mixers and MLPs (except for Mamba, which has no MLP) followed by LayerNorm at the end.
We experiment with the following architectures: Attention [Vaswani et al., 2017], BaseConv Arora et al. [2024a], Based Arora et al. [2024b], H3 Fu et al. [2023], Hyena Poli et al. [2023], and Mamba Gu and Dao [2024]; further details on model configurations are given in [appendix B](https://arxiv.org/html/2505.15105v2#A2).

##### Training.

We minimise cross-entropy loss, and mask the loss on all tokens except the query (the underlined token in the example below). We use the AdamW optimiser with $\beta=(0.9,0.999),\epsilon=10^{-8}$ and no weight decay. We warm up learning rate for the first $10\%$ of training and then follow a cosine decay schedule to 0 0 for the remainder of training. We train for either $16$ epochs (on AR) or $32$ epochs (on ATR) with a batch size of $32$.
Each experiment trains $\approx 200$ models over all hyperparameters. Runtime varies from $0.5$ to $5$ hours, depending on hardware, task, and architecture. Overall, we used $<10,000$ GPU-hours in total, on a cluster with various NVIDIA machines (with GPU memory ranging from $12.3$G to $143.8$G).

##### Behavioural metrics.

We report behavioural metrics given the model’s predicted probabilities over the vocabulary $\hat{\mathbf{y}}\in\mathbb{R}^{\lvert\Sigma\rvert}$ and the index of the single true answer $i$. Our main metric is accuracy: $\mathds{1}[\mathrm{arg\,max}(\hat{\mathbf{y}})=i]$. Additionally, we compute but do not primarily report likelihood $\hat{\mathbf{y}}_{i}$.

### 4 Mechanistic metrics for AR and ATR

Figure: Figure 2: Our interchange intervention setup for analysing AR and ATR. Left: We intervene on input and output representations of whole blocks (1 and 6), sequence mixers (e.g. attention blocks; 2 and 3), and state mixers (4 and 5). Right: An example intervention on AR where we corrupt and attempt to restore the key (A) by intervening at the value token in an intermediate representation. We evaluate the downstream effect on the next-token prediction at the query.
Refer to caption: x2.png

Behavioural metrics provide little information as to why certain architectures succeed or fail on tasks of interest.
Mechanistic metrics, which directly measure how information flows across model components and token positions, can tell us how AR and ATR are being solved by different architectures, and thus help us understand failures. We illustrate our approach in [Figure 2](https://arxiv.org/html/2505.15105v2#S4.F2).

We use interchange interventions [Geiger et al., 2021, 2024] to understand and measure how solutions to AR and ATR are implemented across architectures. We introduce this operation and define the resulting metrics for our tasks below. Our implementation uses the pyvene library Wu et al. [2024].

##### Interchange intervention.

Consider a language model $p(\cdot)$ and some input $\mathbf{b}$. We select a component $f$ inside that model which computes some internal representation $f(\mathbf{b})$ during the LM’s forward pass. Now, consider a counterfactual input $\mathbf{s}$: this produces a counterfactual representation $f(\mathbf{s})$ when processed by $f$. We want to understand what about the output of $p$ is dependent on $f$. Therefore, we perform an intervention which replaces the output $f(\mathbf{b})$ with that of $f(\mathbf{s})$ during the computation of $p(\mathbf{b})$, with the change propagating downstream. The result is notated $p_{f\leftarrow f^{*}}(\mathbf{b},\mathbf{s})$.

##### Concrete setup for AR and ATR.

We take $\mathbf{o}$ to be a ground-truth document from our data distribution and $\mathbf{c}$ to be a version of that document with exactly one important token corrupted: the key (see [Figure 2](https://arxiv.org/html/2505.15105v2#S4.F2)). This corruption significantly reduces task accuracy for both AR and ATR by removing information that is necessary to answer the query.

We intervene at both the input and output each of the following model components $f$: each layer block, each sequence-mixer, and each state-mixer (i.e. MLP, except in Mamba which lacks this component); see [Figure 2](https://arxiv.org/html/2505.15105v2#S4.F2), left. We measure to what extent the intervention can restore the likelihood of the correct answer to the query, i.e. we compare restored likelihood $p_{f\leftarrow f^{*}}(y_{\textrm{true}}\mid\mathbf{c},\mathbf{o})$ with original likelihood $p(y_{\textrm{true}}\mid\mathbf{o})$ and corrupted likelihood $p(y_{\textrm{true}}\mid\mathbf{c})$.

##### Metrics.

Given the above three quantities, we compute
attribution score, or what proportion of the original likelihood was restored by the intervention:

$$ $\displaystyle\mathrm{Attrib}(f)$ $\displaystyle=\frac{p_{f\leftarrow f^{*}}(y_{\textrm{true}}\mid\mathbf{b}, \mathbf{s})-p(y_{\textrm{true}}\mid\mathbf{b})}{p(y_{\textrm{true}}\mid\mathbf {s})-p(y_{\textrm{true}}\mid\mathbf{b})}$ (1) $$

For AR and ATR in particular, there are two choices for $f$ which help us distinguish the mechanism underlying task success. To check whether induction is the underlying mechanism, we compute metrics for $f$ being the layer $1$ block input at the value token. Alternatively, we check whether other tokens at layer $1$ block input mediate information flow, indicating some sort of association-less direct retrieval mechanism: the key, query, and divider.

### 5 Experiments

We now deploy our mechanistic metrics ([section 4](https://arxiv.org/html/2505.15105v2#S4)) on both AR and ATR ([section 3](https://arxiv.org/html/2505.15105v2#S3)). We follow the methodology outlined in [section 3.3](https://arxiv.org/html/2505.15105v2#S3.SS3) to create a variety of AR and ATR datasets and train models with various architectures and hyperparameter configurations. See [appendix D](https://arxiv.org/html/2505.15105v2#A4) for additional experiments not included here.

#### 5.1 (Most) SSMs do not learn induction to solve AR

Figure: (a) Model dimensionality vs. accuracy after tuning learning rate for each setting.
Refer to caption: x3.png

We run experiments on a relatively simple AR task and show that interchange interventions empirically confirm the same mechanisms underlying AR as proposed in existing theoretical work. We fix the total number of unique keys and values in the vocabulary to be $8192$, and present $32$ key–value pairs in context. Our trainset includes $100032$ examples. We vary model dimensionality in $\{16,32,64,128,256\}$ and sweep LR in the range $[3\cdot 10^{-5},3\cdot 10^{-2}]$ for each architecture.

##### Behavioural results.

[Figure 3(a)](https://arxiv.org/html/2505.15105v2#S5.F3.sf1) demonstrates that task accuracy on AR cleanly separates Attention, which achieves $100\%$ accuracy at $d\geq 32$, from nearly all SSMs. Based solves AR near-perfectly with roughly the same dimension-wise scaling curve as Attention, achieving a maximum accuracy of $99.06\%$. However, Mamba is a close third and clearly better than other SSMs at AR, albeit achieving a less-than-perfect $91.25\%$ at $d=256$.

##### Mechanistic analysis.

We compute $\mathrm{Attrib}$ for layer 1 block input at the value token vs. query token for all training runs where $p(y_{\textrm{true}}\mid\mathbf{o})-p(y_{\textrm{true}}\mid\mathbf{c})>0.01$.(^5^55We filter in order to discard low-performing and noisy runs.) A high attribution score on the value token indicates induction as the underlying mechanism while query indicates direct retrieval at the final state, performed in layer 0. Our results in [Figure 3(b)](https://arxiv.org/html/2505.15105v2#S5.F3.sf2) cleanly separate Attention (with nearly all checkpoints with $100.00\%$ attribution at the value) and Based, which only perform induction, from other SSMs, which perform direct retrieval. While only a single BaseConv checkpoint passes our filter, it has the greatest attribution score on the value, indicating an induction mechanism.

SSMs perform direct retrieval at varying layers: the best-performing Mamba, Hyena, and H3 models almost entirely perform direct retrieval at layer 0 via the query token, while worse SSM checkpoints use a mix of query and key tokens, indicating delayed direct retrieval by both layer 0 and layer 1. Jelassi et al. [2024] shows that direct retrieval in SSMs has asymptotically worse capacity than the induction solution, and this is reflected in performance on AR.

#### 5.2 Per-architecture mechanisms are similar between ATR and AR

We consider four initial settings to study models on ATR, over all combinations of $L_{\textrm{max}}=\{5,10\}$ and $\lvert\Sigma\rvert=\{20,8192\}$. We keep all other parameters fixed with settings given in [appendix C](https://arxiv.org/html/2505.15105v2#A3). Varying $L_{\textrm{max}}$ controls the possible distances between keys and values in the PCFG sample without affecting other properties that play a role in task difficulty (e.g. depth). Varying $\lvert\Sigma\rvert$ stresses the state capacity, since more key–value pairs must be tracked, without affecting syntactic complexity. We sweep the same model dimensionalities as in [section 5.1](https://arxiv.org/html/2505.15105v2#S5.SS1), and a smaller learning rate range of $[3\cdot 10^{-5},3\cdot 10^{-3}]$.

##### Behavioural results.

We report results in [Figure 4(a)](https://arxiv.org/html/2505.15105v2#S5.F4.sf1). Surprisingly, Mamba is highly successful at ATR. On the small terminal count setting ($\lvert\Sigma\rvert=20$) Mamba matches or outperforms all other architectures at all model dimensions, particularly with longer production rules ($L_{\textrm{max}}=10$) with performance of $92.19\%$ vs. $80.94\%$ for Attention at $d=256$. This is particularly surprising because longer production rules imply greater positional variation between keys and values, which ought to stress AR-focused SSM designs. Attention only manages to outperform Mamba slightly on the large terminal count setting ($\lvert\Sigma\rvert=8192$) when $d\leq 64$.

##### Mechanistic analysis.

We conduct the same analysis as for AR. We recover the same overall trends but with greater inter-architecture variance: [Figure 4(b)](https://arxiv.org/html/2505.15105v2#S5.F4.sf2) shows that Attention, Based, and BaseConv all primarily learn induction mechanisms, whereas the remaining SSMs perform direct retrieval as on AR, with high attribution scores on either the key (indicating direct retrieval by the layer 1 sequence mixer) or the query (indicating the same but by layer 0).

Intriguingly, [Figure 4(b)](https://arxiv.org/html/2505.15105v2#S5.F4.sf2) shows that different SSMs form different strategies across task difficulties; in particular, all direct-retrieval SSMs favour delaying retrieval to layer 1 when terminal count is large ($\lvert\Sigma\rvert=8192$), but use a mix of layers otherwise. Regardless, the same tendency from AR recurs: SSMs besides Based and BaseConv do not perform induction, but Mamba is still highly performant. Strikingly, as the next section shows, Mamba also achieves high generalization performance on ATR.

Figure: (a) Model dimensionality vs. accuracy after tuning learning rate for each setting.
Refer to caption: x6.png

#### 5.3 Mamba’s solution to ATR does generalise

We reuse the easiest settings from our ATR experiment ($L=5,\lvert\Sigma\rvert=20$) and construct a new dataset with a train–test split on query–answer pairs. Specifically, $80\%$ of possible unique query–answer pairs are provided in the training set, while $20\%$ are only in the test set and thus never trained on. We seek to assess whether models learn a general mechanism for parent–child relations in ATR or if the impressive results of Mamba (as well as Attention and Based) are merely the result of better memorisation of the PCFG parameters.
This setup is akin to Wang et al. [2024]’s technique of train–test split on multi-hop queries; we provide supervision on individual query and answer types, but not on some compositions of them.

##### Behavioural results.

We select the checkpoint with the highest dev accuracy for each architectural and dimensionality setting, after sweeping LR. We plot the dev and test accuracies of each of these checkpoints in [Figure 5(a)](https://arxiv.org/html/2505.15105v2#S5.F5.sf1); all models have much lower test accuracy (e.g. Attention with $d=256$ has $95.62\%$ dev and $68.12\%$ test accuracy). Attention achieves the greatest dev accuracies on $d\geq 32$. Mamba’s relative ranking is lower than on the in-distribution setting in [section 5.2](https://arxiv.org/html/2505.15105v2#S5.SS2), but it still achieves the overall second-highest dev accuracy ($65.00\%$ at $d=128$). Surprisingly, H3 generalises well despite its poor dev accuracy, beating Mamba on test accuracy in $3$ out of $5$ settings.

We compare dev and test accuracies across all LRs in [Figure 5(b)](https://arxiv.org/html/2505.15105v2#S5.F5.sf2). We find that while Mamba does have unusually high dev accuracy given a selected test accuracy (indicating greater memorisation than models with other architectures), its dev accuracy is still generally higher than non-Attention architectures. Interestingly, H3 has nearly Attention-level generalisation while BaseConv exhibits vanishingly little generalisation. Overall, behavioural metrics show that Mamba does nontrivially generalise on ATR, albeit not as well as Attention.

##### Mechanistic analysis.

We report a summary of attribution scores at different tokens (key, query, value), comparing on dev and test sets across all checkpoints in [Figure 5(c)](https://arxiv.org/html/2505.15105v2#S5.F5.sf3). We find largely consistent mechanisms underlying behaviour on both dev and test, and these match attribution scores on ATR without train–test split. The only exception is that BasedConv does induction on the dev set but not nearly as much on the test set; its induction mechanism is more brittle than Attention and Based.

Overall, the induction mechanism is not more general than the direct retrieval mechanism; both Attention and Mamba show greater generalisation than other architectures despite their entirely different solutions, and our mechanistic evaluations confirm that this solution is consistent across in-distribution and out-of-distribution queries.

Figure: (a) Model dimensionality vs. accuracy on checkpoints with highest dev accuracy.
Refer to caption: x8.png

#### 5.4 Short convolutions enable AR and ATR in Mamba and Based

Throughout all our experiments on AR and ATR, we repeatedly observed that Attention, Based, and Mamba are the highest-performing architectures. However, their underlying mechanisms differ: Attention and Based learn induction, a 2-layer mechanism which stores key–value associations at the value token as an intermediate step, whereas Mamba uses direct retrieval, a 1-layer mechanism which directly writes an association to the query token.

Importantly, Based and Mamba share a key architectural component: short convolutions. We hypothesise that this component is necessary(^6^66Since Hyena also has a short convolution, this may not be sufficient for good performance on association.) for performing association (as in AR and ATR) when using a subquadratic sequence mixer. We conduct experiments on AR where we shorten the convolution kernel size in Mamba (from the default $d_{\mathrm{conv}}=4$ to $\{3,2,1\}$, and deleting it) and replace the Based short convolution with implicitly-parametrised long convolution [Poli et al., 2023].

##### Results.

We report results of our ablations in [Figure 6](https://arxiv.org/html/2505.15105v2#S5.F6). On Mamba ([Figure 6(a)](https://arxiv.org/html/2505.15105v2#S5.F6.sf1)), we find a step change in task accuracy when increasing $d_{\mathrm{conv}}$ from $1$ to $2$, which introduces previous token information and thus enables AR. Without short convolution, Mamba fails to learn AR. [Figure 6(b)](https://arxiv.org/html/2505.15105v2#S5.F6.sf2) further shows that larger kernel size leads to earlier (in layer 0) direct retrieval. Finally, besides $d_{\mathrm{conv}}<2$ like Mamba, implicit long convolution in Based also significantly harms AR performance ([Figure 6(c)](https://arxiv.org/html/2505.15105v2#S5.F6.sf3)). Therefore, we conclude that short convolutions are responsible for association on AR in Mamba and Based.

Figure: (a) Accuracy on AR (length $32$) for Mamba when varying the kernel size of the short convolution.
Refer to caption: x11.png

### 6 Discussion

##### Why mechanistic evaluations over behavioural metrics?

Architectural advances on language modelling are largely uncovered and presented in an empirical manner; beyond intuition, we have little justification as to why a modification or innovation improves model performance. Synthetic tasks already regularly inform progress on subquadratic architecture design (such as SSMs), but treating such tasks as another downstream evaluation is loses useful signal; control over task parameters presents an opportunity to explain performance using interpretability.

##### ATR indicates induction is highly general.

We introduced ATR to break the naïve key–value adjacency of AR, and see whether general mechanisms underlying association still emerge across architectures. We find the same induction mechanism, where the association is computed and stored at the value before retrieval, in Attention and Based for both tasks. While Olsson et al. [2022] and later works define induction on adjacent tokens, ATR is evidence that a position-independent and generalising ([section 5.3](https://arxiv.org/html/2505.15105v2#S5.SS3)) notion of association can be implemented by a single attention head. Further investigation of ATR (e.g. multi-hop queries) is necessary to understand the limits of induction.

##### Short convolutions are key to association in SSMs.

We showed that Mamba and Based rely on short convolutions to learn how to associate keys and values on AR and ATR. Several earlier works point to the importance of short convolution: Arora et al. [2024b] empirically show its utility on AR (along with sliding-window attention), Allen-Zhu and Alfarano [2025] introduce a short convolution component (Canon) in various architectures to improve synthetic and real task performance, and Olsson et al. [2022] show that 1-layer attention can learn induction if augmented with a length-2 convolution; further see Liu et al. [2024b], Dolga et al. [2024], Fu et al. [2023], Poli et al. [2023].

### 7 Limitations

While we proposed mechanistic evaluations as a new tool, behavioural metrics like accuracy are still needed to properly contextualise results. Additionally, here we did not perform mechanistic evaluation of subcomponents of sequence mixers (e.g. the selective SSM component within Mamba), due to implementation difficulties when applying interventions within hardware-optimised operators, which are inaccessible via PyTorch hooks. Finally, we focus on synthetic tasks throughout this work; extending our analyses to real-world models would help paint a more complete picture of the differences in capabilities (and underlying mechanisms) of different architectures on real-world tasks.

### 8 Conclusion

In this work, we introduce mechanistic evaluations as a powerful framework for comparing model architectures. This approach goes beyond high-level behavioural metrics, revealing substantive differences between architectures. Through analysis of synthetic in-context retrieval tasks, we uncover the underlying mechanisms that explain the success and failure points of various architectures. Mechanistic evaluations thus provide a useful tool for architecture design and analysis, as well as a new opportunity for interpretability research to open the blackbox of progress in AI.

### Acknowledgements

We would especially like to thank Zhengxuan Wu, Qinan Yu, Atticus Geiger, and all other attendees of the #weekly-interp-meeting at Stanford who gave feedback on an early version of this project. We also thank Yanzhe ‘Sanju’ Zhang, Harshit Joshi, Rohan Pandey, Justus Mattern, Ken Ziyu Liu, Julie Kallini, Chenglei Si, Bradley Brown, Jordan Juravsky, Arjun Vikram, and Christine Ye for helpful discussion and feedback at various stages of the project.

This research is supported in part by grants from Google and Open Philanthropy.

### References

- Akyürek et al. [2024]
Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas.
In-context language learning: Architectures and algorithms.
In *Forty-first International Conference on Machine Learning, ICML 2024*, Vienna, Austria, 2024. OpenReview.net.
URL [https://openreview.net/forum?id=3Z9CRr5srL](https://openreview.net/forum?id=3Z9CRr5srL).
- Allen-Zhu and Alfarano [2025]
Zeyuan Allen-Zhu and Alberto Alfarano.
Physics of Language Models: Part 4.1, Architecture design and the magic of Canon layers.
*SSRN*, 2025.
URL [https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5240330](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5240330).
- Allen-Zhu and Li [2024]
Zeyuan Allen-Zhu and Yuanzhi Li.
Physics of Language Models: Part 1, Learning hierarchical language structures.
*arXiv:2305.13673*, 2024.
URL [https://arxiv.org/abs/2305.13673](https://arxiv.org/abs/2305.13673).
- Arora et al. [2024a]
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré.
Zoology: Measuring and improving recall in efficient language models.
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024a.
URL [https://openreview.net/forum?id=LY3ukUANko](https://openreview.net/forum?id=LY3ukUANko).
- Arora et al. [2024b]
Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Ré.
Simple linear attention language models balance the recall-throughput tradeoff.
In *Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024*. OpenReview.net, 2024b.
URL [https://openreview.net/forum?id=e93ffDcpH3](https://openreview.net/forum?id=e93ffDcpH3).
- Ba et al. [2016]
Jimmy Ba, Geoffrey E. Hinton, Volodymyr Mnih, Joel Z. Leibo, and Catalin Ionescu.
Using fast weights to attend to the recent past.
In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, *Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain*, pages 4331–4339, 2016.
URL [https://proceedings.neurips.cc/paper/2016/hash/9f44e956e3a2b7b5598c625fcc802c36-Abstract.html](https://proceedings.neurips.cc/paper/2016/hash/9f44e956e3a2b7b5598c625fcc802c36-Abstract.html).
- Bick et al. [2025]
Aviv Bick, Eric Xing, and Albert Gu.
Understanding the skill gap in recurrent language models: The role of the gather-and-aggregate mechanism.
*arXiv:2504.18574*, 2025.
URL [https://arxiv.org/abs/2504.18574](https://arxiv.org/abs/2504.18574).
- Brinkmann et al. [2024]
Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt.
A mechanistic analysis of a transformer trained on a symbolic multi-step reasoning task.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, *Findings of the Association for Computational Linguistics: ACL 2024*, pages 4082–4102, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-acl.242.
URL [https://aclanthology.org/2024.findings-acl.242/](https://aclanthology.org/2024.findings-acl.242/).
- Danihelka et al. [2016]
Ivo Danihelka, Greg Wayne, Benigno Uria, Nal Kalchbrenner, and Alex Graves.
Associative long short-term memory.
In Maria-Florina Balcan and Kilian Q. Weinberger, editors, *Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016*, volume 48 of *JMLR Workshop and Conference Proceedings*, pages 1986–1994. JMLR.org, 2016.
URL [http://proceedings.mlr.press/v48/danihelka16.html](http://proceedings.mlr.press/v48/danihelka16.html).
- Dao and Gu [2024]
Tri Dao and Albert Gu.
Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality.
*arXiv:2405.21060*, 2024.
URL [https://arxiv.org/abs/2405.21060](https://arxiv.org/abs/2405.21060).
- Dolga et al. [2024]
Rares Dolga, Lucas Maystre, Marius Cobzarenco, and David Barber.
Latte: Latent attention for linear time transformers.
*arXiv:2402.17512*, 2024.
URL [https://arxiv.org/abs/2402.17512](https://arxiv.org/abs/2402.17512).
- Edelman et al. [2024]
Ezra Edelman, Nikolaos Tsilivis, Benjamin L. Edelman, Eran Malach, and Surbhi Goel.
The evolution of statistical induction heads: In-context learning markov chains.
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024*, Vancouver, BC, Canada, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/75b0edb869e2cd509d64d0e8ff446bc1-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2024/hash/75b0edb869e2cd509d64d0e8ff446bc1-Abstract-Conference.html).
- Elhage et al. [2021]
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah.
A mathematical framework for transformer circuits.
*Transformer Circuits Thread*, 2021.
URL [https://transformer-circuits.pub/2021/framework/index.html](https://transformer-circuits.pub/2021/framework/index.html).
- Feng and Steinhardt [2024]
Jiahai Feng and Jacob Steinhardt.
How do language models bind entities in context?
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=zb3b6oKO77](https://openreview.net/forum?id=zb3b6oKO77).
- Fu et al. [2023]
Daniel Y. Fu, Tri Dao, Khaled Kamal Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré.
Hungry hungry hippos: Towards language modeling with state space models.
In *The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023*. OpenReview.net, 2023.
URL [https://openreview.net/forum?id=COZDy0WYGg](https://openreview.net/forum?id=COZDy0WYGg).
- Geiger et al. [2021]
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts.
Causal abstractions of neural networks.
In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, *Advances in Neural Information Processing Systems*, volume 34, pages 9574–9586. Curran Associates, Inc., 2021.
URL [https://proceedings.neurips.cc/paper_files/paper/2021/file/4f5c422f4d49a5a807eda27434231040-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2021/file/4f5c422f4d49a5a807eda27434231040-Paper.pdf).
- Geiger et al. [2024]
Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard.
Causal abstraction: A theoretical foundation for mechanistic interpretability.
*arXiv:2301.04709*, 2024.
URL [https://arxiv.org/abs/2301.04709](https://arxiv.org/abs/2301.04709).
- Graves et al. [2014]
Alex Graves, Greg Wayne, and Ivo Danihelka.
Neural turing machines.
*arXiv:1410.5401*, 2014.
URL [https://arxiv.org/abs/1410.5401](https://arxiv.org/abs/1410.5401).
- Greff et al. [2020]
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber.
On the binding problem in artificial neural networks.
*arXiv:2012.05208*, 2020.
URL [https://arxiv.org/abs/2012.05208](https://arxiv.org/abs/2012.05208).
- Gu and Dao [2024]
Albert Gu and Tri Dao.
Mamba: Linear-time sequence modeling with selective state spaces.
*arXiv:2312.00752*, 2024.
URL [https://arxiv.org/abs/2312.00752](https://arxiv.org/abs/2312.00752).
- Hahn and Goyal [2023]
Michael Hahn and Navin Goyal.
A theory of emergent in-context learning as implicit structure induction.
*arXiv:2303.07971*, 2023.
URL [https://arxiv.org/abs/2303.07971](https://arxiv.org/abs/2303.07971).
- Jelassi et al. [2024]
Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach.
Repeat after me: Transformers are better than state space models at copying.
In *Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=duRRoGeoQT](https://openreview.net/forum?id=duRRoGeoQT).
- Kim and Schuster [2023]
Najoung Kim and Sebastian Schuster.
Entity tracking in language models.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 3835–3855, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.acl-long.213.
URL [https://aclanthology.org/2023.acl-long.213/](https://aclanthology.org/2023.acl-long.213/).
- Li et al. [2025a]
Belinda Z. Li, Zifan Carl Guo, and Jacob Andreas.
(How) do language models track state?
*arXiv:2503.02854*, 2025a.
URL [https://arxiv.org/abs/2503.02854](https://arxiv.org/abs/2503.02854).
- Li et al. [2025b]
Mingchen Li, Xuechen Zhang, Yixiao Huang, and Samet Oymak.
On the power of convolution-augmented transformer.
In Toby Walsh, Julie Shah, and Zico Kolter, editors, *AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4, 2025, Philadelphia, PA, USA*, pages 18393–18402. AAAI Press, 2025b.
doi: 10.1609/AAAI.V39I17.34024.
URL [https://doi.org/10.1609/aaai.v39i17.34024](https://doi.org/10.1609/aaai.v39i17.34024).
- Lieberum et al. [2023]
Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik.
Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla.
*arXiv preprint arXiv:2307.09458*, 2023.
- Liu et al. [2024a]
Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu.
Longhorn: State space models are amortized online learners.
*arXiv:2407.14207*, 2024a.
URL [https://arxiv.org/abs/2407.14207](https://arxiv.org/abs/2407.14207).
- Liu et al. [2024b]
Zicheng Liu, Siyuan Li, Li Wang, Zedong Wang, Yunfan Liu, and Stan Z. Li.
Short-long convolutions help hardware-efficient linear attention to focus on long sequences.
In *Forty-first International Conference on Machine Learning, ICML 2024*, Vienna, Austria, 2024b. OpenReview.net.
URL [https://openreview.net/forum?id=TRrXkVdhwi](https://openreview.net/forum?id=TRrXkVdhwi).
- Lubana et al. [2024]
Ekdeep Singh Lubana, Kyogo Kawaguchi, Robert P. Dick, and Hidenori Tanaka.
A percolation model of emergence: Analyzing transformers trained on a formal language.
*arXiv:2408.12578*, 2024.
URL [https://arxiv.org/abs/2408.12578](https://arxiv.org/abs/2408.12578).
- Lutati et al. [2023]
Shahar Lutati, Itamar Zimerman, and Lior Wolf.
Focus your attention (with adaptive IIR filters).
In Houda Bouamor, Juan Pino, and Kalika Bali, editors, *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 12538–12549, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.772.
URL [https://aclanthology.org/2023.emnlp-main.772/](https://aclanthology.org/2023.emnlp-main.772/).
- Nanda et al. [2023]
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt.
Progress measures for grokking via mechanistic interpretability.
In *The Eleventh International Conference on Learning Representations, ICLR 2023*, Kigali, Rwanda, 2023. OpenReview.net.
URL [https://openreview.net/forum?id=9XFSbDPmdW](https://openreview.net/forum?id=9XFSbDPmdW).
- Nowak and Cotterell [2023]
Franz Nowak and Ryan Cotterell.
A fast algorithm for computing prefix probabilities.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 57–69, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.acl-short.6.
URL [https://aclanthology.org/2023.acl-short.6/](https://aclanthology.org/2023.acl-short.6/).
- Okpekpe and Orvieto [2025]
Destiny Okpekpe and Antonio Orvieto.
Revisiting associative recall in modern recurrent models.
In *First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models*, 2025.
URL [https://openreview.net/pdf?id=CcqAd5RPk5](https://openreview.net/pdf?id=CcqAd5RPk5).
- Olsson et al. [2022]
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah.
In-context learning and induction heads.
*arXiv:2209.11895*, 2022.
URL [https://arxiv.org/abs/2209.11895](https://arxiv.org/abs/2209.11895).
- Pandey [2024]
Rohan Pandey.
gzip predicts data-dependent scaling laws.
*arXiv:2405.16684*, 2024.
URL [https://arxiv.org/abs/2405.16684](https://arxiv.org/abs/2405.16684).
- Poli et al. [2023]
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré.
Hyena hierarchy: Towards larger convolutional language models.
In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, *International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA*, volume 202 of *Proceedings of Machine Learning Research*, pages 28043–28078. PMLR, 2023.
URL [https://proceedings.mlr.press/v202/poli23a.html](https://proceedings.mlr.press/v202/poli23a.html).
- Prakash et al. [2024]
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau.
Fine-tuning enhances existing mechanisms: A case study on entity tracking.
In *The Twelfth International Conference on Learning Representations, ICLR 2024*, Vienna, Austria, 2024. OpenReview.net.
URL [https://openreview.net/forum?id=8sKcAWOf2D](https://openreview.net/forum?id=8sKcAWOf2D).
- Prakash et al. [2025]
Nikhil Prakash, Natalie Shapira, Arnab Sen Sharma, Christoph Riedl, Yonatan Belinkov, Tamar Rott Shaham, David Bau, and Atticus Geiger.
Language models use lookbacks to track beliefs.
*arXiv:2505.14685*, 2025.
URL [https://arxiv.org/abs/2505.14685](https://arxiv.org/abs/2505.14685).
- Reddy [2024]
Gautam Reddy.
The mechanistic basis of data dependence and abrupt learning in an in-context classification task.
In *The Twelfth International Conference on Learning Representations, ICLR 2024*, Vienna, Austria, 2024. OpenReview.net.
URL [https://openreview.net/forum?id=aN4Jf6Cx69](https://openreview.net/forum?id=aN4Jf6Cx69).
- Rodkin et al. [2025]
Ivan Rodkin, Yuri Kuratov, Aydar Bulatov, and Mikhail Burtsev.
Associative recurrent memory transformer.
*arXiv:2407.04841*, 2025.
URL [https://arxiv.org/abs/2407.04841](https://arxiv.org/abs/2407.04841).
- Singh et al. [2024]
Aaditya K. Singh, Ted Moskovitz, Felix Hill, Stephanie C. Y. Chan, and Andrew M. Saxe.
What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation.
In *Forty-first International Conference on Machine Learning, ICML 2024*, Vienna, Austria, 2024. OpenReview.net.
URL [https://openreview.net/forum?id=O8rrXl71D5](https://openreview.net/forum?id=O8rrXl71D5).
- Strobl et al. [2024]
Lena Strobl, William Merrill, Gail Weiss, David Chiang, and Dana Angluin.
What formal languages can transformers express? A survey.
*Transactions of the Association for Computational Linguistics*, 12:543–561, 2024.
doi: 10.1162/tacl_a_00663.
URL [https://aclanthology.org/2024.tacl-1.30/](https://aclanthology.org/2024.tacl-1.30/).
- Tigges et al. [2024]
Curt Tigges, Michael Hanna, Qinan Yu, and Stella Biderman.
LLM circuit analyses are consistent across training and scale.
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024*, Vancouver, BC, Canada, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/47c7edadfee365b394b2a3bd416048da-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2024/hash/47c7edadfee365b394b2a3bd416048da-Abstract-Conference.html).
- Trockman et al. [2024]
Asher Trockman, Hrayr Harutyunyan, J. Zico Kolter, Sanjiv Kumar, and Srinadh Bhojanapalli.
Mimetic initialization helps state space models learn to recall.
*arXiv:2410.11135*, 2024.
URL [https://arxiv.org/abs/2410.11135](https://arxiv.org/abs/2410.11135).
- Valvoda et al. [2022]
Josef Valvoda, Naomi Saphra, Jonathan Rawski, Adina Williams, and Ryan Cotterell.
Benchmarking compositionality with formal languages.
In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na, editors, *Proceedings of the 29th International Conference on Computational Linguistics*, pages 6007–6018, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.
URL [https://aclanthology.org/2022.coling-1.525/](https://aclanthology.org/2022.coling-1.525/).
- Vaswani et al. [2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, *Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017*, pages 5998–6008, Long Beach, CA, USA, 2017.
URL [https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html](https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html).
- Waleffe et al. [2024]
Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, and Bryan Catanzaro.
An empirical study of Mamba-based language models.
*arXiv:2406.07887*, 2024.
URL [https://arxiv.org/abs/2406.07887](https://arxiv.org/abs/2406.07887).
- Wang et al. [2024]
Boshi Wang, Xiang Yue, Yu Su, and Huan Sun.
Grokking of implicit reasoning in transformers: A mechanistic journey to the edge of generalization.
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024*, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/ad217e0c7fecc71bdf48660ad6714b07-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2024/hash/ad217e0c7fecc71bdf48660ad6714b07-Abstract-Conference.html).
- Wang et al. [2025]
Ke Alexander Wang, Jiaxin Shi, and Emily B. Fox.
Test-time regression: a unifying framework for designing sequence models with associative memory.
*arXiv:2501.12352*, 2025.
URL [https://arxiv.org/abs/2501.12352](https://arxiv.org/abs/2501.12352).
- Wen et al. [2024]
Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu.
RNNs are not transformers (yet): The key bottleneck on in-context retrieval.
*arXiv:2402.18510*, 2024.
URL [https://arxiv.org/abs/2402.18510](https://arxiv.org/abs/2402.18510).
- White and Cotterell [2021]
Jennifer C. White and Ryan Cotterell.
Examining the inductive bias of neural language models with artificial languages.
In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 454–463, Online, August 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.acl-long.38.
URL [https://aclanthology.org/2021.acl-long.38/](https://aclanthology.org/2021.acl-long.38/).
- Wiegreffe et al. [2025]
Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, and Ashish Sabharwal.
Answer, assemble, ace: Understanding how LMs answer multiple choice questions.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=6NNA0MxhCH](https://openreview.net/forum?id=6NNA0MxhCH).
- Wu et al. [2024]
Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christopher Manning, and Christopher Potts.
pyvene: A library for understanding and improving PyTorch models via interventions.
In Kai-Wei Chang, Annie Lee, and Nazneen Rajani, editors, *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations)*, pages 158–165, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.naacl-demo.16.
URL [https://aclanthology.org/2024.naacl-demo.16/](https://aclanthology.org/2024.naacl-demo.16/).
- Yin and Steinhardt [2025]
Kayo Yin and Jacob Steinhardt.
Which attention heads matter for in-context learning?
*arXiv:2502.14010*, 2025.
URL [https://arxiv.org/abs/2502.14010](https://arxiv.org/abs/2502.14010).
- Zhang and Zhou [2017]
Wei Zhang and Bowen Zhou.
Learning to update auto-associative memory in recurrent neural networks for improving sequence memorization.
*arXiv:1709.06493*, 2017.
URL [https://arxiv.org/abs/1709.06493](https://arxiv.org/abs/1709.06493).

## Appendix

### Appendix A Formal definitions and details for ATR

For reference, we provide formal definitions for PCFGs and the normal form we use in ATR.(^7^77We use similar formalisations of PCFGs as previous work in NLP, e.g. Nowak and Cotterell [2023].)

###### Definition A.1 .

A probabilistic context-free grammar is a tuple $\mathcal{G}=\langle\mathcal{N},\Sigma,\mathrm{S},\mathcal{R},p\rangle$ where:

- •
$\mathcal{N}$ is a finite set of non-terminal symbols;
- •
$\Sigma$ is an alphabet of terminal symbols;
- •
$\mathrm{S}\in\mathcal{N}$ is a start symbol;
- •
$\mathcal{R}\subset\mathcal{N}\times(\mathcal{N}\cup\Sigma)^{*}$ is a finite set of production rules, mapping a left-hand side symbol $\mathrm{N}\in\mathcal{N}$ to a string of symbols that may be either terminals or nonterminals; each such rule is written as $\mathrm{X}\to\bm{\alpha}$;
- •
$p:\mathcal{R}\to[0,1]$ is a weighting function which assigns a probability to each production rule for a nonterminal; this function is locally normalised, meaning $\{\sum_{\mathrm{X}\to\bm{\alpha}}{p(\mathrm{X}\to\bm{\alpha})}=1\mid\mathrm{X}
\in\mathcal{N}\}$.

###### Definition A.2 .

A PCFG $\mathcal{G}=\langle\mathcal{N},\Sigma,\mathrm{S},\mathcal{R},p\rangle$ is in Greibach normal form (GNF) if each production rule in $\mathcal{R}$ is of the form $\mathrm{X}\to a~{}\mathrm{X_{1}}~{}\ldots~{}\mathrm{X_{n}}$,
where $\mathrm{X}_{1},\ldots,\mathrm{X}_{n}\in\mathcal{N}$ and $n$ may be 0 0. Similarly, a PCFG is in right-Greibach normal form if each rule is of the form $\mathrm{X}\to\mathrm{X_{1}}~{}\ldots~{}\mathrm{X_{n}}~{}a$.

For ATR, the PCFG is in Greibach normal form if the head is the leftmost symbol of the production rule’s righthand side; similarly, if the PCFG is right-headed, it is in right-Greibach normal form.

###### Definition A.3 .

A derivation step $\bm{\alpha}\Rightarrow\bm{\beta}$ is an operation where, given strings of symbols $\bm{\alpha},\bm{\beta}\in(\mathcal{N}\cup\Sigma)^{*}$, the leftmost nonterminal $\mathrm{X}\in\mathcal{N}$ in $\bm{\alpha}$ is rewritten using the right-hand side of a production rule $\mathrm{X}\to\ldots\in\mathcal{R}$ to obtain $\bm{\beta}$.

###### Definition A.4 .

A derivation under the PCFG $\mathcal{G}$ is a sequence of strings $[\bm{\alpha}_{0},\ldots,\bm{\alpha}_{m}]$ where $\bm{\alpha}_{0}\in\mathcal{N}$ and each step $\bm{\alpha}_{i+1}$ is formed by a derivation step on $\bm{\alpha}_{i}$. The final string $\bm{\alpha}_{m}\in\Sigma^{*}$ is the yield of the derivation.

Each ATR document is the yield of a derivation sampled under the GNF PCFG $\mathcal{G}$.

###### Definition A.1 .

###### Definition A.2 .

###### Definition A.3 .

###### Definition A.4 .

#### A.1 Additional details on ATR

##### Parent terminals in GNF.

We set the left/right-most terminal in each production rule (which leads to the GNF property) the parent of all other generated terminals. This terminal is sampled specially: for each nonterminal, we independently sample a distribution over terminals from a uniform Dirichlet, and for all production rules with that nonterminal on the lefthand side we use that distribution to sample the parent terminal. This simulates how heads of phrases in natural language (analogous to our parent terminals) decide the type of the phrase they head (analogous to our nonterminals).

##### Maximum depth.

To enforce maximum depth, we first assign a uniformly random depth score $d:\mathcal{N}\to\mathbb{N}\in\{1,\ldots,\texttt{max_depth}\}$ to each nonterminal in the vocabulary. Then, for each production rule for each nonterminal $\mathrm{X}$, we only allow nonterminals $\mathrm{Y}$ with $d(\mathrm{Y})>d(\mathrm{X})$ on the right-hand side. Note that this means no recursion is possible.

### Appendix B Model configurations

**Table 2: Default model configurations across all architectures. In experiments, we sweep learning rate and embedding dimension, reporting results from the instance with highest accuracy.**
| Parameter | Values |
| --- | --- |
| dropout | $0.0$ |
| num_heads | $1$ |

### Appendix C Task hyperparameters

**Table 3: Task hyperparameters used for constructing key–value sets for AR and PCFGs for ATR. For a description of each parameter, see Table [1](https://arxiv.org/html/2505.15105v2#S3.T1).**
| Parameter | Values |
| --- | --- |
| $L_{\textrm{max}}$ | $32$ |
| $L_{\textrm{min}}$ | $32$ |
| $\lvert\Sigma\rvert$ | $\{8192\}$ |

### Appendix D Additional experiments on AR and ATR

Many parameters of synthetic tasks like AR and ATR and the model architectures we tested have interesting effects on behavioural and mechanistic metrics, but not all experiments could fit in our main text. Therefore, we include additional interesting observations in this appendix.

#### D.1 Attention needs position embeddings

Figure: Figure 7: Position embedding: Model accuracy on AR and two ATR settings with and without absolute position embeddings.
Refer to caption: x14.png

Due to an initial configuration mistake, we accidentally trained all architectures with absolute position embeddings; in the zoology codebase [Arora et al., 2024a], only Attention is meant to be trained in this way. Fortuitously, this resulted in an interesting ablation: do SSMs, which are usually trained without it, also benefit from position embeddings?

##### Behavioural results.

Our results in [Figure 7](https://arxiv.org/html/2505.15105v2#A4.F7) resoundingly show no: SSMs generally perform worse with position embeddings (PE). Attention is highly dependent on PE; performance on AR drops from $100.00\%$ to $5.62\%$ at $d=256$ with NoPE. Attention lacks recurrence, unlike SSMs, so this is not surprising. However, on ATR, at smaller dimensionalities NoPE actually outperforms PE Attention. Further ablations ought to consider alternative PE methods such as RoPE and Alibi.

#### D.2 1-layer SSMs learn direct retrieval on AR and ATR

Figure: (a) Accuracy of 1-layer vs. 2-layer models on AR, 32 key–value pairs. 1-layer induction models fail.
Refer to caption: x15.png

Throughout our experiments on AR and ATR, we have claimed that SSMs (except for Based and possibly BaseConv) learn a direct retrieval mechanism which does not require an intermediate step like attention, i.e. only a single SSM layer is needed to learn AR and ATR. To verify this, we repeat AR and $L=5,\lvert\Sigma\rvert=20$ ATR experiments (without train–test split) with 1-layer models.

##### Behavioural results.

We find comparable performance for direct retrieval models between 1-layer and 2-layer settings on AR ([Figure 8(a)](https://arxiv.org/html/2505.15105v2#A4.F8.sf1)). In fact, at $d=256$, 1-layer Mamba ($96.25\%$) outperforms 2-layer Mamba ($91.25\%$), as does Hyena ($31.56\%$ vs. $29.69\%$). 1-layer Based and BaseConv are architecturally identical, so we only report one; that architecture and Attention, both relying on induction in the 2-layer case, fail to learn AR with one layer. On ATR ([Figure 8(b)](https://arxiv.org/html/2505.15105v2#A4.F8.sf2)), we see a more noticeable difference with layer count on all architectures, but again Attention, Based, and BaseConv become the worst architectures with one layer (e.g. $96.25\%\to 74.69\%$ for Attention at $d=256$).

#### D.3 SSMs prefer layer 0 to perform AR

Figure: (a) Accuracy with 1–3 layers on AR.
Refer to caption: x17.png

Since we have confirmed that the direct retrieval mechanism in SSMs requires only a single layer, we are curious which layer this mechanism forms in if more than two layers are present. We train models with up to three layers on AR and report results.

##### Behavioural results.

3-layer models perform about the same on AR as 2-layer models across architectures ([Figure 9(a)](https://arxiv.org/html/2505.15105v2#A4.F9.sf1)), except for a large drop in performance for Mamba when $d=32$; this may just be an optimisation failure.

##### Mechanistic analysis.

For our mechanistic metric, instead of intervening on each block, we intervene at the sequence mixer’s output to the query token in each layer; this tells us if that layer is directly responsible for writing the answer to the output position. We apply the same filter as in [section 5.1](https://arxiv.org/html/2505.15105v2#S5.SS1), with a threshold of $0.01$. [Figure 9(b)](https://arxiv.org/html/2505.15105v2#A4.F9.sf2) shows that among performant models, Hyena and Mamba prefer layer 0 for performing AR no matter the layer count; however, some Mamba checkpoints learn the mechanism in the final layer as well (but never layer 1 in a 3-layer model). Attention, Based, and BaseConv prefer layer 1, which is expected since this is the second step of the induction mechanism. However, some checkpoints of Attention and Based also have non-zero attribution score at layer 2 in the 3-layer setting.

#### D.4 Rightmost sibling queries are trivial for all architectures

Figure: Figure 10: Sibling queries: Accuracy across models with 1–3 layers on ATR ($L_{\textrm{max}}=5,\lvert\Sigma\rvert=20$)
Refer to caption: x19.png

Since ATR has hierarchical structure, we attempted an initial experiment with multihop queries; specifically, we present queries where the answer is that terminal’s rightmost sibling terminal. Models are only trained on this type of query, not standard parent queries as reported in the main text. We train with the same settings in [section 3.3](https://arxiv.org/html/2505.15105v2#S3.SS3).

##### Behavioural results.

In [Figure 10](https://arxiv.org/html/2505.15105v2#A4.F10) we show that all models (except Based and BasedConv with $1$ layer, where they only have local convolutions) achieve greater than $80\%$ accuracy at the task at all dimensionalities. We see slight improvement from 1-layer to 2-layer models but at this point performance is saturated and 3-layer does not help. Clearly, this task is extremely simple for all models, even more so than parent queries, and thus does not provide useful signal for comparing architectures.

##### Why are sibling queries easy?

Parent nodes are guaranteed to be special terminals in our GNF which are sampled from a nonterminal-dependent distribution (see [appendix A](https://arxiv.org/html/2505.15105v2#A1)). However, siblings have a large chance of being fixed terminals specified by the production rule. Additionally, the rightmost sibling of a particular terminal may be itself, if it is the rightmost terminal of its production rule. We speculate that these factors combined make sibling queries easier than parent queries, and thus not a suitable testbed for multihop reasoning.

##### Future work.

The appropriate analogue to study multihop reasoning in ATR is grandparent relations (or higher up ancestors in the tree), since the grandparent is always a special head terminal (like the parent) and is always to the right of the parent and thus different from the query terminal. We leave further experiments on this to future work.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - Please correct me if I misunderstood, but ATR induces unequal pair frequencies, which may induce a generative prior, allowing the corrupted-key accuracy to stay above 0 in most cases. 
- Replacing Based's short convolution kernel with an implicit long convolution kernel significantly harms AR. Why might this be? Providing heatmaps for the long convolution kernel case to show where association fails could help better understand this phenomenon. Also a more detailed discussion about this phenomenon would strengthen this manuscript.
- The authors note that Hyena includes a short convolution kernel but it performs poorly on AR. This signals that a convolution kernel is not sufficient to solve this task of AR. Which downstream component could be preventing the convolution kernel from implementing the association step?
soundness: 3
strengths: - The intervention protocol pinpoints the mechanism - more specifically, the QKV positioned restorations at layer input/outputs disambiguate induction from direct retrieval.
- The kernel size ablation study demonstrates that short receptive-field convolution kernels implement the association needed for AR. 
- With the new task (ATR), the paper verifies that the mechanism transfer to a harder task without positional dependence.
summary: This paper introduces mechanistic evaluations (causal interchange interventions) to analyze the impact of different architectural components in solving Associative Recall (AR) task, along with a new retrieval task, called Associative Treecall (ATR). The authors show that on AR, Attention and Based architectures exhibit induction while Mamba and DeltaNet perform direct retrieval. The main ablation revolves around the size of the convolution kernels, where the ablations show that short convolution kernels are critical for AR on Mamba and Based architectures. With a detailed analysis on the two tasks, the authors claim that comparable accuracies can hide different internal mechanisms.
weaknesses: - The experiment results seem quite noisy. For figure 4, in particular, it is unclear why model dim 128 and Mamba Conv. = 2 suddenly fails. Also, it is unclear why Restored @ Key value for this configuration is notably high compared to other entries. A similar trend is observed for Figure 5, where model configurations that perform well can drastically fail, (~0% accuracy) with learning rates that are slightly modified. This result, unless it can be justified empirically or theoretically, raises a serious concern whether the results are valid conclusions or are due to insufficient sweep in the hyperparameter space.
- Potential confounds with the parameter count on the models. The models have different parameter counts and FLOPs, and hence presenting the results in one of these dimensions would be much more valuable.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. can you provide an example of how mechanic metric helps the design of architecture?
soundness: 3
strengths: 1. comprehensively and mechanically evaluate common linear models and transformer.
2. clear and good writing.
summary: The paper introduces a generalized Associative Recall task named Associative Treecall and provide mechanic interpretability for the success of Mamba.
weaknesses: 1. mechanic metric are not used to provide guidance for the design of architecture but can only help understanding. thus its use is limited.

---

confidence: 5
contribution: 2
initial_rating: 2
presentation: 2
questions: See weaknesses.
soundness: 2
strengths: Studying basic performance tasks on transformers vs new sequence models is interesting. The authors train a large set of models and carry out the analysis also on ATR, which is a less common setup that I did not know before, but I find quite insightful. 
The paper is also pleasant to read and schematic, which helps deliver the message. Plots are clear.
summary: The paper at hand presents a study of associative recall (AR) and associative tree recall (ATR) on various sequence models: from simple convolutions to Mamba and Transformers. The authors claim these models use different methods to solve AR, which they study by intervening on input sequences.
weaknesses: The findings in the paper have quite a few overlaps with previous works:

- Zoology: https://arxiv.org/abs/2312.04927
- Based: https://arxiv.org/abs/2402.18668
- Convolution-augmented transformers: https://arxiv.org/abs/2407.05591
- Revisiting Associative Recall: https://openreview.net/pdf/f7e9f322ba15e88dcc818ab70866648650a5e319.pdf
- H3 : https://arxiv.org/pdf/2212.14052

In light of the findings in the papers above, I did not find the paper very surprising. The authors cite all the papers above, but do not discuss or compare their results to previous literature:

1) performance on AR is reported in (1) "Zoology", and (2) with a much finer LR grid in "Revisiting Associative Recall". In the latter, the authors claim that LR sensitivity is a big issue in Mamba and Hyena. The authors do not discuss this issue nor seem to take any action to perform careful evaluations. Additionally, betas = (0.9, 0.999) in Adam is the default, but it is not what people typically use in language models. beta2 is too high (0.95 is default in many repos). How do you make sure your results depict what "each model can achieve"?

2) It is a bit hard to follow what the authors mean by "induction". I think, despite 1k years of philosophical debates, the definition can be a bit arbitrary. I was confused while reading your claims. Please define what you mean! I had an approximate idea at the end of the paper, but this is not formal enough.

3) The reader is not prompted to read the figures correctly, and the tasks are not well defined. Let us consider Figure 2: if you change A to B, the eval always returns A, and 2 returns "???". This is very unclear to me. I do not understand the ground truth and I find little explanations in the text. Furthermore, Figure 3 has a similar issue: you never formally define any of the tasks; what is "Restored @ Key"? You discuss how these tasks resemble interventions, but I cannot determine their respective importance. What are they individually supposed to test?

4) The role of convolution has been studied thoroughly, in "power of convolution-augmented transformers" but also in the H3 paper. In H3, they place a shift + gate exactly to enhance recall (https://arxiv.org/pdf/2212.14052, Fig 1). Again, I do not find surprising the claim about convolution, given also the induction head standard mechanism where the first transformer layer indeed represents a shift (e.g. proof in the Jelassi paper on the copy task, and Figure 1 above).

All in all, I do not see the level of novelty here to be at the level of acceptance. I ask the authors to please specify which new insights are presented in the paper, and to clarify what their causal interventions are precisely testing.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Sec 4.4 Convolution moves information to the next key (line 317-322): This claim arises from the results in Fig.5, where restoring next key at layer 0 gives the best result. But why does the restoration necessarily imply the convolution component moves information to the next key? If my understanding of restoration is correct (which is not very precisely defined in paper, but loosely on Fig.2), restoration means we can arbitrary set any token at the next-key position. Then one easy solution is to set the next-key to be the corrupted key, which does not rely on convolution moving information to the next key. In addition, if the next key is arbitrary far away from the previous key, a short-convolution by definition cannot move information to the next key. Can the authors clarify?

 2. I appreciate the interesting use of causal intervention to mechanistically evaluate the AR mechanisms, but the findings seem to mostly corroborate the existing known results (e.g., short convolution are key to association in SSMs as mentioned by the authors in related work), including some provable solution mechanisms in missing related works [1] [2]. Does the mechanistic study offer novel insights (e.g., identify unknown mechanisms, provide more fine-grained analysis on how  the AR mechanisms interact with the choice of architecture and optimization set-up)?

 References:

 [1] Bietti et al. "Birth of a transformer: A memory viewpoint." NeurIPS 2023.

 [2] Huang et al. "Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacity." ICML 2025.
soundness: 2
strengths: 1. It is an interesting idea to use causal intervention as a mechanistic understanding tool to compare SSMs with transformers

2. The paper is clearly organized, with illustrative figures supporting the main claims.
summary: The paper performs a mechanistic study on how SSMs and Transformers solve associative recall (AR) and associative treecall (ATR) tasks. The authors use causal interventions as the main tool. They find that for AR, Transformers and Based use induction mechanism, whereas SSMs such as Mamba rely on the convolution component --- without which Mamba fails to learn AR. They also find that similar mechanisms hold for ATR, except that Mamba can also use the Attention-like two-layer induction mechanisms in absence of the convolution component.
weaknesses: 1. Some claims are not well supported. For example, in line 75-76, the authors claim that ``(Mamba) fails to learn AR at all without this (short convolution component)'', but the experiments (Fig.4) only investigate the Mamba convolution kernel size no greater than $4$, unclear whether Mamba fails with long(er) convolution kernel. 

2. Lack of architecture details and analysis. Although the authors provide source code, the paper lacks concrete definitions of the architectures used for study, and thus provides little explanation why certain mechanisms present in one architecture but not the others.