Design Principles for Sequence Models via Coefficient Dynamics

## Abstract

Abstract Deep sequence models, ranging from Transformers and State Space Models (SSMs) to more recent approaches such as gated linear RNNs, fundamentally compute outputs as linear combinations of past value vectors. To draw insights and systematically compare such architectures, we develop a unified framework that makes this output operation explicit, by casting the linear combination coefficients as the outputs of autonomous linear dynamical systems driven by impulse inputs . This viewpoint, in spirit substantially different from approaches focusing on connecting linear RNNs with linear attention, reveals a common mathematical theme across diverse architectures and crucially captures softmax attention, on top of RNNs, SSMs, and related models. In contrast to new model proposals that are commonly evaluated on benchmarks, we derive design principles linking architectural choices to model properties. Thereby identifying tradeoffs between expressivity and efficient implementation, geometric constraints on input selectivity, and stability conditions for numerically stable training and information retention. By connecting several insights and observations from recent literature, the framework both explains empirical successes of recent designs and provides guiding principles for systematically designing new sequence model architectures.

## 1 Introduction

Sequence modeling lies at the core of modern machine learning, powering advances in natural language processing, computer vision, speech recognition, and robotics. At the heart of this field is a deceptively simple question [Elman, 1990]: how should a model combine past information to produce meaningful representations and make predictions about the future? Classical approaches such as recurrent neural networks (RNNs; Hochreiter and Schmidhuber [1997]) framed this as a recursive state-update problem, whereas the Transformer [Vaswani et al., 2017] introduced attention mechanisms that compute adaptive linear combinations over past tokens. More recently, state space models (SSMs) [Gu et al., 2022a, Gu and Dao, 2023] or equivalently linear attention models [Katharopoulos et al., 2020, Dao and Gu, 2024, Schlag et al., 2021] have further broadened the design space by importing ideas from dynamical systems, delivering foundation models with linear sequence length complexity.
Despite their differences in formulation, computational complexity, and performance on crucial benchmark tasks [Arora et al., 2023, Jelassi et al., 2024], both linear and softmax attention models share a common thread: their core sequence mixing mechanism computes a linear combination of value vectors with coefficients determined by the interaction of queries, keys, and additional learned parameters. For attention, these coefficients are derived through similarity functions and normalizations; for SSMs and (gated) linear attention, they emerge from the evolution of hidden states under linear dynamics. Yet, while this unifying perspective is informally recognized, most architectural innovations are still introduced and validated primarily through benchmarks. This benchmark-driven approach has fueled rapid empirical progress, but complicates the disentanglement of which design elements – e.g. normalization, gating, or state-update dynamics – truly account for observed improvements. What is lacking is a set of general design principles that explain why specific models succeed and guide the systematic development of new ones, beyond approaches focusing on ad-hoc analyses of single-task setups [Jelassi et al., 2024, Merrill et al., 2024, Arora et al., 2024].

In this work, we develop a principled framework for sequence model design grounded in dynamical systems theory. Specifically, we show that the coefficients governing linear combinations of value vectors, can be expressed as the output of autonomous linear dynamical systems subject to impulse inputs. Our approach reveals the common mathematical structure underlying diverse architectures and provides a principled foundation for designing new sequence models. Since the present study is restricted to single-layer models, multi-layer composition can produce richer behaviors – such as formation of induction heads [Olsson et al., 2022] – beyond what our base framework captures. Nevertheless, by laying out the core design principles at the layer level, we aim to establish the foundations for understanding and designing more complex systems. Our *contributions* are as follows:

- •
Design principles for sequence models that stem from a unifying theory on the dynamic computations in sequence models, clarifying the role of sequence model components.
- •
Analysis of tradeoffs and constraints concerning expressivity, efficient implementation, input selectivity, and stability requirements in sequence models.
- •
A unifying formalization of sequence models that captures most recent architectures as special cases (Table [1](https://arxiv.org/html/2510.09389v1#S3.T1)), enabling systematic comparison without reliance on experimental benchmarks.
- •
Empirical validation demonstrates that our theory translates into practice.

## 2 Related Work

We have recently witnessed a revived interest in efficient RNNs and linear attention mechanisms in language modeling as well as image/video processing [Liu et al., 2024] and DNA modeling [Nguyen et al., 2023, Schiff et al., 2024]. Stemming from the seminal S4 work by Gu et al. [2020, 2022a], early SSMs [Gu et al., 2022b, Smith et al., 2023] and linear recurrent networks [Orvieto et al., 2023], outperformed attention on pattern-recognition and long-range reasoning benchmarks [Tay et al., 2021] and were rapidly adapted to the language domain [Wang et al., 2022, Fu et al., 2023]. RetNet [Sun et al., 2023] and GateLoop [Katsch, 2023] paved the way for Mamba [Gu and Dao, 2023], which combined input selectivity with efficient training (linear in sequence length) to achieve improved performance in language modeling. Drawing inspiration from the known connection between RNNs and linear attention [Katharopoulos et al., 2020, Schlag et al., 2021], advancements such as GLA, Deltanet, xLSTM [Yang et al., 2023, 2024, Beck et al., 2024], and Mamba2 [Dao and Gu, 2024] have aided the unification of modern efficient sequence modeling approaches in terms of latent efficient matrix multiplications [Yang and Zhang, 2024]. On top of this unified framework – which crucially does not contain softmax attention – several improvements were recently proposed, such as Gated Deltanet [Yang et al., 2025a], DeltaProduct [Siems et al., 2025], and Log-Linear Attention [Guo et al., 2025]. We have also witnessed cross-fertilization between new linear RNN/attention ideas (forget gate, positional encodings) and the softmax world, with examples such as FoX [Lin et al., 2025] and Path [Yang et al., 2025b].

As flagship models are starting to adopt hybrid solutions based on mixing linear and softmax attention to enhance training and inference efficiency [AI21 Labs, 2024, Nano, 2025, Qwen Team, 2025], additional work is needed to further understand which architectural properties impact performance depending, on the specific nature (e.g. recall, associativity, memorization, compression) of the task at hand [Poli et al., 2024]. While early studies show promise across benchmarks, especially for hybrid models at high pretraining budgets [Waleffe et al., 2024], others point to fundamental limitations of fixed-memory processing in recall, copy [Arora et al., 2023, Jelassi et al., 2024] and retrieval tasks [Guo et al., 2025]. Expressivity of new linear RNN layers is well-studied [Orvieto et al., 2024, Cirone et al., 2024, Merrill et al., 2024], yet less is known about how architectural details impact model capabilities – especially as linear RNNs can be substantially more challenging to optimize [Zucchet and Orvieto, 2024, Okpekpe and Orvieto, 2025], – and we lack a unified framework able to capture both mechanisms under the same formalism, without resorting to infinite-dimensional expansions [Sieber et al., 2024]. Our coefficient dynamics are inspired by this issue.

## 3 Linear Combinations with Temporal Dynamics

We consider a one-layer causal sequence model (e.g. attention, S6), i.e. a parametric mechanism that defines $\{x_{j}\}_{j=1}^{i}\mapsto y_{i}$, where $i$ denotes the (time) position index $i=1,\dots,L$ and $x_{i}\in\mathbb{R}^{d},y_{i}\in\mathbb{R}^{d_{v}}$ are the input and output respectively. In the following, we provide a formulation that encompasses most existing sequence model architectures using what we name coefficient dynamics. Following the standard transformer notation [Vaswani et al., 2017], and given the duality between attention, SSMs, and other recurrent architectures [Dao and Gu, 2024, Sieber et al., 2024], we define:

$$ $q_{i}=W_{Q}x_{i},\quad k_{i}=W_{K}x_{i},\quad v_{i}=W_{V}x_{i},$ (1) $$

where $W_{Q}\in\mathbb{R}^{n\times d},\ W_{K}\in\mathbb{R}^{n\times d},\ \text{and}\ W_{V}\in\mathbb{R}^{d_{v}\times d}$ are the learned parameters of the model.

Let $\{v_{j}\}_{j=1}^{i}$ denote the value vectors at (time) position $i$,
then most existing sequence model can be fundamentally viewed as computing linear combinations of these value vectors:(^1^11For simplicity, we ignore potential output projections from the linear combination to the output.)

$$ $y_{i}=\sum_{j=1}^{i}\frac{\alpha_{i,j}}{\eta_{i}}v_{j},$ (2) $$

where each coefficient $\alpha_{i,j}$ represents the contribution of the value vector at position $j$ to the output at position $i$ and $\eta_{i}:\mathbb{R}\rightarrow\mathbb{R}$ denotes the normalization factor. In standard architectural proposals, such as softmax attention or linear attention, coefficients $\alpha_{i,j}$ are typically computed using queries, keys, and additional learned parameters. For consistency with existing literature, we refer to $\alpha_{i,j}$ as the *attention coefficients*.

In our framework, we interpret the coefficients $\alpha_{i,j}$ as the output measurement of a latent linear dynamical system with impulse input, which we call the *coefficient dynamics*. In contrast to previous works, which study how tokens are mixed as the iteration counts progress [Ali et al., 2024], we consider the tokens individually. This orthogonal perspective allows to precisely model both softmax and linear attention, with no need for infinite dimensional approximations. Specifically, for each key position $j$, we define an autonomous dynamical system with initial state $h_{i-1,j}=0,\forall i\leq j$, i.e.,

[Uncaptioned image]: x1.png

where in standard dynamical systems’ terminology, $h_{i,j}\in\mathbb{R}^{n}$, $u_{i}\in\mathbb{R}^{n}$, and $\alpha_{i,j}\in\mathbb{R}$ are the state, input, and output of the system at time $i$ with an impulse input at time $j$, respectively. The readout map is given by $g_{i}(x)\coloneqq\phi\left(q_{i}^{\top}x\right)$, where $q_{i}\in\mathbb{R}^{n}$ is the query, $k_{j}\in\mathbb{R}^{n}$ is the key, and $\phi(\cdot):\mathbb{R}\to\mathbb{R}$ is a general nonlinear map. $A_{i}\in\mathbb{R}^{n\times n}$ and $b_{i}\in\mathbb{R}$ are the evolution matrix and the input scaling, respectively.
For $A_{i}$ we focus on scalar matrices, $A_{i}=\lambda_{i}\mathbb{I}_{n}$, diagonal matrices, $A_{i}=\textrm{diag}(\lambda_{i})$, and Householder matrices.(^2^22A Householder matrix is an orthogonal matrix of the form $A_{i}=I-2z_{i}z_{i}^{\top}/\|z_{i}\|^{2}$, with $z_{i}$ a nonzero vector; and represents a reflection through the hyperplane orthogonal to $v_{i}$.) However, generally $A_{i}$ can have no particular structure. We consider the *autonomous* evolution of dynamical system ([3](https://arxiv.org/html/2510.09389v1#S3.E3)), i.e., given a zero initial condition, we study the system’s evolution when it receives an impulse input at time $j$ and then evolves without further input.

###### Remark 1 .

This formulation differs from existing state space models (e.g., Mamba) in two key aspects: (i) the system receives input only at a single time step $i=j$, and (ii) the hidden state is zero immediately before the impulse, not at global time $i=-1$.

It is possible to recover an analytic expression for the coefficients $\alpha_{i,j}$ in ([2](https://arxiv.org/html/2510.09389v1#S3.E2)) by writing the convolution expression associated with ([3](https://arxiv.org/html/2510.09389v1#S3.E3)). In particular, the state (or “transformed key”) evolves, for $j\leq i$, as $h_{i,j}=\left(\prod_{t=j+1}^{i}A_{t}\right)b_{j}k_{j}$. Hence, we obtain the form:

$$ $\alpha_{i,j}=\phi(x_{i}^{\top}T_{q}^{\top}T_{h,i}x_{j}),\qquad\text{with}\quad T_{h,i}:=\big(\textstyle\prod_{t=j+1}^{i}A_{t}\big)b_{j}W_{K},\quad T_{q}:=W_{Q}.$ (4) $$

This interpretation allows for the identification of four classes of parameters with distinct roles:

- •
Readout map ($\phi(\cdot)$): Applies pointwise (often nonlinear) transformations to dot product $q_{i}^{\top}h_{i,j}$.
- •
Evolution matrices ($A_{t}$): Control the temporal evolution of keys through scaling (diagonal matrices) or rotation (orthogonal matrices). Importantly, the evolution matrices define a sequence of transformation on the keys between input time $j$ and readout time $i$.
- •
Scaling parameters ($b_{j}$): Scale individual keys at their input time $j$.
- •
Normalization factors ($\eta_{i}$): Normalize the $\alpha_{i,j}$ coefficients, before the linear combination.

This framework provides a unified lens for understanding existing architectural proposals. We show in Table [1](https://arxiv.org/html/2510.09389v1#S3.T1) that popular architectures can be recovered as special cases of this framework.

**Table 1: Coefficient dynamics of popular architectures; the derivations are provided in Appendix [B](https://arxiv.org/html/2510.09389v1#A2).**
| Architecture | Parameters | Coefficients |  |  |  |
| --- | --- | --- | --- | --- | --- |
| $A_{t}$ | $b_{j}$ | $\phi(\cdot)$ | $\eta_{i}$ | $\alpha_{i,j}$ |  |
| Softmax Attn.<br>[Vaswani et al., 2017] | $\mathbb{I}_{n}$ | $\frac{1}{\sqrt{n}}$ | $\exp(\cdot)$ | $\sum_{j=0}^{i}\alpha_{i,j}$ | $\frac{\exp(\nicefrac{{q_{i}^{\top}k_{j}}}{{\sqrt{n}}})}{{\sum_{j=1}^{i}\exp(\nicefrac{{q_{i}^{\top}k_{j}}}{{\sqrt{n}}})}}$ |
| Linear Attn.<br>[Katharopoulos et al., 2020] | $\mathbb{I}_{n}$ | $\frac{1}{\sqrt{n}}$ | $\psi(\cdot)\psi(\cdot)$ | $\sum_{j=0}^{i}\alpha_{i,j}$ | $\frac{\psi(q_{i})^{\top}\psi(\nicefrac{{k_{j}}}{{\sqrt{n}}})}{{\psi(q_{i})^{\top}\sum_{j=1}^{i}\psi(\nicefrac{{k_{j}}}{{\sqrt{n}}})}}$ |
| Normalized Attn.<br>[Sieber et al., 2024] | $\mathbb{I}_{n}$ | $\frac{1}{\sqrt{n}}$ | $\textrm{Id}(\cdot)$ | $\eta_{i}$ | $\frac{q_{i}^{\top}k_{j}}{\eta_{i}\sqrt{n}}$ |
| GLA<br>[Yang et al., 2023] | $\textrm{diag}(\alpha_{i})$ | $\frac{1}{\sqrt{n}}$ | $\textrm{Id}(\cdot)$ | $1$ | $q_{i}^{\top}\!\left(\prod_{t=j+1}^{i}\textrm{diag}(\alpha_{t})\right)\frac{k_{j}}{\sqrt{n}}$ |
| Mamba-2<br>[Dao and Gu, 2024] | $e^{-\Delta_{t}A}$ | $\Delta_{j}$ | $\text{Id}(\cdot)$ | $1$ | $q_{i}^{\top}\!\left(\prod_{t=j+1}^{i}e^{(-\Delta_{t}A)}\right)\Delta_{j}k_{j}$ |
| DeltaNet<br>[Schlag et al., 2021] | $\mathbb{I}\!-\!\beta_{t}k_{t}k_{t}^{\top}$ | $\frac{\beta_{j}}{\sqrt{n}}$ | $\textrm{Id}(\cdot)$ | $1$ | $q_{i}^{\top}\!\left(\prod_{t=j+1}^{i}\mathbb{I}\!-\!\beta_{t}k_{t}k_{t}^{\top}\right)\!\frac{\beta_{j}k_{j}}{\sqrt{n}}$ |
| Gated DeltaNet<br>[Yang et al., 2025a] | $\alpha_{t}(\mathbb{I}\!-\!\beta_{t}k_{t}k_{t}^{\top})$ | $\frac{\beta_{j}}{\sqrt{n}}$ | $\textrm{Id}(\cdot)$ | $1$ | $q_{i}^{\top}\!\left(\prod_{t=j+1}^{i}\alpha_{t}(\mathbb{I}\!-\!\beta_{t}k_{t}k_{t}^{\top})\right)\!\frac{\beta_{j}k_{j}}{\sqrt{n}}$ |
| mLSTM<br>[Beck et al., 2024] | $e^{f_{t}}$ | $\frac{e^{i_{j}}}{\sqrt{n}}$ | $\textrm{Id}(\cdot)$ | $\frac{\eta_{i}}{o_{i}}$ | $\frac{o_{i}}{\eta_{i}}q_{i}^{\top}\!\left(\prod_{t=j+1}^{i}e^{f_{t}}\right)\frac{e^{i_{j}}}{\sqrt{n}}k_{j}$ |

###### Remark 1 .

## 4 Design Principles for Sequence Models

In the following, we provide principles that can inform the architecture design for sequence models. These principles are grounded in theoretical results for the coefficient dynamics (as described in ([2](https://arxiv.org/html/2510.09389v1#S3.E2)) & ([3](https://arxiv.org/html/2510.09389v1#S3.E3))) and followed by a discussion of their practical applicability. Formal proofs for the lemmas and corollaries are provided in Appendices [C.1](https://arxiv.org/html/2510.09389v1#A3.SS1) and  [C.2](https://arxiv.org/html/2510.09389v1#A3.SS2), respectively.

### 4.1 On choosing the Readout Map ϕ ​ ( ⋅ ) \phi(\cdot)

A fundamental issue with softmax attention is its quadratic complexity in sequence length, directly linked to its nonlinear readout map $\phi(\cdot)=\exp(\cdot)$. This fact is captured by the following principle.
Principle 1.
On modern hardware, a sequence model can only be efficiently computed using a recurrent formulation (e.g. parallel scan) if the readout function $\phi(\cdot)$ is linear.

###### Lemma 1 .

A recurrent formulation of ([3](https://arxiv.org/html/2510.09389v1#S3.E3)) with finite memory (state) in $\mathbb{R}^{n\times d_{v}}$, which allows simultaneous computation of $\alpha_{i,j}$, exists if and only if $\phi(\cdot):\mathbb{R}\to\mathbb{R}$ is a linear map.

###### Proof.

If $\phi(\cdot)$ is linear, its action distributes over the summation in ([2](https://arxiv.org/html/2510.09389v1#S3.E2)), allowing all past contributions to be aggregated into a fixed-dimensional recurrent state.
Conversely, if $\phi(\cdot)$ is nonlinear, the recurrence requires storing unboundedly many nonlinear combinations of past inputs, which cannot be represented by any finite-dimensional state. See Appendix [C.1](https://arxiv.org/html/2510.09389v1#A3.SS1) for a formal proof.
∎

Various linear attention (e.g., Katharopoulos et al. [2020], Choromanski et al. [2021]) and SSM/RNN (e.g., Dao and Gu [2024], Beck et al. [2024], De et al. [2024]) proposals implicitly utilize this principle to design sequence models with linear complexity in sequence length. However, choosing a linear readout map $\phi$ has several geometric implications, as discussed below, which directly affect performance on crucial recall and retrieval tasks [Arora et al., 2024, Guo et al., 2025].

Beyond implementation considerations, in domains such as language processing, an important feature of sequence models is their ability to selectively attend to different tokens in the input sequence. This is often referred to as input selectivity. Mathematically, this translates to the model’s capacity to set coefficients $\alpha_{i,j}$ equal to zero (or near-zero in practice) for uninformative tokens and to high values for informative tokens. Crucially, how informative a token is depends on the query, as is clear for tasks such as associative recall [Arora et al., 2023].

Principle 2.
The capability of a model to robustly set coefficients to zero (or near-zero) depends on the geometry of the zero-level set of the readout map $\phi(\cdot)$. Intuitively, if the zero (or near-zero) set has large measure in $\mathbb{R}$, values can be suppressed more robustly. Conversely, if the zero set is thin, suppression is fragile to learn.

###### Lemma 2 .

Let $\phi:\mathbb{R}\to\mathbb{R}$ be the readout map in ([3](https://arxiv.org/html/2510.09389v1#S3.E3)) with connected zero-level set $\mathcal{Z}=\{z\in\mathbb{R}\mid\phi(z)=0\}$, and let $z=q_{i}^{\top}h_{i,j}$, with $T_{q}$, $T_{h,i}$ defined in ([4](https://arxiv.org/html/2510.09389v1#S3.E4)). Let $\mathcal{T}$ be the set of linear transformations that achieve the zero-level set, i.e., $\mathcal{T}=\{(T_{q},T_{h,i})\mid q_{i}^{\top}h_{i,j}\in\mathcal{Z}\ s.t.\ q_{i}=T_{q}x_{i}$, $h_{i,j}=T_{h,i}x_{j}\}$ for any pair of inputs $x_{i},x_{j}\in\mathbb{R}^{d}$. Then, the measure of $\mathcal{T}$ is directly proportional to the measure of the zero-level set $\mathcal{Z}$, i.e., $\lvert\mathcal{T}\rvert=c\lvert\mathcal{Z}\rvert$ with $c>0$.

###### Proof.

Given that $\mathcal{Z}$ is an interval, one can construct a set of linear combinations $\mathcal{T}_{z}=\{(T_{q},T_{h,i})\mid q_{i}^{\top}h_{i,j}=z\}$ that achieves $z$ for each $z\in\mathcal{Z}$. A larger set $\mathcal{Z}$ then implies that $\mathcal{T}$ is constructed from more subsets $\mathcal{T}_{z}$. The complete proof is provided in Appendix [C.1](https://arxiv.org/html/2510.09389v1#A3.SS1).
∎

Although Lemma [2](https://arxiv.org/html/2510.09389v1#Thmlemma2) only considers zero-level sets, large near-zero level sets $\{x\mid-\epsilon\leq\phi(x)\leq\epsilon\}$ with small $\epsilon>0$, provide effective input selectivity in practice. Intuitively, the proof follows from the fact that for an interval $\mathcal{Z}$, we can construct a set of linear combinations $\mathcal{T}_{z}=\{(T_{q},T_{h,i})\mid q_{i}^{\top}h_{i,j}=z\}$ that achieves $z$ for each $z\in\mathcal{Z}$. A larger set $\mathcal{Z}$ then implies that $\mathcal{T}$ is constructed from more subsets $\mathcal{T}_{z}$. Hence, when the zero-level set $\mathcal{Z}$ has large measure in $\mathbb{R}$ ($|\mathcal{Z}|\gg 0$), many linear transformations $(T_{q},T_{h,i})$ can achieve $\alpha_{i,j}=0$, enabling robust input selectivity. While our analysis establishes the existence of these beneficial configurations, determining whether gradient descent actually converges to any transformation in $\mathcal{T}$, requires a deeper analysis of specific optimization dynamics, loss functions, and architectures. However, as argued in Goodfellow et al. [2016], the abundance of high-dimensional parameter configurations, yielding a desired behavior, increases the likelihood that gradient descent will converge to such parameters during training.

###### Example 1 (Standard nonlinear readout maps) .

The dominant choice for readout map $\phi(\cdot)$ is the exponential function $\exp(\cdot)$ (as in softmax attention), which is a positive monotonic non-decreasing function that is close to zero on the negative domain, i.e., $\phi(x)\to 0$ for $x<0$. In this case, $\alpha_{i,j}\approx 0$ can be achieved by $q_{i}^{\top}h_{i,j}<0$, which happens if the angle between the two vectors $q_{i},h_{i,j}$ lies in $[\frac{\pi}{2}+n\pi,\frac{3\pi}{2}+n\pi]$ for any $n\in\mathbb{Z}^{+}$. This results in a large set $\mathcal{T}$, since many combinations of $q_{i},h_{i,j}$ will lead to coefficients being approximately zero.

Together, Lemmas [1](https://arxiv.org/html/2510.09389v1#Thmlemma1) and [2](https://arxiv.org/html/2510.09389v1#Thmlemma2) show the tradeoff in the choice of $\phi(\cdot)$ and offer a new perspective on the tradeoff between SSMs and Transformers [Gu, 2025]: on the one hand, a nonlinear choice facilitates input selectivity; on the other, a linear choice facilitates efficient computation. To address this trade-off, various kernel approximation methods, i.e. $\phi\approx\psi_{q}(\cdot)\psi_{k}(\cdot)$, have been proposed in the literature, e.g. [Katharopoulos et al., 2020, Tsai et al., 2019, Arora et al., 2024], which try to approximate the behavior of $\phi(\cdot)=\exp(\cdot)$ as closely as possible, while striving to fulfill Principle [2](https://arxiv.org/html/2510.09389v1#Thmprinciple2). However, regardless of how exact the approximation is, there is a fundamental geometric issue with these approaches, as outlined in the next principle.

Principle 2.1.
If the readout map is of the form $\phi(\cdot)=\psi_{q}(\cdot)\psi_{k}(\cdot)$ (e.g.,kernel approximation), its zero-level set is of measure zero. Hence, by Principle [2](https://arxiv.org/html/2510.09389v1#Thmprinciple2), input selectivity is fragile to learn.

###### Corollary 2.1 .

Consider the kernel approximation of $\phi(\cdot):\mathbb{R}^{n}\to\mathbb{R}^{q}$ to be $\tilde{\phi}(q_{i}^{\top}h_{i,j})=\psi_{q}(q_{i})^{\top}\psi_{h}(h_{i,j})$, with $\psi_{q}:\mathbb{R}^{n}\to\mathbb{R}^{q}$ and $\psi_{h}:\mathbb{R}^{n}\to\mathbb{R}^{q}$, such that it approximates the readout map $\phi(\cdot)\approx\tilde{\phi}(\cdot)$ in ([3](https://arxiv.org/html/2510.09389v1#S3.E3)). Then, the zero-level set of the approximation $\tilde{\phi}(\cdot)$ is the singleton $\mathcal{Z}=\{0\}$ and by Lemma [2](https://arxiv.org/html/2510.09389v1#Thmlemma2) the set of linear transformations $\mathcal{T}$ is small.

###### Proof.

If the approximation functions $\psi_{q}(\cdot),\,\psi_{h}(\cdot)$ are thought of as a preprocessing step to compute $q_{i},\,h_{i,j}$, the approximate readout map can be written as $\tilde{\phi}(q_{i}^{\top}h_{i,j})=\tilde{q}_{i}^{\top}\tilde{h}_{i,j}$, which is equivalent to the identity map $\textrm{Id}(\cdot)$. The formal proof is provided in Appendix [C.2](https://arxiv.org/html/2510.09389v1#A3.SS2).
∎

By Principle [2](https://arxiv.org/html/2510.09389v1#Thmprinciple2) and the results from Lemma [2](https://arxiv.org/html/2510.09389v1#Thmlemma2), choosing kernel approximations as readout maps hinders the capacity for input selectivity. To overcome this limitation, the kernel approximations $\psi_{q}(\cdot),\psi_{k}(\cdot)$ need to be designed carefully. While, setting $\psi_{q},\psi_{h}$ to positive functions, e.g. $\psi_{q}(x)=\psi_{h}(x)=\textrm{elu}(x)+1$ [Katharopoulos et al., 2020], preserves the positivity of the coefficients, it eliminates much of the crucial geometric information stored in $q_{i}$ and $h_{i,j}$, since the angle between these vectors is not preserved. To alleviate this issue, some approximation proposals induce a bias towards near-zero coefficients by explicitly using orthogonal feature maps, e.g. FAVOR+ [Choromanski et al., 2021] or random feature maps [Peng et al., 2021].

We note that Principle [2](https://arxiv.org/html/2510.09389v1#Thmprinciple2) presents general considerations for driving coefficients $\alpha_{i,j}$ to zero. Yet, by ([2](https://arxiv.org/html/2510.09389v1#S3.E2)), $i$ coefficients are simultaneously computed using a single query $q_{i}$ and $i$ states (or transformed keys) $h_{i,j}$ with $j=1,\dots,i$. In what follows, we illustrate that achieving near-zero coefficients $\alpha_{i,j}$ *simultaneously* is a more restrictive setup than the general setting of Principle [2](https://arxiv.org/html/2510.09389v1#Thmprinciple2).

Principle 2.2.
If the readout map is the identity $\phi(\cdot)=\operatorname{Id}(\cdot)$, the number of coefficients that can be simultaneously suppressed by a nonzero query vector, is limited to $n-1$. Beyond this limit, suppression necessarily collapses to the trivial (zero) query vector.

###### Corollary 2.2 .

Let $y_{L}\in\mathbb{R}^{d_{v}}$ be the solution to ([2](https://arxiv.org/html/2510.09389v1#S3.E2)), where $\alpha_{L,j}:\mathbb{R}^{n}\rightarrow\mathbb{R};\ q_{L}\mapsto\alpha_{L,j}(q_{L})$ is defined in ([3](https://arxiv.org/html/2510.09389v1#S3.E3)) with identity readout map and normalization factor, i.e., $\phi(\cdot)=\operatorname{Id}(\cdot)$, $\eta_{L}=1$. Consider the set of linearly independent states $\mathcal{H}=\{h_{L,t}\mid c_{1}h_{L,1}+\dots+c_{t}h_{L,t}=0\ \implies\ c_{1}=\dots=c_{t}=0\}$. Then, a nonzero $q_{L}$ that achieves $\alpha_{L,j}=0,\,\forall h_{L,j}\in\mathcal{H}$ exists if and only if $\dim\big(\operatorname{span}\{h_{L,j}\in\mathcal{H}\}\big)<n$. In particular, given a nonzero $q_{L}$, the measure $\lvert\mathcal{H}\rvert\leq n-1$.

###### Proof.

Finding simultaneous zero coefficients $\alpha_{L,j}=0,\,j\in\mathcal{J}$ is equivalent to computing a solution of a linear homogeneous system, for which existence results exist via the rank-nullity theorem. The full proof is provided in Appendix [C.2](https://arxiv.org/html/2510.09389v1#A3.SS2).
∎

Principle [2.2](https://arxiv.org/html/2510.09389v1#Thmprinciple2.Thmsubprinciple2) is primarily a limiting factor for identity readout maps,(^3^33Cor. [2.2](https://arxiv.org/html/2510.09389v1#Thmlemma2.Thmcor2) can be extended to linear readout maps; we limit it to identity readout maps for ease of exposition.) since zero coefficients $\alpha_{i,j}$ are only achieved for $q_{i}^{\top}h_{i,j}=0$, i.e., $q_{i}$ and $h_{i,j}$ are orthogonal. For nonlinear readout maps (see Example 1), many configurations of $q_{i}$ and $h_{i,j}$ achieve $\alpha_{i,j}=0$.

###### Principle 1 .

###### Lemma 1 .

###### Proof.

###### Principle 2 .

###### Lemma 2 .

###### Proof.

###### Example 1 (Standard nonlinear readout maps) .

###### Principle 2.1 .

###### Corollary 2.1 .

###### Proof.

###### Principle 2.2 .

###### Corollary 2.2 .

###### Proof.

##### Can learnable parameters save us?

We established that identity readout maps $\phi(\cdot)=\textrm{Id}(\cdot)$ can suppress coefficients – if $q_{i}$, $h_{i,j}$ are orthogonal – but results in a much smaller set $\mathcal{T}$ than for nonlinear $\phi(\cdot)$, which in principle makes the learning problem harder. However, since $T_{h,i}$ in ([4](https://arxiv.org/html/2510.09389v1#S3.E4)) is a function of $A_{i,j}\coloneqq\prod_{t=j+1}^{i}A_{t}$ and $b_{j}$, these can be carefully parametrized and leveraged to achieve $h_{i,j}=0$, which also results in zero coefficients. This is the central design paradigm of many SSM proposals, e.g. Dao and Gu [2024], Yang et al. [2025a]. We elaborate more on how these choices can compensate for a linear readout map in the next section. However, it is important to note that although the choices of $A_{i,j}$ and $b_{j}$ can be used to suppress certain tokens, these are typically designed for input selectivity on the keys. For example, choosing $A_{t}=1-\lambda_{t},\,b_{j}=\lambda_{j}$ (as in GRU Cho et al. [2014]),(^4^44A similar inverse relationship between $A_{t}$ and $b_{j}$ is found in many SSM proposals, see e.g. Dao and Gu [2024], De et al. [2024], Schlag et al. [2021].) enables selection of the state or key (([3](https://arxiv.org/html/2510.09389v1#S3.E3))). While this is straightforward to understand in an SSM setting, where the state aggregates the keys over time, our framework directly links this gating behavior to the coefficients $\alpha_{i,j}$. Choosing $\lambda_{j}=0$ for a key $k_{j}$ results in suppression of this key in all future time steps, i.e., $\alpha_{i,j}=0,\forall i\geq j$. Additionally, choosing $\lambda_{t}=1$ or equivalently $A_{t}=0$ for any $t\in[j,i]$, results in a zero state $h_{i,j}$ for all $i\geq t$, thus suppressing the coefficients. This shows that input selectivity design in SSMs, i.e., design of $A_{t}$, $b_{j}$, is a special case of the more general input selectivity design for coefficients considered in this work.

### 4.2 On choosing the Evolution Matrices A t A_{t}

It is well-known that softmax attention cannot distinguish between two identical inputs that appear at different (time) positions in the sequence [Yun et al., 2020]. The reason for this is the evolution matrix ($A_{t}=\mathbb{I}$) used in softmax attention and is commonly mitigated via positional embeddings (e.g. Su et al. [2024]). However, careful design of the evolution matrix enables processing of positional information in the attention mechanism, as evidenced by SSMs [Dao and Gu, 2024].

Principle 3.
If $A_{t}=\mathbb{I}$, no positional information is contained in the coefficients $\alpha_{i,j}$. This means for a sequence model choosing $A_{t}$ to be the identity, positional embeddings are needed.

###### Definition 1 (Positional Information) .

$\alpha_{i,j}$ in ([4](https://arxiv.org/html/2510.09389v1#S3.E4)) are said to have positional information, if for two identical inputs $x_{j}=x_{\bar{j}}=\bar{x}$, the resulting coefficients are not identical, i.e., $\alpha_{i,j}\neq\alpha_{i,\bar{j}}$.

###### Lemma 3 .

The coefficients $\alpha_{i,j}$ in ([4](https://arxiv.org/html/2510.09389v1#S3.E4)) have positional information if and only if $A_{t}\neq\mathbb{I}_{n},\forall t$.

###### Proof.

The proof relies on the fact that two keys $k_{j}$, $k_{\bar{j}}$ computed from the same input $x_{j}$ result in the same coefficients $\alpha_{i,j}$ if and only if $A_{i}=\mathbb{I}_{n}$. The formal proof is provided in Appendix [C.1](https://arxiv.org/html/2510.09389v1#A3.SS1).
∎

By Lemma [3](https://arxiv.org/html/2510.09389v1#Thmlemma3), positional information can be embedded in the coefficients by designing $A_{t}$ appropriately, further strengthening our intuition why linear RNN/SSM proposals (such as Mamba-2, DeltaNet, mLSTM) do not require positional embeddings to learn in-context recall tasks [Arora et al., 2023, Poli et al., 2024, Okpekpe and Orvieto, 2025] or language [Gu and Dao, 2023]. However, despite positional embeddings not being strictly necessary, using positional embeddings for RNNs and SSMs ($A_{t}\neq\mathbb{I}$) can be beneficial in practice [Morita, 2024].

Beyond enabling processing of positional information, the choices of $A_{t}$ can selectively suppress tokens under some circumstances. Specifically, as discussed in Principle [2](https://arxiv.org/html/2510.09389v1#Thmprinciple2), when the readout map $\phi(\cdot)$ is a linear function, $\alpha_{i,j}=0$ is only achieved when $q_{i}$ and $h_{i,j}$ are orthogonal, or if one of the vectors $q_{i}^{\top},h_{i,j}$ (or both) is zero. Since relying on the orthogonality of $q_{i}^{\top}$ and $h_{i,j}$ leads to harder learning problem (per Principle [2](https://arxiv.org/html/2510.09389v1#Thmprinciple2)), many architectures that use a linear readout map for computational reasons (as per Principle [1](https://arxiv.org/html/2510.09389v1#Thmprinciple1)), rely on dynamics ([3](https://arxiv.org/html/2510.09389v1#S3.E3)) to achieve selectivity, e.g. Mamba-2. In these models, the right choice of the dynamics components $A_{t}$ and $b_{j}$ is crucial to ensure that $h_{i,j}$ can be effectively driven to zero. Additionally, the evolution matrices $A_{t}$ can be used to rotate non-orthogonal keys over time, to achieve $q_{i}^{\top}h_{i,j}=0$, despite $q_{i}^{\top}k_{j}\neq 0$.
Principle 4.
Assuming a linear readout map $\phi(\cdot)$, the structure imposed on the evolution matrices $A_{t}$ limits the operations (e.g. scaling, rotation) that can be performed on the keys.

###### Lemma 4 .

Imposing any of the following structures on $A_{t}$ in ([3](https://arxiv.org/html/2510.09389v1#S3.E3)), results in the corresponding allowed transformations for the keys $\{k_{j}\}_{j=1}^{i}$:

- a)
$A_{t}=\lambda_{t}\mathbb{I}_{n},\lambda_{t}\in\mathbb{R},\lvert\lambda_{t}\rvert\leq 1$; allows scaling of keys (including flipping, if $\lambda_{t}<0$) uniformly along all dimensions $n$,
- b)
$A_{t}=\textrm{diag}(\lambda_{t}),\lambda_{t}\in\mathbb{R}^{n},\lvert\lambda^{(r)}_{t}\rvert\leq 1$ where $\lambda^{(r)}$ denotes the $r$-th entry of $\lambda_{t}$; allows scaling of keys (including flipping, if $\lambda^{(r)}_{t}<0$) separately along dimensions $n$,
- c)
$A_{t}=\mathbb{I}_{n}-\beta_{t}\lambda_{t}\lambda_{t}^{\top},\beta_{t}\in[0,2],\lambda_{t}\in\mathbb{R}^{n},$ (Householder matrix); allows scaling and specific rotations of keys.

###### Proof.

The results follow directly from an elementary eigenvalue analysis connected to their respective geometric interpretation. The formal proof is provided in Appendix [C.1](https://arxiv.org/html/2510.09389v1#A3.SS1).
∎

Lemma [4](https://arxiv.org/html/2510.09389v1#Thmlemma4) provides guidance on how to design the evolution matrices $A_{t}$, depending on the transformations that should be enabled. Note that it is possible to combine these structures, e.g., combine a) and c) in Lemma [4](https://arxiv.org/html/2510.09389v1#Thmlemma4) to obtain a scaled Householder matrix [Yang et al., 2025a]. This list is not exhaustive and more general structures can be imposed on $A_{t}$ to achieve other transformations. Yet, these structures might be prohibited by current CUDA kernel implementations [Siems et al., 2025].

###### Principle 3 .

###### Definition 1 (Positional Information) .

###### Lemma 3 .

###### Proof.

###### Principle 4 .

###### Lemma 4 .

###### Proof.

### 4.3 On choosing the Scaling Parameters b j b_{j}

As discussed in Section [4.1](https://arxiv.org/html/2510.09389v1#S4.SS1), the scaling parameters $b_{j}$ are often linked to the evolution matrices $A_{t}$ (see Table [1](https://arxiv.org/html/2510.09389v1#S3.T1)). While the need for scaling/normalization of keys has been established in previous architectural proposals, there is a lack of consensus on how to design $b_{j}$. The following result is based on classical signal propagation analyses [Glorot and Bengio, 2010, He et al., 2015].
Principle 5.
Choosing the scaling parameter $b_{j}=\mathcal{O}(\nicefrac{{1}}{{\sqrt{n}}})$ ensures that the dot product $q_{i}^{\top}h_{i,j}$ has $\mathcal{O}(1)$ variance; anything larger than this scale lets the variance grow with $n$.

###### Lemma 5 .

Consider two i.i.d. normally distributed random vectors $x_{i},x_{j}\sim\mathcal{N}(0,\Sigma)$. Then, the dot product $q_{i}^{\top}h_{i,j}$ with $q_{i},h_{i,j}$ defined in ([4](https://arxiv.org/html/2510.09389v1#S3.E4)), has zero-mean and variance $\textrm{Var}(q_{i}^{\top}h_{i,j})=\textrm{tr}(\Sigma_{q}\Sigma_{h})$ with $\Sigma_{q}=T_{q}\Sigma T_{q}^{\top}$, $\Sigma_{h}=T_{h,i}\Sigma T_{h,i}^{\top}$.
Assuming $\Sigma=\sigma\mathbb{I}_{d}$, both $\Sigma_{q}$ and $\Sigma_{h}$ are positive semi-definite and the variance of the dot product scales as $\textrm{Var}(q_{i}^{\top}h_{i,j})=\mathcal{O}(n)$.

###### Proof.

The formal proof is provided in Appendix [C.1](https://arxiv.org/html/2510.09389v1#A3.SS1).
∎

Note that in Vaswani et al. [2017, Section 3.2.1], the statement of Lemma [5](https://arxiv.org/html/2510.09389v1#Thmlemma5) is discussed informally. While the scaling factor choice $b_{j}=\nicefrac{{1}}{{\sqrt{n}}}$ is widely used for attention-based models, in SSMs or RNNs, $b_{j}$ is more often designed in combination with evolution matrices $A_{t}$. Since in these models, $A_{t}$ and $b_{j}$ have an inverse relationship and $A_{t}$ is typically designed to have eigenvalues close to the unit circle, this forces $b_{j}$ to be small. If the entangling of $A_{t}$ and $b_{j}$ is abandoned, Principle [5](https://arxiv.org/html/2510.09389v1#Thmprinciple5) provides a minimum scaling level that is essential for stable training of a sequence model.

###### Example 2 .

Mamba-2 parameterizes both $A_{t}$ and $b_{j}$ with $\Delta_{j}$ (see Table [1](https://arxiv.org/html/2510.09389v1#S3.T1)), which is biased to lie in the range $\Delta_{j}\in[0.001,0.1]$. This range can be thought of as $\nicefrac{{1}}{{\sqrt{n}}}$ for $n\in[1e2,1e6]$, thus fulfilling Principle [5](https://arxiv.org/html/2510.09389v1#Thmprinciple5) for a wide range of dimensions $n$.

###### Principle 5 .

###### Lemma 5 .

###### Proof.

###### Example 2 .

### 4.4 On choosing the Normalization Factors η i \eta_{i}

While the scaling factors $b_{j}$ scale the keys, the normalization factors $\eta_{i}$ directly scale the $\alpha_{i,j}$s. Therefore, the normalization factors offer a way to re-scale all coefficients at time index $i$ ( amplifying large, reducing small) or normalize the linear combination at every time index individually.

Principle 6.
If the readout map $\phi(\cdot)$ is unbounded and/or $A_{t}$ are not stable, the normalization factors $\eta_{i}$ need to be designed to counteract the growth in the coefficients.

###### Lemma 6 .

Consider a fixed index $j$ and let $s_{i}\coloneqq q_{i}^{\top}h_{i,j}$, $\alpha_{i}=\phi(s_{i})$ in ([3](https://arxiv.org/html/2510.09389v1#S3.E3)). Let $\phi:\mathbb{R}\to[0,\infty)$ be nondecreasing and unbounded, and let $g:[0,\infty)\to(0,\infty)$ be an increasing and unbounded comparison function, such that $L\coloneqq\limsup_{i\to\infty}\frac{\phi(s_{i})}{g(s_{i})}<\infty$. If $\eta_{i}$ is chosen such that $\liminf_{i\to\infty}\frac{\eta_{i}}{g(s_{i})}\coloneqq m>0$, then the normalized coefficients are bounded: $\limsup_{i\to\infty}\frac{\alpha_{i}}{\eta_{i}}\leq\frac{L}{m}$.

###### Proof.

The formal proof is provided in Appendix [C.1](https://arxiv.org/html/2510.09389v1#A3.SS1).
∎

As an example, consider the mLSTM architecture [Beck et al., 2024], which allows unstable evolution matrices $A_{t}$ with eigenvalues outside the unit circle and an identity readout map. To fulfill Principle [6](https://arxiv.org/html/2510.09389v1#Thmprinciple6), the normalization factors $\eta_{i}$ need to be designed such that they grow linearly (or superlinearly) with the state $h_{i,j}$. The model does this by relying on a separate normalization state that uses the same dynamics as the state (see Table [1](https://arxiv.org/html/2510.09389v1#S3.T1)).
An alternative option to design the normalization factors, is to choose $\eta_{i}$ as a function of the coefficients $\alpha_{i,j}$, i.e., $\eta_{i}=f(\alpha_{i,j})$. In this case, only linear (or superlinear) growth of $f$ is required to fulfill Principle [6](https://arxiv.org/html/2510.09389v1#Thmprinciple6) (see Remark [5](https://arxiv.org/html/2510.09389v1#Thmremark5)). A simple choice of a superlinear $f$ is used in softmax attention, and is discussed in the following corollary.

###### Corollary 6.1 .

Choosing $\eta_{i}=\sum_{j=1}^{i}\alpha_{i,j}$ in ([2](https://arxiv.org/html/2510.09389v1#S3.E2)) constrains the normalized coefficients to $\tilde{\alpha}_{i,j}=\nicefrac{{\alpha_{i,j}}}{{\eta_{i}}}\in[0,1]$ and imposes $\sum_{j=1}^{i}\tilde{\alpha}_{i,j}=1$.

Corollary [6.1](https://arxiv.org/html/2510.09389v1#Thmlemma6.Thmcor1) additionally imposes a constraint on the linear combination in ([2](https://arxiv.org/html/2510.09389v1#S3.E2)), which restricts the outputs $y_{i}$ to lie in a well-defined geometric space spanned by the value vectors. This fact and its implications are further discussed in Appendix [A](https://arxiv.org/html/2510.09389v1#A1).

###### Principle 6 .

###### Lemma 6 .

###### Proof.

###### Corollary 6.1 .

## 5 Experimental Validation

In this section, we empirically validate the principles discussed in Section [4](https://arxiv.org/html/2510.09389v1#S4) on selected tasks of the MAD benchmark [Poli et al., 2024]. Additional experiments are reported in Appendix [D](https://arxiv.org/html/2510.09389v1#A4) and all details of the experimental setup are discussed in Appendix [E](https://arxiv.org/html/2510.09389v1#A5). The code for reproducing the experiments is available here: [https://git.sieber.io/mdl-design](https://git.sieber.io/mdl-design).

##### Principle 1

Given that the principle concerns the implementation of sequence models, we show its implication on the model’s throughput in Appendix [D](https://arxiv.org/html/2510.09389v1#A4).

##### Principle 2

Figure [1](https://arxiv.org/html/2510.09389v1#S5.F1) shows the accuracy of different readout maps $\phi(\cdot)$ on the fuzzy in-context recall and selective copying tasks of MAD. The other parameters are fixed to $A_{t}=\mathbb{I}$, $b_{j}=\nicefrac{{1}}{{\sqrt{n}}}$, $\eta_{i}=\sum_{j}\alpha_{i,j}$, across all readout maps. Since input selectivity is crucial for both tasks, we plot accuracy against the fraction of coefficients close to 0 (near-zero set with $\epsilon=0.001$) as well as the theoretical near-zero set of each $\phi(\cdot)$. *The results show, that more coefficients are set close to zero as the near-zero set of the readout map grows. For both tasks this increases performance*. Additional results and the results for Principles [2.1](https://arxiv.org/html/2510.09389v1#Thmprinciple2.Thmsubprinciple1) & [2.2](https://arxiv.org/html/2510.09389v1#Thmprinciple2.Thmsubprinciple2) are provided in Appendix [D](https://arxiv.org/html/2510.09389v1#A4).

Figure: Figure 1: (Principle 2) Performance of different readout maps $\phi(\cdot)$ on two MAD tasks against the fraction of coefficients with near zero values ($\lvert\alpha\rvert\leq 0.001$; left & middle) and the theoretical near-zero sets of each readout map ($\lvert\phi(x)\rvert\leq 0.001$; right). The other parameters $A_{t}$, $b_{j}$, $\eta_{i}$ are fixed, thus the setting for ^† is equivalent to softmax attention.
Refer to caption: x2.png

Figure: Figure 2: (Principle 3) Performance of two $A_{t}$ choices with and without positional embeddings (PE) on the noisy in-context recall task.
Refer to caption: x3.png

##### Principle 3

We show that $A_{t}\neq\mathbb{I}_{n}$ can replace positional embeddings (PE) for the simplest choice of $A_{t}$. Figure [2](https://arxiv.org/html/2510.09389v1#S5.F2) shows the accuracy of four models on the noisy in-context recall of MAD, where we set $b_{j}=\nicefrac{{1}}{{\sqrt{n}}}$, $\eta_{i}=\sum_{j}\alpha_{i,j}$, and vary $A_{t}\in\{\mathbb{I}_{n},0.95\,\mathbb{I}_{n}\}$, $\phi(\cdot)=\{\exp(\cdot),\textrm{softplus}(\cdot),\textrm{ReLU}(\cdot)\}$, and wether the models have PE or not. We choose a constant $A_{t}\!=\!0.95\,\mathbb{I}_{n}$, since it directly connects to ALiBi [Press et al., 2022]. While ALiBi enables positional information via additive biases to the attention matrix, $A_{t}=0.95\,\mathbb{I}_{n}$ does the same but via multiplication. *The experiment shows that noisy in-context recall is solved by both PE and $A_{t}\neq\mathbb{I}_{n}$ for all readout maps. For $A_{t}\!=\!\mathbb{I}_{n}$ without PE, only $\phi(\cdot)=\textrm{ReLU}(\cdot)$ achieves non-random performance but does not perform perfectly.*

##### Principle 4

To show the effect of allowed transformations in $A_{t}$ on the performance, we ablate four structures imposed on $A_{t}$ – scalar, diagonal, Householder with keys $k_{t}$, and Householder with a learned vector $z_{t}$ – on the fuzzy in-context recall and selective copying tasks of MAD; the other parameters are fixed to $\phi(\cdot)=\textrm{Id}(\cdot)$, $\eta_{i}=1$, $b_{j}=\nicefrac{{1}}{{\sqrt{n}}}$ across all choices of $A_{t}$. For each $A_{t}$, we additionally vary how the scalar and diagonal are parameterized (either using GLA [Yang et al., 2023] or Mamba-2 [Gu and Dao, 2023] parameterizations), and the scaling factor of the rotation vector in the Householder matrix (either $\beta_{t}=2$ or learned from the input). *On both tasks, the performance generally increases as more transformations are allowed in $A_{t}$ (blue line). However, the parameterization of the involved parameters is important and good parameterizations can considerably improve performance (orange line).*

##### Principles 5 & 6

These principles are mainly concerned with training stability, which we show by ablating various choices of $b_{j}$ and $\eta_{i}$ in Appendix [D](https://arxiv.org/html/2510.09389v1#A4).

Figure: Figure 3: (Principle 4) Performance of four $A_{t}$ choices (scalar, diagonal, Householder with $k_{t}$, Householder with learned $z_{t}$) on two MAD tasks, with all other parameters fixed. The scalar/diagonal parameter(s) $\lambda_{t}$ are using either the GLA (gray) or Mamba-2 (magenta) parameterization and the Householder scaling $\beta_{t}$ is either fixed (gray) or learned (magenta).
Refer to caption: x4.png

## 6 Conclusions

This paper studies sequence models with a focus on the *coefficient dynamics* that multiply value vectors for a single layer. We view the sequence model outputs as linear combinations of past values with coefficients produced by autonomous linear dynamics with impulse inputs given by the keys. This view captures a broad class of existing sequence models, which can be recovered as a special case of this formalization. Furthermore, we show how studying the *coefficient dynamics* sheds light onto how normalization, geometric operations, and state updates influence input selectivity, efficient implementations, and training stability. These insights, formalized as mathematical results, unlock six design principles for sequence models that enable the tailored design of models. Experimental results validate these principles.

##### Limitations:

The present study is limited to single-layer setups and more work is needed to translate the proposed principles to multi-layer models. It also does not cover optimization considerations and the role of other components in the architecture, such as convolutions, gates, and specific positional embeddings, which could inform new principles or strengthen existing ones. Finally, the principles are validated on synthetic datasets and require additional experiments on real-world applications, such as language modeling.

## References

- Elman [1990]
Jeffrey L Elman.
Finding structure in time.
*Cognitive science*, 14(2):179–211, 1990.
- Hochreiter and Schmidhuber [1997]
Sepp Hochreiter and Jürgen Schmidhuber.
Long short-term memory.
*Neural computation*, 9(8):1735–1780, 1997.
- Vaswani et al. [2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.
Attention is All you Need.
In *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc., 2017.
- Gu et al. [2022a]
Albert Gu, Karan Goel, and Christopher Ré.
Efficiently Modeling Long Sequences with Structured State Spaces.
In *The International Conference on Learning Representations (ICLR)*, 2022a.
- Gu and Dao [2023]
Albert Gu and Tri Dao.
Mamba: Linear-Time Sequence Modeling with Selective State Spaces, 2023.
URL [https://arxiv.org/abs/2312.00752](https://arxiv.org/abs/2312.00752).
- Katharopoulos et al. [2020]
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.
Transformers are RNNs: fast autoregressive transformers with linear attention.
In *Proceedings of the 37th International Conference on Machine Learning*, ICML’20. JMLR.org, 2020.
- Dao and Gu [2024]
Tri Dao and Albert Gu.
Transformers are SSMs: Generalized Models and Efficient Algorithms with Structured State Space Duality.
In *ICML 2024*, 2024.
- Schlag et al. [2021]
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber.
Linear transformers are secretly fast weight programmers.
In *International Conference on Machine Learning*, pages 9355–9366. PMLR, 2021.
- Arora et al. [2023]
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré.
Zoology: Measuring and Improving Recall in Efficient Language Models.
*arXiv:2312.04927*, 2023.
- Jelassi et al. [2024]
Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach.
Repeat after me: Transformers are better than state space models at copying.
*arXiv preprint arXiv:2402.01032*, 2024.
- Merrill et al. [2024]
William Merrill, Jackson Petty, and Ashish Sabharwal.
The illusion of state in state-space models.
*arXiv preprint arXiv:2404.08819*, 2024.
- Arora et al. [2024]
Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Re.
Simple linear attention language models balance the recall-throughput tradeoff.
In *Forty-first International Conference on Machine Learning*, 2024.
- Olsson et al. [2022]
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al.
In-context learning and induction heads.
*arXiv preprint arXiv:2209.11895*, 2022.
- Liu et al. [2024]
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu.
Vmamba: Visual state space model.
*arXiv preprint arXiv:2401.10166*, 2024.
- Nguyen et al. [2023]
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al.
Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution.
*Advances in neural information processing systems*, 36:43177–43201, 2023.
- Schiff et al. [2024]
Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov.
Caduceus: Bi-directional equivariant long-range dna sequence modeling.
*arXiv preprint arXiv:2403.03234*, 2024.
- Gu et al. [2020]
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré.
HiPPO: Recurrent Memory with Optimal Polynomial Projections.
In *Advances in Neural Information Processing Systems*, volume 33, pages 1474–1487. Curran Associates, Inc., 2020.
- Gu et al. [2022b]
Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré.
On the Parameterization and Initialization of Diagonal State Space Models, 2022b.
URL [https://arxiv.org/abs/2206.11893](https://arxiv.org/abs/2206.11893).
- Smith et al. [2023]
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman.
Simplified State Space Layers for Sequence Modeling.
In *The Eleventh International Conference on Learning Representations*, 2023.
- Orvieto et al. [2023]
Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De.
Resurrecting Recurrent Neural Networks for Long Sequences.
In *Proceedings of the 40th International Conference on Machine Learning*, volume 202, pages 26670–26698. PMLR, 23–29 Jul 2023.
- Tay et al. [2021]
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler.
Long Range Arena : A Benchmark for Efficient Transformers.
In *International Conference on Learning Representations (ICLR)*, 2021.
- Wang et al. [2022]
Junxiong Wang, Jing Nathan Yan, Albert Gu, and Alexander M Rush.
Pretraining without attention.
*arXiv preprint arXiv:2212.10544*, 2022.
- Fu et al. [2023]
Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré.
Hungry Hungry Hippos: Towards Language Modeling with State Space Models, 2023.
URL [https://arxiv.org/abs/2212.14052](https://arxiv.org/abs/2212.14052).
- Sun et al. [2023]
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei.
Retentive network: A successor to transformer for large language models, 2023.
- Katsch [2023]
Tobias Katsch.
Gateloop: Fully data-controlled linear recurrence for sequence modeling, 2023.
- Yang et al. [2023]
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim.
Gated linear attention transformers with hardware-efficient training.
*arXiv preprint arXiv:2312.06635*, 2023.
- Yang et al. [2024]
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim.
Parallelizing linear transformers with the delta rule over sequence length.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
URL [https://openreview.net/forum?id=y8Rm4VNRPH](https://openreview.net/forum?id=y8Rm4VNRPH).
- Beck et al. [2024]
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter.
xLSTM: Extended Long Short-Term Memory.
*arXiv preprint arXiv:2405.04517*, 2024.
- Yang and Zhang [2024]
Songlin Yang and Yu Zhang.
Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024.
URL [https://github.com/sustcsonglin/flash-linear-attention](https://github.com/sustcsonglin/flash-linear-attention).
- Yang et al. [2025a]
Songlin Yang, Jan Kautz, and Ali Hatamizadeh.
Gated delta networks: Improving mamba2 with delta rule.
In *The Thirteenth International Conference on Learning Representations*, 2025a.
URL [https://openreview.net/forum?id=r8H7xhYPwz](https://openreview.net/forum?id=r8H7xhYPwz).
- Siems et al. [2025]
Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi.
Deltaproduct: Improving state-tracking in linear rnns via householder products.
*arXiv preprint arXiv:2502.10297*, 2025.
- Guo et al. [2025]
Han Guo, Songlin Yang, Tarushii Goel, Eric P Xing, Tri Dao, and Yoon Kim.
Log-linear attention.
*arXiv preprint arXiv:2506.04761*, 2025.
- Lin et al. [2025]
Zhixuan Lin, Evgenii Nikishin, Xu Owen He, and Aaron Courville.
Forgetting transformer: Softmax attention with a forget gate.
*arXiv preprint arXiv:2503.02130*, 2025.
- Yang et al. [2025b]
Songlin Yang, Yikang Shen, Kaiyue Wen, Shawn Tan, Mayank Mishra, Liliang Ren, Rameswar Panda, and Yoon Kim.
Path attention: Position encoding via accumulating householder transformations.
*arXiv preprint arXiv:2505.16381*, 2025b.
- AI21 Labs [2024]
AI21 Labs.
Ai21-jamba-large-1.7.
[https://huggingface.co/ai21labs/AI21-Jamba-Large-1.7](https://huggingface.co/ai21labs/AI21-Jamba-Large-1.7), 2024.
Accessed: 2025-09-22.
- Nano [2025]
NVIDIA Nemotron Nano.
Efficient hybrid mamba-transformer reasoning model.
*arXiv preprint arXiv:2508.14444*, 2025.
- Qwen Team [2025]
Qwen Team.
Qwen3-next: Towards ultimate training and inference efficiency.
[https://qwen.ai/blog?id=e34c4305036ce60d55a0791b170337c2b70ae51d&from=home.latest-research-list](https://qwen.ai/blog?id=e34c4305036ce60d55a0791b170337c2b70ae51d&from=home.latest-research-list), 2025.
Accessed: 2025-09-22.
- Poli et al. [2024]
Michael Poli, Armin W. Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, Ce Zhang, and Stefano Massaroli.
Mechanistic design and scaling of hybrid architectures.
In *ICML*, 2024.
URL [https://openreview.net/forum?id=GDp7Gyd9nf](https://openreview.net/forum?id=GDp7Gyd9nf).
- Waleffe et al. [2024]
Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al.
An empirical study of mamba-based language models.
*arXiv preprint arXiv:2406.07887*, 2024.
- Orvieto et al. [2024]
Antonio Orvieto, Soham De, Caglar Gulcehre, Razvan Pascanu, and Samuel L Smith.
Universality of linear recurrences followed by non-linear projections: Finite-width guarantees and benefits of complex eigenvalues.
*International Conference on Machine Learning*, 2024.
- Cirone et al. [2024]
Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, and Terry Lyons.
Theoretical foundations of deep selective state-space models.
*arXiv preprint arXiv:2402.19047*, 2024.
- Zucchet and Orvieto [2024]
Nicolas Zucchet and Antonio Orvieto.
Recurrent neural networks: vanishing and exploding gradients are not the end of the story.
*Advances in Neural Information Processing Systems*, 37:139402–139443, 2024.
- Okpekpe and Orvieto [2025]
Destiny Okpekpe and Antonio Orvieto.
Revisiting associative recall in modern recurrent models.
In *First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models*, 2025.
- Sieber et al. [2024]
Jerome Sieber, Carmen A Alonso, Alexandre Didier, Melanie N Zeilinger, and Antonio Orvieto.
Understanding the differences in foundation models: Attention, state space models, and recurrent neural networks.
*Advances in Neural Information Processing Systems*, 37:134534–134566, 2024.
- Ali et al. [2024]
Ameen Ali, Itamar Zimerman, and Lior Wolf.
The hidden attention of mamba models.
*arXiv preprint arXiv:2403.01590*, 2024.
- Choromanski et al. [2021]
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller.
Rethinking Attention with Performers.
In *International Conference on Learning Representations*, 2021.
- De et al. [2024]
Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre.
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models, 2024.
URL [https://arxiv.org/abs/2402.19427](https://arxiv.org/abs/2402.19427).
- Goodfellow et al. [2016]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
*Deep Learning*.
MIT Press, 2016.
[http://www.deeplearningbook.org](http://www.deeplearningbook.org).
- Gu [2025]
Albert Gu.
On the tradeoffs of state space models and transformers, 2025.
URL [https://goombalab.github.io/blog/2025/tradeoffs/](https://goombalab.github.io/blog/2025/tradeoffs/).
- Tsai et al. [2019]
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov.
Transformer dissection: a unified understanding of transformer’s attention via the lens of kernel.
*arXiv preprint arXiv:1908.11775*, 2019.
- Peng et al. [2021]
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong.
Random feature attention.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=QtTKTdVrFBB](https://openreview.net/forum?id=QtTKTdVrFBB).
- Cho et al. [2014]
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio.
On the properties of neural machine translation: Encoder–decoder approaches.
In Dekai Wu, Marine Carpuat, Xavier Carreras, and Eva Maria Vecchi, editors, *Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation*, pages 103–111, Doha, Qatar, October 2014. Association for Computational Linguistics.
doi: 10.3115/v1/W14-4012.
URL [https://aclanthology.org/W14-4012/](https://aclanthology.org/W14-4012/).
- Yun et al. [2020]
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar.
Are transformers universal approximators of sequence-to-sequence functions?
In *International Conference on Learning Representations*, 2020.
URL [https://openreview.net/forum?id=ByxRM0Ntvr](https://openreview.net/forum?id=ByxRM0Ntvr).
- Su et al. [2024]
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.
Roformer: Enhanced transformer with rotary position embedding.
*Neurocomputing*, 568:127063, 2024.
ISSN 0925-2312.
doi: https://doi.org/10.1016/j.neucom.2023.127063.
URL [https://www.sciencedirect.com/science/article/pii/S0925231223011864](https://www.sciencedirect.com/science/article/pii/S0925231223011864).
- Morita [2024]
Takashi Morita.
Positional encoding helps recurrent neural networks handle a large vocabulary.
*Transactions on Machine Learning Research*, 2024.
ISSN 2835-8856.
URL [https://openreview.net/forum?id=PtnwXd13SF](https://openreview.net/forum?id=PtnwXd13SF).
- Glorot and Bengio [2010]
Xavier Glorot and Yoshua Bengio.
Understanding the difficulty of training deep feedforward neural networks.
In Yee Whye Teh and Mike Titterington, editors, *Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics*, volume 9 of *Proceedings of Machine Learning Research*, pages 249–256, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. PMLR.
URL [https://proceedings.mlr.press/v9/glorot10a.html](https://proceedings.mlr.press/v9/glorot10a.html).
- He et al. [2015]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.
In *Proceedings of the IEEE international conference on computer vision*, pages 1026–1034, 2015.
- Press et al. [2022]
Ofir Press, Noah Smith, and Mike Lewis.
Train short, test long: Attention with linear biases enables input length extrapolation.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=R8sQPpGCv0](https://openreview.net/forum?id=R8sQPpGCv0).
- Heo and Choi [2024]
DongNyeong Heo and Heeyoul Choi.
Generalized probabilistic attention mechanism in transformers, 2024.
URL [https://arxiv.org/abs/2410.15578](https://arxiv.org/abs/2410.15578).
- von Oswald et al. [2023]
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov.
Transformers learn in-context by gradient descent.
pages 35151–35174, 2023.
URL [https://arxiv.org/abs/2212.07677](https://arxiv.org/abs/2212.07677).
- Richter and Wattenhofer [2020]
Oliver Richter and Roger Wattenhofer.
Normalized attention without probability cage, 2020.
URL [https://arxiv.org/abs/2005.09561](https://arxiv.org/abs/2005.09561).
- Federer [1959]
Herbert Federer.
Curvature measures.
*Transactions of the American Mathematical Society*, 93(3):418–491, 1959.
- Friedberg et al. [1997]
Stephen H Friedberg, Arnold J Insel, and Lawrence E Spence.
*Linear algebra*.
Prentice Hall, 1997.
- Dao [2023]
Tri Dao.
FlashAttention-2: Faster attention with better parallelism and work partitioning.
2023.
- Merity et al. [2016]
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.
Pointer sentinel mixture models, 2016.
- Britz et al. [2017]
Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc Le.
Massive exploration of neural machine translation architectures.
In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, *Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing*, pages 1442–1451, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
doi: 10.18653/v1/D17-1151.
URL [https://aclanthology.org/D17-1151/](https://aclanthology.org/D17-1151/).
- Shazeer [2020]
Noam Shazeer.
Glu variants improve transformer, 2020.
URL [https://arxiv.org/abs/2002.05202](https://arxiv.org/abs/2002.05202).

## Appendix A Interpretation of linear combination classes

Our perspective of viewing a sequence model’s output as a linear combination of value vectors (([2](https://arxiv.org/html/2510.09389v1#S3.E2))) has important considerations. In particular, restrictions imposed on the normalized coefficients $\tilde{\alpha}_{i,j}=\frac{\alpha_{i,j}}{\eta_{i}}$ fully determine the output space of the model, as spanned by the value vectors. Specifically, any linear combination of the form ([2](https://arxiv.org/html/2510.09389v1#S3.E2)) can be divided into four different classes depending on the restrictions imposed. These classes together with their restrictions, output space, and example architectures are provided in Table [2](https://arxiv.org/html/2510.09389v1#A1.T2). The output space of these classes are also visualized in Figure [A](https://arxiv.org/html/2510.09389v1#A1).

**Table 2: Summary of linear combination classes depending on the choice of coefficients.**
| Type | Restrictions | Output Space | Example |
| --- | --- | --- | --- |
| Convex comb. | $\tilde{\alpha}_{i,j}\geq 0$ & $\sum_{j}\tilde{\alpha}_{i,j}=1$ | Simplex | Softmax Attention |
| Conical comb. | $\tilde{\alpha}_{i,j}\geq 0$ | Positive Orthant | Normalized Attention |
| Affine comb. | $\sum_{j}\tilde{\alpha}_{i,j}=1$ | Hyperplane | GPAM [Heo and Choi, 2024] |
| Linear comb. | None | $\mathbb{R}^{d_{v}}$ | Mamba-2, DeltaNet |

The structure of the output space defined by the normalized coefficients $\tilde{\alpha}_{i,j}$ and the value vectors $\{v_{1},\dots,v_{i}\}$ can be linked to a model’s performance, however this area remains underexplored. For instance, the output of softmax attention lies in the convex hull (simplex) of previous value vectors, which has been linked to various performance traits, such as in-context learning [von Oswald et al., 2023],(^5^55Their derivation of the in-context gradient step hinges on the probability simplex imposed by the softmax operation.) and failure to represent basic logical operators (XOR) [Richter and Wattenhofer, 2020]. More research into the role of linear combination classes and their effect on a model’s performance or properties is needed to better understand the implications, but we believe this paper has shown the advantages of viewing sequence models through the lens of linear combinations.

## Appendix B Linear Combination Coefficients in Existing Architectures

In the following, we derive the coefficient dynamics of all architectures listed in Table [1](https://arxiv.org/html/2510.09389v1#S3.T1). To do this, we make use of a model’s representation for a single output token (e.g. ([5](https://arxiv.org/html/2510.09389v1#A2.E5))) and ([4](https://arxiv.org/html/2510.09389v1#S3.E4)) to transfer each model into a linear combination (([2](https://arxiv.org/html/2510.09389v1#S3.E2))).

### B.1 Softmax Attention

###### Lemma 7 (Coefficient Dynamics of Softmax Attention) .

Softmax attention [Vaswani et al., 2017, eq. 1]

$$ $y_{i}=\sum_{j=0}^{i}\frac{\exp(\frac{q_{i}^{\top}k_{j}}{\sqrt{n}})v_{j}}{\sum_{j=1}^{i}\exp(\frac{q_{i}^{\top}k_{j}}{\sqrt{n}})}$ (5) $$

can be written in the form of ([2](https://arxiv.org/html/2510.09389v1#S3.E2)), by setting $A_{i}=\mathbb{I}_{n}$, $b_{j}=\frac{1}{\sqrt{n}}$, $\phi(\cdot)=\exp(\cdot)$, and $\eta_{i}=\sum_{j=1}^{i}\exp(\frac{q_{i}^{\top}k_{j}}{\sqrt{n}})$. The resulting normalized coefficients satisfy $\tilde{\alpha}_{i,j}\geq 0$ and $\sum_{j=1}^{i}\tilde{\alpha}_{i,j}=1$, placing the output in the convex hull of the value vectors.

###### Proof.

First, identify the following parameters, by comparing ([5](https://arxiv.org/html/2510.09389v1#A2.E5)) to ([2](https://arxiv.org/html/2510.09389v1#S3.E2)):

$$ $\alpha_{i,j}=\exp(\frac{q_{i}^{\top}k_{j}}{\sqrt{n}}),\quad\eta_{i}=\sum_{j=1}^{i}\exp(\frac{q_{i}^{\top}k_{j}}{\sqrt{n}}).$ $$

Then, the parameter choices $A_{i}=\mathbb{I}_{n}$, $b_{j}=\frac{1}{\sqrt{n}}$, and $\phi(\cdot)=\exp(\cdot)$ ensure the coefficients $\alpha_{i,j}$ align with ([4](https://arxiv.org/html/2510.09389v1#S3.E4)).

The properties $\tilde{\alpha}_{i,j}\geq 0$ follow from $\exp(\cdot)\geq 0$, and $\sum_{j=1}^{i}\tilde{\alpha}_{i,j}=1$ follows by construction of the normalization factor $\eta_{i}$. Therefore, the output $y_{i}=\sum_{j=1}^{i}\tilde{\alpha}_{i,j}v_{j}$ lies in the convex hull of $\{v_{1},\ldots,v_{i}\}$.
∎

###### Lemma 7 (Coefficient Dynamics of Softmax Attention) .

###### Proof.

### B.2 Linear Attention

###### Lemma 8 (Coefficient Dynamics of Linear Attention) .

Linear attention [Katharopoulos et al., 2020, eq. 4]

$$ $y_{i}=\sum_{j=1}^{i}\frac{\psi(q_{i})^{\top}\psi(\nicefrac{{k_{j}}}{{\sqrt{n}}})v_{j}}{\sum_{j=0}^{i}\psi(q_{i})^{\top}\psi(\nicefrac{{k_{j}}}{{\sqrt{n}}})}$ (6) $$

with $\psi(\cdot)\geq 0$, can be written in the form of ([2](https://arxiv.org/html/2510.09389v1#S3.E2)) by setting $A_{i}=\mathbb{I}_{n}$, $b_{j}=\frac{1}{\sqrt{n}}$, $\phi(\cdot)=\psi(\cdot)\psi(\cdot)$, and $\eta_{i}=\sum_{j=1}^{i}\psi(q_{i})^{\top}\psi(\nicefrac{{k_{j}}}{{\sqrt{n}}})$. The resulting normalized coefficients satisfy $\tilde{\alpha}_{i,j}\geq 0$ and $\sum_{j=1}^{i}\tilde{\alpha}_{i,j}=1$, placing the output in the convex hull of the value vectors.

###### Proof.

First, identify the following parameters, by comparing ([6](https://arxiv.org/html/2510.09389v1#A2.E6)) to ([2](https://arxiv.org/html/2510.09389v1#S3.E2)):

$$ $\alpha_{i,j}=\psi(q_{i})^{\top}\psi(\frac{k_{j}}{\sqrt{n}}),\quad\eta_{i}=\sum_{j=1}^{i}\psi(q_{i})^{\top}\psi(\nicefrac{{k_{j}}}{{\sqrt{n}}}).$ $$

Then, the parameter choices $A_{i}=\mathbb{I}_{n}$, $b_{j}=\frac{1}{\sqrt{n}}$, and $\phi(\cdot)=\psi(\cdot)\psi(\cdot)$(^6^66Katharopoulos et al. [2020] uses $\psi(x)=\textrm{elu}(x)+1$, but there exist many more proposals for $\psi(\cdot)$ in the literature.) ensure the coefficients $\alpha_{i,j}$ align with ([4](https://arxiv.org/html/2510.09389v1#S3.E4)). Note that $\psi(\cdot)$ can be seen as preprocessing steps of the queries and keys, therefore allowing to write the coefficients as $\alpha_{i,j}=\nicefrac{{\tilde{q}_{i}^{\top}\tilde{k}_{j}}}{{\sqrt{n}}}$ with $\tilde{q}_{i}=\psi(q_{i})$ and $\tilde{k}_{j}=\psi(k_{j})$, which reveals that the readout map is effectively the identity as discussed in Section [4.1](https://arxiv.org/html/2510.09389v1#S4.SS1).

The properties $\tilde{\alpha}_{i,j}\geq 0$ follow from $\psi(\cdot)\geq 0$, and $\sum_{j=1}^{i}\tilde{\alpha}_{i,j}=1$ follows by construction of the normalization factor $\eta_{i}$. Therefore, the output $y_{i}=\sum_{j=1}^{i}\tilde{\alpha}_{i,j}v_{j}$ lies in the convex hull of $\{v_{1},\ldots,v_{i}\}$.
∎

###### Lemma 8 (Coefficient Dynamics of Linear Attention) .

###### Proof.

### B.3 Normalized Attention

###### Lemma 9 (Coefficient Dynamics of Normalized Attention) .

Normalized attention [Sieber et al., 2024, eq. 22]

$$ $y_{i}=\sum_{j=1}^{i}\frac{\frac{q_{i}^{\top}k_{j}}{\sqrt{n}}v_{j}}{\exp(W_{\eta}x_{i})}$ (7) $$

can be written in the form of ([2](https://arxiv.org/html/2510.09389v1#S3.E2)) by setting $A_{i}=\mathbb{I}_{n}$, $b_{j}=\frac{1}{\sqrt{n}}$, $\phi(\cdot)=\textrm{Id}(\cdot)$, and $\eta_{i}=\exp(W_{\eta}x_{i})$. The resulting normalized coefficients impose no additional constraints on the output.

###### Proof.

First, identify the following parameters, by comparing ([7](https://arxiv.org/html/2510.09389v1#A2.E7)) to ([2](https://arxiv.org/html/2510.09389v1#S3.E2)):

$$ $\alpha_{i,j}=\frac{q_{i}^{\top}k_{j}}{\sqrt{n}},\quad\eta_{i}=\exp(W_{\eta}x_{i}).$ $$

Then, the parameter choices $A_{i}=\mathbb{I}_{n}$, $b_{j}=\frac{1}{\sqrt{n}}$, and $\phi(\cdot)=\textrm{Id}(\cdot)$ ensure the coefficients $\alpha_{i,j}$ align with ([4](https://arxiv.org/html/2510.09389v1#S3.E4)).

Note that positive preprocessing functions as in linear attention ($\psi(\cdot)\geq 0$) are allowed, which would constrain the normalized coefficients to $\tilde{\alpha}_{i,j}\geq 0$. Otherwise, there are no restrictions on the resulting normalized coefficients $\tilde{\alpha}_{i,j}$, placing the output in the full space $\mathbb{R}^{d_{v}}$.
∎

###### Lemma 9 (Coefficient Dynamics of Normalized Attention) .

###### Proof.

### B.4 Gated Linear Attention (GLA)

###### Lemma 10 (Coefficient Dynamics of Gated Linear Attention (GLA)) .

Gated linear attention [Yang et al., 2023, eq. 3](^7^77Here, we assume the vector product $k_{t}v_{t}$ is well-defined. This is explicit in Yang et al. [2023], but only needed if recurrences are computed, which is not the case in this paper. Therefore, assume here that $k_{t}v_{t}$ computes $k_{t}v_{t}^{\top}$ and that the product $q_{i}^{\top}s_{i}$ is transposed after summation. Then, this is the equivalent setting to [Yang et al., 2023, eq. 3] but with all quantities transposed.)

$$ $s_{t}=\textrm{diag}(\alpha_{t})s_{t-1}+k_{t}v_{t}$ (8) $$

can be written in the form of ([2](https://arxiv.org/html/2510.09389v1#S3.E2)) by setting $A_{i}=\textrm{diag}(\alpha_{i})$, $b_{j}=\frac{1}{\sqrt{n}}$, $\phi(\cdot)=\textrm{Id}(\cdot)$, and $\eta_{i}=1$. The resulting normalized coefficients impose no additional constraints on the output.

###### Proof.

First, we need to unroll ([8](https://arxiv.org/html/2510.09389v1#A2.E8)) to get an elementwise convolutional representation (see Sieber et al. [2024], Dao and Gu [2024]):

$$ $s_{i}=\sum_{j=1}^{i}\left(\prod_{t=j+1}^{i}\textrm{diag}(\alpha_{t})\right)k_{j}v_{j},$ $$

before computing the output by multiplying the queries:

$$ $o_{i}=q_{i}^{\top}\sum_{j=1}^{i}\left(\prod_{t=j+1}^{i}\textrm{diag}(\alpha_{t})\right)k_{j}v_{j}.$ $$

Note that Yang et al. [2023] implicitly assumes that the keys are scaled by $\frac{1}{\sqrt{n}}$,(^8^88See codebase: [https://github.com/fla-org/flash-linear-attention](https://github.com/fla-org/flash-linear-attention).) and rearranging above equation yields

$$ $y_{i}=\sum_{j=1}^{i}q_{i}^{\top}\left(\prod_{t=j+1}^{i}\textrm{diag}(\alpha_{t})\right)\frac{k_{j}}{\sqrt{n}}v_{j}.$ $$

We identify the following parameters by comparing to ([2](https://arxiv.org/html/2510.09389v1#S3.E2))

$$ $\alpha_{i,j}=q_{i}^{\top}\left(\prod_{t=j+1}^{i}\textrm{diag}(\alpha_{t})\right)\frac{k_{j}}{\sqrt{n}},\quad\eta_{i}=1.$ $$

Then, the parameter choices $A_{i}=\textrm{diag}(\alpha_{i})$,(^9^99In Yang et al. [2023] the evolution matrix is computed as the low-rank parameterization $\alpha_{i}=\sigma(W_{\alpha}^{2}W_{\alpha}^{1}x_{i}+b_{\alpha})^{\frac{1}{\tau}}$ with $\tau=16$.) $b_{j}=\frac{1}{\sqrt{n}}$, and $\phi(\cdot)=\textrm{Id}(\cdot)$ ensure the coefficients $\alpha_{i,j}$ align with ([4](https://arxiv.org/html/2510.09389v1#S3.E4)). The resulting normalized coefficients $\tilde{\alpha}_{i,j}$ are unrestricted in sign and magnitude, placing the output in the full space $\mathbb{R}^{d_{v}}$.
∎

###### Lemma 10 (Coefficient Dynamics of Gated Linear Attention (GLA)) .

###### Proof.

### B.5 Mamba-2

###### Lemma 11 (Coefficient Dynamics of Mamba-2) .

Mamba-2 [Dao and Gu, 2024]

$$ $y_{i}=\sum_{j=1}^{i}q_{i}^{\top}\left(\prod_{t=j+1}^{i}\exp(-\Delta_{t}A)\right)\Delta_{j}k_{j}v_{j}$ (9) $$

can be written in the form of ([2](https://arxiv.org/html/2510.09389v1#S3.E2)) by setting $A_{i}=\exp(-\Delta_{i}A)$, $b_{j}=\Delta_{j}$, $\phi(\cdot)=\textrm{Id}(\cdot)$, and $\eta_{i}=1$. The resulting normalized coefficients impose no additional constraints on the output.

###### Proof.

First, note that ([9](https://arxiv.org/html/2510.09389v1#A2.E9)) is directly obtained from state-space duality (SSD; see Sieber et al. [2024], Dao and Gu [2024]). We identify the following parameters by comparing to ([2](https://arxiv.org/html/2510.09389v1#S3.E2))

$$ $\alpha_{i,j}=q_{i}^{\top}\left(\prod_{t=j+1}^{i}\exp(-\Delta_{t}A)\right)\Delta_{j}k_{j},\quad\eta_{i}=1.$ $$

Then, the parameter choices $A_{i}=\exp(-\Delta_{i}A)$, $b_{j}=\Delta_{j}$, and $\phi(\cdot)=\textrm{Id}(\cdot)$ ensure the coefficients $\alpha_{i,j}$ align with ([4](https://arxiv.org/html/2510.09389v1#S3.E4)).

Note that the *discretization step* $\Delta_{j}$ is computed from the input as $\Delta_{j}=\textrm{softplus}(W_{\Delta}x_{i}+\beta)$. The resulting normalized coefficients $\tilde{\alpha}_{i,j}$ are unrestricted in sign and magnitude, placing the output in the full space $\mathbb{R}^{d_{v}}$.
∎

###### Lemma 11 (Coefficient Dynamics of Mamba-2) .

###### Proof.

### B.6 DeltaNet

###### Lemma 12 (Coefficient Dynamics of DeltaNet) .

DeltaNet [Schlag et al., 2021, eq. 24](^10^1010Here, we assume the vector product $k_{t}v_{t}$ is well-defined. This is explicit in Schlag et al. [2021], but only needed if recurrences are computed, which is not the case in this paper. Therefore, assume here that $k_{t}v_{t}$ computes $k_{t}v_{t}^{\top}$ and that the product $q_{i}^{\top}s_{i}$ is transposed after summation. Then, this is the equivalent setting to [Schlag et al., 2021, eq. 24] but with all quantities transposed.)

$$ $s_{t}=s_{t-1}+\beta_{t}k_{t}(v_{t}-k_{t}^{\top}s_{t-1})=(\mathbb{I}-\beta_{t}k_{t}k_{t}^{\top})s_{t-1}+\beta_{t}k_{t}v_{t},$ (10) $$

can be written in the form of ([2](https://arxiv.org/html/2510.09389v1#S3.E2)) by setting $A_{i}=\mathbb{I}-\beta_{i}k_{i}k_{i}^{\top}$, $b_{j}=\frac{\beta_{j}}{\sqrt{n}}$, $\phi(\cdot)=\textrm{Id}(\cdot)$, and $\eta_{i}=1$. The resulting normalized coefficients impose no additional constraints on the output.

###### Proof.

First, we need to unroll ([10](https://arxiv.org/html/2510.09389v1#A2.E10)) to get an elementwise convolutional representation (see Sieber et al. [2024], Dao and Gu [2024]):

$$ $s_{i}=\sum_{j=1}^{i}\left(\prod_{t=j+1}^{i}\mathbb{I}-\beta_{t}k_{t}k_{t}^{\top}\right)\beta_{j}k_{j}v_{j},$ $$

before computing the output by multiplying the queries:

$$ $o_{i}=q_{i}^{\top}\sum_{j=1}^{i}\left(\prod_{t=j+1}^{i}\mathbb{I}-\beta_{t}k_{t}k_{t}^{\top}\right)\beta_{j}k_{j}v_{j}.$ $$

Note that Schlag et al. [2021, Section 3.1] assumes that the keys are scaled by $\frac{1}{\sqrt{n}}$, and rearranging above equation yields

$$ $y_{i}=\sum_{j=1}^{i}q_{i}^{\top}\left(\prod_{t=j+1}^{i}\mathbb{I}-\beta_{t}k_{t}k_{t}^{\top}\right)\frac{\beta_{j}}{\sqrt{n}}k_{j}v_{j}.$ $$

We identify the following parameters by comparing to ([2](https://arxiv.org/html/2510.09389v1#S3.E2))

$$ $\alpha_{i,j}=q_{i}^{\top}\left(\prod_{t=j+1}^{i}\mathbb{I}-\beta_{t}k_{t}k_{t}^{\top}\right)\frac{\beta_{j}}{\sqrt{n}}k_{j},\quad\eta_{i}=1.$ $$

Then, the parameter choices $A_{i}=\mathbb{I}-\beta_{i}k_{i}k_{i}^{\top}$, $b_{j}=\frac{\beta_{j}}{\sqrt{n}}$,(^11^1111The *writing strength* is computed as $\beta_{i}=\sigma(W_{\beta})$, with $\sigma(\cdot)$ the sigmoid function. It can be multiplied by $2$ to allow negative eigenvalues of $A_{i}$.) and $\phi(\cdot)=\textrm{Id}(\cdot)$ ensure the coefficients $\alpha_{i,j}$ align with ([4](https://arxiv.org/html/2510.09389v1#S3.E4)).

As discussed in Schlag et al. [2021, Section 4.2], the keys $k_{i}$ need to be normalized (i.e. $\|k_{i}\|=1$), otherwise the evolution matrix $A_{i}$ is not a proper Householder matrix and its eigenvalues can lie outside the unit circle leading to instability. This normalization can be seen as a preprocessing step of the keys. Additionally, DeltaNet allows preprocessing of keys and queries with positive functions (similar to linear attention, i.e., $\psi(\cdot)\geq 0$), which would constrain the normalized coefficients to $\tilde{\alpha}_{i,j}\geq 0$. Otherwise, there are no restrictions on the resulting normalized coefficients $\tilde{\alpha}_{i,j}$, placing the output in the full space $\mathbb{R}^{d_{v}}$.
∎

###### Lemma 12 (Coefficient Dynamics of DeltaNet) .

###### Proof.

### B.7 Gated DeltaNet

###### Lemma 13 (Coefficient Dynamics of Gated DeltaNet) .

Gated DeltaNet [Yang et al., 2025a, eq. 10](^12^1212Here, we assume the vector product $k_{t}v_{t}$ is well-defined. This is explicit in Yang et al. [2025a], but only needed if recurrences are computed, which is not the case in this paper. Therefore, assume here that $k_{t}v_{t}$ computes $k_{t}v_{t}^{\top}$ and that the product $q_{i}^{\top}s_{i}$ is transposed after summation. Then, this is the equivalent setting to [Yang et al., 2025a, eq. 10] but with all quantities transposed.)

$$ $s_{t}=\alpha_{t}(\mathbb{I}-\beta_{t}k_{t}k_{t}^{\top})s_{t-1}+\beta_{t}k_{t}v_{t},$ (11) $$

can be written in the form of ([2](https://arxiv.org/html/2510.09389v1#S3.E2)) by setting $A_{i}=\alpha_{t}\big(\mathbb{I}-\beta_{i}k_{i}k_{i}^{\top}\big)$, $b_{j}=\frac{\beta_{j}}{\sqrt{n}}$, $\phi(\cdot)=\textrm{Id}(\cdot)$, and $\eta_{i}=1$. The resulting normalized coefficients impose no additional constraints on the output.

###### Proof.

First, we need to unroll ([11](https://arxiv.org/html/2510.09389v1#A2.E11)) to get an elementwise convolutional representation (see Sieber et al. [2024], Dao and Gu [2024]):

$$ $s_{i}=\sum_{j=1}^{i}\left(\prod_{t=j+1}^{i}\alpha_{t}(\mathbb{I}-\beta_{t}k_{t}k_{t}^{\top})\right)\beta_{j}k_{j}v_{j},$ $$

before computing the output by multiplying the queries:

$$ $o_{i}=q_{i}^{\top}\sum_{j=1}^{i}\left(\prod_{t=j+1}^{i}\alpha_{t}(\mathbb{I}-\beta_{t}k_{t}k_{t}^{\top})\right)\beta_{j}k_{j}v_{j}.$ $$

Note that Yang et al. [2025a] implicitly assumes that the keys are scaled by $\frac{1}{\sqrt{n}}$,(^13^1313See codebase: [https://github.com/fla-org/flash-linear-attention](https://github.com/fla-org/flash-linear-attention).) and rearranging above equation yields

$$ $y_{i}=\sum_{j=1}^{i}q_{i}^{\top}\left(\prod_{t=j+1}^{i}\alpha_{t}(\mathbb{I}-\beta_{t}k_{t}k_{t}^{\top})\right)\frac{\beta_{j}}{\sqrt{n}}k_{j}v_{j}.$ $$

We identify the following parameters by comparing to ([2](https://arxiv.org/html/2510.09389v1#S3.E2))

$$ $\alpha_{i,j}=q_{i}^{\top}\left(\prod_{t=j+1}^{i}\alpha_{t}(\mathbb{I}-\beta_{t}k_{t}k_{t}^{\top})\right)\frac{\beta_{j}}{\sqrt{n}}k_{j},\quad\eta_{i}=1.$ $$

Then, the parameter choices $A_{i}=\alpha_{i}(\mathbb{I}-\beta_{i}k_{i}k_{i}^{\top})$,(^14^1414$\alpha_{i}$ is computed equivalently to $\exp(-\Delta_{i}A)$ in Mamba-2.) $b_{j}=\frac{\beta_{j}}{\sqrt{n}}$,(^15^1515The *writing strength* is computed as $\beta_{i}=\sigma(W_{\beta})$, with $\sigma(\cdot)$ the sigmoid function. It can be multiplied by $2$ to allow negative eigenvalues of $A_{i}$.) and $\phi(\cdot)=\textrm{Id}(\cdot)$ ensure the coefficients $\alpha_{i,j}$ align with ([4](https://arxiv.org/html/2510.09389v1#S3.E4)).

As discussed in DeltaNet, the keys $k_{i}$ need to be normalized (i.e. $\|k_{i}\|=1$), otherwise the evolution matrix $A_{i}$ is not a proper Householder matrix and its eigenvalues can lie outside the unit circle leading to instability. This normalization can be seen as a preprocessing step of the keys. The resulting normalized coefficients $\tilde{\alpha}_{i,j}$ are unrestricted in sign and magnitude, placing the output in the full space $\mathbb{R}^{d_{v}}$.
∎

###### Lemma 13 (Coefficient Dynamics of Gated DeltaNet) .

###### Proof.

### B.8 mLSTM

###### Lemma 14 (Coefficient Dynamics of mLSTM) .

mLSTM [Beck et al., 2024, eq. 19 - 27](^16^1616Here, we assume the vector product $k_{t}v_{t}$ is well-defined. This is explicit in Beck et al. [2024], but only needed if recurrences are computed, which is not the case in this paper. Therefore, assume here that $k_{t}v_{t}$ computes $k_{t}v_{t}^{\top}$ and that the product $q_{i}^{\top}s_{i}$ is transposed after summation. Then, this is the equivalent setting to [Beck et al., 2024, eq. 19] but with all quantities transposed.)

$$ $\displaystyle s_{t}$ $\displaystyle=\exp(f_{t})s_{t-1}+\exp(i_{t})k_{t}v_{t},$ (12a) $\displaystyle y_{t}$ $\displaystyle=\frac{o_{t}}{\eta_{t}}q_{t}^{\top}s_{t}$ (12b) $$

can be written in the form pf ([2](https://arxiv.org/html/2510.09389v1#S3.E2)) by setting $A_{i}=\exp(f_{t})$, $b_{j}=\frac{\exp(i_{t})}{\sqrt{n}}$, $\phi(\cdot)=\textrm{Id}(\cdot)$, and $\eta_{i}=\frac{\eta_{i}}{o_{i}}$. The resulting normalized coefficients impose no additional constraints on the output.

###### Proof.

First, we need to unroll ([12a](https://arxiv.org/html/2510.09389v1#A2.E12.1)) to get an elementwise convolutional representation (see Sieber et al. [2024], Dao and Gu [2024]):

$$ $s_{i}=\sum_{j=1}^{i}\left(\prod_{t=j+1}^{i}\exp(f_{t})\right)\exp(i_{j})k_{j}v_{j},$ $$

before computing the output as

$$ $y_{i}=\frac{o_{i}}{\eta_{i}}q_{i}^{\top}\sum_{j=1}^{i}\left(\prod_{t=j+1}^{i}\exp(f_{t})\right)\exp(i_{j})k_{j}v_{j}.$ $$

Note that Beck et al. [2024, eq. 23] assumes that the keys are scaled by $\frac{1}{\sqrt{n}}$, and rearranging above equation yields

$$ $y_{i}=\sum_{j=1}^{i}\frac{o_{i}}{\eta_{i}}q_{i}^{\top}\left(\prod_{t=j+1}^{i}\exp(f_{t})\right)\frac{\exp(i_{j})}{\sqrt{n}}k_{j}v_{j}.$ $$

We identify the following parameters by comparing to ([2](https://arxiv.org/html/2510.09389v1#S3.E2))

$$ $\alpha_{i,j}=q_{i}^{\top}\left(\prod_{t=j+1}^{i}\exp(f_{t})\right)\frac{\exp(i_{j})}{\sqrt{n}}k_{j},\quad\eta_{i}=\frac{\eta_{i}}{o_{i}}.$ $$

Then, the parameter choices $A_{i}=\exp(f_{i})$, $b_{j}=\frac{\exp(i_{j})}{\sqrt{n}}$, and $\phi(\cdot)=\textrm{Id}(\cdot)$ ensure the coefficients $\alpha_{i,j}$ align with ([4](https://arxiv.org/html/2510.09389v1#S3.E4)).

Note that the forget and input gates $f_{i}$, $i_{i}$ are computed by linear input projections and that $\exp(f_{i})$, $\exp(i_{i})$ need to be regularized for numerical stability, since both can grow exponentially fast. In mLSTM this is achieved by multiplying $\exp(m_{i-1}-m_{i})$ and $\exp(m_{i})$ to the forget and input gates, respectively, where $m_{i}$ is computed via another recurrence (for all details see Beck et al. [2024]). The resulting normalized coefficients $\tilde{\alpha}_{i,j}$ are unrestricted in sign and magnitude, placing the output in the full space $\mathbb{R}^{d_{v}}$.
∎

###### Lemma 14 (Coefficient Dynamics of mLSTM) .

###### Proof.

## Appendix C Proofs

This appendix is split into two parts: Section [C.1](https://arxiv.org/html/2510.09389v1#A3.SS1) provides the proofs of all lemmas in the main text and Section [C.2](https://arxiv.org/html/2510.09389v1#A3.SS2) provides the proofs of all corollaries in the main text. For convenience, the lemmas and corollaries are copied before the proofs.

### C.1 Lemmas

#### C.1.1 Lemma 1

###### Lemma 1 .

A recurrent formulation of ([3](https://arxiv.org/html/2510.09389v1#S3.E3)) with finite memory (state) in $\mathbb{R}^{n\times d_{v}}$, which allows simultaneous computation of $\alpha_{i,j}$, exists if and only if $\phi(\cdot):\mathbb{R}\to\mathbb{R}$ is a linear map.

###### Proof.

We prove the two directions, i.e., sufficiency and necessity, separately.

(Necessity.)
If $\phi(\cdot)$ is linear, i.e., $\phi(x)=ax,\,a\in\mathbb{R}$, then

$$ $y_{i}=\sum_{j=1}^{i}\frac{a\,q_{i}^{\top}h_{i,j}}{\eta_{i}}\,v_{j}=\frac{a\,q_{i}^{\top}}{\eta_{i}}\sum_{j=1}^{i}h_{i,j}\,v_{j}^{\top},$ $$

where the second equality is due to linearity and equivalent to the reformulation of linear attention in Katharopoulos et al. [2020].
Then, expanding $h_{i,j}$ using ([3](https://arxiv.org/html/2510.09389v1#S3.E3)) yields

$$ $y_{i}=\frac{a\,q_{i}^{\top}}{\eta_{i}}\sum_{j=1}^{i}\left(\prod_{t=j+1}^{i}A_{t}\right)b_{j}k_{j}\,v_{j}^{\top}.$ $$

Define the $n\times d_{v}$ matrix $S_{i}\coloneqq k_{i}v_{i}^{\top}$ and note that
$S_{i}$ admits recurrent updates to compute the sum, i.e.,

$$ $\displaystyle S_{i}$ $\displaystyle=A_{i}S_{i-1}+b_{i}k_{i}v_{i}^{\top},$ $\displaystyle y_{i}$ $\displaystyle=\frac{a\,q_{i}^{\top}}{\eta_{i}}S_{i},$ $$

Therefore, a recurrent implementation with finite state (memory) $S_{i}\in\mathbb{R}^{n\times d_{v}}$ exists if $\phi(\cdot)$ is linear.

(Sufficiency.)
Assume a finite state (memory) recurrent implementation exists, i.e.,

$$ $\displaystyle S_{i}$ $\displaystyle=A_{i}S_{i-1}+b_{i}k_{i}v_{i}^{\top},$ (13a) $\displaystyle y_{i}$ $\displaystyle=\frac{a\,q_{i}^{\top}}{\eta_{i}}S_{i},$ (13b) $$

where $a\in\mathbb{R}$ is a scaling factor. We prove that $\phi(\cdot)$ must be linear by contradiction, thus suppose $\phi(\cdot)$ is any nonlinear function on $\mathbb{R}$, such that the output $y_{i}$ can be computed as

$$ $y_{i}=\sum_{j=1}^{i}\frac{\phi(q_{i}^{\top}h_{i,j})}{\eta_{i}}\,v_{j}.$ (14) $$

Hence, the expression from $y_{i}$ derived from rolling out ([13](https://arxiv.org/html/2510.09389v1#A3.E13)) must be equal to ([14](https://arxiv.org/html/2510.09389v1#A3.E14)):

$$ $\frac{a\,q_{i}^{\top}}{\eta_{i}}\sum_{j=1}^{i}\left(\prod_{t=j+1}^{i}A_{t}\right)b_{j}k_{j}\,v_{j}^{\top}=\sum_{j=1}^{i}\frac{phi(q_{i}^{\top}h_{i,j})}{\eta_{i}}\,v_{j},$ $$

and by simplifying the equation above with the definition of $h_{i,j}$ (([3](https://arxiv.org/html/2510.09389v1#S3.E3))) we have

$$ $\sum_{j=1}^{i}\frac{a\,q_{i}^{\top}h_{i,j}}{\eta_{i}}v_{j}^{\top}=\sum_{j=1}^{i}\frac{\phi(q_{i}^{\top}h_{i,j})}{\eta_{i}}\,v_{j}.$ $$

This equality does not hold for any nonlinear function $\phi(\cdot)$, but only a linear one, thus concluding the proof.
∎

###### Lemma 1 .

###### Proof.

#### C.1.2 Lemma 2

###### Lemma 2 .

Let $\phi:\mathbb{R}\to\mathbb{R}$ be the readout map in ([3](https://arxiv.org/html/2510.09389v1#S3.E3)) with connected zero-level set $\mathcal{Z}=\{z\in\mathbb{R}\mid\phi(z)=0\}$, and let $z=q_{i}^{\top}h_{i,j}$, with $T_{q}$, $T_{h,i}$ defined in ([4](https://arxiv.org/html/2510.09389v1#S3.E4)). Let $\mathcal{T}$ be the set of linear transformations that achieve the zero-level set, i.e., $\mathcal{T}=\{(T_{q},T_{h,i})\mid q_{i}^{\top}h_{i,j}\in\mathcal{Z}\ s.t.\ q_{i}=T_{q}x_{i}$, $h_{i,j}=T_{h,i}x_{j}\}$ for any pair of inputs $x_{i},x_{j}\in\mathbb{R}^{d}$. Then, the measure of $\mathcal{T}$ is directly proportional to the measure of the zero-level set $\mathcal{Z}$, i.e., $\lvert\mathcal{T}\rvert=c\lvert\mathcal{Z}\rvert$ with $c>0$.

###### Proof.

Given that the set $\mathcal{Z}$ is connected on $\mathbb{R}$, the set is defined by the interval $\mathcal{Z}=[\underline{z},\overline{z}]\subset\mathbb{R}$. Then, since $T_{q},T_{h,i}\in\mathbb{R}^{n\times d}$, we consider the Euclidean space $E^{2nd}$ of pairs $(T_{q},T_{h,i})$, equipped with the Lebesgue measure $\lambda(E)$. We define the map

$$ $F:E\to\mathbb{R},\qquad F(T_{q},T_{h,i})=(T_{q}x_{i})^{\top}(T_{h,i}x_{j}),$ $$

which is smooth and its derivative $\nabla F(T_{q},T_{h,i})$ is non-zero for $x_{i}\neq 0$ and $x_{j}\neq 0$; we treat the trivial case of either $x_{i}=0$ or $x_{j}=0$ at the end of the proof.

Then, for each $z\in[\underline{z},\overline{z}]$, we construct the subset $\mathcal{T}_{z}$ as a fiber(^17^1717The fiber $f^{-1}(y)$ of a function $f$ is the preimage of the singleton $y$.) of $F$, i.e.,

$$ $\mathcal{T}_{z}:=F^{-1}(\{z\})=\{(T_{q},T_{h,i})\in E\mid(T_{q}x_{i})^{\top}(T_{h,i}x_{j})=z\}.$ $$

Therefore, the set of all linear transformations that achieve a value in $\mathcal{Z}$ is

$$ $\mathcal{T}=F^{-1}(\mathcal{Z})=\bigcup\limits_{z\in\mathcal{Z}}\mathcal{T}_{z}.$ $$

To prove the lemma, we need to compute the Lebesgue measure of $\mathcal{T}$(^18^1818Intuitively, the volume of the set $\mathcal{T}$.), i.e., $\lvert\mathcal{T}\rvert=\int_{E}\mathbf{1}_{F^{-1}(\mathcal{Z})}\lambda(t)$, where $t=(T_{q},T_{h,i})$ and $\mathbf{1}_{F^{-1}(\mathcal{Z})}$ is the indicator function selecting all $t\in\mathcal{T}$. Since $x_{i},x_{j}\neq 0$, $F$ is smooth and $\nabla F\neq 0$ on $E$, we can apply the coarea formula [Federer, 1959], i.e.,

$$ $\int_{E}\mathbf{1}_{F^{-1}(\mathcal{Z})}(t)\,\lambda(t)=\int_{\mathcal{Z}}\left(\int_{F^{-1}(z)}\frac{1}{\|\nabla F(t)\|}\,dH^{2nd-1}(t)\right)dz$ $$

where $H^{2nd-1}$ is the $(2nd-1)$-dimensional Hausdorff measure and $\|\nabla F(t)\|$ is the Euclidean norm of the gradient of $F$ at $t$. We then define

$$ $g(z)\coloneqq\int_{F^{-1}(z)}\frac{1}{|\nabla F(t)|}\,d\mathcal{H}^{2nd-1}(t),$ $$

which denotes the size of each subset (fiber) $\mathcal{T}_{z}$. Therefore, the measure of $\mathcal{T}$ is

$$ $\lvert\mathcal{T}\rvert=\int_{\mathcal{Z}}g(z)dz,$ $$

and by choosing the constant $c$ as

$$ $c=\frac{1}{\lvert\mathcal{Z}\rvert}\int_{\mathcal{Z}}g(z)dz,$ $$

gives the desired relation $\lvert\mathcal{T}\rvert=c\lvert\mathcal{Z}\rvert$.

In case $x_{i},x_{j}=0$, the dot product $q_{i}^{\top}h_{i,j}$ is trivially zero, thus $F(t)=0,\,\forall t=(T_{q},T_{h,i})\in E$. In this case, $\lvert\mathcal{T}\rvert=0$, if $0\notin\mathcal{Z}$, i.e., empty because $z=0$ is not contained in the zero-level set, or $\lvert\mathcal{T}\rvert=\lvert E\rvert$, i.e., the measure of the Euclidean space $E$.
∎

###### Remark 2 .

To intuitively understand Lemma [2](https://arxiv.org/html/2510.09389v1#Thmlemma2) and its proof, consider the simplest case: $q_{i}=T_{q}x_{i}=ax_{i}\in\mathbb{R}$, $h_{i,j}=T_{h,i}x_{j}=bx_{j}\in\mathbb{R}$, thus all variables are scalars. Additionally, consider a readout function $\phi(\cdot)$ with zero-level set $\mathcal{Z}=[-1,0]$. In this setting, we want to find all pairs $t=(a,b)$ such that $ab(x_{i}x_{j})=z$ and we define $X=x_{i}x_{j}\neq 0$. Then, all $t$ that achieve $z$ in the zero-level set, need to lie on the hyperbola

$$ $a\cdot b=\frac{z}{X}.$ $$

This hyperbola exactly describes the set $\mathcal{T}_{z}$ in the proof and there are infinitely many combinations $(a,b)$ that lie in this set. To get to $\mathcal{T}$, we need to integrate over the zero-level set $\mathcal{Z}$ to account for all possible $z$. These sets (hyperbolas) are also visualized in the figure on the right.

###### Lemma 2 .

###### Proof.

###### Remark 2 .

#### C.1.3 Lemma 3

###### Lemma 3 .

The coefficients $\alpha_{i,j}$ in ([4](https://arxiv.org/html/2510.09389v1#S3.E4)) have positional information if and only if $A_{i}\neq\mathbb{I}_{n},\forall i$.

###### Proof.

Consider the definition of the coefficients (([4](https://arxiv.org/html/2510.09389v1#S3.E4))):

$$ $\alpha_{i,j}=\phi\left(q_{i}^{\top}\left(\prod_{t=j+1}^{i}A_{t}\right)b_{j}k_{j}\right).$ $$

We first consider the case $A_{t}=\mathbb{I}_{n},\forall t$, then

$$ $\alpha_{i,j}=\phi\left(q_{i}^{\top}b_{j}k_{j}\right).$ $$

Assuming the scaling factors are computed from the input $b_{j}=f(x_{j})$, the factors $b_{j}$ contain no positional information. Specifically, because $b_{j}$ is computed from the same input $x_{j}$ as $k_{j}$, we can rewrite the key using a change of variable $\tilde{k}_{j}=b_{j}k_{j}=g(x_{j})$, i.e., the new key is computed from a single input $x_{j}$, albeit in a nonlinear fashion. Therefore, we consider the simplified coefficients

$$ $\bar{\alpha}_{i,j}=\phi\left(q_{i}^{\top}\tilde{k}_{j}\right).$ $$

Given two keys $\tilde{k}_{j}$, $\tilde{k}_{\bar{j}}$ computed from the same input $x_{j}$ (i.e., $\tilde{k}_{j}=\tilde{k}_{\bar{j}}$), the coefficients for these two keys are equivalent $\bar{\alpha}_{i,j}=\bar{\alpha}_{i,\bar{j}}$ and thus the coefficients have no positional information, i.e., the coefficients cannot be discriminated depending on the index $j$.

Now, consider the case $A_{t}\neq\mathbb{I}_{n},\forall t$, then

$$ $\alpha_{i,j}=\phi\left(q_{i}^{\top}\left(\prod_{t=j+1}^{i}A_{t}\right)b_{j}k_{j}\right).$ $$

Using the same simplification for $b_{j}$ as above and let $A_{i,j}=\prod_{t=j+1}^{i}A_{t}$, we get

$$ $\bar{\alpha}_{i,j}=\phi\left(q_{i}^{\top}A_{i,j}\tilde{k}_{j}\right).$ $$

The evolution $A_{i,j}$ contains information about the (time) index evolution from $j$ to $i$, since it defines a map $f:\mathbb{R}^{n}\to\mathbb{R}^{n}$ that maps any vector $a_{j}\in\mathbb{R}^{n}$ to $a_{i}=A_{i}\cdot\dots\cdot A_{j+1}a_{j}$, i.e., describes the evolution of $a_{j}$ from (time) index $j$ to $i$. Consider again two keys $\tilde{k}_{j}$, $\tilde{k}_{\bar{j}}$ computed from the same input $x_{j}$ (i.e., $\tilde{k}_{j}=\tilde{k}_{\bar{j}}$), then their respective coefficients are not equivalent due to $A_{i,j}$:

$$ $\bar{\alpha}_{i,j}=\phi\left(q_{i}^{\top}A_{i,j}\tilde{k}_{j}\right)\neq\phi\left(q_{i}^{\top}A_{i,\bar{j}}\tilde{k}_{\bar{j}}\right)=\bar{\alpha}_{i,\bar{j}}.$ $$

Specifically, if $\bar{j}>j$, then the following relation holds

$$ $A_{i,j}=\prod_{t=j+1}^{i}A_{t}=\prod_{t=\bar{j}+1}^{i}A_{t}\prod_{t=j+1}^{\bar{j}}A_{t}=A_{i,\bar{j}}A_{\bar{j},j},$ $$

and thus there exists a linear relation between $A_{i,j}$ and $A_{i,\bar{j}}$. Therefore, the coefficients have positional information, quantified by $A_{\bar{j},j}$.
∎

###### Remark 3 .

The proof of Lemma [3](https://arxiv.org/html/2510.09389v1#Thmlemma3) relies on the fact that keys and parameters such as $b_{j}$ are computed from a single input $x_{j}$. However, this is not true if the inputs are preprocessed by a short convolution, i.e., $\{\bar{x}_{j}\}_{j=1}^{i}=\Phi\star\{x_{j}\}_{j=1}^{i}$, where $\Phi$ denotes the convolution weights. In this case, $\bar{x}_{j}$ contains scaled copies of previous inputs, depending on the convolution dimension, e.g., $\bar{x}_{j}=\Phi_{0}x_{j}+\Phi_{1}x_{j-1}+\Phi_{2}x_{j-2}+\Phi_{3}x_{j-3}$. Therefore, preprocessing the inputs with a convolution can offer another way to embed positional information in the coefficients $\alpha_{i,j}$. However, note that this "moving averaging" of inputs might not work for certain repeating input signals.

###### Lemma 3 .

###### Proof.

###### Remark 3 .

#### C.1.4 Proof of Lemma 4

###### Lemma 4 .

Imposing any of the following structures on $A_{i}$ in ([3](https://arxiv.org/html/2510.09389v1#S3.E3)), results in the corresponding allowed transformations:

- a)
$A_{i}=\lambda_{i}\mathbb{I}_{n},\lambda_{i}\in\mathbb{R},\lvert\lambda_{i}\rvert\leq 1$; allows scaling of keys (including flipping, if $\lambda_{i}<0$) uniformly along all dimensions $n$,
- b)
$A_{i}=\textrm{diag}(\lambda_{i}),\lambda_{i}\in\mathbb{R}^{n},\lvert\lambda^{(r)}_{i}\rvert\leq 1$ where $\lambda^{(r)}$ denotes the $r$-th entry of $\lambda_{i}$; allows scaling of keys (including flipping, if $\lambda^{(r)}_{i}<0$) separately along dimensions $n$,
- c)
$A_{i}=\mathbb{I}_{n}-\beta_{i}\lambda_{i}\lambda_{i}^{\top},\beta_{i}\in[0,2],\lambda_{i}\in\mathbb{R}^{n},$ (Householder matrix); allows scaling and specific rotations of keys.

###### Proof.

All three items follow by a direct computation of $A_{i}$’s action on an arbitrary key vector $k_{j}\in\mathbb{R}^{n}$ and by inspecting eigenvalues/eigenvectors.

(a) Uniform scaling. If $A_{i}=\lambda_{i}\mathbb{I}_{n}$ then for any key $k_{j}$ we have

$$ $A_{i}k_{j}=\lambda_{i}\mathbb{I}_{n}k_{j}=\lambda_{i}k_{j},$ $$

so $A_{i}$ scales $k_{j}$ uniformly in every coordinate by the scalar $\lambda_{i}$. If $\lvert\lambda_{i}\rvert\leq 1$ the operator is a contraction; if $\lambda_{i}<0$ it also flips the sign of every component (a global reflection through the origin).

(b) Coordinate-wise scaling. If $A_{i}=\operatorname{diag}(\lambda_{i}^{(1)},\dots,\lambda_{i}^{(n)})$, then

$$ $(A_{i}k_{j})^{(r)}=\lambda_{i}^{(r)}\,k_{j}^{(r)},\qquad r=1,\dots,n.$ $$

Thus each coordinate is scaled independently by $\lambda_{i}^{(r)}$. The constraints $\lvert\lambda_{i}^{(r)}\rvert\leq 1$ render the operation a contraction in each coordinate; negative entries produce sign flips on the corresponding coordinate only.

(c) Rank-one update / Householder-type transform.
Given $A_{i}=\mathbb{I}_{n}-\beta_{i}\lambda_{i}\lambda_{i}^{\top}$, we decompose any $k_{j}\in\mathbb{R}^{n}$ into components parallel and orthogonal to $\lambda_{i}$:

$$ $k_{j}=k_{j}^{\parallel}+k_{j}^{\perp},\qquad k_{j}^{\parallel}=\frac{\lambda_{i}^{\top}k_{j}}{\|\lambda_{i}\|^{2}}\,\lambda_{i},\quad\lambda_{i}^{\top}k_{j}^{\perp}=0,$ $$

where $\|\cdot\|$ as the Euclidean in $\mathbb{R}^{n}$. Then

$$ $A_{i}k_{j}=(\mathbb{I}_{n}-\beta_{i}\lambda_{i}\lambda_{i}^{\top})(k_{j}^{\parallel}+k_{j}^{\perp})=(1-\beta_{i}\|\lambda_{i}\|^{2})k_{j}^{\parallel}+k_{j}^{\perp}.$ $$

Hence $k_{j}^{\perp}$ (every vector orthogonal to $\lambda_{i}$) is an eigenvector with eigenvalue $1$, and the direction $\lambda_{i}$ itself is an eigenvector with eigenvalue $1-\beta_{i}\|\lambda_{i}\|^{2}$. Therefore, $A_{i}$ acts by scaling the component along $\lambda_{i}$ by the factor $1-\beta_{i}\|\lambda_{i}\|^{2}$, while leaving the orthogonal complement unchanged.

Special cases and remarks:

- •
If one chooses $\beta_{i}=\nicefrac{{2}}{{\|\lambda_{i}\|^{2}}}$ (and $\lambda_{i}\neq 0$), then $1-\beta_{i}\|\lambda_{i}\|^{2}=-1$ and
$A_{i}=\mathbb{I}_{n}-\frac{2}{\|\lambda_{i}\|^{2}}\,\lambda_{i}\lambda_{i}^{\top}$
is a classical Householder reflection (an orthogonal matrix with determinant $-1$) that reflects across the hyperplane orthogonal to $\lambda_{i}$.
- •
For general $\beta_{i}\in[0,2]$ and $\|\lambda_{i}\|$ normalized (or when $\beta_{i}\|\lambda_{i}\|^{2}\in[0,2]$), the eigenvalue $1-\beta_{i}\|\lambda_{i}\|^{2}$ lies in $[-1,1]$, so the operator is a contraction on the $\lambda_{i}$-direction and may flip the sign (reflection), if the eigenvalue is negative.
- •
Although a single rank-one transform of this form does not produce an arbitrary rotation, compositions of Householder reflections (each of which is of the form $\mathbb{I}_{n}-2\lambda\lambda^{\top}$ with unit $\lambda\in\mathbb{R}^{n}$ can generate any orthogonal matrix (rotations and reflections). Hence, by composing appropriate special cases of the matrices in (c), orthogonal rotations can be realized.

Combining these elementary computations proves that each structural constraint on $A_{i}$ yields the stated class of allowed transformations.
∎

###### Remark 4 .

In Lemma [4](https://arxiv.org/html/2510.09389v1#Thmlemma4) and the proof above, we make no assumption on the magnitude of $\lambda_{i}$ in a Householder-type evolution matrix. However, as shown in the proof, assuming unit $\lambda_{i}$ is necessary to define all transformations. This is the reason why the keys $k_{j}$ and queries $q_{i}$ are L2 normalized in Yang et al. [2024, 2025a] for the computation of $A_{i}$.

###### Lemma 4 .

###### Proof.

###### Remark 4 .

#### C.1.5 Proof of Lemma 5

###### Lemma 5 .

Consider two i.i.d. normally distributed random vectors $x_{i},x_{j}\sim\mathcal{N}(0,\Sigma)$. Then, the dot product $q_{i}^{\top}h_{i,j}$ with $q_{i},h_{i,j}$ defined in ([4](https://arxiv.org/html/2510.09389v1#S3.E4)), has zero-mean and variance $\textrm{Var}(q_{i}^{\top}h_{i,j})=\textrm{tr}(\Sigma_{q}\Sigma_{h})$ with $\Sigma_{q}=T_{q}\Sigma T_{q}^{\top}$, $\Sigma_{h}=T_{h,i}\Sigma T_{h,i}^{\top}$.
Assuming $\Sigma=\sigma\mathbb{I}_{d}$, both $\Sigma_{q}$ and $\Sigma_{h}$ are positive semi-definite and the variance of the dot product scales as $\textrm{Var}(q_{i}^{\top}h_{i,j})=\mathcal{O}(n)$.

###### Proof.

We proof the lemma in three parts, first zero mean, then the variance relation, and finally the scaling under the isotropy assumption.

Mean.
Since $x_{i}$ and $x_{j}$ are independent and zero-mean,

$$ $\mathbb{E}[q_{i}^{\top}h_{i,j}]=\mathbb{E}[(T_{q}x_{i})^{\top}(T_{h,i}x_{j})]=\mathbb{E}_{x_{i}}\mathbb{E}_{x_{j}}[x_{i}^{\top}T_{q}^{\top}T_{h,i}x_{j}]=0,$ $$

hence the mean is zero.

Variance.
Since the mean is zero,

$$ $\mathrm{Var}(q_{i}^{\top}h_{i,j})=\mathbb{E}\!\left[(q_{i}^{\top}h_{i,j})^{2}\right]=\mathbb{E}\!\left[(T_{q}x_{i})^{\top}(T_{h,i}x_{j})(T_{h,i}x_{j})^{\top}(T_{q}x_{i})\right].$ $$

Conditioning on $x_{i}$ and taking expectation over $x_{j}$ first,

$$ $\mathbb{E}_{x_{j}}\!\left[(T_{h,i}x_{j})(T_{h,i}x_{j})^{\top}\right]=T_{h,i}\,\mathbb{E}[x_{j}x_{j}^{\top}]\,T_{h,i}^{\top}=T_{h,i}\Sigma T_{h,i}^{\top}\coloneqq\Sigma_{h},$ $$

and due to symmetry $\mathbb{E}_{x_{i}}\!\left[(T_{q}x_{i})(T_{q}x_{i})^{\top}\right]=T_{q}\Sigma T_{q}^{\top}=\Sigma_{q}$. Therefore,

$$ $\mathbb{E}\!\left[(q_{i}^{\top}h_{i,j})^{2}\right]=\mathbb{E}_{x_{i}}\!\left[(T_{q}x_{i})^{\top}\Sigma_{h}(T_{q}x_{i})\right]=\mathbb{E}_{x_{i}}\!\left[\mathrm{tr}\!\big(\Sigma_{h}(T_{q}x_{i})(T_{q}x_{i})^{\top}\big)\right]=\mathrm{tr}\!\big(\Sigma_{q}\Sigma_{h}\big),$ $$

due to the previous relations and the cyclic property of the trace operator.

Isotropy.
If $\Sigma=\sigma\mathbb{I}_{d}$, then

$$ $\Sigma_{q}=\sigma T_{q}T_{q}^{\top},\qquad\Sigma_{h}=\sigma T_{h}T_{h}^{\top},$ $$

which are symmetric positive semidefinite by definition of matrix outer products [Friedberg et al., 1997]. Therefore,

$$ $\mathrm{Var}(q_{i}^{\top}h_{i,j})=\mathrm{tr}\!\big(\Sigma_{q}\Sigma_{h}\big)=\sigma^{2}\mathrm{tr}\!\big(T_{q}T_{q}^{\top}T_{h,i}T_{h,i}^{\top}\big)=\sigma^{2}\|T_{q}^{\top}T_{h,i}\|^{2}_{F},$ $$

where we used the cyclic property of the trace and the identity $\|M\|_{F}=\mathrm{tr}(MM^{\top})$ [Friedberg et al., 1997], with $\|\cdot\|_{F}$ denoting the Frobenius norm. To prove the scaling statement, we assume $T_{q}$, $T_{h,i}$ are properly normalized weight matrices (i.e. the matrices have entries $\mathcal{O}(\nicefrac{{1}}{{\sqrt{n}}})$, such that the output variance remains $\mathcal{O}(1)$). This is assumption is common in practice. Therefore, we have $\|T_{q}\|_{F}=\mathcal{O}(\sqrt{n})$, $\|T_{h,i}\|_{F}=\mathcal{O}(\sqrt{n})$ and thus $\|T_{q}^{\top}T_{h,i}\|^{2}_{F}=\mathcal{O}(n)$ and $\mathrm{Var}(q_{i}^{\top}h_{i,j})=\mathcal{O}(n)$. Importantly, given common normalization choices for $T_{q}$, $T_{h,i}$, which keep per-coordinate variances to $\mathcal{O}(1)$, the dot product variance grows linearly in $n$.
∎

###### Lemma 5 .

###### Proof.

#### C.1.6 Proof of Lemma 6

###### Lemma 6 .

Consider a fixed index $j$ and let $s_{i}\coloneqq q_{i}^{\top}h_{i,j}$, $\alpha_{i}=\phi(s_{i})$ in ([3](https://arxiv.org/html/2510.09389v1#S3.E3)). Let $\phi:\mathbb{R}\to[0,\infty)$ be nondecreasing and unbounded, and let $g:[0,\infty)\to(0,\infty)$ be an increasing and unbounded comparison function, such that $L\coloneqq\limsup_{i\to\infty}\frac{\phi(s_{i})}{g(s_{i})}<\infty$. If $\eta_{i}$ is chosen such that $\liminf_{i\to\infty}\frac{\eta_{i}}{g(s_{i})}\coloneqq m>0$, then the normalized coefficients are bounded: $\limsup_{i\to\infty}\frac{\alpha_{i}}{\eta_{i}}\leq\frac{L}{m}$.

###### Proof.

Fix $\varepsilon\in\bigl(0,\min\{1,m\}\bigr)$. By the definition of the limit superior, there exists $N_{1}\in\mathbb{N}$ such that for all $i\geq N_{1}$,

$$ $\frac{\phi(s_{i})}{g(s_{i})}\;\leq\;L+\varepsilon.$ $$

By the definition of the limit inferior, there exists $N_{2}\in\mathbb{N}$ such that for all $i\geq N_{2}$,

$$ $\frac{\eta_{i}}{g(s_{i})}\;\geq\;m-\varepsilon.$ $$

Hence for all $i\geq N:=\max\{N_{1},N_{2}\}$ we have

$$ $\frac{\alpha_{i}}{\eta_{i}}=\frac{\phi(s_{i})}{\eta_{i}}=\frac{\phi(s_{i})}{g(s_{i})}\cdot\frac{g(s_{i})}{\eta_{i}}\leq\frac{L+\varepsilon}{m-\varepsilon}.$ $$

Since finitely many initial indices do not affect the limit superior, letting $\varepsilon\to 0$ yields

$$ $\limsup_{i\to\infty}\frac{\alpha_{i}}{\eta_{i}}\leq\frac{L}{m},$ $$

which proves the lemma.
∎

###### Remark 5 .

Assuming the normalization factor $\eta_{i}$ is a function of the coefficients themselves, i.e., $\eta_{i}=f(\alpha_{i,j})$, simplifies Lemma [6](https://arxiv.org/html/2510.09389v1#Thmlemma6) significantly, since we only need to analyze $\frac{\alpha_{i}}{f(\alpha_{i})}$ (without comparison function $g$). Hence, if $f(\cdot)$ grows linearly, $\nicefrac{{\alpha_{i}}}{{f(\alpha_{i})}}$ converges to a constant $m>0$; if $f(\cdot)$ grows superlinearly, $\nicefrac{{\alpha_{i}}}{{f(\alpha_{i})}}$ converges to 0; and $f(\cdot)$ grows sublinearly, $\nicefrac{{\alpha_{i}}}{{f(\alpha_{i})}}$ blows up.

###### Lemma 6 .

###### Proof.

###### Remark 5 .

### C.2 Corollaries

#### C.2.1 Proof of Corollary 2.1

###### Corollary 2.1 .

Consider the kernel approximation of $\phi(\cdot):\mathbb{R}^{n}\to\mathbb{R}^{q}$ to be $\tilde{\phi}(q_{i}^{\top}h_{i,j})=\psi_{q}(q_{i})^{\top}\psi_{h}(h_{i,j})$, with $\psi_{q}:\mathbb{R}^{n}\to\mathbb{R}^{q}$ and $\psi_{h}:\mathbb{R}^{n}\to\mathbb{R}^{q}$, such that it approximates the readout map $\phi(\cdot)\approx\tilde{\phi}(\cdot)$ in ([3](https://arxiv.org/html/2510.09389v1#S3.E3)). Then, the zero-level set of the approximation $\tilde{\phi}(\cdot)$ is the singleton $\mathcal{Z}=\{0\}$ and by Lemma [2](https://arxiv.org/html/2510.09389v1#Thmlemma2) the set of linear transformations $\mathcal{T}$ is small.

###### Proof.

Given that the kernel approximation is defined as $\tilde{\phi}(q_{i}^{\top}h_{i,j})=\psi_{q}(q_{i})^{\top}\psi_{h}(h_{i,j})$, we can rewrite this approximation using a change of variables as

$$ $\tilde{\phi}(q_{i}^{\top}h_{i,j})=\tilde{q}_{i}^{\top}\tilde{h}_{i},j,\quad\textrm{with}\;\tilde{q}_{i}\coloneqq\psi_{q}(q_{i}),\ \tilde{h}_{i,j}\coloneqq\psi_{h}(h_{i},j).$ (15) $$

Note that ([15](https://arxiv.org/html/2510.09389v1#A3.E15)) is equivalent to a dot product with an identity readout map $\textrm{Id}(\cdot)$, albeit in different dimensions than the untransformed dot product $q_{i}^{\top}h_{i,j}$ ($\tilde{q}_{i}\in\mathbb{R}^{q}$ compared to $q_{i}\in\mathbb{R}^{n}$). Given that the zero-level set of a dot product is the singleton $\{0\}$, the zero-level set of $\tilde{\phi}(\cdot)$ is $\mathcal{Z}=\{0\}$. Hence, $\lvert\mathcal{Z}\rvert=0$ and the measure of the set of linear transformations reduces to $\lvert\mathcal{T}\rvert=g(\{0\})$ (see proof of Lemma [2](https://arxiv.org/html/2510.09389v1#Thmlemma2) for the exact definitions). Therefore, $\mathcal{T}$ is reduced to a single fiber, i.e., $\mathcal{T}=\{(T_{q},T_{h,i})\mid(T_{q}x_{i})^{\top}(T_{h,i}x_{j})=0\}$.
∎

###### Corollary 2.1 .

###### Proof.

#### C.2.2 Proof of Corollary 2.2

###### Corollary 2.2 .

Let $y_{L}\in\mathbb{R}^{d_{v}}$ be the solution to ([2](https://arxiv.org/html/2510.09389v1#S3.E2)), where $\alpha_{L,j}:\mathbb{R}^{n}\rightarrow\mathbb{R};\ q_{L}\mapsto\alpha_{L,j}(q_{L})$ is defined in ([3](https://arxiv.org/html/2510.09389v1#S3.E3)) with identity readout map and normalization factor, i.e., $\phi(\cdot)=\operatorname{Id}(\cdot)$, $\eta_{L}=1$. Consider the set of linearly independent states $\mathcal{H}=\{h_{L,t}\mid c_{1}h_{L,1}+\dots+c_{t}h_{L,t}=0\ \implies\ c_{1}=\dots=c_{t}=0\}$. Then, a nonzero $q_{L}$ that achieves $\alpha_{L,j}=0,\,\forall h_{L,j}\in\mathcal{H}$ exists if and only if $\dim\big(\operatorname{span}\{h_{L,j}\in\mathcal{H}\}\big)<n$. In particular, given a nonzero $q_{L}$, the measure $\lvert\mathcal{H}\rvert\leq n-1$.

###### Proof.

The conditions $\alpha_{L,j}=q_{L}^{\top}h_{L,j}=0,\ \forall h_{L,j}\in\mathcal{H}$ are equivalent to the linear homogeneous system

$$ $H^{\top}q_{L}=0,$ (16) $$

where $H\in\mathbb{R}^{n\times\lvert\mathcal{H}\rvert}$ with $\lvert\mathcal{H}\rvert$ the measure of set $\mathcal{H}$,(^19^1919This means the number of vectors $h_{L,j}$ contained in the set.) is a matrix whose columns consist of the vectors $h_{L,j}\in\mathcal{H}$. The solution space for $q_{L}$ is the nullspace of $H^{\top}$. By the rank–nullity theorem [Friedberg et al., 1997, Theorem 2.3], we get

$$ $\dim\big(\ker(H^{\top})\big)=n-\operatorname{rank}(H^{\top})=n-\operatorname{rank}(H).$ $$

Hence a nonzero $q_{L}$ solving ([16](https://arxiv.org/html/2510.09389v1#A3.E16)) exists if and only if $\operatorname{rank}(H)<n$, i.e., if and only if the vectors $h_{L,j}\in\mathcal{H}$ span a proper subspace of $\mathbb{R}^{n}$.

Consequently, the largest possible number of linearly independent constraints (equivalently, independent $h_{L,j}$) that admit a nonzero solution is $n-1$. Equivalently, at most $n-1$ independent coefficients $\alpha_{L,j}$ can be forced to zero simultaneously, with a nonzero choice of $q_{L}$.
∎

###### Corollary 2.2 .

###### Proof.

#### C.2.3 Proof of Corollary 6.1

###### Corollary 6.1 .

Choosing $\eta_{i}=\sum_{j=1}^{i}\alpha_{i,j}$ in ([2](https://arxiv.org/html/2510.09389v1#S3.E2)) constrains the normalized coefficients to $\tilde{\alpha}_{i,j}=\nicefrac{{\alpha_{i,j}}}{{\eta_{i}}}\in[0,1]$ and imposes $\sum_{j=1}^{i}\tilde{\alpha}_{i,j}=1$.

###### Proof.

Consider the linear combination (([2](https://arxiv.org/html/2510.09389v1#S3.E2)))

$$ $y_{i}=\sum_{j=1}^{i}\frac{\alpha_{i,j}}{\eta_{i}}v_{j}.$ $$

Then, setting $\eta_{i}=\sum_{j=1}^{i}\alpha_{i,j}$, results in

$$ $y_{i}=\frac{\sum_{j=1}^{i}\alpha_{i,j}v_{j}}{\sum_{j=1}^{i}\alpha_{i,j}}=\sum_{j=1}^{i}\tilde{\alpha}_{i,j}v_{j},$ $$

with $\tilde{\alpha}_{i,j}=\nicefrac{{\alpha_{i,j}}}{{\eta_{i}}}$ as used in the statement. Therefore, the coefficients $\tilde{\alpha}_{i,j}$ trivially sum to $1$. Additionally, this restricts the coefficients and specializes the linear combination to a convex combination (if $\tilde{\alpha}_{i,j}\geq 0$) or an affine combination, as discussed in Appendix [A](https://arxiv.org/html/2510.09389v1#A1). Therefore, the output $y_{i}$ lies in either the simplex or a hyperplane spanned by the value vectors $v_{j}$ [Friedberg et al., 1997].
∎

Figure: Figure 5: (Principle 1) Computation time of a recurrent (magenta) and non-recurrent (gray) implementation against sequence length.
Refer to caption: x5.png

###### Corollary 6.1 .

###### Proof.

## Appendix D Extended Experimental Validation

In this appendix, we provide additional experiments to validate the principles stated in the main text.

### D.1 Principle 1

Since Principle [1](https://arxiv.org/html/2510.09389v1#Thmprinciple1) and Lemma [1](https://arxiv.org/html/2510.09389v1#Thmlemma1) are existence results, the principle cannot be validated experimentally. However, the principle has profound implications on the efficiency of sequence models as shown in Figure [5](https://arxiv.org/html/2510.09389v1#A3.F5). With respect to sequence length, a recurrent implementation has linear complexity, while a non-recurrent implementation has quadratic complexity. Figure [5](https://arxiv.org/html/2510.09389v1#A3.F5) shows the computation time of FlashAttention-2 [Dao, 2023] – a non-recurrent implementation – and SSD [Dao and Gu, 2024] – a reccurent implementation – against sequence length.(^20^2020The data for this plot was obtained using [https://github.com/state-spaces/mamba](https://github.com/state-spaces/mamba).)

### D.2 Principle 2 and Sub-Principles

Figure [6](https://arxiv.org/html/2510.09389v1#A4.F6) adds to Figure [1](https://arxiv.org/html/2510.09389v1#S5.F1) in the main text, by adding the memorization task and additionally showing the behavior of the investigated readout maps $\phi(\cdot)$ close to zero, which highlights how the theoretical near-zero sets are computed.

Figure: Figure 6: (Principle 2) Performance of different readout maps $\phi(\cdot)$ on the memorization task of MAD against the fraction of coefficients with near zero values ($\lvert\alpha\rvert\leq 0.001$; left). The other parameters $A_{i}$, $b_{j}$, $\eta_{i}$ are fixed, thus the setting for ^† is equivalent to softmax attention. Behavior of the investigated readout maps $\phi(\cdot)$ close to zero (middle) and the theoretical near-zero sets of each readout map ($\lvert\phi(x)\rvert\leq 0.001$; right).
Refer to caption: x6.png

Figure: Figure 7: (Principle 2) Performance of different readout maps $\phi(\cdot)$ on three MAD tasks with respect to the ratio of coefficients $\alpha_{i,j}$ set to large ($\approx 1$) and near zero values ($\approx 0$). The other parameters $A_{i}$, $b_{j}$, $\eta_{i}$ are fixed, thus the setting for ^† is equivalent to softmax attention.
Refer to caption: x7.png

Figure [7](https://arxiv.org/html/2510.09389v1#A4.F7) additionally validates Principle [2](https://arxiv.org/html/2510.09389v1#Thmprinciple2) by showing performance against the ratio of coefficients $\alpha_{i,j}$ set to large values ($\alpha\in[0.9,1]$) and coefficients contained in the near-zero set ($\lvert\alpha\rvert\leq 0.001$); since the ratio is small, we plot it on a log-scale. This ratio captures not only a readout map’s ability to suppress coefficients, but also its ability to select coefficients close to $1$. *The results in Figure [7](https://arxiv.org/html/2510.09389v1#A4.F7) suggest that as the ability of a readout map to set coefficients to zero increases, the model becomes more selective (also setting more coefficients to $1$), increasing the performance of all three tasks*.

##### Principle 2.1

To validate Principle [2.1](https://arxiv.org/html/2510.09389v1#Thmprinciple2.Thmsubprinciple1), we ablate four kernel approximation maps $\psi(\cdot)\in\{\textrm{ReLU}(\cdot),\textrm{ELU}(\cdot)+1,\textrm{softplus}(\cdot),\exp(\cdot)\}$ that approximate the readout map as $\phi(\cdot)\approx\psi(\cdot)\psi(\cdot)$. The other coefficient dynamics parameters are fixed to $A_{i}=\mathbb{I}_{n}$, $b_{j}=\nicefrac{{1}}{{\sqrt{n}}}$, and $\eta_{i}=\sum_{j}\alpha_{i,j}$. In Figure [8](https://arxiv.org/html/2510.09389v1#A4.F8), we report the performance of all approximation maps on the fuzzy in-context recall and selective copying tasks of MAD, against the fraction of coefficients in the near-zero set (equivalent to Figure [1](https://arxiv.org/html/2510.09389v1#S5.F1)). Additionally, we provide the performance of softmax attention ($\phi(\cdot)=\exp(\cdot)$) and pure linear attention ($\phi(\cdot)=\textrm{Id}(\cdot)$) as baselines. As dictated by the principle, the kernel approximations set a smaller fraction of coefficients in the near-zero set – compared to the readout maps in Figure [1](https://arxiv.org/html/2510.09389v1#S5.F1) – thus deteriorating performance on both tasks. However, note that most kernel approximations ($\psi(\cdot)\in\{\textrm{ELU}(\cdot)+1,\textrm{softplus}(\cdot),\exp(\cdot)\}$) are more efficient at setting coefficients close to zero than the identity readout map.

Figure: Figure 8: (Principle 2.1) Performance of different kernel approximations $\psi(\cdot)$ of the readout map $\phi(\cdot)\approx\psi(\cdot)\psi(\cdot)$ on two MAD tasks against the fraction of coefficients with near zero values ($\lvert\alpha\rvert\leq 0.001$). The other parameters $A_{i}$, $b_{j}$, $\eta_{i}$ are fixed. Dotted lines indicate the performance level of softmax attention ($\phi(\cdot)=\exp(\cdot)$) and pure linear attention ($\phi(\cdot)=\textrm{Id}(\cdot)$) from Figure [1](https://arxiv.org/html/2510.09389v1#S5.F1), as a baseline.
Refer to caption: x8.png

##### Principle 2.2

To validate Principle [2.2](https://arxiv.org/html/2510.09389v1#Thmprinciple2.Thmsubprinciple2), we investigate the number of coefficients that are set to zero per time index $i$. Since no coefficient $\alpha_{i,j}$ is exactly set to zero due to numerical precision, we treat $\lvert\alpha_{i,j}\rvert\leq 1e{-}6$ as zero. Using two models trained for Figure [8](https://arxiv.org/html/2510.09389v1#A4.F8) (Principle [2.1](https://arxiv.org/html/2510.09389v1#Thmprinciple2.Thmsubprinciple1)), i.e., readout approximations $\psi(\cdot)\in\{\textrm{softplus}(\cdot),\exp(\cdot)\}$, we compute for each time index $i$ and each head, the average number of zero coefficients over a sample input batch of size $64$. Since the two models have state dimension $n=128$ and $16$ heads, the effective state dimension(^21^2121In a multi-head setting, the state dimension $n$ in Principle [2.2](https://arxiv.org/html/2510.09389v1#Thmprinciple2.Thmsubprinciple2) needs to be replaced by the head dimension $\tilde{n}=\nicefrac{{n}}{{\textrm{nr. of heads}}}$.) is $\tilde{n}=\frac{n}{16}=8$, therefore we expect no more than $\tilde{n}-1=7$ zero coefficients per time index and head (Principle [2.2](https://arxiv.org/html/2510.09389v1#Thmprinciple2.Thmsubprinciple2)). The results are shown in Figure [9](https://arxiv.org/html/2510.09389v1#A4.F9) for one example head of each model on both tasks. The figure shows that the allowed number of zero coefficients is never exceeded and in general longer sequence lengths allow to set more coefficients to zero (compare the two tasks with different sequence lenghts). This can be explained by the *linear independence* of states $h_{i,j}$ required by Corollary [6.1](https://arxiv.org/html/2510.09389v1#Thmlemma6.Thmcor1): for longer sequences, there exist more linearly independent states, which facilitates setting more coefficients to zero.

Figure: Figure 9: (Principle 2.2) Number of zero coefficients per time index $i$, for one head of two models $\psi(\cdot)\in\{\textrm{softplus}(\cdot),\exp(\cdot)\}$ trained on fuzzy in-context recall and selective copying (same models as in Fig. [8](https://arxiv.org/html/2510.09389v1#A4.F8)).
Refer to caption: x9.png

### D.3 Principle 5

To validate Principle [5](https://arxiv.org/html/2510.09389v1#Thmprinciple5), we ablate four choices of scaling factors $b_{j}$, on four combinations of readout maps $\phi(\cdot)$ and evolution matrices $A_{i}$. We choose two scaling factors that adhere to the principle ($b_{j}=\nicefrac{{1}}{{\sqrt{n}}}$, $b_{j}=\nicefrac{{z_{j}}}{{\sqrt{n}}}$) and two that do not ($b_{j}=1$, $b_{j}=z_{j}$), where $z_{j}$ is input-dependent, i.e., $z_{j}=\sigma(W_{z}x_{j})$ with $\sigma(\cdot)$ the sigmoid function. For readout maps and evolution matrices, we choose the combinations $(\phi(\cdot),A_{i})=\{(\exp(\cdot),\mathbb{I}_{n}),(\textrm{ReLU}(\cdot),\mathbb{I}_{n}),(\textrm{Id}(\cdot),\alpha_{i}\mathbb{I}_{n}),(\textrm{Id}(\cdot),\mathbb{I}_{n}-\beta_{i}k_{i}k_{i}^{\top})\}$, where $\alpha_{i}$ is parameterized equivalently to $\exp(-\Delta_{i}A)$ in Mamba-2 [Dao and Gu, 2024] and $\beta_{i}$ is input-dependent, i.e., $\beta=2\sigma(W_{\beta}x_{i})$ with $\sigma(\cdot)$ the sigmoid function. These models are trained on the fuzzy in-context recall task of MAD and the results are shown in Figure [10](https://arxiv.org/html/2510.09389v1#A4.F10). We plot the accuracy on the task, across three different seeds, against the effective head dimension, i.e., $\nicefrac{{n}}{{\textrm{nr. of heads}}}$.(^22^2222In a multi-head setting, the state dimension $n$ in $\nicefrac{{1}}{{\sqrt{n}}}$ is replaced by the head dimension $\tilde{n}=\nicefrac{{n}}{{\textrm{nr. of heads}}}$.) The results in Figure [10](https://arxiv.org/html/2510.09389v1#A4.F10) suggest that for scaling factors not adhering to the principle ($b_{j}=1$, $b_{j}=z_{j}$), the performance depends on the random seed, as we notice large performance fluctuations. Conversely, for scaling factors adhering to the principle ($b_{j}=\nicefrac{{1}}{{\sqrt{n}}}$, $b_{j}=\nicefrac{{z_{j}}}{{\sqrt{n}}}$), the performance remains largely constant across seeds. This behavior is expected from Principle [5](https://arxiv.org/html/2510.09389v1#Thmprinciple5), since the dot product $q_{i}^{\top}h_{i,j}$ has larger variance for $b_{j}$ not adhering to the principle. Note that the performance fluctuations are largest for readout map $\phi(\cdot)=\exp(\cdot)$. We hypothesize that this is due to the exponential growth of the readout map, as dot products $q_{i}^{\top}h_{i,j}$ with large variance are either pushed to zero or large values. This is necessarily less pronounced for the other three combinations, although performance fluctuations are still more pronounced for scaling factors not adhering to the principle. While there are indications that performance fluctuates more as the head dimension increases (see e.g. "scalar" for $b_{j}=1$) – which is expected per Principle [5](https://arxiv.org/html/2510.09389v1#Thmprinciple5) – this pattern is not consistent. We hypothesize that the head dimension is too small for this pattern to emerge consistently.

**Table 3: Perplexity of sequence models with different choices of scaling factor $b_{j}$ pretrained on Wikitext-103.**
|  | Mamba-2 | Softmax Attention |  |  |  |
| --- | --- | --- | --- | --- | --- |
| scaling parameter | $b_{j}=1$ | $b_{j}=\nicefrac{{1}}{{\sqrt{n}}}$ | $b_{j}=\Delta_{j}$ | $b_{j}=1$ | $b_{j}=\nicefrac{{1}}{{\sqrt{n}}}$ |
| parameter count | $110\,$M | $110\,$M | $110\,$M | $120\,$M | $120\,$M |
| perplexity | $29.23$ | $28.84$ | $28.87$ | $32.69$ | $31.73$ |

We also validate Principle [5](https://arxiv.org/html/2510.09389v1#Thmprinciple5) on a more realistic dataset than the previous synthetic one. For this, we train a standard softmax attention model and a standard Mamba-2 model on Wikitext-103 [Merity et al., 2016]. For both models, we modify the scaling factor $b_{j}$ and fix all other parameters to the standard choices of each model. Both models, have the same size but softmax attention is equipped with additional positional embeddings, which explains the higher parameter count in Table [3](https://arxiv.org/html/2510.09389v1#A4.T3). The results in Table [3](https://arxiv.org/html/2510.09389v1#A4.T3) suggest that there exists a performance gap, yet small, between scaling factors adhering to the principle ($b_{j}=\nicefrac{{1}}{{\sqrt{n}}}$, $b_{j}=\Delta_{j}$)(^23^2323In Mamba-2, $\Delta_{j}$ adheres to the principle as discussed in Example [2](https://arxiv.org/html/2510.09389v1#Thmexample2).) and scaling factors that do not ($b_{j}=1$). The finding that softmax attention without scaling of the keys can perform well in practice, has also been observed in Britz et al. [2017]. Finally, note that decoupling the parameters $A_{i}$ and $b_{j}$ in Mamba-2 does not hurt performance, with the scaling factor $b_{j}=\nicefrac{{1}}{{\sqrt{n}}}$ (with $\Delta_{i}$ still used to compute $A_{i}$) performing on par with $b_{j}=\Delta_{j}$.

Figure: Figure 10: (Principle 5) Minimal and maximal performance across three seeds on the fuzzy in-context recall task, for different choices of scaling factor $b_{j}$ and combinations $(\phi(\cdot),A_{i})$. The legend refers to the combinations $(\phi(\cdot),A_{i})=\{(\exp(\cdot),\mathbb{I}_{n}),(\textrm{ReLU}(\cdot),\mathbb{I}_{n}),(\textrm{Id}(\cdot),\alpha_{i}\mathbb{I}_{n}),(\textrm{Id}(\cdot),\mathbb{I}_{n}-\beta_{i}k_{i}k_{i}^{\top})\}$ and each column shows the results for a single $b_{j}$.
Refer to caption: x10.png

### D.4 Principle 6

Principle [6](https://arxiv.org/html/2510.09389v1#Thmprinciple6) is mainly concerned with training stability related to the evolution matrices $A_{i}$ and normalization factors $\eta_{i}$. Due to Corollary [6.1](https://arxiv.org/html/2510.09389v1#Thmlemma6.Thmcor1), choosing $\eta_{i}=\sum_{j}\alpha_{i,j}$ guarantees stable training and has additional implications on the output space of the sequence model (see Appendix [A](https://arxiv.org/html/2510.09389v1#A1)). We hence focus on evolution matrices to experimentally validate the principle. It is widely known, that stable $A_{i}$ (all eigenvalues lie in the unit circle of the complex plane) guarantee numerically stable training, as evidenced by the design of $A_{i}$ in e.g. S4 [Gu et al., 2022a], LRU [Orvieto et al., 2023], Mamba-2 [Dao and Gu, 2024], GLA [Yang et al., 2023], and DeltaNet [Schlag et al., 2021]. To validate the principle, we therefore consider the simple unstable evolution matrix $A_{i}=1.05\,\mathbb{I}_{n}$ and show that with appropriate design of $\eta_{i}$, we are able to solve noisy in-context recall. We fix $b_{j}=1$, $A_{i}=1.05\,\mathbb{I}_{n}$ and ablate the readout maps $\phi(\cdot)\in\{\exp(\cdot),\textrm{softplus}(\cdot),\textrm{ReLU}(\cdot)\}$ and normalization factors $\eta_{i}\in\{1,1.05^{i},\sum_{j}\alpha_{i,j}\}$; results are shown in Figure [11](https://arxiv.org/html/2510.09389v1#A4.F11). As expected by Corollary [6.1](https://arxiv.org/html/2510.09389v1#Thmlemma6.Thmcor1), the normalization factor $\eta_{i}=\sum_{j}\alpha_{i,j}$ ensures stable training and achieves perfect accuracy. The results of the other two choices need additional discussion.

Figure: Figure 11: (Principle 6) Performance of three normalization factor choices $\eta_{i}$ on the noisy in-context recall task, for unstable evolution matrix $A_{i}=1.05\,\mathbb{I}_{n}$, $b_{j}=1$, and readout maps $\phi(\cdot)\in\{\exp(\cdot),\textrm{softplus}(\cdot),\textrm{ReLU}(\cdot)\}$.
Refer to caption: x11.png

First, consider $\eta_{i}=1$: for readout map $\exp(\cdot)$, training is unstable and the model fails to learn anything, yet for readout maps $\textrm{softplus}(\cdot),\textrm{ReLU}(\cdot)$ training is stable and the model achieves good performance, albeit suboptimal. This is explained by the growth of the readout maps, while $\exp(\cdot)$ grows superlinearly, $\textrm{softplus}(\cdot),\textrm{ReLU}(\cdot)$ grow linearly, thus the coefficients $\alpha_{i,j}$ grow faster for $\phi(\cdot)=\exp(\cdot)$ than the other two maps, as the state $h_{i,j}$ grows due to $\prod_{t=j+1}^{i}1.05\,\mathbb{I}_{n}$ (see ([4](https://arxiv.org/html/2510.09389v1#S3.E4))). Since the task has short sequence length ($128$) and the chosen evolution matrix is mildly unstable ($1.05$), readout maps with linear growth can still achieve good performance.

Now consider $\eta_{i}=1.05^{i}$: for readout map $\exp(\cdot)$, training is unstable and the model fails to learn anything, yet for readout maps $\textrm{softplus}(\cdot),\textrm{ReLU}(\cdot)$ training is stable and the model achieves perfect performance. This is again explained by the growth of the readout maps, when considering the normalized coefficients $\frac{\alpha_{i,j}}{\eta_{i}}$. For $\phi(\cdot)=\exp(\cdot)$, the coefficients $\alpha_{i,j}$ grow faster than the normalization factors $\eta_{i}$, leading to instability. For the other two readout maps, the coefficients $\alpha_{i,j}$ grow as fast as the normalization factors $\eta_{i}$, thus stabilizing training. Further experimental validation of the principle, is given by the experimental results in Beck et al. [2024], since the model could not have been trained without proper design of the normalization factors $\eta_{i}$.

## Appendix E Details on Experimental Setup

The experimental results provided in Section [5](https://arxiv.org/html/2510.09389v1#S5) and Appendix [D](https://arxiv.org/html/2510.09389v1#A4) are performed on the mechanistic architecture design (MAD) benchmark [Poli et al., 2024]. To perform the experiments, we modified the MAD code base(^24^2424[https://github.com/athms/mad-lab](https://github.com/athms/mad-lab)) and integrated it in our existing code base.(^25^2525[https://git.sieber.io/mdl-design](https://git.sieber.io/mdl-design)) For pretraining in Appendix [D.3](https://arxiv.org/html/2510.09389v1#A4.SS3), we used the Wikitext-103 [Merity et al., 2016] dataset. All experiments were run on a cluster equipped with $8$ NVIDIA RTX 4090 GPUs.

### E.1 Global settings

##### Wikitext

The Wikitext-103 dataset is tokenized using the standard GPT-2 tokenizer (vocab_size: $50257$) and chunked into blocks of sequence length $1024$, before being processed by the model.

##### MAD Tasks

We perform experiments on the following four tasks of MAD: selective copying, memorization, noisy in-context recall, and fuzzy in-context recall. For a detailed exposition of these tasks, we refer to Poli et al. [2024, Section 3.1]. Below, we report the dataset parameters used for our experiments; we refer to Poli et al. [2024, Appendix B] for a detailed explanation of each parameter and its effect on the task difficulty.

- •
Selective Copying: vocab_size: $64$, seq_len: $512$, num_tokens_to_copy: $32$, selective: True, num_train_examples: $3200$.
- •
Memorization: vocab_size: $4096$, seq_len: $32$, frac_noise: $0.0$ (no noise), num_train_examples: $256$.
- •
Noisy In-context Recall: vocab_size: $32$, seq_len: $128$, multi_query: True, frac_noise: $0.2$ (noisy), noise_vocab_size: $16$, num_train_examples: $3200$.
- •
Fuzzy In-context Recall: vocab_size: $32$, seq_len: $128$, multi_query: True, frac_noise: $0.0$ (no noise), num_train_examples: $12800$.

##### Training

For MAD tasks, we use the parameters shown in Table [4](https://arxiv.org/html/2510.09389v1#A5.T4). These are the same parameters as proposed in Poli et al. [2024], but we sweep a larger range of learning rates. All performances/accuracies reported in this paper, correspond to the best performance/accuracy across the learning rate and weight decay sweep reported in Table [4](https://arxiv.org/html/2510.09389v1#A5.T4).

**Table 4: Training parameters used on the MAD benchmark.**
| Optimizer | AdamW |
| --- | --- |
| Optimizer Momentum | $(\beta_{1},\beta_{2})=(0.9,0.98)$ |
| Dropout | None |
| Batch Size | $128$ |
| Training Epochs | $200$ |
| Learning Rate Schedule | Cosine Decay |
| Learning Rate Warmup | None |
| Number of Test Samples | $1280$ |
| Learning Rate | $[0.0001,0.0005,0.001,0.005,0.01]$ |
| Weight Decay | $[0.0,0.1]$ |

For pretraining on Wikitext, we use the training parameters stated in Table [5](https://arxiv.org/html/2510.09389v1#A5.T5).

**Table 5: Training parameters used for pretraining on Wikitext.**
| Optimizer | AdamW |
| --- | --- |
| Optimizer Momentum | $(\beta_{1},\beta_{2})=(0.9,0.95)$ |
| Dropout | None |
| Batch Size | $8$ |
| Training Steps | $130$k |
| Learning Rate Schedule | Cosine Decay |
| Warmup Steps | $3$k |
| Learning Rate | $0.0006$ |
| Weight Decay | $0.1$ |

##### Models

The model parameters for each experiment are reported individually in the following section. We use the two block designs shown in Figure [12](https://arxiv.org/html/2510.09389v1#A5.F12). Generally, we use the Transformer style block for attention-based models (i.e. readout map $\phi(\cdot)\neq\textrm{Id}(\cdot)$) and the Gated DeltaNet style block for SSMs/RNNs (i.e. readout map $\phi(\cdot)=\textrm{Id}(\cdot)$). Additionally, we typically use interleaved MLP blocks after each block (either Type 1 or 2), which do not count towards the number of layers. For example, a two layer model with interleaved MLP blocks, would effectively consist of four layers, two sequence mixing layers and two MLP layers. The MLP block consists of two dense layers with SwiGLU activation [Shazeer, 2020] and inner dimension reported per experiment.

Figure: Figure 12: Block designs used in experiments: Transformer style (left; Type 1) and Gated DeltaNet style (right, Type2) from [Yang et al., 2025a, Fig. 1]. Dashed parts show the optional SwiGLU MLP.

### E.2 Model Parameters per Experiment

In the following, we detail the model parameters used to experimentally validate each principle individually.

- •
The model parameters used to obtain Figures [1](https://arxiv.org/html/2510.09389v1#S5.F1), [6](https://arxiv.org/html/2510.09389v1#A4.F6) & [7](https://arxiv.org/html/2510.09389v1#A4.F7) and validate Principle [2](https://arxiv.org/html/2510.09389v1#Thmprinciple2) are provided in Table [6](https://arxiv.org/html/2510.09389v1#A5.T6).
- •
The model parameters used to obtain Figures [8](https://arxiv.org/html/2510.09389v1#A4.F8) & [9](https://arxiv.org/html/2510.09389v1#A4.F9) and validate Principles [2.1](https://arxiv.org/html/2510.09389v1#Thmprinciple2.Thmsubprinciple1) and [2.2](https://arxiv.org/html/2510.09389v1#Thmprinciple2.Thmsubprinciple2) are provided in Table [7](https://arxiv.org/html/2510.09389v1#A5.T7).
- •
The model parameters used to obtain Figure [2](https://arxiv.org/html/2510.09389v1#S5.F2) and validate Principle [3](https://arxiv.org/html/2510.09389v1#Thmprinciple3) are provided in Table [8](https://arxiv.org/html/2510.09389v1#A5.T8).
- •
The model parameters used to obtain Figure [3](https://arxiv.org/html/2510.09389v1#S5.F3) and validate Principle [4](https://arxiv.org/html/2510.09389v1#Thmprinciple4) are provided in Table [9](https://arxiv.org/html/2510.09389v1#A5.T9).
- •
The model parameters used to obtain Figure [10](https://arxiv.org/html/2510.09389v1#A4.F10) & Table [3](https://arxiv.org/html/2510.09389v1#A4.T3) and validate Principle [5](https://arxiv.org/html/2510.09389v1#Thmprinciple5) are provided in Table [10](https://arxiv.org/html/2510.09389v1#A5.T10) and Table [11](https://arxiv.org/html/2510.09389v1#A5.T11), respectively.
- •
The model parameters used to obtain Figure [11](https://arxiv.org/html/2510.09389v1#A4.F11) and validate Principle [6](https://arxiv.org/html/2510.09389v1#Thmprinciple6) are provided in Table [12](https://arxiv.org/html/2510.09389v1#A5.T12).

**Table 6: Model parameters used for Figures [1](https://arxiv.org/html/2510.09389v1#S5.F1), [6](https://arxiv.org/html/2510.09389v1#A4.F6) & [7](https://arxiv.org/html/2510.09389v1#A4.F7). These are the same parameters used in Poli et al. [2024].**
| Coefficient Dynamics Parameters | Detailed in Section [5](https://arxiv.org/html/2510.09389v1#S5) |
| --- | --- |
| Key ($k_{j}$) & Querry ($q_{i}$) Preprocessing | None |
| Block Design | Transformer style (Type 1) |
| Number of Layers | $2$ |
| Embedding Dimension $d$ | $128$ |
| Inner (State) Dimension $n$ | $128$ |
| Value Dimension $d_{v}$ | $128$ |
| Number of Heads | $16$ |
| Interleaved MLP Layers | Yes (dim $=256$) |
| Positional Embedding | Yes |

**Table 7: Model parameters used for Figures [8](https://arxiv.org/html/2510.09389v1#A4.F8) & [9](https://arxiv.org/html/2510.09389v1#A4.F9). These are the same parameters used in Poli et al. [2024].**
| Coefficient Dynamics Parameters | Detailed in Appendix [D.2](https://arxiv.org/html/2510.09389v1#A4.SS2) |
| --- | --- |
| Key ($k_{j}$) & Querry ($q_{i}$) Preprocessing | Detailed in Appendix [D.2](https://arxiv.org/html/2510.09389v1#A4.SS2) |
| Block Design | Transformer style (Type 1) |
| Number of Layers | $2$ |
| Embedding Dimension $d$ | $128$ |
| Inner (State) Dimension $n$ | $128$ |
| Value Dimension $d_{v}$ | $128$ |
| Number of Heads | $16$ |
| Interleaved MLP Layers | Yes (dim $=256$) |
| Positional Embedding | Yes |

**Table 8: Model parameters used for Figure [2](https://arxiv.org/html/2510.09389v1#S5.F2). These are the same parameters used in Poli et al. [2024].**
| Coefficient Dynamics Parameters | Detailed in Section [5](https://arxiv.org/html/2510.09389v1#S5) |
| --- | --- |
| Key ($k_{j}$) & Querry ($q_{i}$) Preprocessing | None |
| Block Design | Transformer style (Type 1) |
| Number of Layers | $2$ |
| Embedding Dimension $d$ | $128$ |
| Inner (State) Dimension $n$ | $128$ |
| Value Dimension $d_{v}$ | $128$ |
| Number of Heads | $16$ |
| Interleaved MLP Layers | Yes (dim $=256$) |
| Positional Embedding | Detailed in Section [5](https://arxiv.org/html/2510.09389v1#S5) |

**Table 9: Model parameters used for Figure [3](https://arxiv.org/html/2510.09389v1#S5.F3).**
| Coefficient Dynamics Parameters | Detailed in Section [5](https://arxiv.org/html/2510.09389v1#S5) |
| --- | --- |
| Key ($k_{j}$) & Querry ($q_{i}$) Preprocessing | ShortConv(dim$=4$) |
| Block Design | Gated DeltaNet style (Type 2) |
| Number of Layers | $2$ |
| Embedding Dimension $d$ | $128$ |
| Inner (State) Dimension $n$ | $128$ |
| Value Dimension $d_{v}$ | $128$ |
| Number of Heads | $8$ |
| Interleaved MLP Layers | Yes (dim $=256$) |
| Positional Embedding | No |

**Table 10: Model parameters used for Figure [10](https://arxiv.org/html/2510.09389v1#A4.F10). Below, if a parameter is declared with "or", the first option is for attention style models and the second for SSM/RNN style models.**
| Coefficient Dynamics Parameters | Detailed in Appendix [D.3](https://arxiv.org/html/2510.09389v1#A4.SS3) |
| --- | --- |
| Key ($k_{j}$) & Querry ($q_{i}$) Preprocessing | None or ShortConv(dim$=4$) |
| Block Design | Type 1 or Type 2 |
| Number of Layers | $2$ |
| Embedding Dimension $d$ | $128$ |
| Inner (State) Dimension $n$ | $[128,256,512,1024]$ |
| Value Dimension $d_{v}$ | $128$ |
| Number of Heads | $4$ |
| Interleaved MLP Layers | Yes (dim $=256$) |
| Positional Embedding | Yes or No |

**Table 11: Model parameters used for Table [3](https://arxiv.org/html/2510.09389v1#A4.T3). Below, if a parameter is declared with "or", the first option is for softmax attention and the second for Mamba-2.**
| Coefficient Dynamics Parameters | Softmax attention and Mamba-2 standard; varying $b_{j}$ |
| --- | --- |
| Key ($k_{j}$) & Querry ($q_{i}$) Preprocessing | None or ShortConv(dim$=4$) |
| Block Design | Type 1 or Type 2 |
| Number of Layers | $8$ |
| Embedding Dimension $d$ | $768$ |
| Inner (State) Dimension $n$ | $768$ |
| Value Dimension $d_{v}$ | $768$ |
| Number of Heads | $12$ |
| Interleaved MLP Layers | Yes (dim $=512$) |
| Positional Embedding | Yes or No |

**Table 12: Model parameters used for Figure [11](https://arxiv.org/html/2510.09389v1#A4.F11).**
| Coefficient Dynamics Parameters | Detailed in Appendix [D.4](https://arxiv.org/html/2510.09389v1#A4.SS4) |
| --- | --- |
| Key ($k_{j}$) & Querry ($q_{i}$) Preprocessing | None |
| Block Design | Transformer style (Type 1) |
| Number of Layers | $2$ |
| Embedding Dimension $d$ | $128$ |
| Inner (State) Dimension $n$ | $128$ |
| Value Dimension $d_{v}$ | $128$ |
| Number of Heads | $16$ |
| Interleaved MLP Layers | Yes (dim $=256$) |
| Positional Embedding | Yes |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: N/A
soundness: 3
strengths: 1. The paper is clearly written and the mathematical formulation is rigorous.
2. Unifies existing insights on SSMs, RNNs, and attention into a single framework summarizing core principles like linearity, efficiency, input selectivity, normalization, and stability. This is pedagogically useful for newcomers to the field.
summary: 1. The paper proposes a unified theoretical framework (Coefficient Dynamics) for analyzing sequence models, including Transformers, linear attention, and State Space Models (SSMs). The framework formalizes the fact that these models compute outputs as linear combinations of past tokens, and interprets the outputs of *linear dynamical systems driven by impulse inputs*.

2. Unlike prior unification approaches, this formulation explicitly introduces a per-token index, j, representing each previous key/value, conceptually similar to a KV-cache, and shows Transformers, SSMs, Linear Attention-like architectures can be expressed as special cases.


Building on this framework, the authors derive a set of design principles:

  1. **Linearity of $\phi$:** Only linear readout maps permit parallel recurrent computation.
  2. **Input selectivity:** Nonlinear $\phi$ enables sparse coefficients aka selectivity.
  3. **Positional information:** Setting $A_t \neq I$ embeds position into coefficients; $A_t = I$ requires positional embeddings.
  4. **Evolution structure:** The choice of $A_t$ (scalar, diagonal, Householder) allows for transformations such as scaling or rotation.
  5. **Scaling of $b_j$:** Proper scaling of $b_j$, prevents variance blow-up with increasing hidden state.
  6. **Normalization $\eta_i$:** Stable training requires normalization of coefficient magnitudes.
weaknesses: ### **On the framework**

The “coefficient dynamics” construction, builds on standard frameworks like Dynamical Systems Framework with the new ingredient being the explicit *per token j index*, which is equivalent to maintaining a key–value (KV) cache. In my opinion this viewpoint is not novel in a theoretical sense: prior work (e.g., Dao & Gu, 2024; Sieber et al., 2024) already expresses attention and SSMs as linear recurrences or matrix multiplications over past states. The main change, vis-a-vis previous works, is the KV cache-like formulation to unify attention without using infinite state sizes. 

---

### **On Principle 1 ($\phi$ must be linear for parallelization)**

In my opinion, this is a well-known result in the subquadratic. Specifically, it is known that only linear readouts permit associative-scan like formulations required for efficient recurrent computation. This principle has been used in multiple works (Linear Attention, Mamba, Mamba-2, Gated Deltanet). While the lemma is correct, its inclusion as a “new principle” adds little beyond reiterating that *linear functions yield linear complexity*.

---

### **On Principle 2 (Input selectivity and geometry of $\phi$)**

The main idea—that nonlinear ϕ enables suppression of uninformative tokens while linear ϕ limits selectivity—is sound and nicely presented. However, this observation is well known and connects to classic results on **associative memory capacity**, where linear associative memories can store only ~n patterns in dimension n. 

As a nit remark: the follow-up discussion "Can learnable parameters save us?" is technically correct but does not logically follow from the principle, since modifying $A_t$ or $b_j$ changes only the key–value dynamics, not the query-dependent coefficients $\alpha_{ij}$.

Remark: Authors claim that Linear Attention has a readout of the form $\psi(\cdot)\psi(\cdot)$, but this is not really a function acting post the readout as defined and hence does not fit with the framework definition. It is better viewed as a preprocessing trick rather than a part of the framework.

---

### **On Principle 3 (Positional Information)**

The result that $A_t = I$, which implies that per-query token the sequence mixing process is permutation invariant and hence requires position embeddings is also well known and has been discussed in prior works. Authors of Mamba mention that due to the decay, the operation is no longer permutation invariant and hence does not require position embeddings. In my opinion, the lemma correctly states this but adds no new insight.

---

### **On Principle 4 (Structure of $A_t$)**

The fact that the structure of the state transition matrix $A_t$ (scalar, diagonal, Householder) limits the operation that can be performed on the keys is a tautological statement for me as the state transition matrix is what acts on the keys to produce the output. In the Lemma, authors simple summarize the actions performed by scalar, diagonal, or Householder matrix operators on the keys being acted upon.

---

### **On Principles 5 & 6 (Scaling and normalization)**

The discussion on $b_j = O(1/\sqrt{n})$ to maintain $O(1)$ variance and the normalization of coefficients $\alpha_{ij}$ to avoid exploding norms repeats standard initialization theory (Glorot & Bengio, 2010; Vaswani et al., 2017). While correctly stated, these are rules of thumbs which are widely used in architecture design to ensure that variance remains bounded in deep models. In my opinion, their inclusion as novel “principles” is overstated.

---

My overall assessment of novelty is that the paper’s strength lies in gathering well-known rules of thumbs under a shared formalism, but every individual principle has been studied or applied before.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. Based on the design principles of the sequence modeling architecture proposed in the paper, what characteristics should the optimal model architecture for sequence modeling?
2. Although these design principles can guide the design of the optimal sequence modeling architecture, for linear RNN architectures, the design of some components may not be conducive to hardware efficient parallel training of RNN. In this case, would a more general and expressive matrix gate be less useful than a simpler, more efficiently trained diagonal/scaler gate?
soundness: 3
strengths: 1. The design principles for sequence modeling architecture proposed in this paper are instructive.
2. The analysis combining theory and experiment is convincing.
summary: This paper proposes a unified theoretical framework to describe sequence models including Transformer and RNNs, and puts forward a series of design principles and experimental verifications to guide sequence modeling.
weaknesses: A unified sequence modeling framework for transformer and RNN model architectures has been mentioned in various works, such as MetaLA [1], PaTH Attention [2], and log-linear-attention [3], which may diminish the contribution of this paper. Therefore, further comparison and discussion with similar related works will help highlight the contribution of this paper.

[1] Yuhong Chou, et al. MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map. NeurIPS, 2024.
[2] Songlin Yang, et al. PaTH Attention: Position Encoding via Accumulating Householder Transformations. NeurIPS, 2025.
[3] Han Guo, et al. Log-Linear Attention. arXiv, 2025.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - What would be the model with all the principles applied, and how does it perform on language modeling?

- Is Principle 3 true?
soundness: 2
strengths: The paper is well structured, proposing several ideas that not only help designing new architectures but also understand existing methods. The principles that are introduced in the paper are simple, thus can be easily adopted. Each proposal is backed up with an experiment.
summary: This paper proposes a number of principles for designing sequence modeling layers. To come up with such proposals, the paper provides a survey of previous works, and mechanistically separate parts into A, b, \phi, \eta, and \alpha, highlighting major different parts. Coming up with six principles, the authors provide ablations for each, validating their claims.
weaknesses: My major concern of this paper is that the ideas are only validated using synthetic benchmarks as these benchmarks cannot model all real-world issues, and quite vulnerable to training setups such as weight decay and learning rates. Since the paper comes up with the principles that each should improve the model, I believe the authors should have used some real-world datasets (say language modeling) and benchmark at least their best model that all the principles are applied.

Additionally, I also have concerns with some principles. For example, Principle 1 and 5 are quite trivial which are already well known among the community, and Principle 6 is not specific enough (what specifically is an unstable A?). Also, I wonder if Principle 3 is actually correct: for instance, attention-based autoregressive models without positional embeddings (i.e., NoPE) has shown promising results.

---

confidence: 5
contribution: 1
initial_rating: 2
presentation: 3
questions: 1. **Scope of derivations:** Are any of your truly lemmas new, or are they reinterpretations of existing SSM analyses?
2. **Experimental depth:** Can you test your design principles on real-world data (e.g., language modeling or speech) to demonstrate their usefulness beyond toy MAD benchmarks?
3. **Relation to Mamba and Selective SSMs:** Since these models already embody your principles (learned (A_t,b_j), stability constraints), how does your framework offer new insights beyond theirs?
4. **Extension to multi-layer architectures:** The study is single-layer. How do the principles scale in multi-layer or cross-attention settings?
soundness: 2
strengths: * Clear and mathematically consistent formulation.
* Provides a clean pedagogical summary connecting RNNs, attention, and SSMs under one algebraic form.
* Well-presented with readable equations and illustrative diagrams.
* Serves as a potential tutorial reference for newcomers to the field.
summary: This paper proposes a unified theoretical framework for viewing sequence models (including Transformers, recurrent neural networks (RNNs), and state-space models (SSMs)) through the lens of **coefficient dynamics**.
The authors model the coefficients alpha in the linear combination as outputs of an autonomous linear dynamical system driven by *impulse inputs*.
They claim this representation reveals shared mathematical structure among diverse sequence architectures, enabling the derivation of six **“design principles”** related to:

1. Linear vs. nonlinear readout maps and their efficiency tradeoffs
2. Input selectivity through the geometry of zero set
3. Encoding positional information via non-identity evolution matrices
4. Structured choices of A_t matrix (e.g., diagonal or Householder)
5. Proper scaling of injection parameters (b_j)
6. Normalization factors to ensure stability.

Experiments on synthetic tasks (MAD benchmark) empirically test these principles, showing expected patterns such as improved selectivity with larger zero sets and the sufficiency of (A_t that is not equal to I) to encode position without embeddings.
weaknesses: * **No novel theoretical result:** All lemmas rederive existing intuitions without advancing formal understanding.
* **Weak experimental validation:** Evaluations on simple synthetic tasks (MAD) do not test scalability, language modeling, or real-world data.
* **Limited empirical novelty:** Trends (e.g., gating helps, non-identity (A_t) adds positional info) are already widely known.
* **Incomplete discussion of prior work:** The paper fails to acknowledge prior theoretical frameworks that make similar connections. No point is not already known. I don't see anything in this paper that is novel and that we did not already know from training deep sequence models.


Decision: Reject