Understanding the Differences in Foundation Models: Attention, State  Space Models, and Recurrent Neural Networks
Authors: Jerome Sieber
Carmen Amo Alonso
Alexandre Didier
Melanie Zeilinger
Antonio Orvieto
Venue: NeurIPS 2024 poster (2024)
Keywords: deep learning architectures
softmax attention
linear attention
state space models
recurrent neural networks
dynamical system models
Primary area: deep_learning_architectures

Abstract: Softmax attention is the principle backbone of foundation models for various artificial intelligence applications, yet its quadratic complexity in sequence length can limit its inference throughput in long-context settings. To address this challenge, alternative architectures such as linear attention, State Space Models (SSMs), and Recurrent Neural Networks (RNNs) have been considered as more efficient alternatives. While connections between these approaches exist, such models are commonly developed in isolation and there is a lack of theoretical understanding of the shared principles underpinning these architectures and their subtle differences, greatly influencing performance and scalability. In this paper, we introduce the Dynamical Systems Framework (DSF), which allows a principled investigation of all these architectures in a common representation. Our framework facilitates rigorous comparisons, providing new insights on the distinctive characteristics of each model class. For instance, we compare linear attention and selective SSMs, detailing their differences and conditions under which both are equivalent. We also provide principled comparisons between softmax attention and other model classes, discussing the theoretical conditions under which softmax attention can be approximated. Additionally, we substantiate these new insights with empirical validations and mathematical arguments. This shows the DSF's potential to guide the systematic development of future more efficient and scalable foundation models.

Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks
Jerome Sieber* ETH Zurich Zurich, Switzerland j ssiebereetzz.co ch
Carmen Amo Alonso* ETH Zurich Zurich, Switzerland camoalonso@ethz.com ch
Alexandre Didier ETH Zurich Zurich, Switzerland adi adidierGethz. ch
Melanie N. Zeilinger
Antonio Orvieto
ETH Zurich
ELLIS Institute Tübingen Tübingen, Germany antonioštue. ellis.er eu
Zurich, Switzerland
mzeilinger@ethz.ch
Abstract
Softmax attention is the principle backbone of foundation models for various artifi- cial intelligence applications, yet its quadratic complexity sequence length can limit its inference throughput in long-context settings. To address this challenge, alternative architectures such as linear attention, State Space Models (SSMs), and Recurrent Neural Networks (RNNs) have been considered as more efficient al- ternatives. While connections between these approaches exist, such models are commonly developed in isolation and there is lack of theoretical understanding of the shared principles underpinning these architectures and their subtle differences, greatly influencing performance and scalability. In this paper, introduce the Dynamical Systems Framework (DSF), which allows a principled investigation of all these architectures in common representation. Our framework facilitates rigorous comparisons, providing new insights on the distinctive characteristics each model class. For instance, compare linear attention and selective SSMs, detailing their differences and conditions under which both are equivalent. We also provide principled comparisons between softmax attention and other model classes, discussing the theoretical conditions under which softmax attention can be approximated. Additionally, we substantiate these new insights with empirical vali- dations and mathematical arguments. This shows the DSF's potential to guide the systematic development of future more efficient and scalable foundation models.
Introduction
Foundation models serve as the backbone for a wide range of tasks across Artificial Intelligence due to their ability learn complex interactions in large datasets [1]. In recent years, the attention mechanism 2 has been the dominating token-mixing strategy foundation models. However, its major computational bottleneck, i.e., the quadratic complexity with context length, has posed challenge to scaling and deploying these models beyond moderate context lengths [3]. In order to mitigate these issues, attention-free architectures have been proposed: prominent examples of these are the novel State Space Models (SSMs) [ as well as recent efforts to enhance Recurrent Neural Networks (RNNs) 12). Although these models show great promise in boosting efficiency, efforts
These authors contributed equally; ordered randomly.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to provide a rigorous theoretical comparison are searce, and current comparisons with attention are merely empirical (see Section [ for an in-depth discussion). Despite the prevalence and ubiquity of foundation models, principled understanding of the similarities and differences among these different design strategies currently lacking.
In order to close this gap, we introduce the Dynamical Systems Framework (DSF) a theoretical framework based on control theoretic perspective that allows evaluate the similarities and differences between different foundation models in principled manner. The DSF serves as powerful tool for approaching theoretical research questions about foundation models, enabling direct comparisons both theoretical and experimental across architectures such as attention mechanisms, SSMs, and RNNs. We believe that the DSF provides new insights on the most relevant features found in current architectures, and can inform a systematic development of future hybrid models. The DSF further simplifies identification of existing computational algorithms to apply to newly developed models. Rather than providing an exhaustive list of insights, the results included below are meant to exemplify important questions that the DSF can answer and guide future research. Specifically, we explore the following questions:
How are attention mechanisms, SSMs, and RNNs related? TL;DR: All three model classes can be represented recurrent models, which can be compared using the proposed DSF.
Can softmax attention be expressed as a recurrent model?
TL:DR: Softmax attention translates to recurrent model within the DSF, however the hidden state dimension needs to be infinite.
Why does state expansion help to improve performance of RNNs and SSMs? TL;DR: This is related to the second question: state expansion increases the dimension of the hidden state thus allowing for an increased expressivity of the model (Lemma
Why do SSMs significantly outperform attention on the LRA benchmark? TL;DR: The performance gap can be explained by the recurrent normalization strategy (discretization step) used by selective SSMs as discussed in Section 4.2.
How closely are linear attention, S6 (i.e. Mamba) related?
TL;DR: The common feature is the coupling of state transition and input matrix via single (normalization) parameter in their recurrent representation. However, the models differ in the parameterization of this parameter, which we analyze experimentally.
What do selective SSMs teach about improving RNN architectures? TL;DR: Replacing the state transition in a RNN variant qLSTM with the state transition of S6 improves performance of the RNN.
We note that the main contribution of our paper is the introduction of the DSF, which is a unifying framework for analysis of attention mechanisms, SSMs, and RNNs. To the best of our knowledge, this is the first unified framework that allows analysis of all three model classes in the same parame- terization and thus allows to identify differences in the models that lead significant performance improvements. While some of the provided results already exist in the literature (e.g that increased state size improves performance). we also provide novel insights unique to the DSF framework in a comprehensive way that enables further analysis with control theoretical tools.
Notation: We use Latin letters in the following way: N is the size of the hidden state in the DSF, the state expansion, the embedding size or model size, and the sequence length. A visual representation of these dimensions is given in Appendix A We use superscripts, e.g. .d, to denote the elements or block-elements of matrix and a blocck-matrix. We use subscripts, e-g. "s, to denote the time index (or input dependency). Specifically, U; represents the value of vector v at time i. We use bold notation to indicate sequences, i.e., V; = [wi...., vi). We use a(-) to denote the sigmoid function. The products @ and @ denote the Hadamard (element-wise) product and the Kronecker (block-wise) product, respectively. T, denotes the identity matrix of size R Generally, we omit stating the bias term for weight matrices unless stating the bias term helps with clarity.

---

2 Preliminaries
In this section, we introduce the key architectural components studied in this work: attention, SSMs, and RNNs. We remark that these components are often the central block considered to be the backbone within complex architecture composed of other blocks and skip connections (see for instance [3].. In what follows, we review exclusively the backbone block, which we denote as f() in = fu), where u E I rxd and E Rixd are the input and output sequences, respectively.
2.1
Attention
The standard self-attention block [2] consists of three matrices: Wo, WK, and Wv, which are the learnt parameters of the model. These matrices, when multiplied with the input u, yield the queries E RR,, keys k E Rdd and values v E respectielly
q= uW k uWK, K. v= uWy.
Keys, queries, and values are then combined in the attention block to produce the output
qk
y=
(2)
Vdx
where ((.) is map RC - R and is applied row-wise. For standard self-attention, the softmax function is used, i.e. ((-) = softmax(-), but given the limitations of the softmax function, alternative formulations have been proposed. We consider two formulations of attention: softmax attention (2)
and linear attention [14]. We focus on masked attention formulations, i.e., the attention matrix ((qk has lower-triangular structure, and to simplify the derivations, we drop the scaling factor vdk.
2.2 State Space Models
Architectures based on a state space parametrization compute the output y through a dynamic recurrence of input signals at each time step i, i.e..
h, = A,h4-1 + B,u
(3a)
3/1 C2hy + D,w,,
(3b)
where hy is the hidden state of the system, and the dynamic matrices of appropriate dimensions A, B,, Cr, D; the learnt model parameters. Different time-varying and timeiinvrriant parameteri- zations for A,, Bi, C, D; have been proposed in the literature (an overview is given in (15).. Here discuss the most prominent one.
S6. The first selective SSM parametrization (S6) was introduced together with the Mamba architec- ture 77. The S6 block parametrizes the recurrence
-WWW, D; = -WDu, with A; softplus(Wa (W,,;; + ba) for every i, WA, W., WB. Wc, WD, and are learnt matrices of appropriate dimensions, and ba is a learnt bias. While SSM models allow for complex- valued matrices A,, B;, C, D. here we restrict ourselves to real-valued matrices as in
A, - A,A B; = A,W.
(4)
2.3 Recurrent Neural Networks
Similar to SSMs, RNNs also parameterize the imput-outpu relationship via a recurrent computation, commonly given by the long short-term memory (LSTM) [16), i.e., at each time step i
I; fa ② Ti-1 + i O ūi,
(5a)
y/ 0; O tanh(x;),
(5b)
where ū; represents the pre-processed raw input u,, i.e.,
ü; tanh(W, ,u, + Uu))-1);
(6)
and fú, in and are the forget gate, the input gate, and the output gate, respectively,
f: (Wy; + Usb i, = oW, Ua) o = oWW4,, + Ua)i-1);
(7)
where W,, W. W, and Uy, U,, U. are the learnt gate parameters. In this paper, we focus on two variants: quasi LSTMs (qLSTM) 19. which removes the output dependence of the gates, and RG-LRU 100, which attempts to integrate ideas from SSMs into RNNs.
3

---

qLSTM.
The qLSTM model is parameterized by recurrence 5 with pre-processed input ü; and
gates fu. ins of
U. = tanh(
fi = oWoa)) i = oW, = oWW).).
(8)
RG-LRU. The RG-LRU model presents a hybrid between qLSTM and a SSM using the recurrence
2; = 0g ① 2 + V 11 of 0 (4
(9a)
y: 3ig
(9b)
with the following gates and no pre-processe pre-prceessing of uy:
Ti = (Wa i = oWau; =- (re-rOoooll)))(A))
(10)
Dynamical Systems Framework for Architecture Comparison
In this section, we introduce the Dynamical Systems Framework (DSF) that allows in-depth analysis of the architectural features of attention, SSMs, and RNNs from a dynamical systems perspective. We use this to rewrite the parametrizations in common framework and provide detailed comparisons.
3.1 Dynamical Systems Framework (DSF)
The DSF relies on dynamical systems representation of the architectures. dynamical system models how system's 's state, here denoted by h, evolves over time according to difference or differential equation. Dynamical systems often evolve under the evolution of some input u, and the observable is output These systems capture time-dependent processes, rendering them suitable for understanding the behavior of sequence models. Here, we choose a recurrent state space representation. This choice is motivated by the widespread use of state space model representations for dynamical systems. Moreover, we show in later sections that this representation encompasses attention, RNNs, and SSMs in a suitable fashion that allows for further analysis. In particular, linear structured time-varying (LTV) dynamical system defined by the recurrence
h = Agh4-1 + Buu,
(1la)
3/ = C,h+ D2u,,
(11b)
where h4 RN is the hidden state initialized with h-1 0. A, E RNN is the diagonal state transition matrix, B; E Nxd and C. E dd are the input and output matrices, respectively, and D, Raxd scaled skip connection. Dynamical system (TT) can alternatively be written in its convolutional representation, i.e., y = ou, where the convolutional kernel & is defined as
Co + Do Bo
C, + Dı
Φ
(12)
Ct. C AgBo
CIArBBI-11 CrBr, + Dr.
Note that the convolution kernel d is of the same dimension the attention matrix C(gk and that these matrices are equivalent, up to scaling factor Wr used self-attention.
Remark 1. This recurrent view yields causal convolution kernel by definition. However, certain models (e.g. non-masked attention) also use non-can con-causal kernels. This can be incorporated in the DSF (II) by modifying the state update aT For the sake of simplicity and consistency with the recent literature, we slick with causal models in the following.
3.2 Architecture Reformulation
In the following, we show how popular architectures based attention, SSMs, and RNNs can be rewritten into the DSF. To do this, all models will be reformulated into recurrence (II). i.c., all resulting DSF representations will have hidden state dimension N Although the parametrization
Thh dimensions used in the following visualized in Appendix A for clarity.

---

of models commonly found in the literature is conductive to efficient computation, here we depart from this convention. The goal of the DSF reformulation is to establish theoretical framework that leads us to mathematical insights on the design of these models. The presented formulations are not intended to be computationally implemented in DSF form, however the framework can be used to identify computational algorithms for new architectures. For instance, the convolutional form of linear attention (12) is efficiently implemented via flash linear attention [17]. However, using the recurrent form derived below it can also be implemented via scan algorithms [18), e.g., parallel scan [5] or accelerated sean [9]] Given that the structural requirements on the model parameterization of the algorithm is met, the DSF allows to identify existing algorithms to apply to new models even the algorithm was designed for another model class.
3.2.1 Attention
In the following, we assume that we can separate the nonlinear map in 2 as
<(u k2) (lu:) ((
(13)
"1 k;)
where o(.): :T R R", v()) R + R", and (7: : I x mx+(())) - R, which is the case for all the considered architectures in this paper. Note that if ((-) is kernel function, the proposed separability is satisfied by construction, as it holds that Φ = s and 1. This allows us to write the self-attention input-output relationship
p((:) ((a)))()))))))))
1/
Wyug,
(14)
(9.,k,)
with qi Wou, R"" = Wvuy R and Wi Rad WK E R Wv E mddd Hence, equation (14) can be reformulated into the DSF (IT) as dynamical system of dimension N = nd, i... with hidden state h, E gnd and dynamic matrices
n(9i-1, k, Ind Inndxdd nig: k,)
A
(15a)
B. ( 7( 1/
Wv Randdd
(15b)
C Tla p(g:) RRdxnd
(15c)
We note that for the recurrence (T). the matrix A, is given an nd nd matrix, where is the number of features in ø and vb, and d is the input dimension. However, due to the scalar structure
of A; in (15a), it can be implemented the scalar multiplication n(g.k;) 1o0=_..kiihh in [TI. Hence, the hidden state is never materialized as such in the computation of the attention scores. Interested readers are referred to Appendix B for detailed derivation.
Linear Attention. In the case of linear attention, both maps <(-) and (b(-) in the DSF parametriza- tion [5) are separable and we use the kernel proposed in [14, i.c.,
o = elu(g:) + 1, (kg)) = clu(k)) +1, n(gi k;) = (elu(g:) + 1) Σ (elu(k)) + 1), (16)
j=0
where elu(-) is the exponential linear unit.
Generalized Linear Attention. We also study generalized linear attention, where we require that the maps ((-), (+))) are linear, but allow for general nonlinear normalization functions 1766 k,), i.e.,
)(g)) ge. (k,)
ký, n7q,, k.).
(17)
Softmax Attention. Softmax attention also satisfies the assumption of separability 13 However, holds that the feature vector representation of the transformed Gaussian kemnel in the softmax function, i.e. e.. k; is infinite dimensional. Hence, the DSF representation (15) of softmax attention 2 and its corresponding hidden state dimension N would also be infinite dimensional. This insight gives further motivation to approximations of the softmax function by using, e.g., Taylor series approximation such in 220). to render the feature vector finie-dimeesional
5

---

Lemma 1. Softmax attention
can be expressed by separable attention
13 with
o(a:) [(kk) - ( - edi ky mla. k:)
Σ c ky
(18)
X j= -i.... is an infinite-dimensiooue feature vector and is where = c- j=1 i
matrix of constant coefficients.
Proof. The exponential in softmax attention e R; can be expressed in terms of its Taylor expansion, which consists of an infinite sum of polynomial kernels of increasing degree decomposable through
the vectors monomials x
qi See Appendix C for a complete proof.
The work in analyzes softmax attention kernel smoother and 14] shows that kemnel-based formulation can lead to linear complexity in sequence length for finite dimensional kernels. In [22), kernel-based formulation of softmax used to propose orthogonal random features model softmax attention with linear complexity. In (20] a Taylor approximation of softmax attention is proposed, also leading to linear complexity. Finally, [23] relates transformer decoders to dynamical systems with increasing state size arising from the masked upper triangular part of the attention matrix. Compared to these works, we analyze how the proposed formulations compare in the recurrence (T1) allowing us to compare SSMs and RNNs in the following sections.
3.2.2 State Space Models
SSM models are straightforward rewrite in the DSF given their intrinsic recurrent linear representa- tion. However, similarly to attention, we slightly rewrite the standard representation introduced in the literature to reveal deeper insights obscured by the standard representation focused on computational efficiency. The detailed derivation can be found in Appendix
S6. The S6 parametrization can be written in the DSF
as
A = é (044))04 E ndxad
(19a)
B. A @ b E Rndxd
(19b)
C = La @ c Rdxnd
(19c)
with A; (liggssofpluussaaaeaa (W..)) ba)) Rddd by WB E R G = Wen R and Wa R2xd Wa dd are weight matrices with < d, and ba E Rd is bias. Note that in formulation (19 the dimensions the matrices are A₃ E Runxxnd B, E R C, E pdxnd ie., n
is the state dimension and is the input dimension in the original formulation
3.2.3 Recurrent Neural Networks
Given their recurrent nature, one can express LSTMs in the DSF with some basic algebraic manipulations (see Appendix(Fforr details). Once again, we slightly rewrite the standard representation since our goal is to obtain mathematical insights as opposed to computational efficiency.
gLSTM. In order to write the qLSTM formulation S in the DSF [T], a small modification is needed. In particular, the tanh functions in the input pre-processing (8) and output gate (5b) need to be dropped. Hence, the reformulated qLSTM in the DSF writes as
A; (Wy.)) Radd
(20a)
B, = di.g(o @ W., E Rdxd,
C, = =idiia((( Raxd
(20b)
where W,, W,, W., Wa E Rdxd are the learnt parameters (8. is important to note that here the dimension of the hidden state h; is equal to the number of input channels d, whereas in attention and SSMs the dimension of the hidden state h; the DSF (TT) nd. For SLSTMs n = 1, which will become relevant in further discussions; refer to the fact that >> 1 as state expansion.

---

RG-LRU, Given the similarities of RG-LRU [0] and SSMs, it is straightforward to reformulate it into the DSF (IT) without the need for modifications besides simple algebraic manipulations. Hence, the RG-LRU can be expressed in the DSF
A = e ( E 12dxd
B, V A2 diag(o(WBu))) E Rdxd C, = La (2la)
where r; = dins() and the function / - - is applied elementwise to A, Similar to the qLSTM and in contrast with the other models, RG-LRU does not have state expansion, i.e. n 1.
4 Architecture Comparison: Theoretical and Experimental Results
In this section, we use the DSF to explore some of the long-standing questions between attention, SSMs, and RNNs. We provide theoretical results and/or numerical experiments substantiate our claims. The experiments presented below are performed on the multi-query associate recall (MQAR) (24] and long range arena (LRA) B] benchmarks using the code bases M provided with the benchmarks. The complete experimental setup and computational resources used are detailed in Appendices D and K respectively, and statistical analysis provided in Appendix L
4.1 Softmax Attention vs. Separable Attention.
Separable attention is used to avoid computation of the query-key matrix qk It allows to compute k v before multiplying the queries q. which reduces the computational complexity from quadratic to linear in sequence length. While the DSF shows how separable attention, and in particular kernelized attention can be rewritten recurrence (T). such reformulation is only practical for a finite state dimension. However, in the case of softmax(-). an infinite-dimensine dimensional kernel is needed, i.e., in the DSF, softmax attention requires good performance observed for softmax attention can only be approximated by separable attention mechanisms, SSMs, RNNs; but other architecture is equivalent. The DSF predicts that softmax can be better approximated by growing n, which we show in the following theoretical result.
oc. This insight can mathematically explain why the
Lemma 2. For two dynamical systems ( with hidden state dimensions N and N with N < N. the dynamical system of state dimension N can always recover the dynamical system with state dimension N.
Proof. The result follows from the fact that the first N states and the output in (1) be chosen to be independent of the additional states. The full proof is given in Appendix D
L: 256, KV-pairs: 16
L: 512, KV-pairs: 64
1.0
Accuracy
0.5
Linear attention (16)
Softmax attention (2)
0.0
32
64
128
256
32
64
128
256
State expansion n
State expansion n
Figure 1: Comparison of linear attention and softmax attention on two MQAR tasks {(L 256, KV-pairs = 16), (L 512, KV-pairs 64)), fixed model size d 512, and varying state expansion n. We report the best result from a learning rate sweep in np mol.oggp (4 -2, 2).
Therefore, holds that the expressivity (and state dimension N = nd), if the rest of the architecture is held constant. As the softmax attention has an infinite hidden state dimension, i.c. n = oo (Lemma [T). we investigate empirically how its performance compares to linear attention (16). with increasing state dimension the MQAR. Figure T shows that with larger n linear attention converges to the performance of softmax attention, which achieves perfect accuracy throughout.
model non-decreasing with increasing state expansion
http://giithi.ccooHHayyeessaooos http:://gthhb.oommoogggeeeeeae/e
7

---

1.0
Softmax att. (2)
SSD (Dao and Go. 2024]
Linear alt. (16)
S6 [Gu and Dan, 2023]
Accuracy
Normalized att. (21)
0.5
Model
LRA
WikiText
Linear Att. T6
53.52
17.42
0.0
Norm. Att. (22
58.08
16,43
Softmax Att. (2)
55.96
13.15
64
128
256
512
S6 4
66.84
N/A
Model size d
Figure 2: Model accuracy with increasing model size for different models: softmax, linear, and normal- ized attention, S6, and SSD. The MQAR task is (L = 512, KV-pairs = 64), we fix n 128, and re- port the best performance of a learning rate sweep np. 1ww.pacecce(cccc (-4, -2, 4).
Table 1: Average accuracy on the LRA
benchmark and training perplexity score for different attention architectures (70M
params) on the WikiText-105 corpus.
4.2 Generalized Linear Attention vs. S6.
By comparing the DSF expressions for both generalized linear attention (15) and S6 (19), we notice that the S6 parameters b; = WBu, E R" C Wen; E IR" directly correspond to the keys and queries attention, i.e. k = b; and qi C.. Moreover, the state expansion n in S6 the same as the hidden dimension in attention. However, while this leads to an equivalent output matrix C₁ in the DSF parametrization for both architectures, there are remarkable differences between the two:
* Number of parameters. The transition matrix A, has d parameters in S6 (9) and only 1 in attention. In attention (15), 17(2, k) is the only parameter in Aç, and it by definition a scalar. In S6, the parameters in A, are determined by A Rd,, which has different parameters. However, it was shown in (8) that the number of parameters in A, can be reduced 1- similar attention without compromising performance. Note that multi-headed attention increases the number of parameters in A; from 1 to the number of heads for more details see Appendix G
Normalization strategy Discretization step. In attention 15 normalization map n(.) is used. This map enters as a fraction in A, and also as a denominator in B, Given that this map is scalar, these two cancel out when computing the output, as the convolution representation (1).. Hence, in attention
in AaB, evaluates k,) Notice that this does not occur S6 [19, since the only shared parameter discretization step A; does not cancel out in A; and B; given their different structure. This impacts the selectivity of the matrices on the input, since some input-dependent features are normalized differently in the two architectures.
While the number parameters the state transition A, does play a role in increasing performance (multi-headed attention typically performs better than single-headed attention (2), the results in [ suggest that this role is small. The larger influence thus lies in the recursive structure of A, and B, and/or the parameterization of normalization 17()). To further investigate this and the role of normalization in attention, we compare S6 and softmax attention to SSD [8]. linear attention [14], and normalized attention on the MQAR 24 and LRA [3] benchmarks and train the three aitention-basee methods on WikiTexxt-109 103. Inspired by S6, we define normalized attention as the attention function
((g;) = ys: (kk)) = ki, r/(u;) e Wya:
(22)
where W,, Rlxd an additional learnt parameter. In Appendix H we discuss two alternatives to (2). The MQAR results are shown in Figure 2 and the average accuracy on the LRA and the training perplexity on WikiText- 103 in Table 1. The MQAR results suggest that proper normalization, ie., using normalization 22 improves the performance of linear attention schemes. This is further supported by the performance of S6 and SSD on the MQAR benchmark, since these two methods also employ input-dependent normalization. Additionally, normalized attention closes part of the gap to softmax attention on the WikiText- 103 dataset (Table [T]. However on LRA, SSM models (S6) still achieve considerably higher performance than attention- based models. While normalized

---

attention outperforms linear and softmax attention on the LRA, it performs significantly worse than S6. This result suggests that while the S6 inspired normalization helps to improve performance, the remaining performance gap is possibly explained by the recurrent normalization strategy employed by selective SSM models. Overall these results warrant further research into normalization strategies for attention-basee models to explain the performance difference to SSMs. The complete experimental results on the MQAR and LRA benchmarks are detailed in Appendices L and M respectively.
4.3 RNNs vs. S6
Comparing RNNs and S6, it is immediate to observe several similarities. In particular, shown in Appendix the state transition matrix A, S6 (99 can be rewritten (assuming A = a -ndd) as
A dieg (W (Wau))") @ l..
(23)
Notice that when state expansion is considered, i.e., 1 and U, 1, this expression almost coincides with the QLSTT state transition (20a), with the only difference that (I) it uses the reversed sigmoid function instead of sigmoid for the forget gate, and (II) there is an additional learnt parameter a in the exponent. Inspired by the subtle difference in the state transition, we compare the original qLSTM state transition (8) and the S6-inspired state transition 23 on the MQAR benchmark. The performance of both models is shown in Figure 3 We note that the reversed sigmoid state transition outperforms the original state transition on all three benchmark tasks, i.e., the performance of qLSTMs can be improved by insights from S6. Considering state expansion (n > 1) for RNNs, the recent xLSTM paper [12] shows that state expanded LSTM; can yield similar performance to S6. This aligns with Lemma 2 and further highlights the importance of state expansion for expressivity. In qLSTM and RG-LRU, state expansion can be easily incorporated by changing the dimensions of the projections W,: W,, W.. where the @ operation in RG-LRU would be replaced by blockwise operations @ Finally, the most apparent difference between the two RNN variants qL.TMM and RG-LRU and S6 is the parameter coupling in A; and B,. While qLSTM does not use a coupling, the couplings in RG-LRU and S6 are performed with different nonlinearities, which is discussed in more detail in [00 Appendix A].
L: 64, KV-pairs:
128, KV-pairs: 8
L: 256, KV-pairs: 16
1.00
1,00
Accurncy
Accuracy
Accuracy
0.75
0.8
0.90
0.50
0.85
QLSTM
4LSTM w/ (23)
0.6
0.25
0.80
0.00
128
256
512
128
256
512
64
128
256
$12
Model
Model
d
Model size
Figure 3: Comparison of QL.STM (8 and qLSTM variant where the original state transition A; replaced by (23)
Related Work
State-space models emerged from the S4 architecture by Gu et al. 25 who developed a new theoreti- cally principled approach to sequence modeling rooted in polynomial approximation theory (26). The result is transformer-like architecture 22. where attention is replaced by a linear recurrent neural network with special reparametrization. The design of $4 got later simplified in [4, (7,27), achieving otate-of-the-art performance on the long-range arena (LRA) K with highly efficient recurrent
mechanism leveraging convolutional views (28, or parallel scans 5 6
The high efficiency of SSMs (lincar processing) makes them particularly appealing when compared to softmax attention based transformers, where both inference time and memory suffer quadratically from sequence length. The S4 architecture found first successful applications in reinforcement
"The presented state expanded LSTM versions, cannot be directly translated into the DSF framework, since the gates not only depend on the inputs u, but also on past outputs 1 However, the used state expansion is essentially d, hence leading to DSF system of size d?

---

learning 29 vision [0), audio B1 as well as online learning [3] Initial attempts in language
modeling 33 34
supported by theoretical investigations 33 36 hint at some necessary architectural
improvements to unlock the NL. domain. Leveraging in-context learning arguments, a few works [37- [39 started incorporating input selectivity mechanisms [40] into SSMs. These efforts culminated
in the Mamba architecture 77], which proposed a highly efficient and light (in terms of parameters)
input selectivity strategy, with drastic improvements when comparing to earlier variants (H3 [33 and
Hyena [41) on text. This approach was also shown to be effective at byte level [42). Beyond text,
Mamba was recently applied to the vision domain 44
with outstanding results compared to
ViTs 45 both in terms of performance and efficiency. Other applications include e.g. genetics 466,
and point clouds [47]. Further, improvements on architectural aspects were proposed in [
The design of Mamba is also strongly supported by theoretical evidence showing precisely its superior expressive power compared to $4 [49]. This boost in computational power is due to Mamba's novel input selectivity mechanism resembling gating, which unlocks content-dependent reasoning 7 40) Interestingly, input selectivity brings SSMs closer to attention: in particular, Ali et al. (50) showed that the particular parametrization of Mamba can be linked to non-normalized softmax operator. This finding is also supported by evidence from language theory - Mamba and Attention can solve a similar class of problems [5]] Beyond Ali et al. [50) the connection between linear attention and
linear RNNs has been illustrated a few times in the literature [4 23 52 (3). Connections between
these architectures have also been carried out using tools from communication complexity in [54] (55). Compared to these works and to Ali et al. [50], this paper offers a more careful comparison identifying some precise distinctions between SSMs, lincar, and softmax attention which play a nontrivial role in practice and can help bring to light interesting architectural improvements.
6 Conclusion
In this paper we presented the DSF, framework based on synamical-systent theory that allows analysis of different deep learning architectures by writing them as linear recurrences in state space. We first showed how to reformulate different architectures into the DSF, and then explored (theoretical and experimental) insights resulting from this analysis, thereby answering the questions posed in the introduction. For instance, we showed that with proper normalization the performance of linear attention can be significantly increased (see Fig. 2). We also show, that the DSF allows to integrate insights from one architecture to another exemplified by Section 4.2 Additionally, the DSF naturally allows analysis of the eigenvalues of the state transition matrix A, which are linked to the exploding/vanishing gradient problem [6]. In the case of SSMs and RNNs, the eigenvalues are constrained to be stable by construction, for attention-based models this is not the case and stability needs to be obtained via normalization. The eigenvalues together with the state expansion also affect a model's long-term memory [6). Both of these aspects can be analyzed via the DSF and should be further investigated in future work. While the training dynamics (especially convergence) can be studied empirically using experiments, the DSF also allows a theoretical analysis. As discussed in Example 2 of [56). gradient-based optimization algorithm (e.g. SGD) be interpreted and written as a dynamical system. Using this viewpoint together with the DSF allows interpretation of the training dynamics as two interacting dynamical systems. Therefore, the training dynamics can be theoretically analyzed using tools from control theory, e.g., via Lyapunov theory for convergence and stability of the training. However, we believe this question requires an in-depth investigation and additional empirical validation of the theoretical findings. We expect that the DSF can serve as a tool for principled analysis and design of deep learning architectures.
Linitations. In terms of limitations, it is important highlight that, while the DSF parametrization allows for a principled comparison between frameworks, architectures written in the DSF do not necessarily enjoy an efficient implementation unless their specific structure can leverage some of the existing algorithms (parallel scan, etc.). In terms of experiments, the insights mentioned above are only verified on two synthetic tasks (MQAR/LRA) and a smaller language task (WikiText-1033, 103). To strengthen the insights, a more detailed analysis is needed on larger and more complex tasks.
Acknowledgments and Disclosure of Funding
Carmen Amo Alonso was partially supported by an ETH AI Center Postdoctoral Fellowship.
10

---

References
[1] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2208.0722: 2021.
[2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[3] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long Range Arena A Benchmark for Efficient Transformers. In International Conference on Learning Representations (ICLR), 2021.
[4] Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the Parameterization and Initialization of Diagonal State Space Models, 2022. URL htt.:ttts///rrx..orrg/bbss
11893
[5] Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified State Space Layers for Sequence Modeling. In The Eleventh International Conference on Learning Representations, 2023.
[6] Antonio Orvieto, Samuel Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting Recurrent Neural Networks for Long Sequences. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 26670-26698. PMLR, 23-29 Jul 2023.
[7] Albert Gu and Tri Dao. Mamba: Linear- Time Sequence Modeling with Selective State Spaces, org/ http//bb/231...0000
2023. URL http:::ttt///aaee
[8] Tri Dao and Albert Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms with Structured State Space Duality. In ICML 2024, 2024.
[9] Aleksandar Stanić, Dylan Ashley, Oleg Serikov, Louis Kirsch, Francesco Faccio, Jürgen Schmidhuber, Thomas Hofmann, and Imanol Schlag. The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute, 2023.
[10] Soham De, Samuel Smith, Anushan Fernando, Alcksandar Botev, George IristianMuurayy Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models, 2024. URL htthhtts::: pattts::/arxt/t //arxiv. org/ hts///bb//240 19427
[11] Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong HGRN2: Gated Linear RNNs with State Expansion. arXiv preprint arXiv:2404. 07904, 2024.
[12] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended Long Short-Term Memory. arXiv preprint -=Xxv:2405.045/7 2024.
[13) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[14] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Flcuret. Transformers are RNNs: fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning. ICML'20. JML,. 2020.
[15] Carmen Amo Alonso, Jerome Sieher, and Melanie N Zeilinger. State space models as foundation models: A control theoretic overview. arXiv preprint arXiv:2403. 16899, 2024.
[16] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()) 1735-1780, 1997.

---

[17] Songlin Yang and Yu Zhang. Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024. URL
flash- Beash-linear--ee atteniionn
[18] Guy E Blelloch. Prefix sums and their applications. 1990.
[19] Volodymyr Kyrylov, Accelerated Scan, January 2024. URL httpss/s/gtttuuu.oo//o/r/o/////e
accelerated-scan
[20] Tobias Christian Nauen, Sebastian Palacio, and Andreas Dengel. Taylorshift: Shifting the complexity of self-attention from squared to linear (and back) using taylor-softmax. arXiv preprint arXiv:2403. 240.00200, 2024.
[21] Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: unified understanding of transformer's attention via the lens of kernel. arXiv preprint arXiv: 1775, 2019.
[22] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
[23] Matanel Oren, Michael Hassid, Yossi Adi, and Roy Schwartz. Transformers are multi-state mns. arXiv preprint arXiv:2401.06104 2024.
[24] Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and Improving Recall in Efficient Language Models. arXiv:2312.04927, 2023.
[25] Albert Gu, Karan Goel, and Christopher Ré. Efficiently Modeling Long Sequences with Structured State Spaces. In The International Conference on Learning Representations (ICLR), 2022.
[26] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. HiPPO: Recurrent Memory with Optimal Polynomial Projections. Advances in Neural Information Processing Systems, volume 33, pages 1474-1488 Curran Associates, Inc., 2020.
[27] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. In Advances in Neural Information Processing Systems, volume 35, pages 22982-22994. Curran Associates, Inc., 2022.
[28] Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. What makes convolu- tional models great on long sequence modeling? arXiv preprint arXiv:2210.09293. 2022.
[29] Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning. Advances in Neural Information Processing Systems, 2023.
[30] Eric Nguyen, Karan Goel, Albert Gu, Gordon W. Downs, Preey Shah, Tri Dao, Stephen A. Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals using state spaces. Advances in Neural Information Processing Systems, 2022.
[31] Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-space models. arXiv preprint arXiv:2202.09729, 2022.
[32] Nicolas Zucchet, Robert Meier, Simon Schug, Asier Mujika, and João Sacramento. Online learning of long-range dependencies, 2023.
[33] Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry Hungry Hippos: Towards I Language Modeling with State Space Models, 2023. URL https://arxiv. org/ abs /2tp./221..cm 14052
[34] Junxiong Wang, Jing Nathan Yan, Albert Gu, and Alexander M Rush. Pretraining without attention. arXiv preprint arXh::2112.10544, 2022.
12

---

[35] Antonio Orvieto, Soham De. Caglar Gulcehre, Razvan Pascanu, and Samuel L Smith. Univer- sality of linear recurrences followed by non-linear projections: Finite-widt guarantees and benefits of complex eigenvalues. International Conference Machine Learning, 2024.
[36] Shida Wang and Beichen Xue. State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory. Advances in Neural Information Processing Systems, 2023.
[37] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing mns for the transformer era. arXiv preprint arXiv:2305. /3048, 2023.
[38] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma. Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models, 2023.
[39] Tobias Katsch. Gateloop: Fully data-controlled linear recurrence for sequence modeling, 2023.
[40] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.1 1895, 2022.
[41] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pages PMLR, 2023.
[42] Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. Mambabyte: Token-free selective state space model. arXiv preprint arXiv:2401.13661 2024.
[43] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401. 2024.
[44] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. 2024.
[45] Hugo Touvron, Matthieu Cord, and Herve Jegou. Deit ii:: Revenge of the vit. arXiv preprint arXiv:2004.07118, 2022.
[46] Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint vrXiv:2403.03234 2024.
[47] Jiuming Liu, Ruiji Yu, Yian Wang, Yu Zheng, Tianchen Deng, Weicai Ye, and Hesheng Wang. Point mamba: A novel point cloud backbone based on state space model with octree-based ordering strategy. arXiv preprint 24Xiv:2403.06444 2024.
[48] Songlin Yang. Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficiem training. arXiv preprint arXiv:2312.06635, 2023.
[49] Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, and Terry Lyons. Theoretical foundations of deep selective state-space models. arXiv preprint 1XXi:24422.19477 2024.
[50] Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models, arXiv preprint arXiv:2403.012 2024.
[51] William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. arXiv preprint Xxx822404000819 2024.
[52] Imanol Schlag, Kazuki Irie, and Jürgen Schnidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, pages 9355-9366. PMLR, 2021.
13

---

[53] Nicolas Zucchet, Seijin Kobayashi, Yassir Akram, Johannes Von Oswald, Maxime Larcher, Angelika Steger, and Joao Sacramento. Gated recurrent neural networks discover attention. arXiv preprini arXiv:2309.01775, 2023.
[54] Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Re. Simple linear attention language models balance the recall- throughput tradcoff. In Forty-first International Conference Machine Learning. 2024.
[55] Satwik Bhattamishra, Michael Hahn, Phil Blunsom, and Varun Kanade. Separations in the representational capabilities of transformers and recurrent architectures. arXiv preprint arXiv:2406.09347, 2024.
[56] Florian Dörfler, Zhiyu He, Giuseppe Belgioioso, Saverio Bolognani, John Lygeros, and Michael Muehlebach. Towards systems theory of algorithms. IEEE Control Systems Letters, 2024.
[57] Amnon Shashua. Introduction to machine learning: Class notes 67577. arXiv preprint arXiv:0904.36644, 2009.
[58] Daniel Sôppez-Sánchez, Angélica González Arricta, and Juan Corchado. Data-independent random projections from the feature-space of the homogencous polynomial kernel. Pattern Recognition, 82:130- 2018.
[59] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, 2019. URL httppss//pppereeieee http:/ppperrviie.neeeffrrrrr
1d=Bkg6Ricqy7
[60]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Bterbert-Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL httph: // http:/tt///coceeeitliiiie neurips.....pp... sec/paper_fillee filestffiee/eeee Siaaper_filespppeeeeeeee http://aeriiileeespppe////////// pappr///22//f/ilee Pww.ee44a-app..ppp.. pdf
[61] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle D'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
14

---

A Visual Representation of the matrix dimensions
Figure represents the dimensions of the recurrence expressed by the linear structured time-varying (LTV) dynamical system described in TTI
h4 A + Ba
where lig E R is the hidden state, A, E IRNNN is the diagonal state transition matrix, and B, Raxd is the input matrix. We highlight the role of the state expansion, where Rd and
E = =2==
A
hi
B
Ui
72
72
N
d
71
state expaiision
N
d
Figure 4: Visual representation of the dimensions considered in the DSF.
B Derivation of Separable Attention into DSF
We consider the layer
y1
Σ j=0
flae, kj, k;)
where we define the sequence of keys as
k; .k....... kj).
We show that this layer can be equivalently written as the LTV system 11. i.e.
C
II Ae
Byuj,
3=0
A=j+1
with A; E Bnnxnd, B₁ E Rdd and C₁ E Raxnd if the function f(,,)) is separable as follows
f(a, k.) (aa:TT((s))
n(25, k;)
where (gg:) E RR, (kyy) E RR, and n(g: k;) E IR can be considered to be normalization function. If the unnormalized part of f( is kernel function, it holds that (kk) o(k,) for some, possibly infinite dimensional, feature vector ő.
We can compare the output formulations
y/ Boun
and
yo f(go, No. Wo)
1 T(go)T n(go. ko) (ko)
15

---

resulting in
Bg 1
Ta ((k)) Wv Ranxd Co Ia @ p(20)" BRdd
1190, ko)
Simlarly, we have
y1 CiBiur + Bou
V1 I(q1 k1, W + f(11, ko W
1 (ko)
= =(q1)) ((q,k ((k)) + o(q))
n(gi, ki)
77900 ko) Und
+ B1
la 8 (ki)) Wv, Ci = La @ )aa)) and A₁ n(11, k1)
(1)
and
1/2 = C2B2u2 + C2A3B1u + C2A2A1
1/2 f92, 12 W2) + flg,, k1 Wyy + f(g.. No, k
1
1/2 x9 T
1 (k2 W +((2))
1992, k2) (kk)) + +
&s (ko)W
1142, k2)
n(92,
7(11, ki)
+ B2
Ta & (b(22)) Wv. C2 = Ila @ 2l92) and A2 7992, k2)
Ind.
(7(2, k2)
Plugging this back in, observe
1/2 C2B2uz C2A2B1u
C2A2A1
1
La 1 T(2))
La (i(kz) Wvuz
1112, k2)
II ((12)
1111, k1)
nd
1
Ta 2 (iki)) Wru
1122; k2)
(1(1,,1)) k1)
n(q1, k1)
Lind n190 ko)
(1(00 ko) ta 8 (o Wyno
1
+14 @ )(g))
1122 k2)
1)(11, k1)
= =(())) T
wb 12 Wv
(l(2)T u2 V2
1
(11) ((92))
((22. k2) (bok)
n(92, k2)
n k2)
= f((2, k2 W2) + (42, k k2) Wru + f(22, ko, k2 Wymo uo This generalizes the dynamical system matrices to
La & ((()))))) Wv,
B nng., k,)
C 1. Φ(q:)
A; 7(91-1, k(
Ind.
"(y;; k;)
C
Proof of Lemma 1: Derivation of Softmax Attention into DSF
The softmax function R + (0, 1mnxm is defined through row-wise normalization and is given by
50,0
50.m
mm
softmax(z)
Fm.m m Ey= 2ni
Pn.D
The attention block is given
V: = (a k)v,
Σ
(uW TWWu,
1=
1=
16

---

where softmax., refers to the element (i,j) of the corresponding matrix. Note that e Ky is kernel, implying that softmax attention can brought into the separable form (3)). order to provide the separable form of esl ki we first consider the Taylor expansion of the kernel, which is given by
(a kj) p!
e
p=0
Each polynomial (q. k,) ) represents itself homogeneous polynomial kernel and its decomposition
into feature vector of T monomials, shown in [57] is given by
(x))
V
p
(24)
n, m.Ep
The feature representation of the exponential kernel is therefore
e9: ky = -1, 2)0(()) T ..... [1, / "
(25)
= =(q)) ((k)).
Note that to attain the monomials in 24 for a given p. one can also use x
==1 x, as given in, c.g., [58
which is equivalent up to the constant coefficients, such that we can use V (z) cp 0
where is matrix of the respective coefficients multiplying each monomial.
D Proof of Lemma 2
Given two dynamical systems of the form IT.. we denote the system of hidden state dimension N with the state h. and the matrices AW, B and DIN and correspondingly. the system of hidden state dimension using the state h and the matrices A B. C and D. We show that the system of dimension N 2 N can recover the system of dimension N by selecting the following system matrices:
AN
0 7
B = B B
A =
A
C = [ON 0,, Dj = D.Y.
It can be seen that the N first states of the system with dimension N propagate equivalently the states of the system of dimension N. The additional states evolve independently given any matrices A, A, of appropriate dimension and do not affect the output, such that both systems are equivalent. The two outputs are then equivalent by setting the corresponding entries of the C. matrix to 0.
E Derivation of S6 into DSF
While A in (4) is represented as a dense matrix of size n X d purely for computational reasons, mathematically A diagonal matrix of size nd x nd. This is evident from the fact that S6 parameterizes a different submatrix Ad E Raxx for each embedding dimension d, leading to
A
A
Ad
To compute A, in (T], the matrix A is multiplied with the selective discretization time A; E [Rdxd which is computed as
A diag(softplus (WA (Wa(W.u) + ba)),
(26)
17

---

where Wa Rexd, Wa R are weight matrices with < d, and by R is bias. Note that we embed the computed discretization times diagonal d matrix simplify the next reformulations. The product of A; and A is performed along the embedding dimension axis, i.e.
A
(A. O.
(27)
A
To arrive at the DSF formulation (9). it only remains to take the exponential function of 27 and state B;, C2 as in with the appropriate dimensions.
F
Derivation of LSTMs into DSF
F.1
RG-LRU
In order to replace the abundant elementwise operations @ in LSTMs with more suitable matrix-vector multiplications for SSMs, we rely on the following observation for 0; E IRd, u, E Rd:
To(aa)
(a:) @ uy
diag(o (a(a))
(a
o (aa))
As with S6 in Appendix E we reformulate some quantities for easier presentation, namely we embed the input-dependent vectors (WWR), Rd (W))) Rd. where a-)) denotes the sigmoid function, in diagonal matrix, i.e.,
r
b
r; (iRu()
b, diag(a(WBu.))
b
The DSF representation
21
is then obtained straightforward manner.
F22 qLSTM
We start by using the same reformulation of the gates as RG-LRU above:
fa = (iag(ooWyull
i = diia((((W:.)))
= di.)(o(W.::)),
where fi is commonly called the forget gate, i; and o; are called the input and output gates, re- spectively, and W,, W,, W. E R By removing the tanh activation function, we effectively eliminated the input activation gate, which now serves standard input to the recurrence (11) i.e., we reformulated the standard QLSTM 8 to the LTV
r; faZt-1 + (ig @ W.)u y: ozh4
G Dynamical System Derivation of Multi-headed Separable Attention
As in Appendix we assume a separable attention function, i.e.,
f(gis ky,k), k.) ((q:) TT) (k(;))
n9gs k,)
Additionally, we consider the multi-headed setting introduced in (2 Section 3.2.2], i.e., different attention operations are performed in parallel. Due to the right multiplication of the input (instead of left multiplication as in the original paper), the output of the different heads is stacked row-wise instead of column-winn wise. Additionally, we assume there is no output mapping after the attention
18

---

operation. This yields the simplified multi-heade layer
(laa) vocks) k.)
plaa) ((()) (().k))
uy
V1 =
==0
ml T (M(( 2 2v. ke)
kl)
T]'
ml9.kk)) 1 Un/s
Σ
j=0
[o(a)) 1
k.). ]" Unso_
[ky))
[oy]"
[uy]]
where - denotes the head index. As is standard for multi-headed attention, reduce the dimensions of the queries, keys, and values by the number of heads, i.e., E Rm/", /, k. E Rm//, and vy E RR//.. Since the different values [uy]* are stacked, this equivalent the single headed version, i.e.,
[uy]
= vy = Wyuj.
[uy]
Above observation is also valid for the queries and keys, i.e., we can e.g. write [p(a)]] using an indicator function (.) on the single headed query qı i.e.
[o(a:)] ((I.(a)) = (WQ,)))
This shows the intuition behind muati-headed attention, which essentially compares parts of the single-headed queries and keys in parallel. Therefore, we can use Appendix to write multi-headed separable attention in the DSF
[nlgi-1, k,- Id/s..... (n( (ike-1,k-1))) ki- Mals [n(g k;)l"
A; diag
In Rddddd
(28a)
[n(ge, k,)]²
B. diag
Tajos-...
1n(a-11,k-1))> Fa/s
@ (II,
Wy Rudxd
(n(9-11,k-1))
(28b)
C. La s (I:))) E BRdxnd
(28c)
Multiple heads therefore extend the single scalar in A₃ (in the single- case) to different scalars, however these only upon a part of the queries and keys k, due to the indicator function.
H Alternative Normalization Schemes
For all experiments in Section 4.2 we use the normalization scheme in (2
The exponential normalization function ry(u;) is inspired by softmax attention and S6, which both use exponential
functions for normalization (see 18 and (19). However, other normalization functions can also be considered e.g.
n(4) (V,tplus(W,,u;).. n(u))) (W,,)),
(29)
(30)
where (-) denotes the sigmoid function. Table 2 shows an experimental comparison of the exponen- normalization function (22) and the two alternatives (2)), 300p on LRA Image and MQAR (I = 512, KV-pairs- = 64) tasks. All three normalization schemes perform similarly on both tasks, however the exponential normalization (22) yields the best performance, which is the reason we choose it for normalized attention throughout the paper.
19

---

Table Accuracy the three normalization functions 22 (L 512, KV-pairs = 64)
29
(30)
on LRA Image and MQAR
Task [%]
Normalization Function
LRA Image
MQAR (L 512, KV-pairs 64)
Exponential
35.96
85.9
Softplus 29
35.27
84.3
Sigmoid
35.80
84.7
I S6 uses reversed sigmoid in state transition matrix
In the following, we show that the state transition matrix A, in S6 is essentially reversed sigmoid of the projected input. To show this, we assume for simplicity that A in A₃ e (A, is a scalar, ie.. A Ind- This assumption simplifies A,
A; -==-0\A
(31)
where each e- itself is a diagonal matrix with n-times e-a on its diagonal. order to analyze this expression, simplify the computation of A, by fusing the two projection matrices with out loss of generality, i.e.
A
W W + ba))
ssottplu (W (W)..
(W)u:)
softplus W (id:nn))
where W denotes the jth of WA- Above reformulation is valid since the softplus(-) function applied elementwise and we note that A = softplus (Waplus(Wa:ru)) in (TT) Using the definition of softplus(-) function, we can show that
aol - woutppluu (W:) (1 e r (W u.)
(32)
where (-) (-) is the reversed sigmoid, i.e., (x)((I)) Ite Since the reversed sigmoid is again applied elementwise to a vector or a matrix, we can write the S6 state transition matrix as
A diag(orev ( (Waw)") la,
where the power is applied elementwise. The assumption A = a Lnd we made in the beginning, can be relaxed any diagonal matrix, however the resulting A; will have more complex representation.
J Experimental Details
The experimental results provided in Section 4 are performed on the multi-query associative re- call (MQAR) benchmark (24), the long range arena (LRA) benchmark (3), and the WikiText- dataset. To obtain the MQAR and LRA results, we modified the Zoology and LR code bases and added the normalized attention model and the selective SSM models, respectively. The code for both benchmarks is provided on GitHub for MQAR and for LR separately.
http:///hhhbbcce//reeeeeettttttt -mqar
http:://gtabbbooo//////////ssatr
20

---

J.1 MQAR experiments
Training Details We evaluate the following three architecture classes:
1. Attention: softmax attention [2], linear attention [14. normalized attention 22 For
all three attention functions, we use a standard GPT-2 style multi-headed Transformer architecture, where we replace the attention block with the respective attention function. The three attention functions are defined in Section 2.1 For all MQAR runs we use single attention head.
2. State space model: S6 77]. SSD 8 For both SSM variants, we use a standard GPT-2 style single-headed Transformer architecture, where we replace the attention block with the respective SSM variant. This means for S6 and SSD, we do not implement the pre- convolution on the input or the SiLU activations; but just the S6 and SSD blocks. We do this to ensure fair comparison of the backbones (sequence mixers) irrespective of the higher-level architecture. The S6 and SSD blocks are defined in Section 2.2 and we use the provided code base implement it.
3. RNN:
qLSTM [9], modified gLSTM. We embed both qLSTM variants in a standard GPT-2 style single-headed Transformer architecture, where we replace the attention block with the gLSTM. We do this to ensure a fair comparison of the backbones (sequence mixers) irrespective of the higher-level architecture. The standard qLSTM defined in Section 2.3 and the modified qLSTM is the same as the standard qLSTM but with modified state transition 23 i.e., a modified forget gate.
For all MQAR runs, we use the following training protocol:
Optimizer and schedule: Weight decay 0.1, linear warmup with duration of 10%, AdamW optimizer [59]. For each run, sweep the learning rates in up apglppppa -2,
2,4 4) and train for 64 epochs. This is the same setup as in [24].
Training duration: We use a global batch size of 512, which we reduce to 256 if sequence length T, 2 128, to 128 if sequence length I. 2 256, and to 64 if sequence length L. 2 512. We do this to keep the memory consumption approximately constant over different tasks.
Width and depth: For all runs, we use two layers (each with sequence model and MLP. interlcaved with layer normalization). The model dimensions d, state expansion n. sequence length L. and number of KV pairs are swept according to the experiment (see Section This is the same setup in (24]
Position information: Positional embeddings 660 are used for the attention and RNN architecture classes, but not for the SSM architecture classes. This is the same setup as in [24).
Data: Each model is trained on 100,000 datapoints and evaluated on 3,000 datapoints. The data and its order are constant for all runs. This is the same setup as in 24
Performed Experiments We run the three attention models and the two state space models on four different MQAR tasks. i.c., {L 64, KV-pairs = 4]. {L 128, KV-pairs 8). { 256, KV-pairs = 16}, and {L = 512, KV-pairs = 64). which progressively increase in complexity. For each model and task, we sweep both the model size d [64, 128, 256, 512] and the state expansion n = |32, 64, 128, 256] To resulting in a total of 320 experiments. We only report the results of the best performing learning rate; the full results of all experiments We run the two qLSTM variants on three different MQAR tasks, i.e., {L = 64, KV-pairs = 4}. 16). For both variants we sweep the model
stated in Appendix
{L 128, KV-pairs 8). and {L 256, KV-pairs size d = [64, 128, 256, 512], resulting in a total of 24 experiments. We only report the results of the best performing learning rate; the full results are reported in Figure
htttpe://ggithub.coom// www./state-spccee/eeeeeeeaaaa
10 We did not increase n further, since the selective scan CUDA kernel provided for S6 and SSD is capped com/ http://www....eetttttepppeeee/aa
256; for more information see htttts://githiie http::////gstuub..mm/sttttttttte
21

---

J.2 LRA experiments
Training Details
We evaluate the following two architecture classes:
1. Attention: softmax attention [2], linear attention [14), normalized attention 22 For all three attention functions, we use a standard GPT-2 style multi-headed Transformer architecture, where we replace the attention block with the respective attention function. The three attention functions are defined in Section To ensure a fair comparison, we keep all hyperparameters of the three attention models constant except the attention function.
2. State space model: S6 77). We use a standard GPT-2 style single-headed Transformer architecture, where we replace the attention block with the S6 block. This means, we do not implement the pre-convolution on the input or the SiLU activations; but just the S6 block. We do this to ensure a fair comparison of the backbones (sequence mixers) irrespective of the higher-level architecture. The block is defined in Section 2.2 and we use the provided code base to implement it.
For all LRA runs, we use the following training protocol:
Optimizer and schedule: Linear warmup with duration of 10% and Adam optimizer [59]
Position information: Positional embeddings [60] are used for the attention architecture classes, but not for the SSM architecture classes.
Data: Each model is trained on the standard datasets provided with the LRA benchmark.
The exact hyperparameters for each LRA task and each model are reported in the publicly available code base Note that we not optimize the hyperparameters, i.c., the reported accuracies might be
lower than in the original LRA paper [3]
Performed Experiments We run the three attention models and the S6 models on the LRA tasks ListOps, Text, Retrieval, Image, and Pathfindes which are summarized below; for the full details we refer to 3
List Operations (ListOps): This task evaluates a model's ability to capture hicrarchical dependencies over long contexts. The goal is to predict the result of mathematical operation consisting of nested mean, median, max, and min operations The task is a ten-way classification task with maximal input lengths of 2k.
2. Text Classification (Text): This task evaluates model's 's ability to capture the of long tokenized texts. The dataset consists of IMDb movie reviews, which need to be classified as negative or positive in tone. The task binary classification task with maximal input lengths of 4k.
3. Document Retrieval (Retrieval): This task evaluates model's ability to compress long sequences into representations that are suitable for similarity matching. The dataset consists of tokenized papers published by the American Academy of Neurology (AAN), which need to be classified in having a citation link or not. The task is binary classification task with maximal input lengths of 8k.
4. Image Classification (Image): This task evaluates model's ability learn 2D spatial relations from a ID vector. The dataset consists of vectorized images, which depict one of possible classes, e.g. a horse or car. The task is ten-way classification task with maximal input lengths of lk.
5. Long-Range Spacial Dependency (Pathfinder -ondder-322 32): This task evaluates a model's ability to learn spacial dependencies depict two circles and multiple dashed paths. The goal is to evaluate whether the two circles are connected by any of the present paths or not. The task is therefore binary classification task with maximal input lengths of 2k.
vectorized image. The dataset consists of images, which
https://github. wwtpsooa/teeeee.e spasteee/ssppeee/ameaeaeea
httpp://gii http://tthuu..cojseee/e/// bennnbbencmaaik
For instance, input: max (4, min(5,6, mean (9, 4, 5))), output: 5.
22

---

J.3 WikiText-103 experiments
Training Details We use the 70M parameter Pythia architecture (Pythia70M) 61
For softmax
attention we use the standard Pythia attention block, while for linear attention [4] and normalized attention (22) we replace the attention block in the Pythia architecture with the respective attention functions defined in Sections 2.1 4.2
For all training
Wiki 103, use the following protocol:
Optimizer and schedule: Weight decay of 0.1, linear warmup with duration of 10%, A optimizer [59] with 8 = (0.9, 0.95), and gradient clipping = For each run, sweep the learning rates in [0.0003, 0.001, 0.003, 0.01] and train for 50 epochs.
Training duration: We use batch size of 128 and train for 50 epochs.
Width and depth: We use a context length of 1024 and the standard Pythia70M configura-
tion, i.e., model size of 512, heads, and layers.
Position information: Positional embeddings 60 used in standard Pythia.
Performed Experiments We train the three attention functions on WikiText-1O3 and sweep the learning rates [0.0003, 0.001, 0.003, 0.01]. For all three attention functions learning rate 0.003 performed best and the corresponding results are reported in Table 1.
K
Computational Resources
All experiments (MQAR, LRA, and WikiText-103) were run on cluster with nodes with the following GPU and CPU specifications:
GPU Model
Nr. of nodes
memory/GPU
GPUs/node
CPUs/node
NVIDIA GTX 1080
11 GB
20
NVIDIA GTX 2080 Ti
11 GB
8
64
NVIDIA GTX 3090
GB
128
NVIDIA GTX 4090
24
128
NVIDIA TITAN RTX
GB
128
NVIDIA Quadro RTX 6000
GB
8
128
NVIDIA V100
32GB
44
NVIDIA A100
40
48
NVIDIA A100
80 GB
10
48
The MQAR and LRA training and test runs were parallelized and assigned to the best available GPU node, while the parallelized training on WikiText-103 was exclusively run on the NVIDIA A100 (80GB) node.
For each learning rate sweep of the MQAR runs described in Appendix P we estimate the average runtime to be 1h T leading to total unparallelized runtime of 54 days for all MQAR tasks. There where approximately 20 runs for debugging and training purposes, which were terminated after few minutes, thus we did not include them in the time estimate.
For each task of the LRA benchmark, estimate the average runtime be 2h for Image, 5h for Text, 6h for ListOps, 30h for Retrieval, and 45h for Pathf inder, leading to a total unparallelized runtime of 31 days for all LRA tasks. Note that to obtain better hyperparameters, they would need to be tuned for each task, which would significantly increase the total runtime. There where approximately 30 runs for debugging and training purposes, which were terminated after few minutes, thus we did not include them in the time estimate.
///tttss//gitthhh com/Eleet hete//eleuhhrrrae I/pythia
15 Obviously, larger model sizes and MQAR tasks with larger sequence length longer than smaller models and tasks with shorter sequence length. However, a more accurate time estimate hard to obtain due to the cluster setup with multiple different GPU models and the fact that we terminated tasks early if the 99% accuracy threshold was achieved.
23

---

For the training WikiText- 103, each learning rate sweep took approximately 14h, leading to total parallelized runtime of 42h for all three attention models. We estimate total of 1h of runtime for luning runs, bringing the total runtime to 43h.
L Extended Results MQAR
Figures 5 and 6 seeds.
show the MQAR results of Figures 2 and 3 but for multiple runs over 10 different
Softmax all. (2)
SSD [Dao and Guu, 2024] (Oa and Dan, 2023]
Lineur all, (16)
Normalized st.
1.0
0.8
0.6
Accuncc
0.4
0.2
0.0
64
128
256
512
Model size
Figure 5: Model accuracy with increasing model size d for different models: softmax, linear, and normalized attention, S6, and SSD. The MQAR task is (L 512, KV-pairs = 64), we fix n 128. and report the best performance of learning rate sweep in np woogspccceeae -4, -2, 4). Solid lines are the average accuracy over 10 different seeds, while the shaded area show the standard deviation.
L: 64, KV-pairs:
128, KV-pairs: 8
L: 256, KV-pairs:
1.00
1.00
Accurncy
0.95
Accuracy
Accuracy
0.75
0.8
0.50
0.85
QLSTM
QLSTM w/ (22)
0.6
0.25
0.80
0.00
64
128
256
512
64
128
256
512
64
128
256
512
Model size
Model size d
Model size d
Figure 6: Comparison of qLSTM (8) and qLSTM variant where the original state transition A, is replace by (23). Solid lines are the average accuracy over 10 different seeds, while the shaded area show the standard deviation.
In Figure we report the complete results of all MQAR experiments detailed in Appendix J A selected subset of these are already presented in Figure and Figure in the main text.
The effect of state expansion can not only be observed for linear attention (Figure T but also for normalized attention (22), S6, and SSD for the task {L 512, KV-pairs = 64). Contrary to this, for small model sizes and larger tasks (e.g. normalized attention, task {L 512, KV-pairs = 64), d = 64) the performance decreases with increased state expansion n shows erratic behaviour. Since this behavior only occurs small d, we hypothesis that this effect due to the model being to small accurately learn the task.
24

---

T.: 64, KV-pairs: 4
L: 128, VV-pairs: 8
T: 256, KV-pairs: 16
T,: 512, KV-pairs: 64
1.00
Softmax att.
64
>99
>99
>.99
>99
>99
>99
>99
>99
1.0
0.75
128
>99
>99
>99
>
2.99
>99
>99
>99
>.99
>99
>99
>99
>99
>99
>99
0.50
256
299
5-99
1999
>.9
2-99
>999
>.99
2.99
5:99
3-99
99
3-99
>:99
2-99
3-99
2.99
0.25
512
2.9
2-99
,>99
>.99
2-99
>99
>>99
2.9
2-99
>.99
>99
209
>99
>*99
>99
2-99
0.00
1.00
64
Linear att.
0.75
128
999
0.50
256
1-99
9999
>99
15.66
3:99
2>99
2:99
28.0
2:99
3-99
4.2
96.2
0.25
512
3-9
1-99
3-99
2-99
2-99
3-99
3-99
3-99
45.7
88.9
2999
3-99
2.1
89.4
96.6
96.4
0.00
1.00
Normalized att.
64
99
93.2
37.4
17.5
0.75
128
2-99
0.50
256
>999
>.99
3-99
>:99
>.99
0.25
512
>-99
>~99
7>99
>>99
2>99
>99
>>99
>>99
2-99
>>99
>>99
>>99
>>99
>>99
2-99
0.00
1.00
64
99
92.6
0.5
0.75
SSD
128
2.99
P
0.50
256
0.25
512
0.00
1.00
64
3-99
>-99
2999
94.0
3-99
3.99
90.7
97.2
88.8
3-99
0.7
0.4
2.2
0.75
128
5.99
999
>999
3.99
5.99
3.99
>.99
>999
68.7
0.50
256
>.99
'9
>.99
>99
62.3
81.0
0.25
512
>-99
9999
0.00
32
64
128
256
32
64
128
256
32
64
128
256
32
64
128
256
Figure 7: Results for softmax attention (2]. linear attention 14] normalized attention 22
S6
and SSD 8 on four different, progressively harder MQAR tasks {L 64, KV-pairs 4) (L
128. KV-pairs = 8), [L 256, KV-pairs 16), and [L = 512, V-pairs = 64). We sweep model size d = [64, 1 128, 256, 512) and the state expansion n [32, 64, 128, 256] for each model and task. We only report the best performance from a learning rate sweep in np 1oggppccee-44. -2,-2,4
measured as accuracy the MQAR task. The accuracy denoted in % in the grid in the figure.
While Figure 2 shows that normalized attention outperforms standard linear attention 14, Figure shows an even more significant performance gap. Additionally, we note that SSD outperforms So. which was already hinted at in [8), and that normalized attention performs on par with S6. Together these results hint at the importance of normalization both for attention and SSMs. The comparison of S6 and SSD shows that reducing the number of parameters the state transition A, from d to scalar does not hurt performance. which is further supported by the findings in [8] These experiments
25

---

also suggest that the recursive structure in A, and B; (present in S6 and SSD but not in normalized attention or linear attention) is less important than proper normalization of the attention scores. Additionally, the results on WikiText- 103 (Table 1) show that better normalization can close 25% of the perplexity gap between linear attention and softmax attention. Together, these results warrant more investigations into new and better normalization techniques for attention-based models.
Finally, we remark that softmax attention performs perfectly accross all sweeps except { 512, KV-pairs = 64). d = 64, and n 32, which is most likely due to too small model or too small learning rate.
M Extended Results LRA
In Table 3 we report the complete results of all LRA experiments detailed in Appendix J The average performance over tasks reported in Table in the main text.
While normalized attention (22) performs slightly better on LRA than the other two attention-based models, there is significant performance gap to the S6 model (a selective SSM variant). The reason for this gap potentially lies in the recurrent normalization employed in selective SSMs (see Section 4.2p. Interestingly, S6 only achieves significantly higher accuracy on the tasks Text and Image, showing that selective SSM models are particularly suited for long- longrrange classification of text and image modalities.
Table 3: Model performance terms of test accuracy on the LRA benchmark. The first entry (Random) represents the performance of random guessing on the task, i.e., indicating the baseline above which a model considered to have learned a meaningful representation.
LRA Task [%]
Model
ListOps
Text
Retrieval
Image
Pathfinder
avg
Random
10.00
50.00
50.00
10.00
50.00
34.00
Softmax Attention
35.72
63.10
77.46
34.22
69.32
55.96
Linear Attention [4]
16.12
64.41
76.44
37.97
72.64
53.52
Normalized Attention
38.24
64.96
79.68
35.96
71.56
58.08
S6 7
38.02
81.34
80.50
65.08
69.26
66.84
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our theoretical analysis and experimental results substantiate the claims made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the proposed framework as well as the computational experi- ments are discussed Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer: [Yes]
Justification: Lemmas T and are properly stated, and their respective proofs are provided
in Appendix and Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We state the complete experimental details in Appendix D and provide a link to the full code base.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We provide a link to the full code base on GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cce
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/tips.cccpublic/gu..dd
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training details are provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Some of the experimental results presented in this paper are accompanied by error bars. Most experiments were performed for fixed seeds and only the best results ablated hyper-parameters are reported. For Figures and and we provide error bars over 10 different seeds in Appendix [ However, obtaining statistical significance data for all performed experiments would have drastically increased the amount of compute and runtime needed, which were both not feasible for us.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The used computational resources are stated in Appendix K
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes].
Justification: Ethical considcrations in accordance with NeurIPS Code Ethics have been respected throughout the research process. Potential limitations discussed in Section 6
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper has not direct socictal impacts. If the presented theoretical frame- work is used to design a new foundation model, there might be societal impacts of the model derived from this work. However, this stage the societal impact is not assessable.
Guidelines:
30

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release new dataset or high risk model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The used code is properly cited in Appendix Jand the licences are appropriately given in the code base accompanying this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
31

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
The paper does not introduce any new assets. The paper analyzes exist- ing models and improvements those, which are guided by the introduced theoretical framework.
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: No human subjects or crowdsourcing were used to conduct this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: Answer: [NA].
Justification: No human subjects crowdsourcing were used to conduct this research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
32

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "AGGFCQf0XY", "reviewer": "Reviewer_BcKL", "summary": "This paper introduces the Dynamical Systems Framework (DSF), a theoretical approach for analyzing and comparing various foundation models in AI. The DSF reformulates attention-based models, State Space Models (SSMs), and Recurrent Neural Networks (RNNs) into a common dynamical systems representation. This allows for principled comparisons and generates new insights into their similarities and differences. The authors leverage the DSF to compare linear attention and selective SSMs, provide conditions for approximating softmax attention, analyze state expansion in RNNs and SSMs, investigate differences between linear attention and S6 (Mamba) models, and apply SSM insights to RNN architectures. The paper combines theoretical results with empirical validations on the MQAR benchmark and WikiText-103 dataset.", "questions": "1. How well might the DSF insights scale to much larger models (billions of parameters) used in state-of-the-art applications?\n2. Are there theoretical limits to the convergence of linear attention to softmax attention performance with increased state expansion?\n3. Have you explored alternative normalization schemes that might bridge the gap between S6 and linear attention while maintaining computational efficiency?\n4. How does the S6-inspired state transition in qLSTM affect its long-term memory capabilities?\n5. Can the theoretical insights from DSF reformulations be leveraged to develop new, efficient implementations?", "limitations": "The authors acknowledge that the DSF parametrization, while theoretically insightful, doesn't necessarily lead to efficient implementations. Empirical validation is limited to synthetic tasks and a relatively small language modeling task, constraining generalizability. The paper also lacks comprehensive analysis of computational efficiency implications for different architectures under the DSF lens.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel unified perspective on different foundation model architectures\n- Enables new theoretical insights and comparisons between previously isolated model classes\n- Sound theoretical analysis with supporting lemmas and proofs\n- Empirical validation on MQAR and WikiText-103 provides practical grounding\n- Clear potential for guiding future development of efficient, scalable models\n- Innovative application of SSM insights to improve RNN architectures", "weaknesses": "- Limited empirical evaluation, focusing primarily on MQAR and a single language modeling task\n- Lack of statistical significance reporting for experimental results\n- Some theoretical results (e.g., Lemma 2) are relatively straightforward\n- Dense theoretical sections may be challenging for broader audience\n- Immediate practical impact on model performance is somewhat limited\n- Doesn't fully explore implications for training or inference efficiency", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the helpful feedback and pointers! Changes or clarifications (as deemed appropriate) for every single point raised will be incorporated in the final version of the paper. In what follows we provide a brief discussion on the points that were raised in the review:\n\n**Empirical evaluation:** We agree that the empirical evaluations of our insights were somewhat limited. To improve this we added additional experiments on the LRA benchmark, which also includes image tasks to broaden the empirical evaluation. The partial results (due to time constraints), i.e., only the sequential CIFAR-10 (image) task are shown in Tab. 1 of the attached pdf, and the results of the full benchmark will be added to the revised manuscript.\n\n**Statistical significance:** Thank you for this comment! We are including error bars to Fig. 2 and 3 in the paper. These results are shown (as shaded regions) in Fig. 1 and 2 of the attached pdf.\n\n**Theoretical results:** We acknowledge that Lemma 2 is somewhat straightforward and that this effect has already been shown empirically. However, this finding is a trivial consequence of the DSF and we wanted to highlight that the DSF provides insights that are consistent with the literature.\n\n**Theoretical sections:** Thank you for this comment! We acknowledge that the theoretical sections might be dense for an audience unfamiliar with dynamical systems. To improve upon this point, we plan to incorporate a brief introduction to the conceptual basics of dynamical systems in the introduction of the revised manuscript. We will also extend Section 3.1 with a more detailed presentation to make it more amenable for a broader audience.\n\n**Practical impact:** We added several more insights and analyses to the revised document as stated in the global response and we believe that the proposed work enables leveraging system theoretic insights in order to generate more specific practical impacts in future works.\n\n**Training and inference efficiency:** Although the primary goal of the DSF is to carry out theoretical comparison and obtain insights on the mathematical representations of foundation models, we believe it can also help in providing pointers for efficient training and inference. For instance, as discussed below in the response to the question regarding efficient implementations, the DSF allows identification of efficient algorithms to implement a specific model. Additionally, the DSF naturally allows analysis of the eigenvalues of the state transition matrix $A$, which are linked to the exploding/vanishing gradient problem ([Orvieto et al. (2023)](https://arxiv.org/abs/2303.06349)). We included a discussion on this to the revised version.\nOn a more theoretical level, it is possible to view the training process as an interaction of two dynamical systems by formulating the optimization algorithm as a separate dynamical system (see e.g. Example 2 in [Dörfler et al. (2024)](https://arxiv.org/abs/2401.14029)). In the revised version, we added a remark on this, but we think elaborating on this requires a more in-depth analysis that is out of scope of the paper.\n\n**Scalability:** In order to explore how the insights (e.g. normalized attention) scale to billions of parameters, an empirical evaluation is vital. Due to the extensive computational demands of this task, we believe this analysis is out of the scope for this paper. In a theoretical analysis, normalized attention scales linearly with context length, similarly to linearized attention and SSD. As such, we anticipate that it will scale similarly to other SSM models that have been demonstrated at a larger scale. \n\n**Softmax approximation:** Thank you for the excellent question! We believe this is a very interesting research question that warrants further investigation, but has no straightforward answer. However, in the special case of polynomial kernel functions (in Lemma 1), softmax-attention can be exactly recovered using infinitely many polynomials. Therefore, in the limit these polynomial kernels can recover the softmax performance with appropriate weights as there is an exact correspondence.\n\n**Alternative normalization schemes:** Yes, we also considered other normalization schemes to the one shown in Eq. (21), i.e.,\n\\begin{align}\n\t\\eta(u_i) &= e^{\\textrm{ReLU}(W_\\eta u_i)} \\\\\\\\\n\t\\eta(u_i) &= \\sigma(W_\\eta u_i)\n\\end{align}\nWe observed that they perform similarly to the one presented in the paper. We included the results on these additional normalization schemes to the appendix of the revised manuscript including a brief discussion.\n\n**Long-term memory:** The long-term memory capabilities of a model from a control theoretic perspective depend on the eigenvalues of the state transition matrix $A$ and the size of the state $x$ and thus the size of $A$ ([Orvieto et al. (2023)](https://arxiv.org/abs/2303.06349)). Therefore, any model reformulated in the DSF can be analyzed in this way. Since the proposed state transition is the same state transition as in S6, the two models have the same theoretical long-term memory capabilities. These capabilities then solely depend on the hyper-parameter $n$ (state size) and the initialization of $A$. However, we agree that this question warrants a more detailed analysis and thorough empirical validation, which is out of scope of this paper.\n\n**Efficient implementations:** Thank you for the excellent question. Yes, the DSF can be leveraged to identify and also develop new efficient implementations. In the revised version, we expand the first paragraph in Section 3.2 that only briefly mentions that the DSF can be used for this. In the appendix, we also added a few examples of how this can be done, e.g., the proposed normalized attention can be efficiently implemented using flash linear attention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5e0RR8I0hB", "reviewer": "Reviewer_US2F", "summary": "This paper provides a dynamical system based framework for principled comparisons between various existing recurrent architectures (linear attention, SSMs, etc.). There are focuses on the formulation and the role of the hidden state dimension. Some experimental results on the multi-query associative recall (MQAR) benchmark and the WikiText-103 dataset are provided to demonstrate the differences between the considered models.", "questions": "See above", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is overall well written and organized\n- Systematic comparisons between the formulations of various sequence models are provided, which are neat and useful for researchers without much background in the area", "weaknesses": "- The two proposed experiments (the MQAR benchmark and the WikiText-103 dataset) seem random and limited. There are no experiments on the more standard benchmark tasks such as the LRA task (focusing on smaller tasks such as the sequential CIFAR task would be valuable) and time series forecasting tasks \n- The theoretical insights are actually quite limited. There are unaddressed natural questions such as:\n\n     (1) one might wonder about finite-dimensional approximation of the softmax attention, \n\n     (2) how Lemma 1 could be related to the kernel method and RKHS \n\n- There are also other aspects that are left out for the comparative studies: e.g., how do the training dynamics differ between the various models (which architecture will converge faster), as well as the stability of the models (how prone are they to the infamous vanishing/exploding gradient problem). Also, can we quantify and compare the long range dependency learning capability of these models? \n- No practical guidance in terms of which architecture is suitable for what kind of task is provided. After all, this should be the main goal of the comparisons", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the helpful feedback and pointers! Changes or clarifications for every single point raised will be incorporated in the final version of the paper. We provide a brief discussion on the points that were raised in the review:\n\n**Benchmark:** Our main goal was to show-case some selected insights the DSF can bring to all three investigated model classes, i.e., transformers, SSMs, and RNNs. Since softmax-attention-based models are known to be the best performing models for language modalities and considerable research focuses on sub-quadratic models to close the performance gap to softmax-attention, we opted to benchmark on language tasks. However, we added the LRA benchmark to the experimental validation of our insights (see global response). The LRA results are consistent with existing empirical results, which show that attention-based models perform worse than SSM-based models on the LRA. Why this is the case is still an open research question and warrants more detailed investigation, but the DSF points to a potential hypothesis for this performance gap: the recurrent normalization discussed in Section 4.2 of the paper. Along with the LRA results we added a discussion on this point to the revised paper.\n\n**Insights:** Thank you for your feedback. We agree that the paper only showed a few theoretical insights. Therefore, we have extended on these insights and added several more as discussed in the general response. Also thank you for the specific suggestions, which we discuss below.\n- *Finite-dimensional approximation:* In the revised manuscript, we expanded on the discussion in Section 3.2.1 on using a Taylor approximation of softmax-attention. Specifically for the Taylor approximation, there exist error bounds that can be used to bound the error between the approximation and softmax-attention on a given interval, e.g., the Lagrange bound (see e.g. Section 3.3.2. in [Chevillard et al. (2008)](https://hal-lara.archives-ouvertes.fr/hal-02102827)). The Lagrange error bound defines the worst-case error of the approximation on the given interval and thus provides a quantitative metric of how well softmax-attention can be approximated. Additionally, we extended the discussion of existing works that use a finite-dimensional approximation of softmax.\n- *Kernel method:* Thank you for the excellent question. The attention function $\\zeta$ in Eq. (13) (later rewritten in Lemma 1) can be interpreted as a kernel if $\\psi = \\phi$ , with softmax-attention using a specific type of exponential kernel function, i.e., $\\exp(x^\\top y)$. This was noticed and further analyzed in [Tsai et al. (2019)](https://arxiv.org/abs/1908.11775) and Katharopoulos et al. (2020). In the revised manuscript, we added a remark on this and refer to the mentioned papers for more details.\n\n**Training dynamics:** While training dynamics (especially convergence) can be studied empirically using experiments, the DSF also allows theoretical analysis. As discussed in Example 2 of [Dörfler et al. (2024)](https://arxiv.org/abs/2401.14029), a gradient-based optimization algorithm (e.g. SGD) can be interpreted and written as a dynamical system. Using this perspective together with the DSF allows interpretation of the training dynamics as two interacting dynamical systems. Thus, the training dynamics can be analyzed theoretically with tools from control theory, e.g., via Lyapunov theory for convergence/stability of the training. However, we believe this question requires an in-depth investigation and additional empirical validation, which are out of scope for this paper.\n\n**Stability:** The exploding/vanishing gradient problem is linked to the eigenvalues of state transition matrix $A$ for SSMs (Orvieto et al. (2023)). Therefore, SSM models actively restrict the eigenvalues to the range [0,1]. Using the DSF the same argumentation can be made for transformers and RNNs. In the revised paper, we included an additional discussion on this.\n\n**Long-range dependencies:** A model's abiltiy to capture long-range dependencies are due to two factors: (i) the modulus of the eigenvalues in dynamic matrix $A$, and (ii) the dimension of the state $x$ (in DSF representation). Other factors include the compression of the input into the state by the encoding architecture, but this is out of the scope of our analysis that focuses on the recurrent block of learning architectures. Given two trained models with numerical values for $A$ and a sufficiently large state dimension, the DSF would be capable of predicting the performance in long-range contexts by looking at the eigenvalues of $A$. Moreover, theory on system theoretic model reduction (e.g. [Obinata & Anderson (2012)](https://dl.acm.org/doi/abs/10.5555/559486)) provides principled ways of comparing the ability of a given state dimension to compress past information. Thus, the DSF can be leveraged to provide a theoretical analysis on the long-range capabilities of the analyzed models. To complement our analysis, we will incorporate this discussion into the revised manuscript. \n\n**Task suitability:** We believe that the question of architecture suitability for specific tasks is a broad open research question, and many factors need to be considered to provide a clear answer. Specifically, the recurrent models presented in this paper are only one aspect of deep learning architectures, but the encoders, decoders, skip connections, etc. also play a role in task suitability. For this reason, we do not aim to answer this question in full in this paper. However, motivated by the comment we have experimentally explored different data modalities. Our findings illustrate (see attached pdf and paper experiments) that transformer architectures (specially with softmax attention) tend to perform better on language tasks and SSMs on tasks that require long-range dependencies such as the LRA benchmark. However, the performance of a model is highly dependent on the specific formulation of the task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZJMOhutwO7", "reviewer": "Reviewer_GS8T", "summary": "This paper shows that many sequence models (including attention, SSMs, and recurrent models) can be viewed as linear time-varying dynamical systems. This is helpful for answering some questions about the differences and similarities between these architectures.", "questions": "It would help to add additional exposition on the results in Fig. 2. What are the takeaways? It looks like exponentiating improves performance, but this isn’t stated explicitly.", "limitations": "Yes the authors discuss limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Importance of unifying framework**. The present state of sequence model research is quite muddy, with tons of new architectures being proposed. On the surface these architectures often look quite complicated and differentiated, when in reality they are closely related to one another.  Because of this, I think unifying frameworks are useful for helping push progress in the space. \n\n**The role of normalization in S6 and Linear Attention.** The paper includes a very interesting discussion of the parameterization of the normalization in S6 and Linear attention.", "weaknesses": "**Presentation and clarity.** The introduction and abstract do not provide any details on how the Dynamical Systems Framework works and assumes a reader is familiar with dynamical systems. To improve the presentation and accessibility, I would recommend including a high-level discussion of what the DSF is and what challenges it addresses. \n\n**Theoretical Novelty.** In the first paragraph of the introduction the authors state, “Although these models show great promise in boosting efficiency, current comparisons with attention are merely empirical.” This isn’t correct: there have been many theoretical studies comparing these new recurrent architectures with Softmax attention. To name a few:\n\n- [Arora *et al.](https://arxiv.org/abs/2402.18668)* include a theoretical analysis grounded in results from communication complexity that highlights differences in models ability to perform associative recall.\n- Using tools from circuit complexity, [Merill *et al.*](https://arxiv.org/pdf/2404.08819) show that Attention and SSMs are in the same complexity class ($\\text{TC}^0$).\n- Using tools from communication complexity, [Bhattamishra *et al.*](https://arxiv.org/pdf/2406.09347)  provide separation results between attention and recurrent architectures on numerous tasks.\n\n**Empirical Novelty.** The main empirical result in this paper is that increased state size (across architectures) leads to improved performance on MQAR and that separable attention matches Softmax attention with sufficiently large recurrent size. This result has already been shown theoretically and empirically in [Arora *et al.*](https://arxiv.org/pdf/2402.18668) (ICML ‘24) - see Figure 2, Theorem 3.1, and Section 3. The work under review builds upon Arora *et al.* by including a few additional architectures to the analysis. But this prior work should be mentioned in the related work and/or the introduction given the significant overlap in results . \n\n**Claims around Lemma 2.** Some of the language around Lemma 2 is a bit incautious. The authors state that because of Lemma 2, “the larger the state expansion n, the more expressivity the architecture has”. In other words, they are claiming that increasing state expansion strictly increases expressivity. To support this claim, the theorem would need to additionally show that a dynamical system of state dimension $N$ ***cannot*** recover all dynamical systems with state dimension $\\hat{N}$.  Additionally, I think its important to add the qualification that this holds assuming everything else about the architecture is held constant. It’s of course possible for an architecture with smaller state size to admit solutions that a larger state size model cannot represent if the parameterizations are different. \n\n**Minor typos and clarifications.**\n\n- Line 32: “framework that allows to evaluate” → “allows us to evaluate”\n- Line 61: “We use $\\sigma(\\cdot)$ to denote is the sigmoid function.\n- In the list of questions answered by the paper (Line 40), I would include hyperlinks to the relevant sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the helpful feedback and pointers! Changes or clarifications (as deemed appropriate) for every single point raised will be incorporated in the final version of the paper. In what follows we provide a brief discussion on the points that were raised in your review:\n\n**High-level discussion:** Thank you for this suggestion! We indeed believe that this would help to make the paper accessible to a broader audience. To address this comment, we plan to add an introduction to the concept of dynamical systems and how it applies in the context of foundation models. We will also delve into the specific dynamical system parametrization chosen by the DSF, which is the canonical state space representation. This choice will be motivated by two reasons: (i) this exists a wide body of literature on state space model representations for dynamical systems, and (ii) it encompasses transformers, RNNs and SSMs in a suitable fashion that allows for further analysis. We will also better explain how expressing foundation models in the DSF allows for a direct comparison between models and enables the use of control theoretical tools for their study.\n\n**Prior theoretical comparative studies:** Thank you for pointing this out, we acknowledge that the statement in the introduction is not correct. We intended to convey that to the best of our knowledge so far there has been no unified approach to theoretically analyze transformers, SSMs, and RNNs, but only empirical studies benchmarking specific models of these classes against each other. In the revised version, we changed this to reflect prior theoretical work comparing various models with attention, including properly referencing the papers that are mentioned in the review.\n\n**Empirical novelty:** Thank you for pointing us at [Arora et al. (2024)](https://arxiv.org/abs/2402.18668). We agree that the insight “increased state size leads to better performance” is not novel and has been shown in previous work. We intended to include this insight, since we believe it is beneficial to show that insights generated from the DSF are consistent with the literature. In the revised manuscript, we appropriately cite [Arora et al. (2024)](https://arxiv.org/abs/2402.18668) and improve the presentation to reflect that this insight has been reported before. Additionally, we improved the presentation of the overall document to better highlight the other two insights generated from the DSF (i.e. normalized attention and the S6-inspired state transition for qLSTMs) as well as the additional insights we added (please see the global response for more details). This change puts more focus on the new insights and explicitly presents the state size finding as established knowledge.\n\n**Lemma 2:** Thank you for the feedback, we acknowledge that the language in and around Lemma 2 is imprecise. As correctly pointed out, Lemma 2 should state that expressivity with a larger state size is at least as good as with lower state size, i.e., not strictly better. We reformulated Lemma 2 and the text surrounding it to reflect this. Additionally, we now explicitly state that this only holds if everything else is held constant as mentioned in the review.\n\n**Results discussion:** Thank you for the feedback. We agree that the key take-aways from Fig. 2 are lacking and are not appropriately discussed in the text. In the revised manuscript, we added the following discussion points:\n- The results in Fig. 2 and Tab. 1 support the conclusions that (i) performance of linear attention can be improved by using a suitable normalization function and (ii) this normalization function does not need to use the keys and queries but can be learnt independently.\n- The difference in parameterization between the proposed normalized attention and SSD mainly lies in the recurrent nature of the normalization parameters in transition matrix $A$ and input matrix $B$ for normalized attention. With the addition of the LRA experiments in the revised manuscript and the performance gap between SSMs and transformers, this warrants the question on the role of this recurrent normalization. In the revised manuscript we added a discussion on this question as detailed in the global response.\n\n**Typos:** Thank you! We will amend these in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xeAp2IXJht", "reviewer": "Reviewer_BUkP", "summary": "The paper concerns using the proposed \"Dynamical Systems Framework\" (DSF) to better understand the differences between linear SSMs, RNNs, linear attention and Softmax attention. The DSF shows a way to write each of these models as a linear time-varying recurrence. The paper aims to use this to answer questions about the differences between the three types of models, the role of state expansion, and how ideas from SSMs can be used to improve RNNs. Empirical experiments on the multi-query associative recall (MQAR) task and Wikitext are used to support the findings.", "questions": "- Perhaps considering data modalities and examples other than language could help to highlight insights that can be drawn from the DSF? E.g. perhaps modalities that more naturally arise from dynamical systems could help to further highlight differences between softmax attention and its more efficient alternatives? Or could help highlight differences between different linear attention/SSM/RNN variants?\n\n- Are the differences between softmax attention and the other methods on the MQAR task, e.g. in Figure 5, simply due to the difference in state size? One can think of the state size of softmax attention as the size of its KV cache required for the sequence length of the task. If you make the recurrent alternatives have a state size that matches the softmax attention KV cache size for the task sequence length, do they then perform as well? Or are there other things happening in Softmax Attention as well? Perhaps exploring this through the lens of the DSF framework could also provide insights?\n\nMinor: \n- It is stated the Matrix in Equation 12 is $L\\times L$, but isn't this only the case when $B$ and $C$ are $N \\times 1$ and $1 \\times N$? The presentation in the paragraph before suggests these are general matrices.\n- Line 203 says \"The DSF makes apparent how a necessary condition for separability is \nfor the map ζ(·) to be expressed by a finite-dimensional kernel, \". But in line 154 it says \"Softmax attention also satisfies the assumption of separability\".  I realize the point that is trying to be made regarding \"practical separability\", but nonetheless, it appears to be a contradiction as currently written.", "limitations": "Adequately discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The goal of better understanding the differences between the different models, and in particular trying to understand the gap between Softmax attention and its more efficient alternatives is important\n- The paper is well-written and clear. I particularly liked Figure 4 in Appendix A which made the dimensions being referred to throughout the paper clear. \n- The different formulations and results in the paper appear to be technically correct\n- Insights are used from SSMs to propose a different normalization scheme for linear attention which appears to improve performance on the tasks considered\n- The empirical results on the MQAR and Wikitext experiments support the claims", "weaknesses": "- The proposed DSF is nice in that it allows comparing each of the different types of models, but it's originality is weakened by the fact that most of the main parts of the framework have been discussed before in prior work\n  - The connections between linear attention and linear RNNs/SSMs are clear and have been discussed in prior works as mentioned in related work. \n  - In addition, the infinite dimensional representation of softmax attention as discussed in Section 3.2.1 is also discussed in Katharopoulos et al. 2020 among others\n  - Softmax attention has also been formulated as a dynamical system in https://arxiv.org/abs/2401.06104 which should be cited\n- The findings that are suggested as a result of the framework also do not seem to be new or particularly surprising\n  - The fact that larger state sizes lead to increased expressivity is not surprising and has been explored before, e.g. in https://arxiv.org/abs/2312.04927, https://arxiv.org/abs/2312.00752\n  - The proposed improvement to RNNs is to change the parameterization of a linear RNN (the quasi LSTM) by replacing its transition parameterization with that of S6. But this seems to just make it more like the already existing linear RNN, RG-LRU (with a slight different in parameterization from S6). This seems to be a much weaker finding than the claimed \"What do selective SSMs teach us about improving RNN architectures\" from line 53 of the Intro.\n   - As a counter to this point I am making, the proposed improvement to the normalization of linear attention appears to be novel and appears to improve performance on the tasks considered.\n- The focus and claims around expressivity and performance in the paper appear to be very language focused. \n  - This is obviously of interest, but perhaps narrows the ability for the DSF to provide new and interesting insights as I mention in the point above.\n  -  In addition, only 2 small tasks are considered to support the claims.  This limits the ability to understand if the claims hold in more general settings.\n  - The authors state in the limitations that to strengthen the insights, a larger and more complex language task is needed. This could be great, however I would suggest perhaps also considering more diverse data modalities could also strengthen the insights of the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the helpful feedback and pointers! Changes or clarifications (as deemed appropriate) for every single point raised will be incorporated in the final version of the paper. In what follows we provide a brief discussion on the points that were raised in the review:\n\n**Prior studies:** Thank you for the feedback and pointing us at [Oren et al. (2024)](https://arxiv.org/abs/2401.06104). We agree that connections between the three model classes have been studied in previous work and are well-established. Our intent is to translate the three model classes specifically as dynamical systems - which is achieved by the DSF - in order to use methods from control theory to analyze the models and to obtain a single form of representation to easily compare certain parameterization choices between the model classes. In the revised version, we made sure to better highlight this intent and we appropriately cited [Oren et al. (2024)](https://arxiv.org/abs/2401.06104).\nWe also acknowledge that the infinite representation of softmax-attention has been discussed previously in [Katharopoulos et al. (2020)](https://arxiv.org/abs/2006.16236) but also in [Nauen et al. (2024)](https://arxiv.org/abs/2403.02920) and [Choromanski et al. (2021)](https://arxiv.org/abs/2009.14794) among others. We believe the discussion on representation of softmax-attention is important in the context of the DSF, which is why we included it. In the revised manuscript, we made sure to include additional related works and improve the presentation to better reflect previous works.\n\n**Empirical novelty:** We acknowledge that the insight on increased performance with larger state sizes is well-established in the literature, we believe it is beneficial to show that insights provided by the DSF are consistent with previous results. In the revised manuscript, we improved the presentation of this insight to clearly state that we are only recovering existing results and cite the relevant works more prominently.\nFor the insights on RNNs, the main difference between the proposed qLSTM variant and the RG-LRU is the shared parameterization in state transition $A$ and input matrix $B$. While RG-LRU shares parameters in both matrices, the qLSTM does not (as briefly mentioned in lines 263-265). Interestingly, any attention-based model also shows this coupling in the DSF. In the revised manuscript, we added a discussion of this point for all three model classes (please also see the global response) and plan to include an empirical analysis for the camera-ready version.\n\n**Task complexity & modalities:** Thank you for the feedback and the suggestion. We agree that more complex and also diverse tasks can strengthen the insights we provide. While we believe in-depth analysis of a specific insight (e.g. the proposed normalized attention) including showing performance on complex tasks are more suited for subsequent works (we only have limited computational resources), we agree that other task modalities offer more insights. Therefore, to improve the empirical validation of our findings, we include experiments on the LRA benchmark, which also contains image tasks, in the revised manuscript. As an example, we included the results of the LRA image task in the attached pdf. In doing this we generated an additional insight (performance gap between SSM models and transformers on LRA) as discussed in the general response.\n\n**Attention and state size:** Thank you for the excellent suggestion. Preliminary experiments we ran show better performance of the other models with their state size increased to the size of the KV cache, but not outperforming softmax attention. Hence, it seems that there is more to softmax attention than just the size of the KV cache. Additionally, we believe there is more potential to such an analysis, since it relates to the question of how information is compressed in the state of the DSF. This is important since running these models with the size of the KV cache is not sustainable in practical applications with larger sequence lengths. We think such an analysis could lead to further follow-up works investigating different aspects of this question. Therefore, we will conduct full experiments for the revised version and perform an in-depth analysis of the results.\n\n**Minor:** Thank you for pointing out the two inconsistencies. It is true that the matrix in Eq. (12) is not $L \\times L$, but should be a block matrix consisting of $L \\times L$ blocks. We will correct this in the revised manuscript. We will also improve the language for practical separability, since it is indeed misleading in the current form.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their time and effort evaluating our paper. We believe the insightful reviews helped us to greatly improve the paper. The main contents of the rebuttal are several added insights we gained from the DSF, extended experiments including the LRA benchmark, and several clarifications in the text regarding existing works and theoretical results.

**Additional experiments:** In response to the reviews, we extended the evaluation of our insights to the Long Range Arena (LRA) benchmark. The LRA benchmark adds an additional data modality (images) to the evaluation and also allows us to investigate the well-known performance gap between attention-based models and SSMs. Additionally, we plan to test our insights on time series forecasting, specifically on the Informer ETT benchmark.

**Additional insights gained from DSF:** In response to all the reviews, we added several additional insights generated from the DSF.
The goal of this paper is to introduce the DSF as a useful framework to approach theoretical research questions about foundation models. The results included are meant to exemplify important questions that the DSF can answer and open future research directions, rather than be a comprehensive list of the insights, each of which will require theoretical derivations in their own right. However, in order to address the concern of limited insights and enhance the theoretical contributions of this work, we added the following insights:
- Investigating the performance of attention and selective SSM models on the LRA benchmark, has led us to investigate the performance gap in more detail (SSMs outperform transformers significantly). We found that the gap can be explained by the recurrent normalization strategy (discretization step) used by selective SSMs as discussed in Section 4.2 of the paper.
- Using the DSF, we extended our study to provide an analysis of the eigenvalues of the state transition matrix $A$ of all three model classes. These eigenvalues are directly linked to the exploding/vanishing gradient problem ([Orvieto et al. (2023)](https://arxiv.org/abs/2303.06349)). In the case of SSMs and RNNs, the absolute value of the eigenvalues are constrained in the range $[0,1]$ by construction. We observe that for attention-based models this is also true empirically due to the normalization used.
- In the DSF formulation of softmax-attention it is revealed that attention-based models share parameters in the state transition matrix $A$ and the input matrix $B$. While we mention this fact in the paper (lines 263-265), we expanded the discussion of this since SSMs and RG-LRU also shared this parameterization. However, this is not the case in more standard RNNs.

**Novelty:** In response to the reviews, we clarified our own theoretical contributions and improved the presentation to appropriately reflect existing results. The main contribution of our paper is the introduction of the DSF, which is a unifying framework for analysis of transformers, SSMs, and RNNs. To the best of our knowledge, this is the first unified framework that allows analysis of all three model classes in the same parameterization and thus allows to identify differences in the models that lead to significant performance differences. While some of the provided results already exist in the literature (e.g that increased state size improves performance), we also provide novel insights unique to the DSF framework in a comprehensive way that enables further analysis with control theoretical tools. For instance, the DSF enabled proposing the normalized attention, which we have extended in the revised version.

**Attached pdf:** The attached pdf contains Figures 2 & 3 of the original paper including error bars, i.e., confidence margins over 10 random seeds, which we aim to increase further as computational constraints permit. Additionally, Table 1 in the attached pdf shows preliminary results on the LRA benchmark, specifically for the image task (sequential CIFAR-10).

Meta-review: The authors introduce the Dynamical Systems Framework (DSF), a unified theoretical approach for comparing attention-based models, State Space Models (SSMs), and Recurrent Neural Networks (RNNs) as linear time-varying dynamical systems. This framework offers insights into the similarities and differences among these architectures, particularly in hidden state dimensions and state expansion, and demonstrates its effectiveness with empirical validations on the MQAR benchmark and WikiText-103 dataset. While the paper is strong in its contributions, it could benefit from improved presentation and clarity, and further emphasis on its theoretical novelty, as some may find the analysis results predictable. Nonetheless, the paper’s contributions are significant, and I recommend its acceptance.
Decision: Accept (poster)