Title: Provable Benefits of Complex Parameterizations for Structured State Space Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural Networks

Theory

Structured State Space Models

Mamba

S4

Complex parametrization
Abstract: Structured state space models (SSMs), the core engine behind prominent neural networks such as S4 and Mamba, are linear dynamical systems adhering to a specified structure, most notably diagonal. In contrast to typical neural network modules, whose parameterizations are real, SSMs often use complex parameterizations. Theoretically explaining the benefits of complex parameterizations for SSMs is an open problem. The current paper takes a step towards its resolution, by establishing formal gaps between real and complex diagonal SSMs. Firstly, we prove that while a moderate dimension suffices in order for a complex SSM to express all mappings of a real SSM, a much higher dimension is needed for a real SSM to express mappings of a complex SSM. Secondly, we prove that even if the dimension of a real SSM is high enough to express a given mapping, typically, doing so requires the parameters of the real SSM to hold exponentially large values, which cannot be learned in practice. In contrast, a complex SSM can express any given mapping with moderate parameter values. Experiments corroborate our theory, and suggest a potential extension of the theory that accounts for selectivity, a new architectural feature yielding state of the art performance.

Provable Benefits of Complex Parameterizations for Structured State Space Models

Abstract

Structured state space models (SSMs), the core engine behind prominent neural networks such as S4 and Mamba, are linear dynamical systems adhering to a specified structure, most notably diagonal. In contrast to typical neural network modules, whose parameterizations are real, SSMs often use complex parameterizations. Theoretically explaining the benefits of complex parameterizations for SSMs is an open problem. The current paper takes a step towards its resolution, by establishing formal gaps between real and complex diagonal SSMs. Firstly, we prove that while a moderate dimension suffices in order for a complex SSM to express all mappings of a real SSM, a much higher dimension is needed for a real SSM to express mappings of a complex SSM. Secondly, we prove that even if the dimension of a real SSM is high enough to express a given mapping, typically, doing so requires the parameters of the real SSM to hold exponentially large values, which cannot be learned in practice. In contrast, a complex SSM can express any given mapping with moderate parameter values. Experiments corroborate our theory, and suggest a potential extension of the theory that accounts for selectivity, a new architectural feature yielding state of the art performance. 1

1 Introduction

Structured state space models ( SSMs ) are the core engine behind prominent neural network architectures such as S4 [21], Mamba [20], LRU [41], Mega [37], S5 [50] and more [23, 31, 38, 34]. In their typical form, SSMs can be thought of as single-input single-output linear dynamical systems, wherein the state transition matrix has a specified structure, most notably diagonal [22, 23, 41, 50, 37, 20]. A salient characteristic of SSMs is that their parameterizations are often complex (take values in C ), in contrast to typical neural network modules whose parameterizations are conventionally real (take values in R ).

There has been mixed evidence regarding benefits of complex parameterizations over real parameterizations for SSMs. Some prior works have demonstrated that complex parameterizations are essential for strong performance [22, 41], whereas others have shown that in various settings real parameterizations lead to comparable (and in some cases better) performance [37, 20]. It was conjectured [20] that in the context of diagonal SSMs (namely, SSMs with diagonal state transition matrix), complex parameterizations are preferable for continuous data modalities ( e.g. , audio, video), whereas for discrete data modalities ( e.g. , text, DNA) real parameterizations suffice. Unfortunately, to date, formal support for this conjecture is lacking. The extent to which complex parameterizations benefit diagonal SSMs remains to be an open question.

In this paper, we take a step towards theoretically addressing the foregoing question. Specifically, we provide two theoretical contributions establishing provable benefits of complex parameterizations

for diagonal SSMs. Our first contribution establishes that, although both real and complex diagonal SSMs are universal -in the sense that both can precisely express any linear time-invariant ( LTI ) mapping up to any time t when their dimensions are equal to or greater than t -there is a strong separation between the SSMs in terms of expressiveness. Namely, denoting the dimensions of the real and complex SSMs by n R and n C , respectively, we prove that for any n C , there are various oscillatory mappings expressible by the complex SSM which cannot be approximately expressed up to time t by the real SSM unless n R is on the order of t , which may be arbitrarily larger than n C . This is in stark contrast to the fact that all mappings expressible by the real SSM can be precisely expressed (up to any time) by the complex SSM whenever n C ≥ n R .

Given the prevalence of overparameterization in machine learning, one may question how consequential the above separation (between the real and complex SSMs) is in practice. Indeed, in an overparameterized regime where a given LTI mapping is to be approximated up to a given time t and n R , n C ≥ t , universality implies that both the real and complex SSMs can precisely express the mapping up to time t . Accordingly, in this overparameterized regime, it is a priori unclear whether the complex SSM offers an advantage over the real SSM. Our second contribution shows that it does. Specifically, we prove a surprising result by which, if the given mapping satisfies a mild condition, then in order to approximately express the mapping up to time t , the real SSM must have dimension or parameter magnitude exponential in t . This is in stark contrast to the complex SSM, which can precisely express the given mapping up to time t with dimension and parameter magnitudes that are at most linear in t . The aforementioned mild condition is satisfied by the canonical copy mapping, by a basic oscillatory mapping, and with high probability by a random (generic) mapping. In such important cases, practical learning of the given mapping necessitates using a complex SSM.

Our theory is corroborated by controlled experiments, demonstrating that complex parameterizations for SSMs significantly improve performance. We also evaluate SSMs with selectivity -a new architectural feature yielding state of the art performance [20, 31, 4, 57]. Our experiments with selectivity portray a more nuanced picture: complex parameterizations are beneficial for some tasks, whereas for others, selectivity allows real parameterizations to achieve comparable (and in some cases better) performance. These findings align with the mixed evidence reported in the literature. Moreover, they suggest a potential extension of our theory that accounts for selectivity and may elucidate this evidence, thereby fully delineating the benefits of complex parameterizations for SSMs.

2 Preliminaries

2.1 Notations

We use non-boldface lowercase letters for denoting scalars ( e.g. α ∈ R , c ∈ C , n ∈ N ), boldface lowercase letters for denoting vectors ( e.g. x ∈ R n , v ∈ C n ), and non-boldface uppercase letters for denoting matrices ( e.g. A ∈ R n,m , B ∈ C n,m ). Series (finite or infinite) of scalars, vectors or matrices are viewed as functions of time and denoted accordingly ( e.g. ( x ( t ) ∈ R n ) t ∈ N , ( A ( t ) ∈ C n,m ) t =1 , 2 ,...,k ). For series of scalars, we also use as notation boldface uppercase letters ( e.g. S = ( s ( t ) ∈ R ) t ∈ N , I = ( i ( t ) ∈ C ) t =1 , 2 ,...,k ). Given k ∈ N and a series of scalars S whose length is greater than or equal to k , we use S k to denote the k th element of S , and S : k to denote the truncation of S to length k ( i.e. the series comprising the first k elements of S ), allowing ourselves to regard this truncated series as a vector of dimension k . For k ∈ N ∪ {∞} , we use [ k ] as shorthand for the set { 1 , 2 , . . . , k } . Given a complex number c ∈ C , we denote its magnitude by | c | ∈ R ≥ 0 , its phase by arg( c ) ∈ [0 , 2 π ) , its real part by ℜ ( c ) ∈ R , and its imaginary part by ℑ ( c ) ∈ R (meaning c = | c | exp( i arg( c )) = ℜ ( c ) + i ℑ ( c ) ). We let 0 and 1 stand for vectors whose entries are all zeros and all ones, respectively, with dimension to be inferred from context. The Hadamard (element-wise) product, defined between vectors or matrices of the same size, and between scalar series of the same length, is denoted by ⊙ . The convolution operator, defined between two (finite or infinite) scalar series, is denoted by ∗ . Namely, given two scalar series S = ( s ( t )) t ∈ [ k ] , ¯ S = (¯ s ( t )) t ∈ [ ¯ k ] of lengths k, ¯ k ∈ N ∪ {∞} respectively, S ∗ ¯ S is the scalar series of length k + ¯ k -1 whose m th element, for m ∈ [ k + ¯ k -1] , is given by ∑ min { m,k } t =max { m -¯ k +1 , 1 } s ( t )¯ s ( m -t +1) .

2.2 Structured State Space Models

Let K = R or K = C . A structured state space model ( SSM ) of dimension n ∈ N is parameterized by three matrices: A ∈ K n,n , a state transition matrix , which adheres to a predefined structure ( e.g. is

constrained to be diagonal); B ∈ K n, 1 , an input matrix ; and C ∈ K 1 ,n , an output matrix . 23 Given the values of A , B and C , the SSM realizes a mapping ϕ n, ( A,B,C ) : R N → R N which receives as input a real scalar series ( u ( t )) t ∈ N , and produces as output a real scalar series ( y ( t )) t ∈ N defined through the following recursive formula:

where ( x ( t ) ∈ K n ) t ∈ N is a vector series of states , and x (0) = 0 ∈ K n . If K = R we say that the SSM is real , and if K = C we say that it is complex . 4 We refer to the SSM as stable if all eigenvalues of A have magnitude strictly smaller than one; otherwise we refer to the SSM as unstable . For convenience, we often identify an SSM with the triplet ( A,B,C ) holding its parameter matrices, and regard the (single column) matrices B and C ⊤ as vectors.

Perhaps the most prominent form of structure imposed on SSMs is stable diagonality , i.e. stability combined with diagonality [22, 23, 41, 37, 20]. Accordingly, unless stated otherwise, we assume that the state transition matrix A of an SSM is diagonal and has entries with magnitude strictly smaller than one.

2.3 Linear Time-Invariant Mappings

Let ϕ : R N → R N be a mapping from the space of (infinite) real scalar series to itself. We say that ϕ ( · ) is linear if for all α ∈ R and S , ¯ S ∈ R N it holds that ϕ ( α S + ¯ S ) = αϕ ( S ) + ϕ ( ¯ S ) . For every k ∈ N , define the k step delay δ k : R N → R N to be the operator that adds k preceding zeros to the series it receives as input. 5 We say that the mapping ϕ ( · ) is linear time-invariant ( LTI ) if it is linear, and it commutes with δ k ( · ) (meaning ϕ ( δ k ( · )) = δ k ( ϕ ( · )) ) for every k ∈ N . It is well known [40] that if ϕ ( · ) is LTI then it is given by ϕ ( S ) = S ∗ ϕ ( I ) , where I := (1 , 0 , 0 , . . . ) ∈ R N is the impulse series, and ϕ ( I ) is referred to as the impulse response of ϕ ( · ) . Conversely, for any R ∈ R N , the mapping defined by S ↦→ S ∗ R is LTI, and its impulse response is R .

We will identify LTI mappings with their impulse responses. More specifically, for any k ∈ N , we identify an LTI mapping up to time k , with the truncation of its impulse response to length k . Accordingly, for any LTI mappings ϕ ( · ) , ¯ ϕ ( · ) and any ϵ ∈ R ≥ 0 , we say that ¯ ϕ ( · ) ϵ -approximates ϕ ( · ) up to time k if ∥ ϕ ( I ) : k -¯ ϕ ( I ) : k ∥ 1 ≤ ϵ . If the latter inequality holds with ϵ = 0 , we also say that ¯ ϕ ( · ) matches ϕ ( · ) up to time k .

Let ( A,B,C ) be an SSM of dimension n ∈ N , realizing the mapping ϕ n, ( A,B,C ) : R N → R N (see Section 2.2). It is straightforward to see that ϕ n, ( A,B,C ) ( · ) is LTI, and that its impulse response is given by:

For real and complex settings ( i.e. for K = R and K = C ), we will study the extent to which varying ( A,B,C ) as well as n , can lead ϕ n, ( A,B,C ) ( · ) to ϵ -approximate different LTI mappings up to different times.

3 Theoretical Analysis

Throughout this section, we consider a real SSM ( A R , B R , C R ) of dimension n R realizing the mapping ϕ n R , ( A R ,B R ,C R ) ( · ) , and a complex SSM ( A C , B C , C C ) of dimension n C realizing the mapping ϕ n C , ( A C ,B C ,C C ) ( · ) (see Section 2.2).

3.1 Universality

It is known (see, e.g. , [14]) that the real SSM is universal , in the sense that it can precisely express any LTI mapping up to any time t when its dimension is equal to or greater than t . Trivially, this implies the same for the complex SSM. Proposition 1 below formalizes these facts for completeness.

Proposition 1. Let ϕ : R N → R N be an arbitrary LTI mapping, and let t ∈ N . Then, the following holds for both K = R and K = C . If n K ≥ t , there exist assignments for ( A K , B K , C K ) with which ϕ n K , ( A K ,B K ,C K ) ( · ) matches ϕ ( · ) up to time t . 6

Proof sketch (proof in Appendix D.1). Beginning with the real SSM ( K = R ), the proof shows that ϕ n R , ( A R ,B R ,C R ) ( · ) matches ϕ ( · ) up to time t if V ( A R )( C ⊤ R ⊙ B R ) = ϕ ( I ) : t , where V ( A R ) is a Vandermonde matrix that has full rank when the diagonal entries of A R are distinct. Assigning A R this way, there must exist v ∈ R n R with which V ( A R ) v = ϕ ( I ) : t . Assigning B R = v and C ⊤ R = 1 concludes the proof for the real SSM. The complex SSM ( K = C ) can be treated analogously.

3.2 Separation in Expressiveness

Proposition 2 and Theorem 1 below together establish that although both the real and complex SSMs are universal (see Section 3.1), there is a strong separation between the two in terms of expressiveness. Proposition 2 formalizes an obvious fact: all mappings expressible by the real SSM can be precisely expressed (up to any time) by the complex SSM whenever n C ≥ n R ( i.e. , whenever the dimension of the complex SSM is equal to or greater than the dimension of the real SSM). Theorem 1 proves a much less obvious result: for any n C , there are various oscillatory mappings ( i.e. mappings with oscillatory impulse responses) expressible by the complex SSM which cannot be approximately expressed up to time t by the real SSM unless n R is on the order of t , which may be arbitrarily larger than n C .

Proposition 2. Consider an arbitrary assignment for ( A R , B R , C R ) , and assume that n C ≥ n R . Then, there exist assignments for ( A C , B C , C C ) with which ϕ n C , ( A C ,B C ,C C ) ( · ) = ϕ n R , ( A R ,B R ,C R ) ( · ) .

Proof. It suffices to prove the sought after result for n C = n R , since one can effectively reduce the dimension of the complex SSM by zeroing out entries of B C (or C C ). Assuming that n C = n R , we may assign to ( A C , B C , C C ) the values of ( A R , B R , C R ) . Under this assignment ϕ n C , ( A C ,B C ,C C ) ( · ) = ϕ n R , ( A R ,B R ,C R ) ( · ) , as required.

Theorem 1. Let t ∈ N and ϵ ∈ R ≥ 0 . Assume without loss of generality that n C = 1 , 7 in which case A C , B C and C C can be regarded as scalars. Suppose | sin(arg( A C )) | ≥ 0 . 2 , | A C | ≥ 0 . 5 1 /t and | B C · C C | ≥ 1 . Then, if ϕ n R , ( A R ,B R ,C R ) ( · ) ϵ -approximates ϕ n C , ( A C ,B C ,C C ) ( · ) up to time t , it must be that n R ≥ ⌊ t/ 9 ⌋ -1 -4 ϵ .

Proof sketch (proof in Appendix D.2). The idea behind the proof is as follows. The complex SSM realizes an oscillatory mapping, in the sense that elements 1 , 3 , . . . , 2 ⌈ t/ 2 ⌉-1 of its impulse response alternate Θ( t ) times between being greater than or equal to 1 / 4 , and being smaller than or equal to -1 / 4 . The real SSM on the other hand is limited in its ability to realize oscillations, insofar as elements 1 , 3 , . . . , 2 ⌈ t/ 2 ⌉ -1 of the impulse response it gives rise to are a linear combination of decaying exponentials, and therefore can change sign at most O ( n R ) times. Combining these two observations leads to the desired result.

3.3 Separation in Practical Learnability

Let ϕ : R N → R N be an LTI mapping with bounded impulse response, which we would like to ϵ -approximate up to time t for some ϵ ∈ R ≥ 0 and t ∈ N . Assume that the dimensions of the real and complex SSMs are greater than or equal to t . By Proposition 1, both the real and complex SSMs can express mappings that match ϕ ( · ) up to time t , and in particular that achieve the desired approximation. The current subsection establishes that despite this parity in terms of expressiveness, there is a strong separation between the real and complex SSMs in terms of practical learnability .

Section 3.3.1 proves that under a mild condition on ϕ ( · ) , in order for the real SSM to achieve the desired approximation, either its dimension or the magnitude of its parameters must be exponential in t . Section 3.3.2 then explains that such exponentiality impedes practical learning via gradient descent. Finally, Section 3.3.3 proves that in stark contrast to the real SSM, the complex SSM can achieve the desired approximation with dimension and parameter magnitudes that are at most linear in t .

3.3.1 Real Parameterizations Suffer from Exponentiality

Definition 1 below formalizes the notion of forward difference for a real scalar series-a discrete analogue of derivative for a differentiable real function. Our main theoretical result, Theorem 2, then establishes that if forward differences associated with ϕ ( I ) -the impulse response of ϕ ( · ) -satisfy a certain condition, then in order for the real SSM to express a mapping that ϵ -approximates ϕ ( · ) up to time t , either the dimension of the real SSM n R or the magnitude of its parameters ( B R , C R ) must be exponential in t . Roughly speaking, the aforementioned condition on forward differences associated with ϕ ( I ) is that there exists some d ∈ Θ( t ) such that the d th forward difference of the restriction of ϕ ( I ) to either odd or even elements has magnitude greater than 2 d ϵ . Perhaps surprisingly, this condition is especially mild, as the magnitude of the d th forward difference of a real scalar series typically scales exponentially with d . Several important cases where the condition is satisfied are presented below.

Definition 1. Let S be a real scalar series of length k ∈ N ∪ {∞} . The forward difference of S , denoted S (1) , is the scalar series of length k -1 whose m th element, for m ∈ [ k -1] , is given by S m +1 -S m . For d ∈ { 2 , 3 , . . . , k -1 } , the d th forward difference of S , denoted S ( d ) , is recursively defined to be the forward difference of S ( d -1) .

Theorem 2. Suppose ϕ n R , ( A R ,B R ,C R ) ( · ) ϵ -approximates ϕ ( · ) up to time t . Then:

where: ϕ ( I ) | odd and ϕ ( I ) | even are the restrictions of the impulse response ϕ ( I ) to odd and even elements, respectively; and ( ϕ ( I ) | odd ) ( d ) m and ( ϕ ( I ) | even ) ( d ) m stand for the m th element of the d th forward difference of ϕ ( I ) | odd and ϕ ( I ) | even , respectively.

Proof sketch (proof in Appendix D.3). The idea behind the proof is as follows. The restrictions of the impulse response of ϕ n R , ( A R ,B R ,C R ) ( · ) to odd and even elementsi.e. , ϕ n R , ( A R ,B R ,C R ) ( I ) | odd and ϕ n R , ( A R ,B R ,C R ) ( I ) | even , respectively-are each a linear combination of n R decaying exponentials, where the coefficients of the linear combination have absolute value no greater than ∥ C R ⊤ ⊙ B R ∥ ∞ . Forward differences of decaying exponentials are exponentially small. Therefore, by linearity of forward differences, requiring ϕ n R , ( A R ,B R ,C R ) ( I ) | odd or ϕ n R , ( A R ,B R ,C R ) ( I ) | even to have a forward difference that is not exponentially small implies an exponentially large lower bound on n R ∥ C R ⊤ ⊙ B R ∥ ∞ . When ϕ n R , ( A R ,B R ,C R ) ( · ) ϵ -approximates ϕ ( · ) up to time t , forward differences of ϕ n R , ( A R ,B R ,C R ) ( I ) | odd and ϕ n R , ( A R ,B R ,C R ) ( I ) | even are close to those of ϕ ( I ) | odd and ϕ ( I ) | even , respectively. We thus conclude that if ϕ ( I ) | odd or ϕ ( I ) | even has a forward difference that is not especially small, then n R ∥ C R ⊤ ⊙ B R ∥ ∞ must be exponentially large. This conclusion is formalized via Equation (3), the sought after result.

Special cases. Theorem 2 implies that the real SSM suffers from exponentiality (namely, its dimension n R or the magnitude of its parameters ( B R , C R ) must be exponential in t in order for it to express a mapping that ϵ -approximates ϕ ( · ) up to time t ) in various important cases. Indeed, Corollaries 1 and 2 below respectively show that the real SSM suffers from exponentiality when ϕ ( · ) is a canonical copy (delay) mapping, and with high probability when ϕ ( · ) is a random (generic) mapping. In light of Theorem 1 (namely, of the inability of the real SSM to compactly approximate various oscillatory mappings expressible by the complex SSM), it is natural to ask if the real SSM suffers from exponentiality in cases where ϕ ( · ) is oscillatory, i.e. where its impulse response oscillates. Corollary 3 below shows that exponentiality indeed transpires in a case where ϕ ( · ) is a basic oscillatory mapping. On the other hand, there are simple cases where ϕ ( · ) is oscillatory yet exponentiality does not take

place, e.g. the case where the impulse response of ϕ ( · ) is (+1 , -1 , +1 , -1 , . . . ) . 8 Precise delineation of the type of oscillations that lead to exponentiality is deferred to future work (see Section 6).

Corollary 1. Suppose t ≥ 9 and ϕ ( · ) = δ ⌊ ( t -1) / 2 ⌋ ( · ) , where as defined in Section 2.3, δ ⌊ ( t -1) / 2 ⌋ ( · ) is the ⌊ ( t -1) / 2 ⌋ step delay mapping. Assume also that ϵ ≤ 1 / ( 8 √ t ) . Then, if ϕ n R , ( A R ,B R ,C R ) ( · ) ϵ -approximates ϕ ( · ) up to time t , it must hold that:

Proof sketch (proof in Appendix D.4). The proof computes forward differences associated with δ ⌊ ( t -1) / 2 ⌋ ( I ) (impulse response of δ ⌊ ( t -1) / 2 ⌋ ( · ) ), and plugs them into Theorem 2 (Equation (3)).

Corollary 2. Let α ∈ R > 0 , and let R ∈ R N be generated by a random process where each element of R is independently drawn from a uniform distribution over the interval [ -α, α ] . Suppose that t ≥ 8 and that ϕ ( · ) is the mapping whose impulse response is R (i.e., ϕ ( · ) is defined by ϕ ( S ) = S ∗ R ). Let p ∈ (0 , 1) , and assume that ϵ ≤ α √ p/t . Then, with probability at least 1 -p , if ϕ n R , ( A R ,B R ,C R ) ( · ) ϵ -approximates ϕ ( · ) up to time t , it must hold that:

Proof sketch (proof in Appendix D.5). The proof derives lower bounds (holding with probability at least 1 -p ) on forward differences associated with R , and plugs them into Theorem 2 (Equation (3)).

Corollary 3. Suppose that ϕ ( I ) = (+1 , 0 , -1 , 0 , +1 , 0 , -1 , 0 , . . . ) and ϵ ≤ 0 . 5 . Then, if ϕ n R , ( A R ,B R ,C R ) ( · ) ϵ -approximates ϕ ( · ) up to time t , it must hold that:

Proof sketch (proof in Appendix D.6). The proof computes forward differences associated with the series (+1 , 0 , -1 , 0 , +1 , 0 , -1 , 0 , . . . ) , and plugs them into Theorem 2 (Equation (3)).

3.3.2 Exponentiality Impedes Practical Learning

For any value of t that is not especially small, exponentiality in t for the real SSM as put forth in Section 3.3.1i.e. , exponentiality in t of the dimension of the real SSM n R or the magnitude of its parameters ( B R , C R ) -impedes practical learning. This impediment is obvious in the case where n R is exponential in t (in this case, it is impractical to even store the parameters of the real SSM, let alone learn them). Appendix B treats the complementary case, i.e. it shows that learning is impractical when the required values for the parameters ( B R , C R ) are exponential in t (this is deferred to an appendix due to space constraints). The results of Section 3.3.1 therefore imply that the real SSM cannot practically learn a mapping that ϵ -approximates ϕ ( · ) up to time t under important choices of ϕ ( · ) .

3.3.3 Complex Parameterizations Do Not Suffer from Exponentiality

Section 3.3.1 established that under a mild condition on ϕ ( · ) , in order for the real SSM to express a mapping that ϵ -approximates ϕ ( · ) up to time t , either the dimension of the real SSM n R or the magnitude of its parameters ( B R , C R ) must be exponential in t . Proposition 3 below proves that in stark contrast, for any choice of ϕ ( · ) (whose impulse response ϕ ( I ) is bounded), the complex SSM can express mappings that match ϕ ( · ) up to time t with dimension n C and magnitude of parameters ( B C , C C ) that are at most linear in t .

Proposition 3. For any choice of n C greater than or equal to t , there exist assignments for ( A C , B C , C C ) with which ϕ n C , ( A C ,B C ,C C ) ( · ) matches ϕ ( · ) up to time t , and wherein: (i) ∥ B C ∥ 2 ≤ 2 ∥ ϕ ( I ) : t ∥ 2 ; and (ii) ∥ C ⊤ C ∥ 2 ≤ 1 . 9

Proof sketch (proof in Appendix D.7). The proof employs the theory of discrete Fourier transform ( DFT ). It begins by assigning (scaled versions of) the t th roots of unity to the diagonal entries of A C . Then, it uses the inverse DFT formula to derive assignments for B C and C C leading ϕ n C , ( A C ,B C ,C C ) ( · ) to match ϕ ( · ) up to time t . Finally, the proof applies Plancheral theorem to show that the derived assignments for B C and C C satisfy the desired criteria.

4 Experiments

This section presents controlled experiments corroborating our theory. Section 4.1 demonstrates that complex parameterizations significantly improve performance of SSMs in the theoretically analyzed setting. Section 4.2 shows that this improvement extends to a real-world setting beyond our theory. Finally, Section 4.3 evaluates SSMs with selectivity -a new architectural feature yielding state of the art performance [20, 31, 4, 57]. The experiments with selectivity portray a nuanced picture: complex parameterizations are beneficial for some tasks, whereas for others, selectivity allows real parameterizations to achieve comparable (and in some cases better) performance. These findings align with mixed evidence reported in the literature (see Section 1). Moreover, they suggest a potential extension of our theory that accounts for selectivity and may elucidate this evidence, thereby fully delineating the benefits of complex parameterizations for SSMs.

For conciseness, we defer some of the details behind our implementation to Appendix F. Code for reproducing our experiments is available at https://github.com/edenlum/ SSMComplexParamBenefits .

4.1 Theoretically Analyzed Setting

To empirically demonstrate our theoretical findings, we trained the analyzed real and complex SSMs (see Section 2.2) to approximate up to time t the mapping ϕ ( · ) (see Section 2.3), with t varying and with the following choices for ϕ ( · ) : the canonical copy (delay) mapping from Corollary 1; the random (generic) mapping from Corollary 2; and the basic oscillatory mapping from Corollary 3. Throughout, the dimension of the real or complex SSM ( n R or n C , respectively) was set to at least t , which, by universality (Section 3.1), implies that the SSM can express a mapping that precisely matches ϕ ( · ) up to time t . Our theory (Section 3.3) establishes that despite this parity between the real and complex SSMs in terms of expressiveness, there is a strong separation between the SSMs in terms of practical learnability. In particular, with the above choices of ϕ ( · ) , there are exponential barriers that apply only to the real SSM, and prevent its training from being able to yield a mapping that closely approximates ϕ ( · ) up to time t . Tables 1 and 2 present results obtained with the real and complex SSMs, respectively. They confirm the predictions of our theory.

4.2 Real-World Setting

To empirically demonstrate the benefits of complex parameterizations for SSMs in settings beyond our theory, we evaluated the prominent S4 neural network architecture [21] on the real-world sequential

CIFAR-10 dataset from the widely recognized Long Range Arena benchmark [52]. Our implementation is based on the official S4 repository, 10 where unless stated otherwise, hyperparameters (pertaining to the neural network architecture and its training) were kept at their default values. A single run with complex parameterization yielded a test accuracy of 89 . 10% , significantly higher than the highest test accuracy of 78 . 27% attained with real parameterization across three random seeds. Modifying the optimizer and initialization scheme with the real parameterization did not improve the test accuracy-see Appendix F.2 for details. The overarching conclusion from our theory-namely, that SSMs benefit from complex parameterizations-thus extends to this real-world setting.

4.3 Selectivity

A new architectural feature for SSMs that yields state of the art performance is selectivity [20, 31, 4, 57]. In its original form-proposed as part of the Mamba neural network architecture [20]selectivity amounts to replacing the parameters B and C (see Section 2.2), as well as an additional discretization parameter ∆ ∈ R > 0 , 2 by certain functions of the input ( u ( t )) t ∈ N . We empirically study the impact of complex parameterizations on SSMs with selectivity by evaluating a Mamba neural network on two synthetic tasks regarded as canonical in the SSM literature [27, 20]: (i) copy , which was shown by our theory (Section 3.3) and earlier experiments (Section 4.1) to pose difficulties for real parameterizations in SSMs with no selectivity; and (ii) induction-head , which can be seen as a generalization of copy in which the delay is input-specified (rather than being fixed). Our implementation is based on a widely adopted Mamba repository easily amenable to modification. 11 Unless stated otherwise, repository hyperparameters (pertaining to the neural network architecture and its training) were kept at their default values. Further details concerning our implementation, including detailed descriptions of the copy and induction-head tasks, can be found in Appendix F.3.

Our first experiment with the Mamba neural network compared real and complex parameterizations for the underlying SSMs, with selectivity included. On the copy task, across three random seeds for each configuration, the highest accuracy attained with the real parameterization was 80 . 17% , whereas the lowest accuracy attained with the complex parameterization was 93 . 05% . This gap in performance in favor of the complex parameterization aligns with our theoretical and empirical findings for SSMs without selectivity. In stark contrast, on the induction-head task, there is no such gap (in fact, there is a slight advantage to the real parameterization): across three random seeds for each configuration, accuracies attained with the real parameterization ranged between 97 . 35% and 98 . 3% , whereas those attained with the complex parameterization ranged between 93 . 93% and 97 . 64% . These results align with mixed evidence reported in the SSM literature, by which complex parameterizations are essential for strong performance on some tasks [22, 41, 22], whereas on others, real parameterizations lead to comparable (and in some cases better) performance [37, 20].

To gain insight into the induction-head task not benefiting from the complex parameterization, we conducted an ablation experiment with partial versions of selectivity ( i.e. , where not all of the parameters B , C and ∆ were replaced by functions of the input). The results of this experiment, reported in Table 3, reveal that when selectivity is fully or partially removed (more precisely, when

B , C or both are not replaced by functions of the input), the complex parameterization regains its advantage. This suggests that selectivity, which is not covered by our theory, may be the key factor enabling real parameterizations to perform as well as complex parameterizations for SSMs on certain tasks. In other words, selectivity may be the dominant factor behind the aforementioned evidence in the SSM literature being mixed. In Section 7 we outline a potential extension of our theory that accounts for selectivity and may elucidate this evidence, thereby fully delineating the benefits of complex parameterizations for SSMs.

5 Related Work

SSMs are closely related to linear dynamical systems-a classic object of study in areas such as systems theory [3] and control theory [51]. Although there exists extensive literature concerning properties of real and complex linear dynamical systems [10, 5, 6, 9], this literature does not readily establish benefits of complex parameterizations for SSMs, primarily due to the following reasons: (i) the output of a complex SSM is turned real by disregarding imaginary components (see Section 2.2), therefore it differs from a complex linear dynamical system; and (ii) the structures typically imposed on state transition matrices of SSMs ( e.g. stable diagonality; see Section 2.2) are generally uncommon in the literature on linear dynamical systems.

SSMs can be viewed as a special case of recurrent neural networks [48], which received significant theoretical attention [46, 39, 25, 11, 53]. In this context, several works focused specifically on SSMs [42, 30, 24, 2, 28, 58, 12, 32, 55]. However, to our knowledge, the only prior work to formally and explicitly treat benefits of complex parameterizations for SSMs is [42]. The treatment of [42] (see Section 4.1 therein) can be viewed as a special case of ours. Indeed, [42] considered a task that, using our notation (see Section 2.2), amounts to linearly reconstructing an input element u ( t ) from the state x ( t ′ ) of an SSM, where t, t ′ ∈ N , t ′ ≥ t . This is equivalent to assigning the output matrix of the SSM C such that the mapping realized by the SSM ϕ n, ( A,B,C ) ( · ) is a canonical copy (delay) mapping. Roughly speaking, [42] showed that this task requires linear operations with exponential parameters if the SSM is real, whereas linear operations with moderate parameters suffice if the SSM is complex. The same result follows from our Corollary 1 and Proposition 3. We stress that our theory goes far beyond this result, for example in that it covers various mappings beyond copy, including a random (generic) mapping-see Section 3.3 for details.

With regards to related empirical work, the literature includes several experimental comparisons between real and complex parameterizations for SSMs [20, 22, 42]. Nonetheless, to our knowledge, the controlled experiments we conducted (see Section 4) are reported herein for the first time.

6 Limitations

While this paper offers meaningful contributions regarding benefits of complex parameterizations for SSMs, it is important to acknowledge several of its limitations. First, while we establish a separation between real and complex parameterizations in terms of expressiveness (Section 3.2), we do not quantify how prevalent this separation is, i.e. , what proportion of the mappings expressible with complex parameterizations cannot be compactly approximated with real parameterizations. Second, while we prove that real parameterizations suffer from exponentiality that impedes practical learning (Sections 3.3.1 and 3.3.2), and that complex parameterizations do not suffer from exponentiality (Section 3.3.3), we do not formally establish practical learnability with complex parameterizationsour evidence for this is purely empirical (Section 4). Third, our theory does not treat all fundamental aspects of learning where real and complex parameterizations may differ, for example it does not treat implicit bias of gradient-based optimization [49]. Fourth, our experiments (Section 4) include only a single real-world setting. Finally, while we establish that a separation between real and complex parameterizations in terms of practical learnability takes place in three important cases (see Corollaries 1 to 3), these cases are likely far from being exhaustive. Indeed, Theorem 2 provides a mild sufficient condition for separation in terms of practical learnability-namely, that certain forward differences are not especially small-and we believe it is possible to apply analytical tools ( e.g. , the Nørlund-Rice integral [17]) for translating this condition into interpretable properties satisfied in various important cases beyond those considered. Pursuing the latter direction, and more broadly, addressing the aforementioned limitations, are regarded as important directions for future work.

7 Discussion

The extent to which complex parameterizations benefit SSMs is an important open question in machine learning. Evidence in the literature is mixed: while some works demonstrate that complex parameterizations are essential for strong performance, others show that in various settings, real parameterizations lead to comparable (and in some cases better) performance. It was conjectured by Gu and Dao [20] that complex parameterizations are preferable for continuous data modalities ( e.g. , audio, video), whereas for discrete data modalities ( e.g. , text, DNA) real parameterizations suffice.

Since a complex SSM includes twice as many parameters as a real SSM of the same dimension, a priori, one might expect that a real SSM would benefit from becoming complex similarly to how it would benefit from doubling its dimension. Our theory showed that this is not the case, and in fact the former benefit far exceeds the latter. Indeed, we established separations between real and complex SSMs, by which a real SSM can only match a complex SSM if either the dimension of the real SSM or the number of iterations required for its training is exponentially large. Experiments corroborated our theory, and suggested that selectivity-a new architectural feature yielding state of the art performance-may be the dominant factor behind the aforementioned evidence in the literature being mixed.

We now outline a potential extension of our theory that accounts for selectivity. Roughly speaking, the separations we established between real and complex SSMs arise from a gap in their ability to express oscillations, i.e. , to express frequency components in their impulse response: while a complex SSM can easily express any frequency, a real SSM struggles to do so. Adding selectivity to a real SSM makes its parameters input-dependent, resulting in what can be viewed as an input-dependent impulse response. We hypothesize that this dependence allows importing frequency components from the input to the impulse response. If confirmed, this hypothesis would imply that when the input data is sufficiently rich in frequency content, selectivity can endow real SSMs with all the benefits we proved for complex SSMs. Such an outcome aligns with the conjecture of Gu and Dao [20]: continuous data modalities often consist of only low frequencies, whereas discrete data modalities typically have a 'whiter spectrum,' i.e. , a more uniform mix of frequencies [54].

We believe that extending our theory as described may elucidate the mixed evidence in the literature, thereby fully delineating the benefits of complex parameterizations for SSMs.

Acknowledgments and Disclosure of Funding

We thank Itamar Zimerman for illuminating discussions. This work was supported the European Research Council (ERC) grants HOLI 819080 and NN4C 101164614, a Google Research Scholar Award, a Google Research Gift, Meta, the Yandex Initiative in Machine Learning, the Israel Science

Foundation (ISF) grant 1780/21, the Tel Aviv University Center for AI and Data Science, the Adelis Research Fund for Artificial Intelligence, Len Blavatnik and the Blavatnik Family Foundation, and Amnon and Anat Shashua.

References

Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/ . Software available from tensorflow.org.

Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models, 2024.

Kathleen T Alligood, Tim D Sauer, James A Yorke, and David Chillingworth. Chaos: an introduction to dynamical systems. SIAM Review , 40(3):732-732, 1998.

Quentin Anthony, Yury Tokpanov, Paolo Glorioso, and Beren Millidge. Blackmamba: Mixture of experts for state-space models, 2024.

Panos J. Antsaklis. Linear systems / Panos J. Antsaklis, Anthony N. Michel. McGraw-Hill' New York, 1997. ISBN 0070414335 (alk. paper).

Masanao Aoki. State space modeling of time series . Springer Science & Business Media, 2013.

Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on edge of stability in deep learning, 2022.

Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein, Nadav Cohen, Amir Globerson, Lior Wolf, and Raja Giryes. Decimamba: Exploring the length extrapolation potential of mamba, 2024. URL https://arxiv.org/abs/2406.14528 .

Roger W Brockett. Finite dimensional linear systems . SIAM, 2015.

John L Casti. Linear dynamical systems . Academic Press Professional, Inc., 1986.

Minshuo Chen, Xingguo Li, and Tuo Zhao. On generalization bounds of a family of recurrent neural networks, 2019. URL https://arxiv.org/abs/1910.12947 .

Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, and Terry Lyons. Theoretical foundations of deep selective state-space models, 2024. URL https://arxiv.org/abs/2402.19047 .

Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. ArXiv , abs/2103.00065, 2021. URL https: //api.semanticscholar.org/CorpusID:232076011 .

Edo Cohen-Karlik, Itamar Menuhin-Gruman, Raja Giryes, Nadav Cohen, and Amir Globerson. Learning low dimensional state spaces with overparameterized recurrent neural nets. International Conference on Learning Representations , 2023.

Alexandru Damian, Eshaan Nichani, and Jason D. Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. ArXiv , abs/2209.15594, 2022. URL https://api.semanticscholar. org/CorpusID:252668622 .

Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URL https://arxiv.org/abs/2405.21060 .

Philippe Flajolet and Robert Sedgewick. Mellin transforms and asymptotics: Finite differences and rice's integrals. Theoretical Computer Science , 144(1):101-124, 1995. ISSN 0304-3975. doi: https://doi.org/ 10.1016/0304-3975(94)00281-M. URL https://www.sciencedirect.com/science/article/pii/ 030439759400281M .

Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations , 2022.

Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-space models, 2022.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023.

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2022.

Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models, 2022.

Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces, 2022.

Joshua Hanson and Maxim Raginsky. Universal approximation of input-output maps by temporal convolutional nets, 2019.

Joshua Hanson and Maxim Raginsky. Universal simulation of stable dynamical systems by recurrent neural nets. In Alexandre M. Bayen, Ali Jadbabaie, George Pappas, Pablo A. Parrilo, Benjamin Recht, Claire Tomlin, and Melanie Zeilinger, editors, Proceedings of the 2nd Conference on Learning for Dynamics and Control , volume 120 of Proceedings of Machine Learning Research , pages 384-392. PMLR, 10-11 Jun 2020. URL https://proceedings.mlr.press/v120/hanson20a.html .

IEEE. 754-2019 - ieee standard for floating-point arithmetic, July 2019.

Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying, 2024. URL https://arxiv.org/abs/2402.01032 .

Sekitoshi Kanai, Yasutoshi Ida, Kazuki Adachi, Mihiro Uchida, Tsukasa Yoshida, and Shin'ya Yamaguchi. Evaluating time-series training dataset through lens of spectrum in deep state space models, 2024. URL https://arxiv.org/abs/2408.16261 .

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.

Zhong Li, Jiequn Han, Weinan E, and Qianxiao Li. Approximation and optimization theory for linear continuous-time recurrent neural networks. Journal of Machine Learning Research , 23(42):1-85, 2022. URL http://jmlr.org/papers/v23/21-0368.html .

Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A hybrid transformer-mamba language model, 2024.

Fusheng Liu and Qianxiao Li. From generalization analysis to optimization designs for state space models, 2024. URL https://arxiv.org/abs/2405.02670 .

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond, 2021. URL https://arxiv.org/abs/1908. 03265 .

Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model, 2024.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2017. URL https://arxiv.org/abs/1608.03983 .

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv. org/abs/1711.05101 .

Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention, 2023.

Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, and Chunting Zhou. Megalodon: Efficient llm pretraining and inference with unlimited context length, 2024. URL https://arxiv.org/abs/2404.08801 .

Yuichi Nakamura and Masahiro Nakagawa. Approximation capability of continuous time recurrent neural networks for non-autonomous dynamical systems. In International Conference on Artificial Neural Networks , 2009. URL https://api.semanticscholar.org/CorpusID:36553223 .

Alan V. Oppenheim and Alan S. Willsky with S. Hamid Nawab. Signals and Systems . Prentice Hall, Upper Saddle River, NJ, 2 edition, 1997. ISBN 978-0138147570.

Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences, 2023.

Antonio Orvieto, Soham De, Caglar Gulcehre, Razvan Pascanu, and Samuel L. Smith. Universality of linear recurrences followed by non-linear projections: Finite-width guarantees and benefits of complex eigenvalues, 2024.

PyTorch Contributors. Pytorch documentation, 2023. URL https://pytorch.org/docs/stable/ . Accessed: 2024-09-22.

Yuval Ran-Milo, Eden Lumbroso, Edo Cohen-Karlik, Raja Giryes, Amir Globerson, and Nadav Cohen. Provable benefits of complex parameterizations for structured state space models, 2024. URL https: //arxiv.org/abs/2410.14067 .

Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics , pages 400-407, 1951.

Anton Maximilian Schäfer and Hans Georg Zimmermann. Recurrent neural networks are universal approximators. In Stefanos D. Kollias, Andreas Stafylopatis, Włodzisław Duch, and Erkki Oja, editors, Artificial Neural Networks - ICANN 2006 , pages 632-640, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-38627-8.

Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling, 2024. URL https://arxiv.org/abs/ 2403.03234 .

Alex Sherstinsky. Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network. Physica D: Nonlinear Phenomena , 404:132306, March 2020. ISSN 0167-2789. doi: 10.1016/j. physd.2019.132306. URL http://dx.doi.org/10.1016/j.physd.2019.132306 .

Yonatan Slutzky, Yotam Alexander, Noam Razin, and Nadav Cohen. The implicit bias of structured state space models can be poisoned with clean labels. arxiv preprint , 2024.

Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling, 2023.

Eduardo D Sontag. Mathematical control theory: deterministic finite dimensional systems , volume 6. Springer Science & Business Media, 2013.

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers, 2020. URL https://arxiv.org/abs/2011.04006 .

Zhuozhuo Tu, Fengxiang He, and Dacheng Tao. Understanding generalization in recurrent neural networks. In International Conference on Learning Representations , 2020. URL https://api.semanticscholar. org/CorpusID:214346647 .

Martin Vetterli, Jelena Kovaˇ cevi´ c, and Vivek K Goyal. Foundations of Signal Processing . Cambridge University Press, 2014.

Shida Wang and Beichen Xue. State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory, 2023. URL https://arxiv.org/abs/2309.13414 .

Bernard Widrow and István Kollár. Quantization Noise: Roundoff Error in Digital Computation, Signal Processing, Control, and Communications . Cambridge University Press, 2008.

Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model, 2024.

Nicolas Zucchet and Antonio Orvieto. Recurrent neural networks: vanishing and exploding gradients are not the end of the story, 2024. URL https://arxiv.org/abs/2405.21064 .

A Extensions of Theoretical Analysis

A.1 Discretization

Various SSMs incorporate discretization [20, 21, 23, 16], which involves replacing parameter matrices with specific transformations that depend on an additional parameter ∆ ∈ R > 0 . With slight modifications, our main theoretical results, Theorem 1 and Theorem 2, apply directly to SSMs with common discretizations. Below, we exemplify this for one of the most common discretizations. Other discretizations can be treated similarly.

One of the most common discretizations is the bilinear discretization [21, 22], in which an SSM ( Ď A K , Ď B K , Ď C K ) is parameterized in the following way:

where ∆ ∈ R > 0 , the matrices A K , B K and C K share the same dimensions and are defined over the same fields as Ď A K , Ď B K and Ď C K , respectively, and the real parts of the diagonal entries of A K are negative. Since the set of functions expressible by a real or complex SSM remains unchanged under this discretization, Theorem 1 holds without modification. Furthermore, Theorem 2 applies to this discretization if the left-hand side of Equation (3) is replaced by n R ∆ ∥ C R ⊤ ⊙ B R ∥ ∞ . This follows directly from Lemma 13, which shows that for any i ∈ [ n R ] , we have | ∆( B R ) i | ≥ | ( Ď B R ) i | . The latter inequality justifies replacing Ď B R with ∆ B R on the left-hand side of Equation (3), leading to the desired result.

A.2 Feedthrough

Some SSMs include an additional feedthrough parameter D ∈ K 1 , 1 [41]. With feedthrough, the expression for y ( t ) in Equation (1) becomes ℜ ( C x ( t ) + Du ( t )) . Our theory essentially applies as is to SSMs with feedthrough, as explained below.

Obviously, an SSM of dimension n without a feedthrough parameter can be emulated by an SSM of dimension n with a feedthrough parameter D by setting D = 0 . Conversely, an SSM ( A,B,C ) of dimension n with a feedthrough parameter D can be emulated by an SSM ( A ′ , B ′ , C ′ ) of dimension n +1 without a feedthrough parameter. This is achieved, for example, by setting the diagonal entries of A ′ to be those of A followed by zero, the entries of B ′ to be those of B followed by one, and the entries of C ′ to be those of C followed by D . For any result where an SSM without feedthrough is assumed, a corresponding result for an SSM with feedthrough holds. Such a result can be attained by either emulating an SSM with feedthrough by an SSM without (while increasing dimension from n to n +1 ) or by emulating an SSM without feedthrough by an SSM with feedthrough, (while maintaining the dimension) depending on the context.

A.3 Complex A with Real B and C

It is possible to consider hybrid SSMs where A is allowed to be complex while B and C are restricted to be real. Below we show that such hybrid SSMs enjoy all provable benefits of complex SSMs ( i.e. , of SSMs where A,B and C can all be complex).

There are four results concerning complex SSMs: Proposition 1, Proposition 2, Theorem 1 and Proposition 3. The first three results can be trivially extended to the setting where only A is complex, as their proofs do not assume complex B or C . An extension of Proposition 3 is also straightforward. Indeed, the proof of Proposition 3 utilizes the fact that for a complex SSM ( A C , B C , C C ) of dimension n C , the impulse response can be any linear combination of n C arbitrary decaying sine and cosine waves, which, by Fourier theory, can approximate any sequence of length n C . If only A C is allowed to be complex, the impulse response can consist of any linear combination of n C arbitrary decaying cosine waves, which can still represent any sequence of length n C via the discrete cosine transform. Thus, the setting remains fundamentally similar, allowing for the extension of Proposition 3 to hybrid SSMs.

B Exponentiality Impedes Practical Learning

In this appendix, we show that learning the parameters of the real SSM, i.e. ( A R , B R , C R ) , is impractical when the required values for ( B R , C R ) are exponential in t . We account for two aspects of this impracticality: the number of iterations required by gradient descent; and the precision (number

of bits) required for representing the parameters. Note that we will not preclude the possibility of overcoming the impracticality through development of new techniques ( e.g. new parameterizations for B R and C R ). We believe our account herein may assist in such developments.

Exponential number of iterations. For the sake of illustration, suppose that training the real SSM to realize a mapping ϕ n R , ( A R ,B R ,C R ) ( · ) that ϵ -approximates ϕ ( · ) up to time t , is implemented via gradient descent over a loss function ℓ ( · ) that measures the squared error of the output at time t , where input elements are drawn independently from the standard normal distribution:

By a simple derivation (see Appendix C) ℓ ( A R , B R , C R ) = ∥ ϕ n R , ( A R ,B R ,C R ) ( I ) : t -ϕ ( I ) : t ∥ 2 2 , therefore sufficient minimization of the loss ℓ ( · ) (namely, minimization of ℓ ( · ) to or below the value ϵ 2 /t ) indeed guarantees that ϕ n R , ( A R ,B R ,C R ) ( · ) ϵ -approximates ϕ ( · ) up to time t . Assume the following regularity conditions on gradient descent: (i) the step sizes (learning rates) for all iterations are upper bounded by a constant (independent of t ); (ii) the step size for each iteration is stable , in the sense that it is upper bounded by 2 /λ max , where λ max represents the maximum eigenvalue of the Hessian of ℓ ( · ) at the respective iteration [13, 7, 15]; and (iii) the values of ℓ ( · ) throughout all iterations are upper bounded by a constant (independent of t ) times the value of ℓ ( · ) at initialization. Proposition 4 below establishes that under said regularity conditions, during gradient descent, the magnitude of the parameters ( B R , C R ) grows at most linearly in the number of iterations. Accordingly, if the required values for ( B R , C R ) are exponential in t and the initialization of ( B R , C R ) is not, attaining the required values necessitates an exponential (in t ) number of iterations.

Proposition 4. Consider an application of gradient descent to the loss ℓ ( · ) in Equation (7) :

where ( A (0) R , B (0) R , C (0) R ) is a chosen initialization, and η ( i ) ∈ R > 0 represents the step size selected for iteration i ∈ N . Assume ∃ c 1 ∈ R > 0 such that ∀ i ∈ N : η ( i ) ≤ c 1 . Assume also:

where λ max ( M ) , for a symmetric matrix M , is the maximum eigenvalue of M . Finally, assume:

Then:

Proof sketch (proof in Appendix D.8). Taking into account the form of ℓ ( · ) (see Appendix C) and the assumptions made, the proof shows that at each iteration of gradient descent, each entry in B R or C R changes by at most (4 c 1 c 2 t · ℓ ( A (0) R , B (0) R , C (0) R )) 0 . 5 . This readily leads to the desired result.

Prohibitive precision. With conventional floating-point representation, a real number ρ is represented as s · k · 2 m , where: s ∈ {-1 , 1 } (the sign) is represented by one bit; k ∈ N ∪ { 0 } (the significand) is represented by b k ∈ N bits; and m ∈ Z (the exponent) is represented by b m ∈ N bits. The precision of the floating-point representation is the total number of bits used for s , k and m , i.e. it is 1 + b k + b m . For example, with the widespread IEEE 754 standard [26]: single precision corresponds to 32 bits with b k = 23 , b m = 8 ; and double precision corresponds to 64 bits with b k = 52 , b m = 11 . It is customary to model quantization error as an additive random variable uniformly distributed over the interval [ -ξ/ 2 , ξ/ 2] , where ξ ∈ R > 0 is the quantization step size [56]. With the floating-point representation of ρ , the best (lowest) achievable quantization step size is on the order of | ρ | · 2 -b k , thus we may model quantization error as a multiplicative random variable uniformly distributed over [1 -q/ 2 , 1 + q/ 2] , where q = Θ(2 -b k ) . Definition 2 below employs this model to formalize the notion of robustness to quantization for values of ( A R , B R , C R ) with which ϕ n R , ( A R ,B R ,C R ) ( · ) ϵ -approximates ϕ ( · ) up to time t . Specifically, Definition 2 defines robustness to q -quantization for such values to be the probability that ϕ n R , ( A R ,B R ,C R ) ( · ) continues

to ϵ -approximate ϕ ( · ) up to time t , when the values are multiplied by random variables uniformly distributed over [1 -q/ 2 , 1 + q/ 2] . Proposition 5 below then proves that robustness to q -quantization is (at most) inversely proportional to q times the magnitude of ( B R , C R ) . Recalling that q = Θ(2 -b k ) , we conclude that if the values of ( B R , C R ) are exponential in t , a non-negligible robustness to quantization necessitates having b k = Ω( t ) , meaning a floating-point precision that scales (at least) linearly in t . The latter requirement is prohibitive, since virtually all computing systems implementing neural networks entail fixed options for floating-point precision (typically 16 , 32 and 64 [1, 43]), all much smaller than what the time t can be (tens of thousands or more [20, 47, 19, 8])

Definition 2. Let q ∈ [0 , 1] , and let Q A R , Q B R and Q C R be random matrices of the same sizes as A R , B R and C R , respectively, wherein each entry is independently drawn from the uniform distribution over [1 -q/ 2 , 1 + q/ 2] . Let ( A ∗ R , B ∗ R , C ∗ R ) be values for ( A R , B R , C R ) with which ϕ n R , ( A R ,B R ,C R ) ( · ) ϵ -approximates ϕ ( · ) up to time t . The robustness to q -quantization of ( A ∗ R , B ∗ R , C ∗ R ) is the probability that ϕ n R , ( A R ,B R ,C R ) ( · ) ϵ -approximates ϕ ( · ) up to time t when ( A R , B R , C R ) take the values ( A ∗ R ⊙ Q A R , B ∗ R ⊙ Q B R , C ∗ R ⊙ Q C R ) .

Proposition 5. Suppose that ϕ n R , ( A R ,B R ,C R ) ( · ) ϵ -approximates ϕ ( · ) up to time t . Then, for any q ∈ [0 , 1] , the robustness to q -quantization of the values held by ( A R , B R , C R ) is at most:

Proof sketch (proof in Appendix D.9). Denote by ( A ∗ R , B ∗ R , C ∗ R ) the values held by ( A R , B R , C R ) , and let Q A R , Q B R and Q C R be random matrices as in Definition 2. When ( A R , B R , C R ) take the values ( A ∗ R ⊙ Q A R , B ∗ R ⊙ Q B R , C ∗ R ⊙ Q C R ) , the first entry of the impulse response of ϕ n R , ( A R ,B R ,C R ) ( · ) equals ( C ∗ R ⊙ Q C R )( B ∗ R ⊙ Q B R ) . It suffices to restrict attention to this first entry, and show that | ( C ∗ R ⊙ Q C R )( B ∗ R ⊙ Q B R ) -ϕ ( I ) 1 | ≤ ϵ with probability at most 2 ϵ/ ( q ∥ C ⊤ R ⊙ B R ∥ ∞ ) . The proof establishes the latter anti-concentration result.

C Loss Functions

Consider a loss function as in our theory (see Equation (7)), applied to both real and complex SSMs:

where K = R or K = C . Below we prove that:

Indeed, as discussed in Section 2.3, for any U ∈ R N :

Thus:

It holds that:

as required.

D Deferred Proofs

D.1 Proof of Proposition 1

Beginning with the real SSM ( K = R ), assume that n R ≥ t . We would like to show that there exist assignments for ( A R , B R , C R ) with which:

where ϕ ( I ) : t stands for the first t elements of the impulse response of ϕ ( · ) . Regarding ϕ ( I ) : t , B R and C ⊤ R as (column) vectors, we may write the sought after equality as V ( A R )( C ⊤ R ⊙ B R ) = ϕ ( I ) : t , where V ( A R ) ∈ R t,n R is a matrix whose ( i, j ) th entry holds the j th diagonal entry of A R exponentiated by i -1 . Notice that V ( A R ) is a Vandermonde matrix comprising powers of the diagonal entries of A R . Any assignment for A R with distinct diagonal entries leads the Vandermonde matrix to have full rank, which in turn means that there exists a vector v ∈ R n R satisfying V ( A R ) v = ϕ ( I ) : t . Assigning B R = v and C ⊤ R = 1 concludes the proof for the real SSM. The complex SSM ( K = C ) can be treated analogously, while assigning real values to A C , B C and C C in order to allow disregarding the fact that only real parts of outputs are taken ( i.e. , disregarding the operator ℜ ( · ) in Equation (2)).

D.2 Proof of Theorem 1

To facilitate the proof of the theorem, we first outline essential definitions and lemmas to quanitify a lower bound for the number of times the impulse response alternates between being greater than or equal to 1 / 4 , and being smaller than or equal to -1 / 4 . Building on this groundwork, we then unpack the main theorem's proof in Appendix D.2.2.

D.2.1 Setup for the proof

Definition 3. For an angle θ in R we will say that θ is in the mostly-positive region if

and we say that θ is in the mostly-negative region if

Remark 1. Under this definition, a non-zero complex number x has that its argument is in the mostly-positive region if and only if R ( x ) ≥ | x | 2 . Conversely, x has that its argument is in the mostly-negative region if and only if R ( x ) ≤ -| x | 2 .

Definition 4. We say that a set is balanced if at least a sixth of its elements are in the mostly-positive region and at least a sixth of its elements are in the mostly-negative region.

Definition 5. The balancing number of a complex number x with argument θ is defined as the minimum p ∈ N such that for any non-zero real number b the set { b + θ, b + 2 θ, . . . , b + pθ } is balanced. If no such p exists, we say that the balancing number of x is infinity. We denote the balancing number of x by β ( x ) .

We will begin by establishing an upper bound on β ( x ) for x in a specific region in the complex plane. Subsequently, we generalize this result to give an upper bound of the balancing number for almost all complex numbers.

Lemma 1. For any natural number p ≥ 4 , if a non-zero complex number x has an argument θ such that θ is in [ 2 π p , 2 π 3 ] , then β ( x ) ≤ p . Moreover, for any natural number p ≥ 9 , if x has an argument θ such that θ is in [ π + 2 π p -1 , 4 π 3 ] then β ( x ) ≤ p .

Proof. Since the mostly-positive and mostly-negative regions are defined by looking at the angle of a complex number mod 2 π , to prove that β ( x ) ≤ p , it is enough to show there exists some number k ≤ p such that for any real b the following set

is balanced.

Indeed, let there be some real b .

If θ is in [ 2 π p , 2 π 3 ] and p ≥ 4 , Let k be the natural number such that 2 π < θk ≤ 2 π + θ . It follows that 4 ≤ k ≤ p and that θ ≤ 2 π k -1 . By Lemma 5 we have that the set

has at least a single point in any continuous interval (in mod 2 π ) of length θ . As such, any continuous interval mod 2 π of length 2 π 3 must contain at least

points, where the last inequality is true for any k ≥ 4 . Since the mostly-positive and mostly-negative regions are continuous intervals (mod 2 π ) of length 2 π 3 , we have that the set

has at least a sixth of its points in the mostly-positive and mostly-negative regions, and is therefore balanced, as needed.

If θ is in [ π 2 + 2 π p -1 , 4 π 3 ] and p ≥ 9 , then 2 θ mod π is in the interval [ 2 π p -1 2 , 2 π 3 ] which is contained in the interval [ 2 π ⌊ p 2 ⌋ , 2 π 3 ] . Therefore, by the previous section, there exists some k ≤ ⌊ p 2 ⌋ such that the following set:

and the set

are both balanced. Overall, their union, the set

is also balanced, showing that the β ( x ) ≤ 2 k ≤ p as needed.

Lemma 2. For any natrual numbers r, θ the following holds:

Proof. For any rational numbers θ and b , and for any natural number p , we will define the set S θ,b,p in the following way:

We will also define the set S θ,p as

Given this notation, we have that β ( e r -iθ ) = p if and only if p is the minimum natural number such that every set in S θ,p is balanced. For any rational numbers θ and b and for any natural number p , we have

(Where minus is taken element-wise mod 2 π ). This immediately gives:

Since the mostly-positive and mostly-negative regions are invariant to taking a minus mod 2 π , we have that all the sets in S θ,p are balanced if and only if so are all the sets of S -θ,p . This directly implies that β ( e r + iθ ) = β ( e r -iθ ) as needed.

Corollary 4. If x ∈ C has that arg( x ) mod π ∈ [ 2 π p -1 , π -2 π p -1 ] for some p ≥ 9 then β ( x ) ≤ p

Proof. Denote θ ≡ arg( x ) . From Lemma 1 if the angle θ lies within the interval

then β ( x ) ≤ p . Additionally, Lemma 2 indicates that the same holds if θ is in the interval -I (where operations are element-wise mod 2 π ). Hence, we have that if θ is contained in interval I ′ , where

then β ( x ) ≤ p . Overall, this means that if θ has that

then β ( x ) ≤ p , which gives the Corollary.

D.2.2 The Proof

Let ϵ > 0 be any positive number. For any sequence X = { X 1 , X 2 , . . . , X t } , let X | 1 denote the subsequence consisting of elements at odd indices, i.e., X | 1 = { X 1 , X 3 , . . . } .

First, we notice that from the assumption that | sin(arg( A C )) | ≥ 0 . 2 we have that arg( A C ) ∈ [ π 16 , π -π 16 ] , and so

This implies, by Corollary 4, that the balancing number of A 2 C is less than 9 . Now, if ϕ n R , ( A R ,B R ,C R ) ( · ) ϵ -approximates ϕ n C , ( A C ,B C ,C C ) ( · ) up to time t , then, by definition,

This implies

Now, we have that

and because the balancing of A 2 C is at most 9 , this implies that there exists a subsequence of 1 , 2 , . . . , ⌊ t/ 2 ⌋ with increasing indices k j of length at least ⌊ t 9 ⌋ -1 such that if j is even then B C C C ( A 2 C ) k j -1 is in the mostly-positive region, which implies

where the first inequality is due to the definition of the mostly-positive region and the second is implied from the assumption that | B C · C C | ≥ 1 and | A C | ≥ 0 . 5 1 /t . Conversely, if j is odd it is in the mostly-negative region, which implies

Next, we will show that the impulse response of any real system, when restricted to even indices, can change its sign at most n R -1 times. This will prove the theorem since

The first inequality results from the fact that there would be at least ⌊ t p ⌋ -n R -1 indices where 1 4 ≤ ∣ ∣ ( ϕ 1 , ( A C ,B C ,C C ) ( I ) : t | 1 ) k ∣ ∣ and ϕ n R , ( A R ,B R ,C R ) ( I ) : t | 1 and ϕ 1 , ( A C ,B C ,C C ) ( I ) : t | 1 disagree on sign, which yields the theorem.

Indeed, let us upper bound the number of sign changes of the real impulse response when restricted to odd indices. We have that

By the mean value theorem, the number of times ϕ n R , ( A R ,B R ,C R ) ( I ) : t | 1 changes signs is bounded by the number of zeros of the following continuous function:

This function is a linear combination of n R exponential functions with a positive base, and so by Lemma 4, it has at most n R -1 zeros, as needed.

D.3 Proof of Theorem 2

Let d, m be natural numbers such that d + m ≤ ⌊ t/ 2 ⌋ , and let σ be in { odd , even } . We will show that

thus proving the theorem.

Let ϕ n R , ( A R ,B R ,C R ) ( · ) be a real system that ϵ -approximates ϕ ( · ) up to time t . For convenience, we will denote ϕ n R , ( A R ,B R ,C R ) ( I ) : t by Y and ϕ ( I ) : t by T . Then, by definition, we have that

Lemma 9 guarantees that there exist some parameters ( ˜ A R , ˜ B R , ˜ C R ) where ˜ A R is non-negative and the following two things hold:

Because ˜ A R is non-negative, Y | σ can be viewed as a linear combination of decaying exponentials with positive coefficients. Since the forward difference is linear, Lemma 6 ensures that the absolute value of the d th forward difference of Y | σ at index m is upper bounded by

which implies that

Using Lemma 12, we get

Lemma 8 implies that

Plugging in | T | σ -Y | σ | ∞ ≤ ϵ and T | σ , we get

Reordering this equation yields the desired result.

D.4 Proof of Corollary 1

For simplicity of notation, we will denote ⌊ t -1 2 ⌋ by k and assume without loss of generality that k is odd. Theorem 2 indicates that

Substituting d = k +1 2 , m = ⌊ k +1 4 ⌋ , σ = even, and ϵ ≤ 1 8 √ t yields:

In the subsequent analysis, we will demonstrate that

Once established, the remainder of the proof will naturally follow since plugging this forthcoming result into Equation (8) will ensure that:

satisfying the requirements for our conclusion. Indeed, we have that for any j ≤ k

Lemma 7 implies that

By Lemma 11, we can assert:

which completes the proof.

D.5 Proof of Corollary 2

Theorem 2 indicates that

For any σ ∈ { odd , even } , plugging in m = ⌊ t 8 ⌋ , d = ⌊ t 4 ⌋ , and ϵ ≤ α √ δ √ t yields:

In the subsequent analysis, we will demonstrate that for any σ ∈ { odd , even } , the following holds with a probability smaller than √ δ :

Once established, the remainder of the proof will naturally follow, as plugging this forthcoming result into Equation (9) will ensure that for any σ ∈ { odd , even } , with probability 1 -√ δ we have that:

satisfying the requirements for our conclusion. Since R : t | even and R : t | odd are independent random variables, we have that the previous statement holds with probability higher than 1 -δ as required.

Now let us proceed with proving the aforementioned result. Assume without loss of generality that σ = even. Lemma 7 implies that

For each j ∈ [⌊ t 4 ⌋] , define the random variable X j as follows:

and denote its PDF by p X j . We have that

which means that

.

Let X be the sum of these random variables X = ∑ ⌊ t 4 ⌋ j =0 X j and denote its PDF by p X . We note that

As a sum of i.i.d. random variables, the maximum of p X is upper bounded by the maximum of the PDF of each of the random variables it is a sum of. Which means that

(where the second inequality results from Lemma 10). Overall we have that

Plugging in ϵ 0 = 2 ⌊ t 4 ⌋ · 2 α √ δ √ t we have that

which concludes the proof.

D.6 Proof of Corollary 3

Theorem 2 demonstrated that

Plugging in σ = odd, we have that for any m ∈ [ ⌊ t/ 2 ⌋ ] the following holds:

Using simple induction, it is easy to show that for any d, m ∈ N such that d + m ≤ ⌊ t/ 2 ⌋ ,

Plugging this into Theorem 2 with σ = odd, d = ⌊ t/ 4 ⌋ , m = ⌊ t/ 4 ⌋ , and ϵ = 0 . 5 yields:

as required.

D.7 Proof of Proposition 3

It is sufficient to prove the proposition for n C = t since the dimension of a diagonal SSM can always be effectively reduced by zeroing out elements of B C .

According to Section 2.3, it is enough to show that there exist assignments for ( A C , B C , C C ) such that the following conditions hold:

For convenience, we will denote A C , B C , C C by A,B,C .

We begin by utilizing the theory of discrete Fourier transform ( DFT ) to allow us to write the truncated impulse response of ϕ of length t as a linear combination of t decaying sine and cosine waves. Specifically, defining ( ˜ ϕ ( I ) : t ) k = ( ϕ ( I ) : t ) k α k -1 , where α ∈ (0 , 1) is some constant, and denoting the DFT of ˜ ϕ ( I ) : t by a + bi ∈ C t we get that

Now, we will derive assignments for the SSM's parameters so that its impulse response will equate this sum. Indeed, we fix the diagonal entries of A to be a scaled version of the t roots of unity, and C to be the inverse of the square root of t as follows:

We know (see Section 2.3) that the impulse response of a complex system at index k is given by:

Which implies that

By the Plancherel theorem the following holds:

Therefore, if we define B = a -ib √ t , we get

and

and by choosing α = ( 1 2 ) 1 t -1 we get the required assignment.

D.8 Proof of Proposition 4

For simplicity of notation, let A ( i ) R , B ( i ) R , C ( i ) R be denoted by A ( i ) , B ( i ) , C ( i ) , respectively, and ℓ ( A ( i ) , B ( i ) , C ( i ) ) by ℓ ( i ) . We begin by writing the loss explicitly (see Appendix C):

Fixing any k ∈ [ n R ] , we have

which, combined with the fact that | A m | ≤ 1 , leads to

This implies

Next, we will demonstrate that

which concludes the proof since this aforementioned result will establish that for all i ∈ N , the following holds:

Where (1) holds because we assume ℓ ( i -1) ≤ c 2 ℓ (0) and (2) is by recursion.

Repeating the same argument for C ( i ) completes the proof.

Finally, let us show that η ( i ) max (∥ ∥ B ( i -1) ∥ ∥ ∞ , ∥ ∥ C ( i -1) ∥ ∥ ∞ ) ≤ √ c 1 . Indeed, Lemma 3 combined with the assumptions that

and ∀ i ∈ N : η ( i ) ≤ c 1 gives that

Considering the cases where max {∥ ∥ B ( i -1) ∥ ∥ ∞ , ∥ ∥ C ( i -1) ∥ ∥ ∞ } is bigger and smaller than 1 / √ c 1 we immediately get the required result.

Lemma 3. In the setting of Proposition 4 we have that for any A R , B R , C R the following holds:

Proof. For simplicity of notation, let A R , B R , C R be denoted by A,B,C , respectively. We will assume, without loss of generality, that ∥ C ∥ ∞ ≥ ∥ B ∥ ∞ , and denote i = arg max i C i and by e i the one-hot vector with 1 at index i and zero elsewhere. We will also denote the following function

by I i ( x ) , and the following loss function:

by ˜ ℓ i . Next, We notice that by definition of the Hessian, we have that

Therefore, λ max ( ∇ 2 ℓ ) at A,B,C is lower-bounded by ˜ ℓ ′′ i (0) . It is therefore sufficient to show that | ˜ ℓ ′′ i (0) | ≥ 2 ∥ C ∥ 2 ∞ . Indeed, by Appendix C, the following holds

as needed.

which concludes the proof.

D.9 Proof of Proposition 5

For simplicity of notation, we will denote A R , B R , C R , n R by A,B,C,n . The robustness to q -quantization is by definition the probability:

Let us denote the random variable ϕ n, ( A ⊙ Q A ,B ⊙ Q B ,C ⊙ Q C ) ( I ) 1 by X , and its PDF by p X . The probability P [ | X -Y | ≤ ϵ ] is equal to the integral ∫ Y + ϵ Y -ϵ p X ( x ) dx ≤ 2 ϵ max x ∈ R p X ( x ) . We get that

and so, to prove the theorem, it suffices to show that

Let us investigate the random variable X :

where q Bi , q Ci ∼ U ( -q/ 2 , q/ 2) . As a sum of independent random variables, the maximum of p X is upper bounded by the maximum of the PDF of each of the random variables it is a sum of. Specifically, for any i ∈ [ n ] , it is bounded by the maximum of the PDF of U ( -B i C i q/ 2 , B i C i q/ 2) which is equal to 1 / ( B i C i q ) . Overall, this means that

E Auxiliary Lemmas

Lemma 4. For any n non-negative numbers a 1 , . . . , a n , and n real numbers b 1 , . . . , b n , the function f defined below

can have at most n -1 zeros.

Proof. We will show this by induction. The claim is obvious for n = 1 , let us assume the induction claim is true for n = k and we will show that it is true for n = k +1 . Indeed, let f be the function defined by

If b 1 = 0 or a 1 = 0 we can immediately use the induction step on f . Otherwise, we can write f in the following way:

Since b 1 a x 1 is always non-zero, the amount of roots of f is equal to the number of roots of the function g defined by

By Rolle's theorem, the number of roots of a function is bounded by the number of roots of its derivative plus one. We have that g ′ ( x ) is given by:

Which, by the induction step, can have at most k roots, as needed.

Lemma 5. Let θ be a real number in the open interval (0 , 2 π ) , let b be a real number in [0 , 2 π ] , and let p be a natural number such that θ · p ≥ 2 π . Then the set

contains at least one point in any closed and continuous-mod2 π interval of length θ within [0 , 2 π ] .

Proof. Let x k = ( b + kθ ) for k = 1 , 2 , . . . , p . Define the intervals

Each interval I k has a length of θ . Since pθ ≥ 2 π we have that

Now, consider any closed and continuous-mod2 π interval J ⊂ [0 , 2 π ) of length θ . Since ⋃ p k =1 I k = [0 , 2 π ] , the interval J must intersect at least one of the intervals I k . That is,

̸

Since J has length θ and overlaps with I k ∗ (which also has length θ ), and since both are continuousmod2 π their intersection must include at least one of the endpoints of I k ∗ which is in the set

This completes the proof.

Lemma 6. Let α be a real number within the interval [0 , 1] , and let n, m be natural numbers with t being a natural number such that t ≥ n + m . The absolute value of the m th forward difference of the sequence X = { 1 , α, α 2 , . . . , α t -1 } at index n , as defined in Definition 1, is bounded above by

Proof. By induction, the m th forward difference of X at index n (when we start at index 0 ), for n ≤ t -m , is α n ( α -1) m . For α ∈ [0 , 1] , this absolute value simplifies to:

To find the maximum value of this expression within [0 , 1] , we differentiate with respect to α and set the derivative to zero:

Solving for α , we find that the critical points are at α ∈ { 0 , 1 } or:

A second derivative test confirms that α = n n + m is a maximum within [0 , 1] . Substituting α max = n n + m into our original expression, we obtain:

thereby completing the proof.

Lemma 7. Given a sequence A = { a 0 , a 1 , . . . , a n -1 } and two natural numbers m<n , the m th forward difference of A (Definition 1) at index n , denoted ( A ( m ) ) n , is given by

Proof. We will show this by induction over m . For m = 0 This is obvious. Let's assume that the Lemma is true for m = k and prove that it is also true for m = k +1 < n .

Indeed

as needed.

Lemma 8. Given two sequences A = { a 0 , a 1 , . . . , a n -1 } and B = { b 0 , b 1 , . . . , b n -1 } of length n , and for any natural number m<n , it holds that

where A ( m ) and A ( m ) are the m th forward difference (Definition 1) of A and B respectively

Proof. We aim to demonstrate that

which directly supports the statement of the lemma.

It suffices to verify this for m = 1 , as the general case can then be established by induction, showing that:

Indeed, for m = 1 , we have that for any i < n -1

Lemma 9. Let ϕ n R , ( A R ,B R ,C R ) ( I ) : t be a truncated impulse response of a real SSM denoted by Y . Each of the following two sequences

are impluse responses of some other real SSM of dimension n R with parameters ( ˜ A R , ˜ B R , ˜ C R ) where ˜ A R is non-negative, and

Proof. We observe that (see Section 2.3)

setting ( ˜ A R , ˜ B R , ˜ C R ) = ( A 2 R , B R , C R ) yields that Y | odd is the impulse response of the real SSM ϕ n R , ( ˜ A R , ˜ B R , ˜ C R ) .

For the even case:

and

setting ( ˜ A R , ˜ B R , ˜ C R ) = ( A 2 R , B R , C R A R ) yields that Y | even is the impulse response of the real SSM ϕ n R , ( ˜ A R , ˜ B R , ˜ C R ) . In both cases, ˜ A R is non-negative, and

since in the even case ( ˜ B R , ˜ C R ) = ( B R , C R ) and in the odd case | ˜ C R j | ≤ | C R j | because | A R j,j | ≤ 1 for all j .

Lemma 10. For any positive integer n , the following inequality holds:

Proof. Using the Stirling's formula for factorials, we know that n ! is bounded by:

which can be simplified to:

we have

We can plug the bound in the factorial form of the binomial coefficients:

Where the last inequality is due to √ πe 1 6 n < 2 for all n ≥ 2 . To complete the proof for all positive integers notice we get equality for n = 1 .

Lemma 11. For any positive integer n , the following inequalitie hold:

To validate the lemma, we consider cases based on the parity of n . For even n ,

and for odd n ,

thereby proving the lemma as required.

Lemma 12. For any positive integers n and m , it holds that

Proof. We establish this inequality through direct analysis:

This last inequality leverages the fact that ( n + m ) 2 -4 nm = ( n -m ) 2 > 0 .

Lemma 13. For any complex number x = a + bi with a ≤ 0 , it holds that

Proof. It suffices to show that | 1 -x | ≥ 1 . Indeed, we have

as required.

Input:

· · ·

c

X

1

X

2

· · ·

X

h

· · ·

c

X

1

· · ·

X

h

-

1

Output:

· · ·

X

1

X

2

· · ·

X

h

2

h

F Implementation Details

Appendices F.1 to F.3 below present implementation details omitted from Sections 4.1 to 4.3, respectively. Source code for reproducing the results of Sections 4.1 and 4.3 can be found at https://github.com/edenlum/SSMComplexParamBenefits . The results of Section 4.2 were obtained using the official S4 repository, available at https://github.com/state-spaces/s4 (Apache-2.0 license). All experiments were conducted on a single NVIDIA A6000 GPU.

F.1 Theoretically Analyzed Setting

In all experiments, we parameterized state transition matrices in a manner that ensures stability, similarly to the LRU architecture [41]. For real SSMs, we performed a grid search for each optimizer, varying learning rates and initialization schemes. Namely, we evaluated learning rates of 1 × 10 -4 , 1 × 10 -5 and 1 × 10 -6 , and randomly initialized the diagonal elements of A R uniformly in [ -1 , 1] or in [ -1 , 0 . 99] ∪ [0 . 99 , 1] . For complex SSMs, we used a learning rate of 1 × 10 -5 and initialized the diagonal elements of A C similarly to [41], by sampling uniformly from the complex ring with radii 0 . 99 to 1 . For all SSMs, we employed a cosine learning rate scheduler [35] and trained for half a million steps.

F.2 Real-World Setting

Our implementation is based on the official S4 repository, available at https://github.com/ state-spaces/s4 (Apache-2.0 license). Unless stated otherwise, repository hyperparameters (pertaining to the neural network architecture and its training) were kept at their default values. The SSM powering the architecture was adapted to a diagonal structure for the state transition matrix A . For real SSMs, we began with the same default configuration as for complex SSMs, and extended the configuration by performing a grid search over different optimizers (Adam [29], AdamW [36], SGD [45]) and initialization schemes (diagonal-real, diagonal-linear, and legs, as defined in the official S4 repository). We ran three random seeds with the default configuration, and a single random seed with all others.

F.3 Selectivity

Copy task. Our version of copy task has a delay of t = 64 . It entails input sequences of length 2 t , where tokens are sampled independently and uniformly as one-hot vectors of dimension 16 . The corresponding output sequences are generated by shifting the input sequences t positions to the right, while introducing zero padding. That is, given an input sequence [ a 1 , a 2 , . . . , a 2 t ] , the corresponding output sequence is [0 , 0 , . . . , 0 , a 1 , a 2 , . . . , a t ] . Training and evaluation (measurement of test accuracy) in this task are based on randomly generated examples, where the input is sampled from a uniform distribution. In training, a fresh batch of examples is generated for each iteration, and the loss corresponding to an example is the cross-entropy loss averaged over the last t tokens of the output. In evaluation, the zero-one loss is averaged over the last t tokens of the output across one thousand freshly generated examples.

Induction-head task. In the induction-head task [18], the goal is to teach a model to identify and copy a specific subsequence of the input. Figure 1 illustrates this task, which in our case has an induction head size of h = 128 . The task entails input sequences of length 3 h , i.e. , input sequences with 3 h tokens. The first (less than h ) tokens in an input sequence are sampled independently and uniformly as one-hot vectors of dimension 16 . These tokens are followed by a special copy token ,

c = 0 . After the copy token comes a sequence of h tokens, sampled as before and denoted X . The 2 h th token is another copy token c , after which the first to penultimate tokens of the sequence X repeat. The tokens between the first appearance of X and the second appearance of c are irrelevant for the task. The output sequence corresponding to the above-described input sequence is equal to the input sequence shifted one position to the left (with the last token of X placed on the right). Training and evaluation (measurement of test accuracy) in this task are based on randomly generated examples, where the following aspects of the input are sampled from uniform distributions: the location of the first copy token c , the sequence X , and the irrelevant tokens. In training, a fresh batch of examples is generated for each iteration, and the loss corresponding to an example is the cross-entropy loss averaged over the last h tokens of the output. In evaluation, the zero-one loss is averaged over the last h tokens of the output across one thousand freshly generated examples.

Hyperparameters. Our implementation is based on a widely adopted Mamba repository, available at https://github.com/alxndrTL/mamba.py (MIT license). Unless stated otherwise, repository hyperparameters (pertaining to the neural network architecture and its training) were kept at their default values. The Mamba neural network was configured to have two layers with a hidden dimension of 64 . We trained the network with a batch size of 8 and a learning rate of 1 × 10 -3 . Training comprised 1 , 000 , 000 steps on the copy task, and 250 , 000 steps on the induction-head task (the latter task warranted a shorter run-time since it was experimented with much more). For complex parameterization, we changed the state transition matrix A , the input matrix B and the output matrix C to be complex, while keeping the discretization parameter ∆ real. To maintain parameter count, the dimension of SSMs with complex parameterization was half the dimension with real parameterization, namely, 8 as opposed to 16 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly differentiate between formal contributions and informal discussions.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper includes a dedicated limitations section (Section 6).

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Every theoretical result is formally stated (as a lemma, proposition, theorem or corollary) and proven.

4. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper provides (in Section 4 and Appendix F) references to code for reproducing experiments.

5. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper provides (in Section 4 and Appendix F) a detailed account of implementation details.

6. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports (in Section 4 and Appendix F) minimum and maximum values across multiple random seeds, while specifying the number of seeds.

7. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper specifies (in Appendix F) the type of hardware used for compute, as well as details determining the computational load of each experiment (number of training steps, neural network architecture etc. ).

8. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper is theoretical in nature, with experiments on synthetic or standard datasets designed to corroborate the theory. As such, it inherently conforms with the NeurIPS Code of Ethics.

9. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is theoretical in nature, with experiments on synthetic or standard datasets designed to corroborate the theory. As such, its societal impacts are self explanatory.

10. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper does not offer data or models that have a high risk for misuse.

11. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper specifies (in Section 4 and Appendix F) all assets used, along with applicable license details.

12. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper is accompanied by code for reproducing experiments ( https:// github.com/edenlum/SSMComplexParamBenefits ), and this code is well documented.

13. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not include crowdsourcing experiments or research with human subjects.

14. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not include research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical analysis of why SSM needs to be parameterized by complex numbers instead of real numbers. It shows that there exist complex LTI systems that could not be well-approximated by real systems of comparable size. Moreover, it proves that certain dynamics cannot be approximated by real LTI systems unless it has an exponentially large state-space dimension or large system matrices; yet, using complex LTI systems resolves this issue.\n\nSTRENGTHS:\n1. The theoretical statements are precisely made. The sketch of the proofs are helpful for understanding the paper.\n2. The comparison between real and complex is fairly thorough, encompassing different perspectives.\n\nWEAKNESSES:\n1. My main concern is about the contribution of this work. While it is true that many ML models use real parameterizations, the diagonal matrix $\\mathbf{A}$ comes from diagonalizing a general state matrix. Therefore, unless one puts restrictions on the matrix to be diagonalized (e.g. Hermitian), it is natural to assume that $\\mathbf{A}$ should be complex-valued. Showing why a real parameterization does not work well sounds like a slightly artificial problem and adds relatively little to the SSM community.\n2. The experiments could not strongly corroborate the theory. In addition to showing the performance, maybe some synthetic experiments would be helpful to show the exponential gap in Theorem 2 and Proposition 3.\n\nQUESTIONS:\n1. You formulate the problem using the discrete LTI system. Of course, in SSMs, the parameters are from the continuous-time LTI systems. While this does not change the basic question you are exploring because the real axis in the left half-plane gets mapped to the real axis in the unit disk under virtually all discretization schemes, would Theorem 2 be changed if you take the discretization into account?\n2. The paper studies two cases: $\\mathbf{A}$, $\\mathbf{B}$, and $\\mathbf{C}$ are either all restricted to real or all allowed to be complex. Intuitively, however, the important thing is that $\\mathbf{A}$ has to be complex. Have you looked into the case where $\\mathbf{A}$ is complex and $\\mathbf{B}$ and $\\mathbf{C}$ are real? In that case, which world would it fall into?\n3. In section 3.3, instead of giving two examples of dynamics that are poorly approximated by real systems, maybe there could be a discussion of what you called forward difference. It would be helpful to relate Theorem 2 to some general (and easily interpretable) properties of the dynamics.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers learning LTI systems with bounded response. It shows that if we restrict to SSMs with diagonal dynamics, both real-valued and complex valued state sizes suffice. However, it shows that if we use only real-valued SSMs, to learn a sequence of length t we will need parameters that scale as $\\exp(t)$, while if we use complex-valued SSMs, we need only parameters that scale as $t$.\n\nSTRENGTHS:\nThis paper did a great job of distinguishing between model expressivity (showing that with large enough state size both real- and complex-valued SSMS are equally expressive) and practical learnability (showing that real- valued SSMs need an impractically large state size). \n\nI also liked how their proof of practical learnability was basically based on the point that for stable real-valued diagonal SSMs, the dynamics pretty have to be decaying exponentials, so we will need exponentially many of them to form a basis for a function. On the other hand, for complex-valued diagonal SSMs, we effectively have a Fourier basis, which is more expressive.\n\nI also like the robust limitations section of the paper, which emphasized that although the paper has demonstrated the need for complex parameterization in the LTI setting, they haven't addressed the selectivity setting, leaving open theoretical contributions to explain why Mamba can get away with using only real parameterization for language tasks. \n\nI also liked the proof technique based on impulse series and impulse response, that was a very nice framing.\n\nThis paper provides an effort grounded in both experiments and theory that attempts to resolve an important question practitioners genuinely care about.\n\nWEAKNESSES:\nThe paper would have been more interesting if it had addressed what is going on in the setting of selectivity. On lines 66-7, the authors write \"We believe our results may serve as an important stepping stone in this pursuit.\" It would be nice if more evidence was given. I can't see how the proof techniques carry over. It would be great if some indication could be given of how this paper could be helpful in addressing selectivity.\n\nFootnote 1 is extremely confusing. It basically contradictions lines 101-2, which state that the SSMs considered in the paper are going to be diagonal. So, the eigenvalues of a real-valued diagonal A are definitely always real. I would just remove footnote 1.\n\nQUESTIONS:\n* I think the first sentence of the paper should name check S5 between S4 and Mamba (which is sometimes referred to in the body of its text as S6). S5 helped to pioneer the use of diagonal complex dynamics in an LTI setting, about which this entire paper is based. I think not namechecking it, and instead lopping in onto the end as \"and more\" does a disservice to a reader trying to learn more about the literature.\n* How do the results in this paper serve as an important stepping stone towards understanding real vs complex parameterization in LTV or selective settings?\n* What happens if we use a feedthrough matrix D in (1), which is common in nearly all SSM parameterizations?\n* What is $\\Omega(t)$ on line 187? Is it defined somewhere in the paper?\n* Is it possible to do better than $n_{\\mathbb{C}} = t$? why or why not?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-13"}

comments: SUMMARY:
This work establishes a formal gap between real and complex parameterizations of stable, diagonal SSMs. While complex parameters can trivially express any real SSM, the converse is not true and real SSMs need an arbitrarily large number of parameters to approximate complex SSMs in at least two important cases.

STRENGTHS:
1. Clear writing and presentation
2. Theoretical results support a clear practical suggestion: use complex parametrizations for your SSM if you don't have input selectivity.

WEAKNESSES:
1. Experimental results on non-synthetic datasets would be great. Especially considering the theorems regarding practical learnability and the exponentiality of real parametrizations for random impulse responses with high probability.

QUESTIONS:
See above.

LIMITATIONS:
The paper has a thorough discussion of the limitations of the analysis.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with an important question: is it possible to show a concrete advantage of complex diagonal RNNs compared to real diagonal RNNs? The motivation is clear, especially for people who studied the SSM literature. While modern SSM variants used in language modeling (e.g. S6-Mamba) do not make use of complex numbers, for some reasoning tasks in the long-range arena, these are necessary. This motivates the question \"which tasks necessitate complex numbers and why?\". The authors start with a useful recap, and correctly point out universality of both real and complex linear RNNs in the approximation of filters. Then show one example of a filter which is represented easily with complex numbers, but requires a huge hidden state if the recurrence is real. Further, the authors discuss a bigger class of filters that comprises random convolutions and shifts: here, real RNN parameters have to explode to reach the correct values. The authors corroborate their findings with experimental validations.\n\nSTRENGTHS:\nI believe in the value of complex numbers in recurrent networks.  I liked reading the paper, and I think many people can find it interesting. However, the theoretical results might not be super surprising for readers who already worked on theory of RNNs (see next part) but experiments are interesting. I think it's good to draw attention to the question the authors study.\nPros: Math is at a good level of formality, notation is precise, and the appendix is well structured. Clarity is good, though a specific thing can be improved (see later). I think with some tweaks, this can be a very good paper. I especially liked some parts of Sec. 3.3.\n\nWEAKNESSES:\nI like the theme of this paper and appreciate the formality and correctness of the results. However, I think some improvements are possible and some things need to be clarified.\n\n- Theorem 1: this is a crucial piece of Section 3. In short, both real and complex diagonal RNNs are universal, but complex diagonal linear RNNs require exponentially fewer parameters to approximate some specific functions. I agree with the proof, which is very simple: you cannot approximate well sin(x) using exponential modes (i.e. real RNNs). It is good to remind this fact, though the result is not novel it helps the discussion. However, while your statement is correct, it might be misleading: it looks like for a big part of the functions of interest in the hypothesis class, complex numbers are crucial. What the theorem actually shows is that for a subset of measure zero in the class of smooth functions, complex valued recurrences are more effective. I agree oscillations are important, but this is not formal and should be toned down. I would phrase this as an example rather than a theorem. It is indeed just a counterexample.\n  \n- The second part of Section 3, Sec 3.3, is completely disconnected: here, you are talking about parameter magnitudes. This is fine, but I was a bit confused by the wall of text at the beginning of page 5. You should be much more schematic. You are now talking about something bit different, and the bridge should be clear but concise.\n  \n- Sec 3.3 is interesting, but I'm not at all convinced that the results you found are orthogonal to those of Orvieto et al. (discussed at the end of page 8). I think your results are stronger, but comparing further makes your paper much more robust: Orvieto et al. shows that to retrieve information from real RNNs hidden states, the readout maps must explode in magnitude. Here, you found something similar, but on the B & C matrices when approximating some filters. I suspect the mechanism is similar, do you agree? I think drawing a better connection is needed, no way those effects have a disjoint cause.\n  \n- Sec 3.3: All bounds you have here should be validated in the simplest toy setting possible. Would it be possible to verify empirically all your bounds in the setting of the propositions? This makes the reader believe much more in the results and makes the bounds clearer and visual.\n\n- Experiments: I like them, but as the authors themselves admit, more efforts are needed. Well my question here is: what would you do next, and why do you think the paper should be accepted without further experiments?\n\nQUESTIONS:\nSee above, + I have another question - something which I might have missed but is my biggest curiosity:\n\n- How do you explain the fact that Mamba on language modeling works better with real recurrences? I am perfectly aware the question is not easy - but given your efforts, I believe your paper should try to answer this question. What you claim in the paper is \"selectivity allows the real parameterization to achieve comparable performance\": this is a bit too quick, on an extremely crucial issue. I expected the discussion to be much more precise on this point.\n\nI will be active in the rebuttal, and will engage in the discussion so to update my borderline score. I thank the authors in advance for their efforts. I know the topic in this paper is not an easy one, but for this exact reason I care the findings are robust and discussion complete.\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}