Title: Theoretical Foundations of Deep Selective State-Space Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: theory

deep learning theory

SSM

RNN

neural controlled differential equations

signatures

kernels
Abstract: Structured state-space models (SSMs) are gaining popularity as effective foundational architectures for sequential data, demonstrating outstanding performance across a diverse set of domains alongside desirable scalability properties. Recent developments show that if the linear recurrence powering SSMs allows for a selectivity mechanism leveraging multiplicative interactions between inputs and hidden states (e.g. Mamba, GLA, Hawk/Griffin, HGRN2), then the resulting architecture can surpass attention-powered foundation models trained on text in both accuracy and efficiency, at scales of billion parameters. In this paper, we give theoretical grounding to the selectivity mechanism, often linked to in-context learning, using tools from Rough Path Theory. We provide a framework for the theoretical analysis of generalized selective SSMs, fully characterizing their expressive power and identifying the gating mechanism as the crucial architectural choice. Our analysis provides a closed-form description of the expressive powers of modern SSMs, such as Mamba, quantifying theoretically the drastic improvement in performance from the previous generation of models, such as S4. Our theory not only motivates the success of modern selective state-space models, but also provides a solid framework to understand the expressive power of future SSM variants. In particular, it suggests cross-channel interactions could play a vital role in future improvements.

Theoretical Foundations of Deep Selective State-Space Models

Abstract

Structured state-space models (SSMs) are gaining popularity as effective foundational architectures for sequential data, demonstrating outstanding performance across a diverse set of domains alongside desirable scalability properties. Recent developments show that if the linear recurrence powering SSMs allows for a selectivity mechanism leveraging multiplicative interactions between inputs and hidden states (e.g. Mamba, GLA, Hawk/Griffin, HGRN2), then the resulting architecture can surpass attention-powered foundation models trained on text in both accuracy and efficiency, at scales of billion parameters. In this paper, we give theoretical grounding to the selectivity mechanism, often linked to in-context learning, using tools from Rough Path Theory. We provide a framework for the theoretical analysis of generalized selective SSMs, fully characterizing their expressive power and identifying the gating mechanism as the crucial architectural choice. Our analysis provides a closed-form description of the expressive powers of modern SSMs, such as Mamba, quantifying theoretically the drastic improvement in performance from the previous generation of models, such as S4. Our theory not only motivates the success of modern selective state-space models, but also provides a solid framework to understand the expressive power of future SSM variants. In particular, it suggests cross-channel interactions could play a vital role in future improvements.

1 Introduction

Sequence-to-sequence blocks are fundamental components of modern deep learning models for language, images, video, audio, time series, and genomics. For the last five years,attention [Vaswani et al., 2017, Dosovitskiy et al., 2020] has been the dominant mechanism powering these architectures. However, competitive results have recently been achieved without attention, by using state-space models (SSMs): GPU-efficient linear recurrent sequence-to-sequence blocks stemming from S4 [Gu et al., 2021]. SSMs achieve state-of-the-art results on long-range-reasoning benchmarks [Tay et al., 2020] and show outstanding performance in various domain including vision [Nguyen et al., 2022], audio [Goel et al., 2022], biological signals [Gu et al., 2021], reinforcement learning [Lu et al., 2023] and online learning [Zucchet et al., 2023]. SSMs recently have gained significant interest in the community since their computational complexity scales linearly in sequence length, while attention scales quadratically; moreover, unlike other recurrent mechanisms such as LSTMs [Hochreiter and Schmidhuber, 1997] and GRUs [Cho et al., 2014], they can be efficiently parallelized on GPUs during training using parallel scans [Martin and Cundy, 2017, Smith et al., 2023].

While standard SSMs were shown to be particularly powerful on signal processing tasks, their computation power is limited: the core sequential mechanism of S4 is equivalent to a convolution (filtering) [Li et al., 2022a]. This represents a drawback in challenging domains such as text and genetics, where the ability to select data efficiently in an input-dependent manner - i.e., perform content-based reasoning - is crucial (see [Wang et al., 2022, Fu et al., 2022, Arora et al., 2023]). Towards reaching this goal with recurrent models, various adaptations of S4 have been proposed in the last few months. Notably, Mamba [Gu and Dao, 2023] implements simple and efficient gating mechanisms on the S4 recurrence, unlocking input selectivity in the memory update. Mamba achieved state-of-the-art performance in various language modeling tasks while greatly improving the inference throughput. Similar ideas can be found in recent developments inspired by attention, such as RWKV [Peng et al., 2023], RetNet [Sun et al., 2023], Gateloop [Katsch, 2023], Gated Linear Attention (GLA) [Yang et al., 2023], and HGRN2 [Qin et al., 2024]. Very recently, De et al. [2024] surpassed the performance of Mamba with a gated RNN architecture - Griffin - based on an improved version of the LRU [Orvieto et al., 2023a], and [Feng et al., 2024] introduced minimal versions of GRU and LSTM as gated SSMs.

Contributions At the core of the models discussed above is a time-varying dynamical system, where reasoning is performed through an efficient and parallelizable update linear in the hidden state. In this paper, we generalize the structure of such models, drawing a direct link to controlled differential equations (CDEs) [Young, 1936, Lyons, 1994, Kidger et al., 2020, Morrill et al., 2021, Fermanian et al., 2021, Salvi et al., 2022, Hoglund et al., 2023, Walker et al., 2024] and use tools from rough path theory [Lyons et al., 2007] to study expressivity.

In Sec. 3.1 we provide a framework for the analysis of (input-controlled) linear (in the hidden state) recurrences such as S4 and Mamba. This framework allows the use of powerful tools and results in the Rough Path Theory literature by casting a large family of SSMs as Linear CDEs driven by the two possibly nonlinear embeddings X ↦→ ω X and X ↦→ ξ X , defining gates. Appendices A and E provide a largely self-contained exposition of the key theoretical tools now available to us.

In Sec. 4 we fully characterize the closure (i.e. the class of functions which can be arbitrarily well approximated) of our generalized models. This provides a generalization of the results by Li et al. [2022b], Orvieto et al. [2023b], Wang and Xue [2023], who only consider the case of S4. The Mamba setting is more rich, complex, and relevant given the rising interest in selective SSMs.

We show (Thm. 4.2) that full expressivity can be obtained by training only a linear layer on a Linear CDE with random parameters, providing a direct link to kernel methods and reservoirs.

We point out (Thm. 4.3) that if the recurrence is diagonal, as the case for Mamba, the closure is strictly smaller than in the general dense case. Interestingly though, the closure is a peculiar set of filters that unlock some specific context-dependent processing. Full expressive power is recovered by stacking multiple SSMs without MLPs in between (Prop. 4.5).

Our framework not only provides significant theoretical insight regarding some recently proposed SSM architectures, but we also envision it to be a useful tool in analysing, and perhaps developing, future architectural advances.

2 State-space Models

We describe here the structure of the main SSMs-based strategies for processing lengthL input sequences of d dimensional tokens: x ∈ R d × L . We denote by x ℓ the ℓ -th column of x (the ℓ -th token) and by x i the i -th row of x (time series for the i -th channel). We will write A · v for matrix-vector multiplication when this enhances comprehension, and use bold letters for 'tensors' of order greater than 2 (such as z ∈ × i R N i × L introduced below).

2.1 Review of Modern SSMs

We start with a quick simplified recap of S4 [Gu et al., 2021], the first SSM proposed in the literature, and then describe recent improved variants such as Mamba (in particular, the S6 block) [Gu and Dao, 2023]. We restrict our focus to the recurrent mechanism and invite the reader to refer to the original papers for a description of the token-wise operations following and preceding each block.

SSM basics and S4. Most 1 SSMs [Gu et al., 2021, 2022] operate independently on input channels. Each time series x i ∈ R L is seen as the result of sampling a latent continuous-time signal X i : [0 , 1] → R at multiples of a channel-dependent stepsize ∆ i > 0 : X i ∆ i ℓ := X i (∆ i ℓ ) = x i ℓ . In S4, each path X i produces a complex-valued hidden state signal Z i : [0 , 1] → C N i as

where A i = diag ( a i, 1 , a i, 2 , . . . a i,N ) is channel-specific diagonal N i × N i complex valued matrix and B ∈ C N i is an input projection shared across input components i ∈ [ d ] . SSMs are based on a stable discretization of the continuous system above: each input sequence channel x i ∈ R L produces a sequence of hidden states z i = [ z i ;1 | z i ;2 | . . . | z i ; L ] ∈ C N i × L as follows:

where ¯ A i and ¯ B i are determined by the discretization technique and the channel-dependent stepsize ∆ i . Under the commonly used Zero-Order Hold discretization 2 ,

Note from (2) that SSMs at inference time are equivalent to linear recurrent neural networks (RNNs). Yet, learning with gradient descent is performed on the continuous-time variables, unlocking stable signal propagation and alleviating vanishing gradients [Orvieto et al., 2023a, Zucchet and Orvieto, 2024]. Finally, at each channel i , the sequence of hidden states is mapped back to real numbers, and linear projections C i : C N i → R are performed to produce an output a sequence of tokens y ∈ R d × L with the same dimensions as x :

To conclude, we point out that the transition matrices A i are often structured, i.e. initialized deterministically through HiPPO theory [Gu et al., 2020] in diagonal form. Common choices [Gu et al., 2022] are a · ,n = -1 2 +i πn (S4D-Lin 3 ) and a · ,n = -1 2 (S4D-Real).

Mamba. As done in practice, let us consider all channels' hidden dimensions N i equal to N . The Selective SSM (S6) powering the Mamba architecture [Gu and Dao, 2023] augments S4 with input-controlled matrices:

where the most crucial component (see in-context learning argument by Gu and Dao [2023]) is the dependency of the diagonal matrix ¯ A i ( x ℓ ) ∈ R N × N at timestamp ℓ on all input channels at timestamp ℓ . This makes the operation ¯ A i ( x ℓ ) · z i ; ℓ -1 effectively a gate . The dependency of ¯ A i : R d → R N × N on the input is achieved efficiently by letting ∆ i in (3) be computed, at step ℓ , as ∆ i ( x ℓ ) where

where · is the scalar product and 4 α i ∈ R d , β i ∈ R . Further, ¯ B i : R d → R N is computed via a, shared between channels, linear map B ∈ R N × d via ¯ B i ( x l ) = ( B · x ℓ ) ∆ i ( x ℓ ) ∈ R N . Finally, each z i ; ℓ ∈ R N is projected to y i ℓ ∈ R via a matrix C i . This step can also be done by means of output gating ( C i function of the input), but we avoid this complication here as it can be seen as an architectural component outside the recurrence.

Remark 2.1 . While each channel evolves separately, the laws of evolution are pointwise determined by all input features: A i and B i can be functions of x ℓ , and not just of x i ℓ . We will discuss this in Sec. 4.3 after presenting our general results.

The RG-LRU [De et al., 2024] works similarly, yet processing all input channels at once with a diagonal recurrence. Gateloop [Katsch, 2023], GLA [Yang et al., 2023], and HGRN2 [Qin et al., 2024] leverage similar ideas, though they differ in parametrization and gating strategies.

2.2 Known properties of (non-linear) recurrences

The expressiveness of standard nonlinear RNNs of the form z ℓ = Aσ ( z ℓ -1 ) + Bx ℓ , where σ is a nonlinearity, has been extensively studied since the seminal work of Siegelmann and Sontag [1992], with recent contributions such as Korsky and Berwick [2019] and Hanson and Raginsky [2020]. In particular, Hanson and Raginsky [2020] proved that wide enough non-linear RNNs can approximate up to vanishing precision non-linear time-homogeneous systems of differential equations driven by input paths. The argument used here is based on the celebrated Barron's theorem [Barron, 1993] for approximation of continuous functions with neural networks with one hidden layer. Indeed, note that non-linear RNNs are recurrent perceptrons with one hidden layer, acting both on the state and the input [Tallec and Ollivier, 2018]. Instead, (selective) SSMs such as S4 and Mamba have transition map which is linear in the state - unlocking parallelization [Smith et al., 2023, Gu and Dao, 2023]. In the context of linear RNNs and non-selective SSMs, many results (classic and new) exist that characterize expressivity. Li et al. [2022b] showed that linear RNNs (i.e. S4-like recurrences) can approximate arbitrary convolution filters in the width limit. Further, Hanson and Raginsky [2019] proved that stacking exponentially (in the sequence length) many temporal convolution filters, chained together with ReLU activations, leads to approximation of arbitrary non-linear filters. Recent works [Orvieto et al., 2023b, Wang and Xue, 2023] prove the universality of linear recurrences (one layer) when equipped with a fixed (timestamp independent) point-wise MLP acting across the recurrence output, with intriguing connections to Volterra series [Boyd and Chua, 1985]. Mamba (alongside with gated linear attention variants e.g. Yang et al. [2023]) falls neither in the linear RNN nor the nonlinear RNN setting: its recurrence is linear on the hidden state (can be parallelized) but unlike S4, it is not linear time-invariant as the input controls the recurrence eigenvalues. In this paper, we are interested in this hybrid setting. It is worth noting that some work exploring Mamba's expressiveness has already been performed to study some interesting toy tasks [Jelassi et al., 2024] and to understand its limitation using the framework of formal language theory [Merrill et al., 2024]. Compared to these works, which outline interesting failure cases, this paper studies a more general class of models, allowing to identify how architectural choices impact expressivity.

3 SSMs as Linear CDEs

The crucial component that unlocks in-context learning and selectivity in modern SSMs is the inputdependent state-to-state transition matrix [Gu and Dao, 2023], gating the hidden state and thus allowing the system to filter out unnecessary context and remember relevant information indefinitely.

At the core of most modern SSMs is a a recurrence which is linear in the hidden state, but potentially non-linear in the input . This class includes many recent SSM-based or inspired models. Crucially, it does not contain classical RNNs, LSTMs, and GRUs - for which results are known and rely on the non-linear dependence of the hidden state in the update rule (Sec. 2.2). As we will shortly see, structure and features of (selective) SSMs can be studied within a unified, convenient, continuous-time framework (Linear Controlled Differential Equations). This allows to answer the following question:

' What is the most one can achieve using a recurrence which is linear in the hidden state and potentially non-linear in the input? '

3.1 Linear CDEs

According to their continuous-time formulation, SSMs process input data sampled from a continuous path X : [0 , 1] → R d , where d is the number of channels. By X i t we denote the input channel i , evaluated at time t ∈ [0 , 1] . More formally, we consider input trajectories in the separable Banach space X = C 1 , 0 ([0 , 1]; R d ) of absolutely continuous R d dimensional paths. In this space, one can write X = ∫ t 0 ˙ X s ds since X ∈ L 1 ([0 , 1]; R d ) ; and the norm is ∥ X ∥ 1;[0 , 1] := ∫ 1 0 | ˙ X s | ds .

To model gates (see Mamba in (4)), we introduce two maps transforming the path X , which output trajectories ω X , ξ X living in potentially higher dimensions: d ω and d ξ

where we used the shorthand notation ω X := ω ( X ) , ξ X = ξ ( X ) . Akin the notation for the i -the channel of X (i.e. X i ), we denote by ω X ,i the i -th channel in ω X . The role of the ξ, ω functions - which we denote gating functions - will be clear very soon.

Definition 3.1 (Linear CDE) . Fix N ∈ N (hidden-state dimension), matrices A 1 , ..., A d ω ∈ R N × N and B ∈ R N × d ξ . The hidden state Z X := Z ( X ) is computed by the Linear CDE through a linear (in the hidden state) differential equation driven by ω, ξ - functions of the input path:

We show that both S4 and Mamba can be written in continuous-time as systems of parallel Linear CDEs. The key ingredient setting the two models apart is the choice of drivers ω and ξ . As in the preceding section, we will use the bold tensor notation to stress the parallel nature of these CDEs.

S4 is a Linear CDE: It is sufficient to consider the setting d = 1 , N = N i . The S4 model [Gu et al., 2021] in this case is dZ t = A · Z t dt + B ( X t dt ) , where A ∈ R N × N is diagonal. This can be written as a Linear CDE with

since dω X t = dt and dξ X t = X t dt . As the reader can promptly notice, here ω X is not a function of X - this will have a crucial role in expressivity (see Sec. 4.2).

Mamba is a Linear CDE: Recall from (4) that the recurrence inside Mamba (i.e. S6), can be written as z i ; ℓ = ¯ A i ( x ℓ ) · z i ; ℓ -1 + ¯ B i ( x ℓ ) x i ℓ where for generic timestamp features x ∈ R d , we have ¯ A i ( x ) = exp(∆ i ( x ) A i ) , ¯ B i ( x ) ≈ ( B · x )∆ i ( x ) and ∆ i ( x ) = softplus ( α i · x + β i ) . Let us introduce a parameter δ > 0 , and consider ˜ α i = α i /δ, ˜ β i = β i /δ . Let us further approximate the softplus function with a ReLU ( σ ( x ) = ReLU ( x ) ≃ log(1 + e x ) ) to obtain ∆ i ( x ) = σ ( δ ˜ α i · x + δ ˜ β i ) = σ (˜ α i · x + ˜ β i ) δ . Therefore, as δ → 0 , one has ¯ A i ( x ) = exp(∆ i ( x ) A i ) = exp( σ (˜ α i · x + ˜ β i ) δA i ) δ → 0 → 1 + σ (˜ α i · x + ˜ β i ) δA i , leading to the recurrence

As we show formally in Appendix F, δ plays the role of the differential dt . The equation above is the Euler discretization of the differential equation

where for each i , Z X i : [0 , 1] → R N . These are Linear CDEs with carefully chosen ω and ξ :

Note that here the ξ i depend on higher powers of X 's dimensions, an intriguing feature connected to input gating [Hochreiter and Schmidhuber, 1997, Cho et al., 2014].

Remark 3.2 (Are hidden state components recurrently mixed?) . In the Linear CDE defined by dZ X t = [ ∑ d ω j =1 A j dω X ,j t ] · Z X t + B · dξ X t channels of the transformed input path ω X are mixed in a linear way in the recurrent step and stored in a shared hidden state Z X . This allows our framework to be more general compared to S4 and Mamba, where each channel of the input is processed individually, and hidden states are later combined . We know already from Merrill et al. [2024] that this distinction between hidden state mixing strategies is crucial for expressivity. Our discussion in Sec. 4.3 provides an in-depth look at the effects of separate channel processing, achieved in our framework by choosing the A i s to be diagonal and non-zero only around a channel-specific portion.

4 Expressivity of Linear CDEs

Having established the connection between SSMs and Linear CDEs, we now provide an explicit characterization of the uniform closure of Linear CDEs, i.e. a description of all the functions from compact subsets of X to R that can be uniformly approximated at an arbitrary precision by a Linear CDE of the form given in (5).

4.1 Characterization of the closure

The proof of the main theorem we present here (Thm. 4.1) is involved and requires tools from Rough Path theory; we provide a full derivation in Appendix B with tools reviewed in the self contained Appendix E, as well as an introduction to the Signature approach in Section 4.5 and Appendix A. While familiarity with these concepts is not necessary to grasp the main results as stated, it is essential for a thorough comprehension.

In this subsection, Linear CDEs are analyzed in full generality - i.e., in the dense setting. While for efficiency reasons non-diagonal recurrences are rarely used in SSMs, our results allow to precisely characterize the Linear CDE hypothesis class (i.e. the closure ), thus to the answer the question 'What is the most we can achieve from recurrences which are linear in the hidden state?' . We show that Linear CDEs can model, at fixed time t , arbitrary continuous functions of the whole seen inputs. Of course, understanding the diagonal setting with separate channel processing is of utmost importance in the current research landscape. The tools and results in this subsection allow us to directly discuss this case and compare it to the dense setting - this is presented in Sec. 4.3.

In this section, for any path γ ∈ C 1 , 0 ([0 , 1] , R d γ ) and any sub-interval [ s, t ] ⊂ [0 , 1] , we denote by γ [ s,t ] ∈ C 1 , 0 ([0 , 1] , R d γ ) the path γ [ s,t ] ( u ) = γ t ∧ u -γ s ∧ u , where a ∧ b = min( a, b ) .

Theorem 4.1. Let X ⊂ C 1 , 0 ([0 , 1] , R d ) be compact and choose continuous gating functions ω, ξ such that 5 ω X , 1 t = t and ω X , 2 t = t 2 . Consider the Linear CDE model (5). Let Ψ , Φ be generic continuous functions from paths to real vectors:

There exist dense matrices A 1 , ..., A d ω , B such that, after a fixed final linear projection C ∈ R 1 × N , the output Y X t = CZ X t is arbitrarily close, uniformly on X × [0 , 1] , to

where · is the scalar product. Moreover Y X t = CZ X t is itself of form (7).

In Theorem 4.1, the A i are dense matrices constructed ad hoc for the proof. We show next that, with high probability, random Glorot-initialized matrices [LeCun et al., 2012] provide enough expressivity - one only has to choose the appropriate matrix C . This is a similar mechanism to the paradigm advocated in reservoir computing [Lukoveviius and Jaeger, 2009, Cuchiero et al., 2021a, Compagnoni et al., 2023]. The next result, however, is novel and of independent interest in Rough Path Theory.

Theorem 4.2. Under the hypothesis of Theorem 4.1, pick [ A j ] n,n ′ iid ∼ N (0 , 1 N ) and [ Z 0 ] n , [ B ] n,j iid ∼ N (0 , 1) . For any functional F : X × [0 , 1] → R of the form (7) and ϵ > 0 it holds that

4.2 Intuition on the closure result: role of gates

Roughly speaking, our result shows that dense Linear CDEs have drastically superior expressive power compared to dense linear RNNs. This contrast is to be attributed completely to the gate ω .

Warmup - Linear RNNs. Thm 4.1 can be seen as a generalization of the Universal Approximation for Linear RNNs presented by Li et al. [2022b][Thm. 7] for generic gates ω, ξ . In fact, their setting is restricted to ω X t = t and ξ X t = ∫ t 0 X s ds . This is also the case for S4, S5 and the LRU - which are linear RNNs at test time. Then the only information contained in ω [ s,t ] is the increment t -s so that family (7) reduces to { ( X,t ) ↦→ ψ ( t ) + ∫ t 0 ϕ ( t -s ) · X s ds } . This is, fundamentally, the set of linear filters on the input. As shown in Gu and Dao [2023], such processing is unable to adapt information flow in-context.

̸

̸

How does a Linear CDE process inputs? Take without loss in generality 6 ω X t = X t . The first term in the function class { Ψ( X [0 ,t ] ) + ∫ t 0 Φ( X [ s,t ] ) · dξ X s } is already enough to establish that the output CZ X t is a nonlinear function of all previously seen inputs X [0 ,t ] -ξ could be set to zero. The term Ψ( X [0 ,t ] ) is however non-trivial only in the Z 0 = 0 case ( cf. Appendix B): picking ξ X t = 0 ∈ R N for all t indeed leads to dZ X t = [ ∑ d ω i =1 A i dω X ,i t ] Z X t , which is evolving only if Z 0 = 0 . While this setting is interesting and suggests a clear direction for future research, a case more similar to Mamba (see Sec. 4.3) is Z 0 = 0 and ξ X t = ∫ t 0 X s ds . Here, we can approximate arbitrarily well outputs of the form

where Φ is any continuous function of the input path, restricted to the portion [ s, t ] . This clearly shows that dense Linear CDEs are capable of context-dependent filtering : the output is again a linear combination of previously seen inputs, but weights are not predetermined as in Linear RNNs (S4) they are a function of the context. A similar yet less powerful processing is happening in the diagonal setting, which we explore next.

4.3 The Diagonal Case

The choice of diagonal weights considerably restricts the family of learnable functionals. Intuitively the diagonal choice corresponds to running N independent 1 -dimensional systems, this absence of mixing between the different hidden dimensions is the main culprit for the loss of expressivity. The full power can however be recovered by chaining the diagonal schemes ( cf. Prop. 4.5).

Theorem 4.3 (Diagonal Case) . If the matrices A 1 , ..., A d ω are constrained to be diagonal, the requirements ω X , 1 t = t , ω X , 2 t = t 2 can be dropped and the closure reduces to

for continuous ψ : R d ω → R and ϕ : R d ω → R d ξ .

Compared to the dense setting, the effect of diagonality is pretty clear. Let us again assume ω X t = X t and ξ X t = ∫ t 0 X s ds for simplicity. While ∫ t 0 Φ( X [ s,t ] ) · X s ds unlocks filtering based on the entire trajectory X [ s,t ] , the diagonal case term ∫ t 0 ϕ ( X t -X s ) · X s ds indicates that filtering coefficients can only be chosen by comparing two elements of the (potentially transformed through ω ) input sequence. While this precise and tight result reveals a pitfall of diagonal recurrence, it also brings about an interesting connection to attention [Vaswani et al., 2017], where only a finite number of tokens are compared at each layer. While a smart choice of gating functions ω, ξ can improve on the learned nonlinear filtering strategy (e.g. based on filtered input versions as in Mamba), our theory reveals the fundamental processing discrepancy compared to the dense setting, a property which was also explored in recent literature [Merrill et al., 2024] using different tools.

As already noted, on top of diagonality, recent SSMs also mix inputs as linear combinations of independently run channel-dependent systems. This slightly modifies the function class as:

Corollary 4.4 (Mamba Case) . In the Mamba setting, the closure reduces to

for continuous ψ i : R → R and ϕ i : R → R .

4.4 Chaining Diagonal CDEs

Fortunately it is possible to re-gain expressivity without sacrificing the computational advantages of diagonal schemes through chaining . This means driving a new Linear CDE by the solution of a previous Linear CDE, and repeating this procedure K times ( cf. Appendix C).

Proposition 4.5. Assume a compact X ⊂ C 1 , 0 ([0 , 1]; R d ) . For any functional F : X × [0 , 1] → R of the form (7) and ϵ > 0 there exists a sequence of linear maps W k ∈ R M k × N k , diagonal weights A ( k ) i and B ( k ) for the following family of chained diagonal Linear CDEs

such that eventually , as k →∞ , there exists C k ∈ R 1 × N k with sup ( X,t ) ∈ X × [0 , 1] | F ( X,t ) -C k Z k, X t | ≤ ϵ .

Intuitively, with chaining one recovers the mixing between the input dimensions which was so important for the expressiveness of dense Linear CDEs. This does not happen immediately but crucially depends on the length of the chain, the result in fact tells us that the recovery holds for long enough chains (and big enough hidden states). In Appendix C.2.1 we argue that the same conclusions hold also in the Mamba setting with non-linear gates.

4.5 Proof Idea - Signature expansion

In this section, we introduce the primary tools, objects, and techniques from Rough Path theory used in our proofs. Given their technical nature, we have chosen to present the main results of this paper without explicit reference to them, allowing readers to understand the work without needing specialized knowledge. For those interested in the finer details, here we provide a brief overview and refer to Appendix A for additional details and references.

To study the expressivity of Linear CDEs it is convenient to introduce the so-called signature transform [Lyons et al., 2007, Kidger et al., 2019, Fermanian et al., 2023], a classical path-transform from stochastic analysis. The main reason for doing so is that, as a simple consequence of the Stone-Weirestrass theorem, linear functionals on the signature provide the essential building blocks (analogous to monomials on Euclidean spaces) to approximate continuous functions on path space.

Consider a path γ ∈ C 1 , 0 ([0 , 1]; R d γ ) and define as W d γ the set of words ( i.e. ordered sequences) in { 1 , . . . , d γ } 7 . The signature transform is the following infinite collection of scalar iterated integrals

A classical result from rough path theory states that a Linear CDE can be expanded explicitly as an (infinite) linear combination of terms in the signature of the driving path.

Proposition 4.6. For any choice of matrices A 1 , ..., A d ω and B , the unique solution Linear CDE (5) is, using the notation A I := A i n ...A i 1 , given by

;

Notice that the previous result does not rely on any assumptions on the nature of Z 0 , A i , and B for any such choice the result is a time-independent linear map on a feature vector T ( X ) 0 ,t :=

Sig ( ω X ) ( I ) 0 ,t , ∫ t 0 Sig ( ω X ) ( I ) s,t dξ X ,i s ) ( I,i ) , where the index ( I, i ) runs over W d ω ×{ 1 , ..., d ξ } . The main takeaway is that any linear projection Y X t := CZ X t is written as an (infinite) linear combination of the terms in T ( X ) 0 ,t . This means that the expressive power of such schemes is almost completely determined by the gate ω , which is the only path of which high order information is taken into consideration through its full signature . It is evident then that the classical choice of input-independent ω (i.e. ω X t = t ) then precludes the use of higher order statistics of X .

5 Path-to-Path Learning

In Section 4, we showed that Linear CDEs (and chained Mamba) can model, at fixed time t , arbitrary continuous functions of the whole seen inputs. Assume wanting to learn a functional of type

( X,t ) ↦→ Ψ t ( ω X [0 ,t ] ) where the map Ψ t is changing with time as well, this is an example of a general continuous path-to-path model. Note in particular how this is a more general family than the one of Theorem 4.1, where the maps Ψ and Φ are fixed once and for all. Here, we discuss how passing the hidden state through a multi-layer perceptron (MLP), instead of just a linear readout (matrix C ), allows us to efficiently approximate this richer class, interpolating between the Ψ t s.

As shown in Orvieto et al. [2023b], classical SSMs followed by an MLP are universal on sequences. Given their nature of input-independent convolutions, their construction defers all reasoning to the MLP acting on the output: in S4, the SSM is simply providing an input compression - with no added reasoning. In our setting, we instead characterized the processing power of input-controlled (dense or diagonal) SSMs precisely, showing how it greatly surpasses linear filtering. For this reason, the computational burden for an MLP action on a general Linear CDE would be greatly diminished and its actual function reduced to an interpolation of the maps Ψ t . We defer the proof to Appendix D.

Proposition 5.1. Fix a compact set K ⊆ X and continuous ω, ξ with ω X , 1 t ≡ t . Then for all ϵ > 0 and all causal 8 continuous mapping G : C 1 , 0 ([0 , 1] , R d ω ) × [0 , 1] → R there exist an integer N ≥ 0 , some MLP F : R N → R , and parameters Z 0 ∈ R N , A i ∈ R N × N , B ∈ R N × d ξ such that

6 Empirical Validation

Code to reproduce all of our experiments can be found at:

https://github.com/Benjamin-Walker/selective-ssms-and-linear-cdes

Datasets. The first task is based on a dataset from Walker et al. [2024] where the aim is to predict terms in the anti-symmetric part of the input path's signature. The dataset's objective aligns with the proofs of Theorem 4.1 and 4.2, which characterise the closure using the path's signature. We created two datasets with dimensions 2 and 3 respectively. The increment in each channel at each step is an integer-rounded sample from a standard Normal distribution. The 2D dataset's target is an area integral ∫ 1 0 ∫ v 0 dX 1 u dX 2 v , and the 3D dataset's target is a volume integral ∫ 1 0 ∫ w 0 ∫ v 0 dX 1 u dX 2 v dX 3 w .

The second task is the A 5 benchmark from Merrill et al. [2024]. It tests models on state-tracking, a crucial ability for tasks involving permutation composition, such as chess. The dataset comprises sequences from the group of even permutations on five elements, A 5 , where the target is the cumulative composition of all preceding permutations. Datasets vary by sequence length, ranging from 3 to 20 .

Models. On the anti-symmetric signature task, we considered seven models: (i-ii) S4 or Mamba recurrence with linear readout, (iii-iv) two stacked S4 or Mamba recurrences with a linear mixing layer in-between and a linear readout, (v-vi) two stacked S4 or Mamba recurrences with a linear mixing layer + GeLU in-between, and a linear readout, (vii) a linear CDE with gates ω X t = ξ X t = ( t, X t ) and a linear readout. All state space models have trainable matrices in their recurrences, whereas the linear CDE is using fixed matrices. All models use a hidden dimension of 256 , with the state space models using a state dimension of 256 . The state space models are trained using gradient descent with a batch size of 32 and Adam with a learning rate of 10 -4 . The output from the linear CDE's recurrence is obtained using the Tsit5 adaptive ODE solver, with an absolute and relative tolerance of 10 -2 . The linear CDEs linear readout is optimised via ordinary least squares.

On the A 5 benchmark, we consider five models: a linear CDE, a RNN, a transformer, and S4 and Mamba recurrences. All models use an embedding layer followed by a series of blocks that combine the sequence-to-sequence model, linear mixing with a non-linear activation function, layer normalization, and a residual connection. Furthermore, all models have trainable matrices in their recurrences and are trained using batch gradient descent with a batch size of 32 and AdamW with a weight decay of 0 . 01 . The RNN, transformer, S4, and Mamba use a hidden dimension of 1024 , with the state space models using a state dimension of 64 and the transformer using 64 heads. Due to memory constraints, the linear CDE uses a hidden dimension of 256 . Note that the IDS4 recurrence introduced by Merrill et al. [2024] corresponds directly to a linear CDE with dense transition matrices, hence we did not include the model as a baseline.

Results. Results on the anti-symmetric signature prediction task (Fig. 1) empirically demonstrate a number of the theoretical results presented in this paper. Firstly, as discussed in Sec. 4.2, recurrences which are linear in the input, such as S4, require a non-linearity inbetween the layers to perform well. Furthermore, as stated in Thm. 4.3 and Prop. 4.5, even if the recurrence is non-linear in the input, such as Mamba, the expressivity of models with diagonal matrices is improved by stacking. Additionally, the inclusion of the non-linearity inbetween the Mamba layers does not improve performance, as the recurrences themselves are expressive enough. Finally, as stated in Thm. 4.2, dense matrices can achieve strong expressivity with random initialization, no stacking, and only a trainable linear readout.

Fig. 2 is a plot of the results on the A 5 benchmark. The figure shows that the number of blocks S4, Mamba, and the transformer require to achieve greater than 90% validation accuracy grows with the sequence length. In fact, given our model architecture, none of these models could achieve greater than 90% validation accuracy for sequences of length twenty 9 . On the other hand, the RNN and Linear CDE are able to achieve greater than 90% validation accuracy for all lengths considered using only one block. These empirical results further validate Thm. 4.3 and Prop. 4.5: there exists a gap in expressivity between diagonal and dense transition matrices, and stacking is required to recover the expressivity. Furthermore, they provide empirical evidence that even for simple state-tracking problems, the number of blocks required can grow quickly with sequence length.

7 Conclusions

This paper explores Linear CDEs, a model family that extends both classical and modern SSM architectures, including recent gated RNNs. Using Rough Paths theory, we have characterized their uniform closure, generalizing the results of Li et al. [2022b] for linear RNNs. We precisely identified the advantages of input-controlled transition dynamics, which allow you to capture highorder statistics of the input as opposed to just the linear ones extracted by convolutions. While dense models reach full expressiveness, imposing diagonality (e.g. Mamba) weakens the model capabilities. This is in direct contrast to S4, where dense and diagonal settings share the same closure. Our analysis lays the theoretical foundation for analyzing the expressive power of future SSM variants and hints at non-diagonality as a potential source of improvements. We believe that light and efficient channel mixing in the recurrent block might already unlock the modeling of higher-order statistics.

Limitations. The current framework examines expressivity from a continuous-time perspective with real-valued inputs. The RNN literature suggests that finite precision often plays a significant role in practice, making it an interesting direction for future exploration. Additionally, although dense transition matrices are shown to be theoretically and empirically more expressive than diagonal ones, their increased computational cost makes them impractical for large-scale models.

0.0

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

10

4

10

3

10

2

10

1

Validation RMSE

Area (2D)

0.0

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

10

4

10

3

10

2

10

1

Volume (3D)

Mamba Depth 1

Mamba Depth 2

Mamba Depth 2 Nonlinear

Predicting Zero

Random Linear CDE

3

4

5

6

7

8

9

10 11 12

14

16

18

20

Sequence Length

1

2

3

4

Minimum # Blocks Required

Mamba

Transformer

RNN

Linear CDE

Acknowledgments

Antonio Orvieto acknowledges the financial support of the Hector Foundation. Terry Lyons was funded in part by the EPSRC [EP/S026347/1], in part by The Alan Turing Institute [EP/N510129/1], in part by the Defence and Security Programme, in part by the Office for National Statistics, and in part by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA). Benjamin Walker was funded by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA). The authors would like to acknowledge the use of the University of Oxford Advanced Research Computing (ARC) facility in carrying out this work. http://dx.doi.org/10.5281/zenodo.22558

References

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 2017.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Neil Houlsby, Sylvain Gelly, Xiaohua Zhang, and Jakob Uszkoreit. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations , 2021.

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations , 2020.

Eric Nguyen, Karan Goel, Albert Gu, Gordon W. Downs, Preey Shah, Tri Dao, Stephen A. Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals using state spaces. Advances in Neural Information Processing Systems , 2022.

Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-space models. International Conference on Machine Learning , 2022.

Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning. Advances in Neural Information Processing Systems , 2023.

Nicolas Zucchet, Robert Meier, Simon Schug, Asier Mujika, and João Sacramento. Online learning of long-range dependencies, 2023.

Sepp Hochreiter and J"urgen Schmidhuber. Long short-term memory. Neural computation , 1997.

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014.

Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. arXiv preprint arXiv:1709.04057 , 2017.

Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling, 2023.

Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. What makes convolutional models great on long sequence modeling? arXiv preprint arXiv:2210.09298 , 2022a.

Junxiong Wang, Jing Nathan Yan, Albert Gu, and Alexander M Rush. Pretraining without attention. arXiv preprint arXiv:2212.10544 , 2022.

Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations , 2022.

Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. arXiv preprint arXiv:2312.04927 , 2023.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023.

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048 , 2023.

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models, 2023.

Tobias Katsch. Gateloop: Fully data-controlled linear recurrence for sequence modeling, 2023.

Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635 , 2023.

Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904 , 2024.

Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. Griffin: Mixing gated linear recurrences with local attention for efficient language models, 2024.

Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. arXiv preprint arXiv:2303.06349 , 2023a.

Leo Feng, Frederick Tung, Mohamed Osama Ahmed, Yoshua Bengio, and Hossein Hajimirsadegh. Were rnns all we needed?, 2024. URL https://arxiv.org/abs/2410.01201 .

Laurence C Young. An inequality of the hölder type, connected with stieltjes integration. 1936.

Terry Lyons. Differential equations driven by rough signals (i): An extension of an inequality of lc young. Mathematical Research Letters , 1(4):451-464, 1994.

Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems , 33:6696-6707, 2020.

James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning , pages 7829-7838. PMLR, 2021.

Adeline Fermanian, Pierre Marion, Jean-Philippe Vert, and Gérard Biau. Framing rnn as a kernel method: A neural ode approach. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 3121-3134. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/p aper_files/paper/2021/file/18a9042b3fc5b02fe3d57fea87d6992f-Paper.pdf .

Cristopher Salvi, Maud Lemercier, and Andris Gerasimovics. Neural stochastic pdes: Resolutioninvariant learning of continuous spatiotemporal dynamics. Advances in Neural Information Processing Systems , 35:1333-1344, 2022.

Melker Hoglund, Emilio Ferrucci, Camilo Hernandez, Aitor Muguruza Gonzalez, Cristopher Salvi, Leandro Sanchez-Betancourt, and Yufei Zhang. A neural rde approach for continuous-time non-markovian stochastic control problems. arXiv preprint arXiv:2306.14258 , 2023.

Benjamin Walker, Andrew D. McLeod, Tiexin Qin, Yichuan Cheng, Haoliang Li, and Terry Lyons. Log neural controlled differential equations: The lie brackets make a difference. International Conference on Machine Learning , 2024.

Terry J Lyons, Michael Caruana, and Thierry Lévy. Differential equations driven by rough paths . Springer, 2007.

Zhong Li, Jiequn Han, E Weinan, and Qianxiao Li. Approximation and optimization theory for linear continuous-time recurrent neural networks. J. Mach. Learn. Res. , 23:42-1, 2022b.

Antonio Orvieto, Soham De, Caglar Gulcehre, Razvan Pascanu, and Samuel L Smith. On the universality of linear recurrences followed by nonlinear projections. arXiv preprint arXiv:2307.11888 , 2023b.

Shida Wang and Beichen Xue. State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory. arXiv preprint arXiv:2309.13414 , 2023.

Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. arXiv preprint arXiv:2206.11893 , 2022.

Nicolas Zucchet and Antonio Orvieto. Recurrent neural networks: vanishing and exploding gradients are not the end of the story. arXiv preprint arXiv:2405.21064 , 2024.

Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems , 33: 1474-1487, 2020.

Hava T Siegelmann and Eduardo D Sontag. On the computational power of neural nets. In Proceedings of the fifth annual workshop on Computational learning theory , pages 440-449, 1992.

Samuel A Korsky and Robert C Berwick. On the computational power of rnns. arXiv preprint arXiv:1906.06349 , 2019.

Joshua Hanson and Maxim Raginsky. Universal simulation of stable dynamical systems by recurrent neural nets. In Learning for Dynamics and Control , pages 384-392. PMLR, 2020.

Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory , 39(3):930-945, 1993.

Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time?, 2018.

Joshua Hanson and Maxim Raginsky. Universal approximation of input-output maps by temporal convolutional nets. Advances in Neural Information Processing Systems , 32, 2019.

Stephen Boyd and Leon Chua. Fading memory and the problem of approximating nonlinear operators with volterra series. IEEE Transactions on circuits and systems , 32(11):1150-1161, 1985.

Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. arXiv preprint arXiv:2402.01032 , 2024.

William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. arXiv preprint arXiv:2404.08819 , 2024.

Ben Hambly and Terry Lyons. Uniqueness for the signature of a path of bounded variation and the reduced path group. Annals of Mathematics , pages 109-167, 2010.

Yann A. LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient BackProp , pages 9-48. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642-35289-8. doi: 10.1007/978-3-642-35289-8_3. URL https://doi.org/10.1007/978-3-642-35289-8_3 .

Mantas Lukoveviius and Herbert Jaeger. Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. , 3:127-149, 2009. URL https://api.semanticscholar.org/Co rpusID:554006 .

Christa Cuchiero, Lukas Gonon, Lyudmila Grigoryeva, Juan-Pablo Ortega, and Josef Teichmann. Discrete-time signatures and randomness in reservoir computing. IEEE Transactions on Neural Networks and Learning Systems , Forthcoming, 04 2021a. doi: 10.1109/TNNLS.2021.3076777.

Enea Monzio Compagnoni, Anna Scampicchio, Luca Biggio, Antonio Orvieto, Thomas Hofmann, and Josef Teichmann. On the effectiveness of randomized signatures as reservoir for learning rough dynamics. In 2023 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2023.

Patrick Kidger, Patric Bonnier, Imanol Perez Arribas, Cristopher Salvi, and Terry Lyons. Deep signature transforms. Advances in Neural Information Processing Systems , 32, 2019.

Adeline Fermanian, Terry Lyons, James Morrill, and Cristopher Salvi. New directions in the applications of rough path theory. IEEE BITS the Information Theory Magazine , 2023.

Thomas Cass and Cristopher Salvi. Lecture notes on rough paths and applications to machine learning. arXiv preprint arXiv:2404.06583 , 2024.

Peter K. Friz and Nicolas B. Victoir. Multidimensional Stochastic Processes as Rough Paths: Theory and Applications . Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2010. doi: 10.1017/CBO9780511845079.

Adeline Fermanian. Embedding and learning with signatures, 2020.

Kuo-Tsai Chen. Integration of paths-a faithful representation of paths by noncommutative formal power series. Transactions of the American Mathematical Society , 89(2):395-407, 1958. ISSN 00029947. URL http://www.jstor.org/stable/1993193 .

Cristopher Salvi, Thomas Cass, James Foster, Terry Lyons, and Weixin Yang. The signature kernel is the solution of a goursat pde. SIAM Journal on Mathematics of Data Science , 3(3):873-899, 2021a. doi: 10.1137/20M1366794. URL https://doi.org/10.1137/20M1366794 .

Berlinet and C. Thomas-Agnan. Reproducing Kernel Hilbert Spaces in Probability and Statistics . Springer US, 2011. ISBN 9781441990969. URL https://books.google.co.uk/books?id= bX3TBwAAQBAJ .

Maud Lemercier, Cristopher Salvi, Theodoros Damoulas, Edwin V. Bonilla, and Terry Lyons. Distribution regression for sequential data, 2021.

Cristopher Salvi, Maud Lemercier, Chong Liu, Blanka Horvath, Theodoros Damoulas, and Terry Lyons. Higher order kernel mean embeddings to capture filtrations of stochastic processes. Advances in Neural Information Processing Systems , 34:16635-16647, 2021b.

Thomas Cochrane, Peter Foster, Varun Chhabra, Maud Lemercier, Terry Lyons, and Cristopher Salvi. Sk-tree: a systematic malware detection algorithm on streaming trees via the signature kernel. In 2021 IEEE international conference on cyber security and resilience (CSR) , pages 35-40. IEEE, 2021.

Cristopher Salvi, Maud Lemercier, Thomas Cass, Edwin V Bonilla, Theodoros Damoulas, and Terry J Lyons. Siggpde: Scaling sparse gaussian processes on sequential data. In International Conference on Machine Learning , pages 6233-6242. PMLR, 2021c.

Nicola Muca Cirone, Maud Lemercier, and Cristopher Salvi. Neural signature kernels as infinitewidth-depth-limits of controlled resnets, 2023.

Zacharia Issa, Blanka Horvath, Maud Lemercier, and Cristopher Salvi. Non-adversarial training of neural sdes with signature kernel scores. Advances in Neural Information Processing Systems , 2023.

Alexandre Pannier and Cristopher Salvi. A path-dependent pde solver based on signature kernels. arXiv preprint arXiv:2403.11738 , 2024.

Georg Manten, Cecilia Casolo, Emilio Ferrucci, Søren Wengel Mogensen, Cristopher Salvi, and Niki Kilbertus. Signature kernel conditional independence tests in causal discovery for stochastic processes. arXiv preprint arXiv:2402.18477 , 2024.

Patrick Kidger. On neural differential equations, 2022.

Guillaume Dubach and Yuval Peled. On words of non-Hermitian random matrices. The Annals of Probability , 49(4):1886 - 1916, 2021. doi: 10.1214/20-AOP1496. URL https://doi.org/10 .1214/20-AOP1496 .

Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of johnson and lindenstrauss. Random Structures & Algorithms , 22, 2003. URL https://api.semanticscholar.org/Corp usID:10327785 .

Christa Cuchiero, Lukas Gonon, Lyudmila Grigoryeva, Juan-Pablo Ortega, and Josef Teichmann. Expressive power of randomized signature. In The Symbiosis of Deep Learning and Differential Equations , 2021b.

A Introduction to Signatures

This initial section of the Appendix is devoted to a brief introduction to the topic of Signature Transform. For a more in-depth account we refer the interested reader to Cass and Salvi [2024].

A.1 Intuition - Controlled Differential Equations

In the simplest setting of smooth paths a CDE is a differential equation of form

where X : [0 , 1] → R d is a known smooth path to which we refer as control , Z 0 the known initial condition and Z : [0 , 1] → R n the unknown solution.

The natural generalization is the following: assume to have two spaces R d x and R d z , X ∈ C 1 ([0 , 1]; R d x ) , Z ∈ C 1 ([0 , 1]; R d z ) , F : R d z → L ( R d x , R d z ) and Z 0 ∈ R d z . We say that ( Z, X, F, Z 0 ) satisfy the CDE

whenever

The theory of Rough Paths has its origins in the study of such types of differential equations and provides a theoretical framework to define and work in rough settings i.e. when X is not kust BV but even α -Hölder for α ∈ (0 , 1) cf. Friz and Victoir [2010].

Assume thus to have the CDE

we get

Keeping with this procedure for N steps we get

where I = ( i 1 , . . . , i k ) runs through the multi-indices, V I f := V i 1 V i 2 . . . V i k f and

for sufficiently regular vector fields V i and X ∈ C 1 ([0 , 1]; R d x ) . Given a smooth f : R n → R , by the change of variable formula ( i.e. fundamental theorem of calculus) we have

where V i f ( z ) := df y [ V i ( z )] . Iterating this procedure on the V i f s, i.e. substituting in the previous equation the analogously obtained equality

As one can imagine, under reasonable regularity assumptions, the remainder goes to 0 as N →∞ and at the limit

This is a remarkable result: to know the solution Z t to the original CDE it suffices to know the quantities V I f for all multi-indices and f in the coordinate maps, together with the iterated integrals

This observation is at the core of Rough Path Analysis , the theory can in a sense be considered an extreme development of it. The collection of iterated integrals, the Signature , will be the main for our analysis.

In Appendix E we expand and make rigorous the arguments of this section in the case of affine vector fields.

A.2 Basic Definitions

Denote by ( R d ) ⊗ n := R d ⊗ · · · ⊗ R d the tensor product of n copies R d , set ( R d ) ⊗ 0 := R . Let T ( R d ) := ⊕ ∞ k =0 ( R d ) ⊗ k be the tensor algebra equipped with sum and tensor product.

Definition A.1. Let { e 1 , . . . , e d } be the canonical basis of R d , then

is a basis of ( R d ) ⊗ k . We equip ( R d ) ⊗ k with the inner product ⟨· , ·⟩ ( R d ) ⊗ k defined on basis elements as

We extend this product to T ( R d ) by

where A = ( a 0 , a 1 , . . . ) and B = ( b 0 , b 1 , . . . )

Note how for any A,B ∈ T ( R d ) we have ⟨ A ⊗ e i , B ⊗ e j ⟩ T ( R d ) = ⟨ A,B ⟩ T ( R d ) ⟨ e i , e j ⟩ R d , we refer to this as to the coproduct property .

Definition A.2 (Infinite Tensor Algebra) . The infinite tensor algebra is defined as the space T (( R d )) := ∏ ∞ k =0 ( R d ) ⊗ k equipped with the operations + and ⊗ which act in the natural algebraic way; its elements are called tensor series .

It is easily seen that ( T (( R d )) , + , ⊗ ) is an algebra with unit 1 = (1 , 0 , 0 , · · · ) and we can endow it with a natural product which inherits the coproduct property.

Another point of view could be taken on the definitions of these spaces, one that we will prefer later on. If we define W d to be the set of words in d letters then T (( R d )) ∼ R W d , T ( R d ) is the subset of such functions with finite support and

where | I | is the length of the word I . The empty word, the only one with length 0 , is denoted by () and corresponds to the basis element of ( R d ) ⊗ 0 . In this view the tensor product coincides with concatenation of words accordingly distributed and the closure of T ( R d ) with respect to its product is just the l 2 space l 2 ( W d ) .

Definition A.3 (Signature) . Given γ ∈ C 1 , 0 ([0 , 1]; R d ) and s, t ∈ [0 , 1] s.t. s ≤ t , the signature Sig ( γ ) s,t ∈ T (( R d )) of the path γ over [ s, t ] is defined as

Equivalently Sig ( γ ) s,t is that element of l 2 ( W d ) defined recursively on words as

A.3 Notable Results

Here we present some notable results of which we will make use through the paper. We omit the proofs if they can be easily found in the suggested references.

The first result is about bounding the norm of Signature entries:

Proposition A.4 (Factorial Decay Rate) . Given γ ∈ C 1 , 0 ([0 , 1]; R d ) , for all k ≥ 1 and s, t ∈ [0 , 1] s.t. s ≤ t one has

The most important fact about Signature is that it acts as the basis for a Taylor expansion in path space. In fact just as finite linear combinations of monomials are dense in the continuous functions with a compact input set, finite linear combinations of Signature entries are dense in continuous functions from compact path-spaces:

Theorem A.5 (Universal Approximation Fermanian [2020]) . Fix K ⊂ C 1 , 0 ([0 , 1]; R d +1 ) compact such that for any γ ∈ K it holds γ 1 t = t . For any F ∈ C 0 ( K ; R ) and ϵ > 0 there is an integer N ≥ 0 such that

for some finite sequence ( α I ) | I |≤ N of real numbers.

Remark A.6 . There is no magic in this result, it is just an application of Stone-Weiestrass enabled by the rich algebraic structure of iterated integrals, studied originally in Chen [1958].

We will need to restrict some paths to sub-intervals of [0 , 1] in such a way to still be able to consider them meaningfully as elements of C 1 , 0 ([0 , 1]; R d ) , this is done in the following way:

Definition A.7. Given any path γ ∈ C 1 , 0 ([0 , 1]; R d ) we define its restriction on a sub-interval [ s, t ] ⊆ [0 , 1] as the path γ [ s,t ] ∈ C 1 , 0 ([0 , 1]; R d ) with values

This definition is such that the following important equation holds

With the right augmentation of the paths one can see that the Signature distinguishes between different sections of paths, this will be crucial for some of the original results presented in this work.

Lemma A.8. Assume ω, γ ∈ C 1 , 0 ([0 , 1]; R d +2 ) with ω 1 = γ 1 ≡ t and ω 2 = γ 2 ≡ t 2

Proof. The if part is follows from Sig ( γ ) s,t = Sig ( γ [ s,t ] ) 0 , 1 . For the only if part, If s = s ′ and t = t ′ the statement holds; this is because if the signatures over the time interval [ s, t ] of two time-augmented paths are equal, then the two paths must be equal on [ s, t ] . We now show that augmenting the path with t 2 and imposing equality of signatures, implies s = s ′ and t = t ′ , which will in turn allow us to conclude the proof by the previous remark. Assume Sig ( ω ) s,t = Sig ( γ ) s ′ ,t ′ , in particular we must have

which reduces to the system

Hence it must be true that t = t ′ and s = s ′ .

B Expressivity

B.1 Model Recap

In the body of the paper we have presented the main results with the simplified assumption of Z X 0 = 0 or at best Z X 0 = Z 0 i.e. with an initial value independent from the input. In this appendix we will carry on the proofs in a more general setting in which Z X 0 is allowed to be input-dependent, as previously discussed the choice of initial value is, in contrast to the classical setting, meaningful inasmuch it allows to approximate linear maps on the signature of ω X [0 , 1] . In order to do so we have to introduce a new gate, the initial value gate, in the form of a map

Despite the notation, there is no reason why ( X ) 0 should be the initial value of the path X , one should think of this map as the one summarizing the data which still matters for the task but which does not have a time-series nature.

To recapitulate, the general setting of our models is the following: a topological input space space X ,

where all the gates are continuous functions on X . The space X does not have to be a space of paths, a topological structure suffices, as long as the gates ( · ) 0 , ω, ξ are well defined and continuous.

Remark B.1 . Typical examples for the choice of gates are X space of paths and

Then the main object of study, "gated" Linear CDEs, are defined as:

Definition B.2. Fix gates ( · ) 0 , ω, ξ as above, N ∈ N , matrices { A i } i =1 ,...,d ω ( A i ∈ R N × N ), B ∈ R N × d ξ , C ∈ R N × d 0 . The corresponding Linear CDE is the functional

B.2 Main Result - Statement and Strategy

Here we present the unified expressivity result in its most general form:

Theorem B.3. For any compact set K ⊆ X and continuous gates ( · ) 0 , ω, ξ with ω X , 1 t ≡ t and ω X , 2 t ≡ t 2 . For any ϵ > 0 and any

where Ψ ∈ C 0 ( C 1 , 0 ; R d 0 ) and Φ ∈ C 0 ( C 1 , 0 ; R d ξ ) , there exist a choice of hidden dimension N ≥ 1 and parameters v ∈ R N , A i ∈ R N × N , B ∈ R N × d ξ , C ∈ R N × d 0 such that

Moreover generic parameters suffice with high probability in the sense that under LeCun initialization

the following holds:

If the A i s are constrained to be diagonal, as often is the case in practice, the requirements ω X , 1 t ≡ t , ω X , 2 t ≡ t 2 can be dropped and the existence result only holds with

for ψ ∈ C 0 ( R d ω ; R d 0 ) and ϕ ∈ C 0 ( R d ω ; R d ξ ) .

Moreover in both the dense and diagonal cases the "reverse" also holds in the sense that, given any choice of matrices A i , B, C there is an ϵ -close map F in the corresponding family.

As one can see the theorem is composed of different sub-results, which we believe are better understood separately from each other. The proof will thus be split in the following steps:

Using the theory developed in Appendix E we see how linear functions on the Z t s can be seen as linear functions on certain terms of the Signature Transform.

Such terms define a feature map T ( X ) 0 ,t which generates a Reproducing Kernel Hilbert Space H ( · ) 0 ,ω,η t . This abstract space acts as an upper bound on expressivity: linear functions on Z t always belong to its closure (in uniform norm), independently of dimension and weights chosen, hence they cannot reach what functions in H ( · ) 0 ,ω,η t can't approximate.

The full expressive range of H ( · ) 0 ,ω,η t is shown to be captured by generic Z t s.

Diagonal systems are shown to be restricted to a subset of the H ( · ) 0 ,ω,η t of which they capture the full expressive range.

B.3 Main Result - Proofs

B.3.1 An expansion for Z X t

Proposition B.4. For any choice of A i ∈ R N × N , B ∈ R N × d ξ and C ∈ R N × d 0 , the unique solution to

is given, using the notation A Ij := A j A I , by

Notice here A I C i , A I B j ∈ R N and X i 0 Sig ( ω X ) I 0 ,t , ∫ t 0 Sig ( ω X ) I s,t dξ X ,j s ∈ R .

Proof. Just apply Theorems (E.2) and (E.6) of Appendix E.

Remark B.5 . A property highlighted by the previous result is the interpretability of these models. After training the CDEs one can compute the matrix multiplications and observe which entries of the signature the model chooses to take into consideration, to attend.

B.3.2 The feature map T and its RKHS

The expression in (30) is a linear map on a feature vector given, at time t , by

This feature vector can be understood as a tensor in the following way:

Definition B.6. Let W d 0 ,d ω ,d ξ be the set of words in the alphabet

Fixed the gates ( · ) 0 , ω, ξ we define T ( X ) : [0 , 1] → l 2 ( W d 0 ,d ω ,d ξ ) ⊆ T (( A d 0 ,d ω ,d ξ )) as the unique solution to:

In fact one readily sees that the only non-zero terms of T ( X ) 0 ,t defined as above are

This is similar to the tensor-valued CDE defining the signature as a tensor i.e. Salvi et al. [2021a]

with the addition of two terms to track X 0 and ξ X . One could also understand T ( X ) s,t as a sub-tensor of

but in doing this one would have to explicitly ignore most of the terms of this vector; the CDE (32) does exactly this, but implicitly. In any case the subtensor view shows that T : X × [0 , 1] 2 → l 2 ( W d 0 ,d ω ,d ξ ) is well defined and continuous.

To the feature map T ( · ) 0 ,t with values in the Hilbert space l 2 ( W d 0 ,d ω ,d ξ ) is then associated a Reproducing Kernel Hilbert Space Berlinet and Thomas-Agnan [2011], where the Kernel is the one induced by the l 2 product, which we denote by

Classical RKHS theory tells us that we can characterize its elements as:

Proposition B.7. A map F ( · ) t : X → R is an element of H ( · ) 0 ,ω,η t if and only if it is of the form

for α i , β j ∈ l 2 ( W d ω ) . Moreover ∥ F ( · ) t ∥ 2 H ( · ) 0 ,ω,η t is equal to the minimal value of

taken over those γ, β for which the above equality holds.

Signature kernels Salvi et al. [2021a] are a class of universal kernels on sequential data which have received attention in recent years thanks to their efficiency in handling path-dependent problems Lemercier et al. [2021], Salvi et al. [2021b], Cochrane et al. [2021], Salvi et al. [2021c], Cirone et al. [2023], Issa et al. [2023], Pannier and Salvi [2024], Manten et al. [2024].

Just as signature kernels, the kernel associated to T ( X ) 0 ,t can be explicitly written as the solution of a two-parameter CDE:

Lemma B.8. Let K X , Y ( s, t ) := ⟨ T ( X ) 0 ,s , T ( Y ) 0 ,t ⟩ l 2 then

or, directly in terms of Signature, also

Proof. The first expression follows immediately from (32) the second one by summing the products of T ( X ) 0 ,s 's and T ( Y ) 0 ,t 's entries given above.

Definition B.9. Define the space H ( · ) 0 ,ω,η [0 , 1] ⊆ C 0 ( X × [0 , 1]; R ) as the space of functions of form

for α i , β j ∈ l 2 ( W d ω ) . Thus for all t ∈ [0 , 1] and F ∈ H ( · ) 0 ,ω,η [0 , 1] it holds F ( · , t ) ∈ H ( · ) 0 ,ω,η t .

B.3.3 Linear maps on Z X t are close to the RKHS

The following proposition will show how linear maps on Z X t cannot be more expressive than elements of the RKHS H ( · ) 0 ,ω,η [0 , 1] since their closure is in the closure of H ( · ) 0 ,ω,η [0 , 1] . In this precise sense these spaces act like upper bounds to expressiveness.

Proposition B.10. Assume X compact. Consider fixed the gates and A i ∈ R N × N , B ∈ R N × d ξ and C ∈ R N × d 0 . Consider a linear readout v ∈ R N . For any ϵ > 0 there exist choices of α i , β j ∈ l 2 ( W d ω ) such that

where F ∈ H ( · ) 0 ,ω,η [0 , 1] . In other words, linear maps on the Z X t are in the uniform closure of H ( · ) 0 ,ω,η [0 , 1] .

Proof. Using (30) we see that ⟨ v, Z X t ⟩ is a linear map on T ( X ) 0 ,t with coefficients

using Cauchy-Schwartz it's moreover easy to see the existence of a constant λ ≥ 0 such that for all I, i, j one has

Since | Sig ( ω ) I s,t | ≤ 1 | I | ! ∥ ω ∥ | I | 1 -var, [ s,t ] we have that, given an integer M ≥ 0 , the bound

where K ≥ 0 is a constant which must exist by compactness of X and continuity of the gates. Since K ∑ ∞ m = M ( λd ω K ) m m ! is just the tail of the taylor expansion of Ke λd ω K there must be an M such that sup t ∈ [0 , 1] R M ( t ) ≤ ϵ . But then the choice

suffices for the required bound.

B.3.4 Uniform closure of the RKHS

Now that we have established the theoretical interest of the H ( · ) 0 ,ω,η [0 , 1] we proceed to characterize which maps ( X,t ) → R can be uniformly approximated through them.

Proposition B.11. Fix a compact input set X and continuous gates ( · ) 0 , ω, ξ with ω X , 1 t ≡ t and ω X , 2 t ≡ t 2 . For any ϵ > 0 and any

where Ψ ∈ C 0 ( C 1 , 0 ; R d 0 ) and Φ ∈ C 0 ( C 1 , 0 ; R d ξ ) , there exist a G ∈ H ( · ) 0 ,ω,η [0 , 1] such that

Proof. Note first that the map

is a continuous map from a compact space, thus the image must be compact too. Moreover by Prop. A.8 the Signature separates the points in this image. Since any G as above has form

the proof follows from the uniform density on compact sets of linear functionals on the (truncated) Signature (Thm. A.5), by also uniformly bounding thanks to compactness and continuity the norms of X 0 and ξ X 1 .

Remark B.12 . The specific restriction of ω to subsets of [0 , 1] is a crucial part of the result. The family of approximable maps does not include all path-to-path causal 11 functions t ↦→ Y X t but a subset of them, of type t ↦→ Y X t := Ψ( ω X [0 ,t ] ) , satisfying the specific time-homogeneity specified by the form of the restriction, akin to that in Li et al. [2022b].

B.3.5 Generic Weights are fully expressive

We have seen how linear maps on Z X t are in the uniform closure of H ( · ) 0 ,ω,η [0 , 1] , and we have explicitly characterized this closure. It is then natural to ask "how much" of this closure the Z X t are able to "explore". The present section not only shows that the Z X t "explore" all the closure, but also that a generic choice of weights is enough to eventually do this with high probability.

The fact that these maps are "universal" in the above sense is not surprising, since it is well known that Linear CDEs are universal for path-to-point tasks cf. Kidger [2022], what is surprising is that this universality can be achieved probabilistically with one of the standard parametrizations used in ML practice (LeCun) 12 .

Theorem B.13. Fix X compact and ϵ > 0 . For all F ∈ H ( · ) 0 ,ω,η [0 , 1] there exist a choice of hidden dimension N ≥ 1 and parameters v ∈ R N , A i ∈ R N × N , B ∈ R N × d ξ , C ∈ R N × d 0 such that

Moreover generic weight choices suffice with high probability, in the sense that under LeCun initialization

the following holds

We propose two proofs, the first one of a deterministic character concerns the first claim in the theorem, the second one is probabilistic and concerns the whole result. The deterministic proof follows the same arguments employed by Kidger [2022] and is included to highlight the main idea of the probabilistic result, which reduces to a "spin" on the central argument of this proof.

Deterministic Proof. Any F ∈ H ( · ) 0 ,ω,η [0 , 1] has form

for fixed α i , β j ∈ l 2 ( W d ω ) . Consider an integer M ≥ 0 such that

where π M is the truncation at length M .

Fix d = d 0 + d ω + d ξ . Consider µ ( M,d ) ∈ N such that R µ ( M,d ) ≃ T M ( R d ) . We are going to write e I ∈ R µ ( M,d ) to mean the image of e I ∈ T M ( R d ) through this identification. Note that ( · ) ⊗ M e k : T M ( R d ) → T M ( R d ) is a linear map, it does then correspond to a matrix Λ k ∈ R µ ( M,d ) × µ ( M,d ) . Write ε ξ j := e d 0 + j for j = 1 , . . . , d ξ and ε ω k := e d 0 + d ξ + k for k = 1 , . . . , d ω . Then the solution to

is the object in R µ ( M,d ) corresponding to the truncated tensor π M ( T ( X ) 0 ,t ) .

This ˜ Z t is of the form Z X t with N = µ ( M,d ) , A k = Λ d 0 + d ξ + k , B = [ ε ξ 1 | · · · | ε ξ d ξ ] and C = [ e 1 | · · · | e d 0 ] .

In particular note how these matrices are such that

since it holds that

and for all | I | > M one has necessarily A I = 0 .

Our strategy is that of using these equaities to create a vector v ∈ R N corresponding to the π M α i and π M β j . Define the vector

Then expanding Z X t as in (30) and using the equalities above one has

proving that for such v and Z X it holds

The crucial ingredient for the success of this proof is the possibility to recreate the space T M ( R d 0 + d ω + d ξ ) as an euclidean space. To do this one needs µ ( M,d 0 + d ω + d ξ ) ∼ ( d 0 + d ω + d ξ ) M orthogonal vectors and a way to express them using the matrices A i , B and C , the essential equations which capture this are given by (45).

The core idea of the following probabilistic proof of this same result is that of allowing for some error in (45), so the idea is that of exhibiting only approximately orthogonal vectors. At the cost of losing exactness, one can leverage results of the Johnson-Lindenstrauss Dasgupta and Gupta [2003] type to find on the order of ∼ e ε 2 N vectors in R N orthogonal up to an ε error, using random projections. This idea in the context of Signature goes back to Cuchiero et al. [2021b], and allows for much smaller hidden dimensions.

Proof. (Probabilistic Proof) Any F ∈ H ( · ) 0 ,ω,η [0 , 1] has form

for fixed α i , β j ∈ l 2 ( W d ω ) . Consider an integer M ≥ 0 such that

where π M is the truncation at length M .

From Cirone et al. [2023][Appendix C] we know that

Our strategy is that of using these bounds to create a vector v ∈ R N "acting" like the π M α i and π M β j . Define, noting that the A i , C i , B j depend on N , the vector

Then expanding Z X t as in (30)

Note how for | I | ≤ M one has and that similarly for | I | > M

Which leads, thanks to the same bounds of Cirone et al. [2023][Appendix C], to

But then by Markov's inequality it holds that

and thus there must be a choice of N, { A i } , B, C such that the inequality holds, and we thus obtain using (50)

and we conclude by arbitrariness of ϵ .

B.4 The Diagonal Case

Here we study the particular, but empirically important, case where the matrices A i are taken to be diagonal 13 .

What we'll discover is that the Z X t cannot differentiate between Sig ( ω X ) I s,t and other Sig ( ω X ) σ ( I ) s,t for any permutation σ of the letters in the word I .

B.4.1 Diagonal Expansion for Z X t

Proposition B.14. For any choice of V ∈ R N × d ω , B ∈ R N × d ξ and C ∈ R N × d 0 , writing A i := diag( V i ) , the unique solution to

which can be expanded as

where

Proof. By Theorem E.1 and Theorem E.6 we know that the solution of

where W s,t is the unique solution to

In case the A i s are commuting matrices one can explicitly write the solution as

since for fixed s one has, using commutativity, that

On the other hand we know, Theorem E.2, that

The two views are reconciled by noticing that the symmetric group S k acts on W k d ω , the space of words of lenth k , by permuting the letters and, by commutativity,

is given by is explicitly given by

Then we have

recalling then how e I 1 · · · e I k = ∑ σ ∈ S k e σ ( I ) we get to

In particular we see how in the commuting case

where

B.4.2 Diagonal Expressiveness

Theorem B.15. Fix a compact input set X and continuous gates ( · ) 0 , ω, ξ . For any ϵ > 0 and any

for ψ ∈ C 0 ( R d ω ; R d 0 ) and ϕ ∈ C 0 ( R d ω ; R d ξ ) , there exist a choice of hidden dimension N ≥ 1 and parameters v ∈ R N , B ∈ R N × d ξ , C ∈ R N × d 0 and diagonal A i ∈ R N × N such that

Moreover the "reverse" also holds i.e. given any choice of matrices A i , B, C there is an ϵ -close map F in the family.

Proof. This is just a repetition of the arguments used for the dense case with little more care to get the uniformity in time.

One defines the subset Sym ( H ( · ) 0 ,ω,η [0 , 1] ) ⊂ H ( · ) 0 ,ω,η [0 , 1] of those F of type (37) defined by α i , β j ∈ l 2 ( W d ω ) such that for any word I and any permutation σ ( I ) of it

The same argument of Proposition B.10 shows that the uniform closure of the space of linear maps on the Z X t is contained in the uniform closure of Sym ( H ( · ) 0 ,ω,η [0 , 1] ) , and the same bounds show that this latter closure is the same as that of its subset composed of those F ∈ Sym ( H ( · ) 0 ,ω,η [0 , 1] ) having entries eventually equal to 0 .

Since

such maps can be expressed exactly in the form

for polynomial maps P, Q fixed in time. The usual compactness and continuity argument, together with an application of Stone-Weiestrass, thus proves that the uniform closure of Sym ( H ( · ) 0 ,ω,η [0 , 1] ) has the form needed.

The final ingredient is the density of the space of linear maps on the Z X t in Sym ( H ( · ) 0 ,ω,η [0 , 1] ) ; this is another consequence of Stone-Weiestrass as seen from Proposition B.18.

Remark B.16 . Notice how here there is no need to augment the paths in creative ways in order to ensure separability of the points. The map ( ω, s, t ) ↦→ ω [ s,t ] ∈ C 1 , 0 ([0 , 1]; R d ω ) is replaced by ( ω, s, t ) ↦→ ω t -ω s ∈ R d ω and the space of polynomials always separates points in R d ω .

Remark B.17 . It is not necessary to pass through Sym ( H ( · ) 0 ,ω,η [0 , 1] ) to prove the previous result, since it directly follows from Proposition B.18. This choice of presentation has been motivated by the conviction of the usefulness of drawing parallels and comparisons.

Proposition B.18. Fix a compact set K ⊂ R d and a d -dimensional convex cone C containing the origin. The space

is uniformly dense in C 0 ( K ; R ) .

Proof. This is an application of Stone-Weiestrass: E is a sub-algebra since

and α, β ∈ C = ⇒ α + β ∈ C by convexity of the cone; E contains the constant function e ⟨ 0 ,x ⟩ = 1 and is clearly point separating since the cone, being d -dimensional, it contains a basis of the whole space.

Remark B.19 . The usefulness of stating the previous result in such a general setting is the following: with this formalism we can, for example, restrict to α ≤ 0 , in this way we would have a method to control the stability ( cf. Appendix C.1) of the Linear CDEs by choosing the gate with a.s. ˙ ω X ≥ 0 .

Corollary B.20 (Mamba Case) . In the Mamba setting, the closure reduces to

for continuous ψ i : R → R and ϕ i : R → R .

Proof. In this setting one runs in parallel d ω diagonal systems and then takes a linear combination of the stacked hidden state. The maps in the closure of the whole system are then just the sums of maps in the closure of the subsystems.

C Stability and Chaining of Diagonal Systems

For this section consider, unless otherwise stated, a fixed N ≥ 0 , compact X and gates ( · ) 0 , ω, ξ .

We will study the stability and chaining of diagonal systems defined by the choice of a matrix V ∈ R N × d ω such that A i := diag ( V i ) , where V = [ V 1 | · · · | V d ω ] .

Note that the present discussion holds even for non-diagonal but commuting matrices, since these can be simultaneously diagonalized (at the cost of considering the complex plane).

C.1 Stability

Here we explore the stability of the dynamical system Z X , thus we need to study the eigenvalues of the W X s,t . Recall how in this setting

Note that because ω X is continuous and of bounded variation, it can be reparameterised to be Lipschitz continuous, hence absolutely continuous. Thus we can assume that ω X is almost everywhere differentiable and its derivative ˙ ω ∈ L 1 .

The stability of the dynamical system then depends on the alignment between ω X t -ω X s and the singular vectors of V . If V ˙ ω X t ≤ 0 for all times, where the inequality is coordinate-wise, then W X s,t has eigenvalues all in [0 , 1] thus the system is stable making training easier Orvieto et al. [2023a].

Consider the singular value decomposition (SVD) of the matrix V

Then, a sufficient condition for stability is that for any k = 1 , ..., K

C.1.1 The case of Mamba

In the case of Mamba Gu and Dao [2023] the matrices are diagonal and

moreover the proposed choices of V are all of type

for some choice of 0 ≤ v ∈ R N . Note that softmax is just a smooth approximation of ReLU and that Im ( ReLU ) ⊆ { w ∈ R d ω : ⟨ 1 d ω , w ⟩ ≥ 0 } hence mamba is implicitly ensuring that the dynamical system is approximately always well-conditioned.

C.2 Chaining

The diagonal case differs from the general one not only in the fact that the class of approximable functions is much weaker but also in the necessity for the presence of ξ X in order to obtain any path-dependence. The term

becomes then a crucial component. At first sight one might think that such a term allows to recover at least level two components of the Signature of ( ω X , ξ X ) , unfortunately things are not as easy as they may seem. Notice how inside of the integral time is "going backwards" from the perspective of ω X , thus we can in general approximate terms of type

which are indeed terms of the Signature, but of the reverse paths ← -ω X r = ω X 1 -r and ← -ξ s = ξ X 1 -s !

Proposition C.1. Fix a compact input set X , continuous gates ( · ) 0 , ω, ξ and X 1 0 = 1 . If the components of ξ X are linear combinations of those of ω X , with time-independent weights, then linear functionals on Z X t can, uniformly in X × [0 , 1] , approximate arbitrarily well the following level 2 terms of Sig (( ω X , ξ X )) 0 ,t :

Proof. Under these hypotheses we know that linear functionals on Z X t are uniformly dense, for continuous ψ, ϕ , in

Assume ξ X ,j s = ⟨ α j , ω X t ⟩ and consider the choices

so that

To conclude note that

hence

If X ∈ C 1 , 0 ([0 , 1]; R d ) we can use the previous result to compute its Signature entries by chaining diagonal Linear CDEs.

Theorem C.2. Assume a compact input set X ⊂ C 1 , 0 ([0 , 1]; R d ) . For any I ∈ W d with | I | ≥ 2 and ϵ > 0 there is a sequence of linear maps W k ∈ R N k × 1 and weights for the following family of chained Linear CDEs

such that for some v ∈ R N | I |-1 one has

Proof. For | I | = 2 we can apply Prop. C.1. Assume the theorem holds for | I | ≤ k and let M := sup X ∈ X ∥ X ∥ 1 -var . Fix | Ij | = k +1 and W k -1 ∈ R N k -1 × 1 such that

Again by Prop. C.1 there are a N k and v ∈ R N k such that

Then

thus concluding the proof.

Then Proposition 4.5 follows as a corollary by running in parallel the systems above to recover simultaneously multiple Signature entries.

C.2.1 ReLU activation choice

Models like Mamba do not only use diagonal matrices but also consider controls of a specific kind:

The choice of ReLU enforces ˙ ω t ≥ 0 for all times as seen above, but could, a priori, destroy information about X which allows for the recovery, after chaining, of its Signature.

Does this choice keep some expressivity? Fortunately almost all of it: since

one can choose a linear map W which allows to linearly recover

from ω X t . By correspondingly modifying the form of ψ and ϕ in (74) such that

one is able, through a similar chaining procedure, to recover arbitrarily deep entries of the Signature of ˜ ω X t = ∫ t 0 X s ds .

D Path-to-Path

Definition D.1. A map G ∈ C 0 ( C 1 , 0 ([0 , 1]; R d ) × [0 , 1]; R ) is causal iff for all t ∈ [0 , 1] and paths ω, ˜ ω ∈ C 1 , 0 ([0 , 1]; R d ) one has

i.e. G is causal if it does not look in the future.

Proposition D.2. Assume a compact input set X , continuous ( · ) 0 , ω, ξ , X 1 0 ≡ 1 and ω X , 1 t ≡ t . Then for all ϵ > 0 and all causal G ∈ C 0 ( C 1 , 0 ([0 , 1]; R d ω ) × [0 , 1]; R ) there exist an integer N ≥ 0 , some Feed Forward neural network F : R N → R , and parameters C ∈ R N × d 0 , A i ∈ R N × N , B ∈ R N × d ξ such that

Proof. Fix ϵ > 0 . By B.7 the space H ( · ) 0 ,ω,η [0 , 1] contains all functionals of form

thus, by the properties of the signature and by compactness of X , for any fixed s 0 ∈ [0 , 1] there is some f ∈ H ( · ) 0 ,ω,η [0 , 1] such that

Using the fact that G ∈ C 0 ([0 , 1]; C 0 ( C 1 , 0 ([0 , 1]; R d ω ); R )) and compactness of [0 , 1] , we find a finite set { 0 ≤ s 0 ≤ · · · ≤ s M ≤ 1 } of points and f 0 , . . . , f M ∈ H ( · ) 0 ,ω,η [0 , 1] such that

for i = 0 , . . . , M -1 . Notice then how for all X ∈ X and s ∈ [ s i -1 , s i +1 ]

It follows that the map F ∈ C 0 ([0 , 1] × X ; R ) linearly interpolating the f i in time satisfies

To conclude note that X being compact, the f i take values in a common compact set K ⊆ R . There exist then a neural network Ψ : [0 , 1] × K M → R such that

which means that

Recalling that ω X , 1 t = t we get that X ↦→{ t ↦→ t } ∈ H ( · ) 0 ,ω,η [0 , 1] so that, given density of linear maps on Z X in the space, Ψ( t, f 0 ( X,t ) , . . . , f M ( X,t )) can be uniformly approximated. Triangular inequality gives finally

which, by arbitrariness of ϵ , gives the thesis.

The non-linearity is crucial for the path-to-path result. A map of type ( ω, t ) ↦→⟨ tα, Sig ( ω ) 0 ,t ⟩ cannot be approximated arbitrarily well by ( ω, t ) ↦→⟨ β, Sig ( ω ) 0 ,t ⟩ .

In any case, note that in the proof the role of the neural network is only that of interpolating the RKHS elements in the right order and at the right time. All the non-linear complexity of learning the particular G is offloaded and taken care of by the RKHS elements.

Remark D.3 . In the proof we have only considered the part of T ( X ) concerning ω X , but T ( X ) t depends linearly on X 0 and ξ X suggesting that neural networks on H ( · ) 0 ,ω,η [0 , 1] have stronger generalization properties. In fact one can prove that it is possible to approximate all continuous G ( X 0 , ω X , ξ X , t ) , this is done by reconstructing X 0 and ξ X [0 , 1] as in the classical SSM case cf. Orvieto et al. [2023b].

E Wronskian Matrix Theory

In this section we obtain a unified theory studying the solutions to general Linear CDEs. The results presented here are not new and can be found in different terms in the literature Friz and Victoir [2010], despite this we have decided to reproduce them from scratch for completeness, notational reasons and to present a self-contained theory.

Theorem E.1. For any choice { A 1 , . . . , A d } ⊆ C 0 ([0 , 1]; R N × N ) and ω ∈ C 1 ([0 , 1]; R d ) there exist a unique map W ∈ C 0 ([0 , 1] × [0 , 1]; R N × N ) solving the following CDE

Proof. We will use Banach fixed point theorem leveraging the completeness of the space Ω := C 0 ([0 , 1] × [0 , 1]; R N × N ) with the uniform norm

Define the map Γ : Ω → Ω as

One has, for X,Y ∈ Ω and k ∈ N setting Γ 0 = Id Ω , that

which iterated gives

where W d is the set of words in the alphabet { 1 , . . . , d } and

By defining M = max { ∥ ∥ A i ∥ ∥ ∞ : i ∈ { 1 , . . . , d }} then one clearly has

thus definitely (in k ) the map Γ k is a contraction. By Banach fixed point there exist a unique fixed point W ∈ Ω .

Theorem E.2. Under the assumptions of the previous theorem one can write W s,t explicitly as

moreover if for all i the matrix-valued maps are constant on all [0 , 1] i.e. A i t ≡ A i then

where Sig ( ω ) I s,t is the Signature of the path ω .

Proof. The second assertion follows from the first by definition of the Signature of a path. Regarding the first notice how the series is absolutely convergent in R N × N , uniformly in s, t since

thus for any s, t ∈ [0 , 1] the series defines an element of ˜ W s,t ∈ R N × N . Using the uniformity of this bound and the fact that for all I ∈ W d one has

as a function of ( s, t ) , which moreover is uniformly continuous

one concludes that ˜ W s,t ∈ Ω . Finally notice that ˜ W s,t is a fixed point of Γ

and conclude by uniqueness.

Proposition E.3. Under the previous conditions, the unique solution of the N -dimensional CDE

Proof. The solutions are unique by standard results Friz and Victoir [2010][Thm. 3.7], moreover

Proposition E.4. The Wronskian matrix has the following properties:

is given by

Proof. Regarding the first statement notice that for all X 0 ∈ R N one has

and by the previous proposition also

thus X t and ˜ X t solve the same CDE and coincide at time t = s . This means, by uniqueness, that X t and ˜ X t coincide for all times; hence W s,t W r,s and W r,t coincide for all times too since for any choice of X 0 one has W s,t W r,s X 0 = W r,t X 0 .

The second statement follows from the previous one setting first r = t and subsequently exchanging s and t .

To prove the third equality note that

hence

Proposition E.5 (Liouville's Formula) . Under the assumptions of the previous theorems, if ω ∈ C 1 ([0 , 1]; R d ) then

Proof. This just follows from the classical case since we can write

We can now state the main result of the section:

Theorem E.6. Under the assumptions of the previous theorems, given continuous functions { B 1 , . . . , B t } ∈ ( R d ) [0 , 1] the unique solution of the N -dimensional CDE

where W s,t ∈ C 0 ([0 , 1] × [0 , 1]; R N × N ) is the Wronskian matrix defined by

is given explicitly by

Proof. Given the unique solution X t one has

hence

F ZOH and Exact Solutions

Consider a Linear CDE as the one of (25)

and recall how the solution can be explicitly written, for times s < t , as

Assume moreover that in the interval [ s, t ] both drivers have constant derivative i.e.

Then if A w := ∑ d ω i =1 A i w i we get that W r,t = e A w ( t -r ) thus

But the integral can be explicitly solved as

leaving us with

which, setting ∆ = t -s , can be rewritten as

i.e. exactly the ZOH scheme.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the Introduction we list our contributions and refer, point by point, to the relevant sections of our work.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed some limitations in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the proofs are found in the Appendices, the specific sections of interest are referred after the statements.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Code to reproduce experiments included in supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code to generate the data used and to run the experiments are included in the supplementary material alongside a README explaining how to run the experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/pu blic/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details can be found in the section titled empirical validation, with additional details for reproducing the results included in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details included in the empirical validation section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This is a theoretical work. We provide information sufficient for the reproduction of empirical results.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This is a theoretical work, not involving crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This is a theoretical work, not involving crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work establishes density results of input-controlled differential equations, which formulate different types of state-space models such as S4 and Mamba in the continuous-time idealization. Under this framework, different closures of these models are derived, indicating their distinct inductive biases in the (universal) approximation sense, particularly implying the advantage of data-dependent modeling.\n\nSTRENGTHS:\n1. This work provides fundamental theoretical justifications on the expressivity of different types of SSMs. \n2. The derived inductive biases clearly distinguish the recent new data-dependent modeling (e.g. Mamba) from the classic original data-independent modeling (e.g. S4) thorough a input-selective framework, which shows rigorously the superiority of the former architecture. \n3. Other theoretical results also lead to useful insights. For instance, diagonal recurrence weakens the approximation capability, but this can be alleviated via stacking. \n4. All these insights are basically verified by numerical experiments.\n\nWEAKNESSES:\n1. In theory, only the density type of results is provided. That is, although models can rates certain targets universally when models' parameters go to infinity, the convergence *rates* are not characterized. This is much more important since the convergence can  be rather slow in certain situations (e.g. curse of dimension), leading to possibly vacuous bounds. The theoretical part would be strengthened and more convincing if authors can discuss more on the approximation rates of SSMs, especially the improvements (in parameter efficiency) of data-dependent modeling. \n2. In experiments, the simulations are conducted on low-dimensional (2 & 3) and synthetic datasets. Is it due to the ability of path signature? Can the path signature be powerful and efficient when handling high-dimensional (and real-world) input data?\n\nQUESTIONS:\n1. Please provide more details of questions raised in the weaknesses section above. \n2. In formulation, the mentioned \"gating\" mechanism in this work seems to have a different meaning as usual. Here, gating refers to the transformation of inputs (i.e. data pre-processing). However, in practice, gating often means the (multiplicative) interaction between the hidden states and readout layers (i.e., to get outputs). Can authors explain more about this?\n3. Minor issues: What is the definition of (linear) NCDE (see line 311, 312, 315, 317 and 643).\n\nLIMITATIONS:\nAs is stated by authors, this work applies a continuous-time idealization in the formulation. The role of concrete discretization schemes is also worthy of explorations, particularly under the case of data-dependent discretization.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

comments: SUMMARY:
This paper proposes a framework for better understanding key features that allow the success of SSMs. To be specific, the authors first show that recent SSM-based models are linear controlled differential equations (CDEs). Then, the expressive power of linear CDEs are explored, depending on whether the matrices $A_i$ are diagonal.

STRENGTHS:
* This paper focuses on timely and important research problem. 
* I could not fully read the proof, but the results seem correct.

WEAKNESSES:
* It would be better to provide preliminaries on rough path theory for readers who are not familiar with that.
* Although the mathematical results are interesting, I am not sure about the implications of the theoretical results in practice. It would be better if the empirical results are designed to provide the messages that are useful in practice.

QUESTIONS:
* How can we derive Eq.3? Why suddenly exponential term appears? Any reference for the zero-order hold discretization? (Actually, I searched for such terminology, but could not make a connection between it and Eq.3)

LIMITATIONS:
* Check the weakness part & questions part
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

comments: SUMMARY:
This paper proposes a framework of using Rough Path Theory to understand the expressivity of SSMs and Mamba. The paper establishes connections to linear CDEs and then uses tools from Rough Path Theory to explain why gates are so powerful in SSM models.

STRENGTHS:
This is a really nice theory explaining SSMs. It gives a new way of looking at the expressivity/quality results that have been observed in prior work. Hopefully this theory can lead to new insights about how to design better SSMs in the future.

Really solid work.

WEAKNESSES:
Not much weaknesses in the work itself. The ultimate test of theory is its predictive power - the paper would be stronger if the theory could "close the loop" and propose a modification to SSM layers that would enable further performance. However, this is a high bar to clear, and I think the work stands on its own as a solid contribution even without a methodological contribution.

QUESTIONS:
Can you think of any improvements to SSM layers/models that the RPT theory would suggest?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work analyses the modeling capability of different SSMs (S4-S6 and others) using Rough Path Theory by viewing SSMs as (input-) controlled differential equations (CDE). To this end, the authors show that SSMs with with dense transition matrices (A) are able to approximate arbitrarily close any continuous functions of the inputs. In contrast, SSMs with diagonal transition matrices lack this property. However, when stacking/chaining multiple diagonal CDEs (as is the case for the deep architectures), full expressivity is gained again as the depth approaches infinity. The work is mostly theoretical, and, hence, there is only a single toy experiment showing RMSE for Mamba and S5 models with depth 1 and 2, showing that the more limited models are not able to solve the toy task, as they cannot approximate the corresponding function.\n\nSTRENGTHS:\n- The paper is generally well written and the authors managed to make the very involved theory and proofs more accessible by walking through the interpretation and takeaways on a more high level. \n- The theoretical results are important contributions for categorizing the expressivity of existing SSM models and may potentially inform about future architecture design of SSMs and related models.\n\nWEAKNESSES:\nDespite the well-described theoretical contributions and insights, it is still not very clear to me what to take away practically.\nIn particular, it is not clear how useful the result regarding chaining diagonal CDEs is in practice: while it is important to have a proof that an infinite number of chains of diagonal CDE with mixing can recover the full expressiveness of the dense CDEs, the infinite case is not practically important. Instead, it would be important for practical considerations (computational efficiency vs model expressiveness) to know whether one should use dense CDEs or diagonal CDEs with mixing to achieve a certain level of expressivity.  However, as far as I understand, there is no result that quantifies e.g. how many diagonal plus mixing layers have a similar expressiveness as a certain number of dense CDEs. The authors speculate in the summary that architectures with non-diagonality might improve performance, but I do not see how this actually follows from this work, since there is no comparison between diagonal+mixing and dense in the regime of finite number of parameters. \n\nThe experimental section is very weak. While this work is mostly theoretical, this work would nevertheless be improved a lot by a greater experimental section that validates the claims. Showing error curves with a single random seed is not that interesting. I would be better to have a table of final performance that includes error bars. Furthermore, it would be great to explore a few different tasks and several ablations such as number of layers. This would make it at least empirically possible to compare dense and diagonal + mixing and random dense + learned C (or MLP) in the finite parameter setting, where its not clear what to take away from the theorems.\n\nQUESTIONS:\n- what is NCDE? This was not introduced.\n- does random linear CDE perform best for Area computation and is also a strong baseline for Volume?  I understand that with high probability we can just take a random initialized A matrix and train only the C matrix (Theorem 4.2) to achieve full expressivity. However, that model should still not be better than a dense CDE where everything is trained such as in case of S5. Or did you choose S5 with diagonal transition?\n\nLIMITATIONS:\nJust a single limitation is mentioned in the conclusion. I am sure, the authors can think of many more limitations, such as my main concern raised above regarding the comparison of different mixed diagonal + mixing vs dense architectures. Or the limited experimental evidence.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}