Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Models
Authors: Benjamin Walker
Lingyi Yang
Nicola Muca Cirone
Cristopher Salvi
Terry Lyons
Venue: NeurIPS 2025 spotlight ()
Keywords: Sequencing Modelling
Theoretical Deep Learning
Mathematical Foundations
Neural Controlled Differential Equations
State Tracking
Model Architecture
Continuous-Time Models
Primary area: deep_learning
TL;DR: Structured Linear Controlled Differential Equations: A unifying framework for sequence models with structured, input-dependent state-transition matrices

Abstract: This work introduces Structured Linear Controlled Differential Equations (SLiCEs), a unifying framework for sequence models with structured, input-dependent state-transition matrices that retain the maximal expressivity of dense matrices whilst being cheaper to compute. The framework encompasses existing architectures, such as input-dependent block-diagonal linear recurrent neural networks and DeltaNet's diagonal-plus-low-rank structure, as well as two novel variants based on sparsity and the Walsh-Hadamard transform. We prove that, unlike the diagonal state-transition matrices of S4D and Mamba, SLiCEs employing block-diagonal, sparse, or Walsh-Hadamard matrices match the maximal expressivity of dense matrices. Empirically, SLiCEs solve the $A_5$ state-tracking benchmark with a single layer, achieve best-in-class length generalisation on regular language tasks among parallel-in-time models, and match the performance of log neural controlled differential equations on six multivariate time-series classification datasets while cutting the average time per training step by a factor of twenty.

Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Models
Benjamin Walker', Lingyi Yang', Nicola Muça Cirone?, Cristopher Salvi?, Terry Lyons 1.2
Mathematicaa Institute, University of Oxford Department of Mathematics, Imperial College London
Abstract
This work introduces Structured Linear Controlled Differential Equations (SLiCEs), unifying framework for sequence models with structured, input-dependent state- transition matrices that retain the maximal expressivity of dense matrices whilst being cheaper to compute. The framework encompasses existing architectures, such input-dependent block-diagonal linear recurrent neural networks and DeltaNet's and the low-nank structure, as well as two novel variants based on sparsity transform. We prove that, unlike the diagonal state- transition matrices of S4D and Mamba, SLiCEs employing block-diagonal, sparse, Walsh- Hadamard matrices match the maximal expressivity of dense matrices. Empirically, SL.CEs solve the A5 state-tracking benchmark with single layer, achieve best-in-class length generalisation regular language tasks among parallel- in-time models, and match the performance of log neural controlled differential equations on six multivariate time-series classification datasets while cutting the average time per training step by a factor of twenty.
Introduction
Parallel-in-time architectures, such as Transformers and Structured State-Space Models (SSMs), have allowed language models to scale to billions parameters [6] 39]. However, theory and
practice agree that these models do not generalise longer sequences state-tracking problems, a task that classical Recurrent Neural Networks (RNNs) handle with ease [65] 66 60]. Linear Neural Controlled Differential Equations (LNCDEs) the state- tratsstttnittionn matrix, or vector field, depends linearly on the input path. This allows for the multiplicative interactions between the hidden state and the input path necessary for gating. Reframing SSMs as LNCDES, becomes clear that using a diagonal state-transition matrix severely restricts expressivity [22] Replacing it with dense matrix restores maximal expressivity and the ability to state-track, but increases the number of parameters and computational cost from (dd)) to Odd)).
continuous time sequence model where
where d,, is the hidden dimension 666 22).
Structured alternatives seek the best of both worlds. Block-diagonal Linear RNNs (LRNNs) 33 and
the Diagonal-Plus- -lus--lll--aaaii (DPLR) structure of DeltaNet [85] [OI 8& reduce computational cost while preserving some expressivity, with the latter recently shown to maximally expressive [72]. We generalise and extend these ideas with Structured Linear Controlled Differential Equations (SLiCEs), a unifying framework for structured, input-dependent state-transition matrices. SLiCEs replace the dense state-transition matrix of LNCDE with efficient structured variant, such as alicoadiagonal, sparse, Haddamadd or DPLR, which maintain the maximal expressivity of dense matrices whilst reducing both the parameter count and computational cost.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Contributions
1. Structured Linear Controlled Differential Equations (SLiCEs) are introduced common framework for models with structured, input-dependent state-transition matrices. SLiCES incorporate SSMs such Mamba (39), LRNNs such DeltaNet [85 and input-dependent and two novel structures based on sparse matrices and the Walsh-Hadaman transform.
block-diagonal LRNN [33]. LNCDEs 22
LRNN
2. Block-diagonal, sparse, and Hadamard SLiCEs are proven to achieve maximal probabilistic expressivity Theorems 4.1 and 4.3 respectively. Previously, such
expressivity had only been shown for dense and DPLR matrices [22 72).
3. A comprehensive empirical evaluation showing that the structure of the state-transition matrix significantly impacts length generalisation state-tracking problems. The block- diagonal structure emerges as a promising option, due to its strong empirical results and favourable parallelisation. Furthermore, on real-world multivariate time-series classi- fication datasets, a block-diagonal SLiCE is shown to match the predictive accuracy of Log-NCDEs, whilst reducing the per-step training time by factor of twenty.
4. Open-source implementations of SLiCEs in both PyTorch and JAX, along with code fully reproduce all experiments from this paper. These are available at httts:://gtthiii com/ htnp:/trrrutttued-ee-ee-----dede cdes (Py Torch) and https: htt///gittte /etgggittuub... com/
Benjaming Waweiinn-wleeerooooooa noraaooo--u----eeee (JAX).
Related work Increasing the expressivity of parallel-in-time sequence models while retaining their computational efficiency is of significant interest, as it would facilitate training large, performant models. One approach parallelises non-linear RNNs by rewriting them as fixed-point problems and applying parallel Newton or quasi-Newton methods to calculate their output [59 36). However, reported wall-clock gains remain limited; parallel autoregressive generation
be up to twice as slow as sequential baselines [36]. There have also been a large number input-dependent
LRNN architectures proposed, including input-dependent block-diagonal LRNN 33,, DeltaNet [000, DeltaProduct [88], Gated DeltaNet [101] Mamba [3), Mamba-2 (28), RWKV-7 76,, HGRN- 2 [79], mLSTM (6), Gated Linear Attention [99]. Gated Random Feature Attention (77], Gated Slot Attention [102), TTT-Linear 192, and Titans 77] These models use either diagonal or DPLR
state-transition matrices. Table in [000 presents a comparison of the architectures for number of these models.
Utilising structured matrices to reduce the computational burden of neural networks extends beyond sequence models. The lottery-ticket hypothesis [35] argues that dense networks contain sparse sub-networks that, when trained in isolation, match the accuracy of the full model. Such sub-networks have been uncovered by pruning before [94), during 19], and after training [43). SLiCEs differ from pruning by imposing structured sparsity initialisation and training the result- ing sparse model directly. Other structured matrix approaches include sparse Transformers [16),
2:4 sparsity in linear layers [67], and Monarch layers, which factorise weight matrices into two block diagonia components 29
Background
2.1 Linear controlled differential equations
Let [0, + Rd be a path with bounded-variation, where w, denotes the value the path at time [o, .7. A linear Controlled Differential Equation (CDE) takes the form
d
dhs
(1)
where A E RRdx xda the linear vector field for each channel i 1, 1,..., dd. and h: [0, 7] Rh. is the solution path. Approximating wa with linear interpolation on the grid 0 to ... . < tn T yields
du Σ
h2 = exp
(W - -wt))' hey
(2)

---

The outputs he, are computable in ([log(n)) parallel steps by composing the flow each interval using an associative parallel scan [9] This approach has been used parallelise linear RNNs [64 and SSMs [99
On each interval [tz 1 uses only the increments of w. providing first-order approximation. The Log-ODE method extends this to higher orders by combining the iterated Lie brackets of the vector field with the log-signature of w [12). See Cass and Salvi [T], Section 3.2.2] for summary description of the algorithm and Appendix C for a description of applying the algorithm to an LNCDE.
2.2 Linear neural controlled differential equations
Let [(4,, H=o denote set of observations from a multivariate time-series and X [Lo - RRdx be a continuous interpolation, such that Xr. = (l2, 24). NCDEs are defined as
hic- = Eo ro), he = lito + [" go(h))dXx:
21 (p
where Eo Rdxx dh and go d - xdxxxxx are neural networks, and Lp : Rd - RR is linear map [33 NCDEs have number of desirable properties, including maximal expressivity and robustness irregular sampling rates. Building on the work of Neural Rough Differential Equations [70]. Log-NCDEs (98] demonstrate that combining NCDEs with the Log-ODE method during training leads to state-of-the-art performance on a range of multivariate time-series modelling benchmarks with up to 50,000 observations.
LNCDEs take the form
h4 heo [ L0
du Σ
hio + S.¹ dh.
dw
Σ Ajdw
ha:
where w (lo. (n] - Rd is a path which depends the input X and the 4 are trainable matrices. As will be discussed further in Section [.2] LNCDEs are maximally expressive [53 22]. Therefore, there exists maximally expressive sequence model whose recurrence can be calculated parallel-in-time using the approach outlined in Section 2.1 However, the number of parameters and computational cost makes this approach infeasible in large models. Independently of LNCDEs, Merrill et al. [66] proposed IDS4, a modification of the S4 layer designed to allow tracking which has the same form as (4). Appendix (eaeudixAApprou AAprovides more detailed introduction to LNCDEs by comparing and contrasting them to SSMs and LRNNs. Additionally, Appendix A contains a toy example demonstrating how the structure of the matrices A affects model expressivity, discussion of how to extend LNCDES to matrix-valued hidden states, and a pseudo-code implementation.
3 Expressivity
3.1 Introduction
Expressivity characterises the set of functions model can approximate, and maximal expressivity (or universal approximation) guarantees that, with suitable parameters, any continuous function compact set can be approximated arbitrarily closely.
Definition 3.1 (Maximal expressivity). Let be a topological space, and let F {ffo X R 0 E } be class of real-valued functions on X, parametrised by some set θ. We say that F is maximally expressive (or universal) if. for every compact set K. C X and every real-valued continuous function [ K. TR, the following property holds:
Vc > 0, 38 θ s.f.
sup f(()) fo(oo) <c aEK
(5)
A classical result is the Universal Approximation Theorem, which states that for x= =R4, single- hidden-layer feed-forward networks with a suitable activation function are maximally expressive 27
471

---

3.2 Maximally expressive models on paths
Let X' denote the space of continuous paths of bounded variation on the interval [0, that start at the same point and contain time as a channel düme-augmented) We endow this space with the 1 --variation topology. Let be the class of LNCDEs defined with linear readout layer l0g. such that So X IR is defined by
du Σ
++ 10 (lee)) ) los heoo"
Ad, h duw
(6)
for w N. In this setting, FF is maximally expressive [51]. Furthermore, LNCDES with diagonal
matrices Ab.. such as S4D [40] and Mamba [39], are not maximally expressive [22]. An alternative
to maximal expressivity is the following probabilistic property.
Definition 3.2 (Maximal Probabilistic Expressivity). Let X be a topological space, E N, and 0 E O be a class of real-valued functions on X defined by
J = ff X + R
f (w) = lo (o)
for X, where PN
- log ERN is a linear readout, 01 ON O2 E ON and ON = ON UON. Given a sequence of probability measures PN on ON with 0, PN, has maximal probabilistic expressivity for every compact set K. c. X and every real-valued continuous function f IC + R. the following property holds:
Big s.t. sup wek. (ww) - (w) s6 <E )
Ve > O,
lim PN N0000
(8)
In the context of machine learning, maximal probabilistic expressivity may be considered more promising property than maximal expressivity, as for large enough N, it implies there exists a significant abundance of parameters 0₃ that are capable of achieving uniformly bounded and arbitrarily low error rates with a linear readout layer. This suggests the parameters should be readily discoverable through standard optimisation methods.
In the case of LNCDEs, = du.
fn dw
fen (w) = 1110
Ag, hodws
and Id, on 0 is a collection of probabilities on matrices Ii with A ~ Pj.. Achieving maximal probabilistic expressivity depends crucially on the choice of 19,,- Building on the work of Cuchiero et al. (26). Cirone al. [22 Theorem B.13] showed that choosing the Ağ to be dense Gaussian matrices with independent entries achieves maximal probabilistic expressivity, Unfortunately, using dense matrices is infeasible in practice due to computational constraints, discussed in Section 2.2
4 Structured linear controlled differential equations
4.1 Introduction
Building on Cirone et al. [22]. we introduce Structured Linear Controlled Differential Equations (SLiCEs), unifying framework for various choices of structured Ag. This section presents three examples based on current models in the literature, diagonal, DPLR, and block-diagonal matrices, as well as two novel examples based on sparse matrices and the Walsh-Hadaman transform. Here, we focus on the recurrent core, with further implementation details given in Appendix A.3
Our main theoretical results are Theorems 4.1 4.2 and which show that SLiCEs with block-
diagonal, sparse, and Halsh-Hdddmmard matrices have maximal probabilistic expressivity. Formal proofs are given in Appendix B These results complement (22 Theorem 4.3], which shows that SLiCES with diagonal matrices are not maximally expressive and 772 Proposition F.2], which shows that SLiCEs with DPLR matrices have maximal probabilistic expressivity.

---

4.2 Diagonal-pans-towrank SLiCEs
SSMs with diagonal state-transing transitantiinn matrices, such as S4D 40 and Mamba [39 are examples of SLiCEs with diagonal matrices, A D Hence, they are not maximally expressive and
underperform on state-tracking benchmarks [22 66]. This limited expressivity motivates the use of
alternative structured state-transition matrices.
DeltaNet, DeltaProduct, and Gated DeltaNet use specific versions of DPLR state-transition matri- ces 85 100 88 101]. The general form of DPLR-SLiCE is
A = D
Ug 1. Σ
(Ug (T)))
(10)
where r the rank. This parameterisation reduces the number of trainable parameters and compu- tational cost of calculating hidden state update from ((d to rdw) Furthermore, [72] Proposition D.2] shows that if + oo as dĄ, + 20, then R-LCCSS have maximal probabilistic expressivity.
4.3 Block-Diagonal SLiCEs
Block-diagonal state-transition matrices were first explored in LRNNs to improve performance on regular-language tasks [33. Block-diagonal SLiCEs (BD-SLiCEs) use the same structure, but make the dependence on the input path linear,
A BlockDiag 1 -.. Bg,x);
(11)
where each Bo, E xb, is a trainable dense block, is the number of blocks, and by are the block- sizes, with d, = E5=1b. This parameterisation reduces the number of trainable parameters and computational cost of caleulating a hidden state-update from (d to Od Σ b3), providing substantial speed-up when each bj < d,, Furthermore, this does not restrict the expressivity.
Theorem 4.1. fax, by - as dn - cc, then block-diagonal SLiCEs have maximal probabilistic expressivity.
Hence, the non-linear dependence of the input-dependent block-diagonal LRNN is not necessary for theoretical expressivity. Because the hidden state factorises into independent parts, BD-SLiCEs can be viewed as multi-head dense LNCDE (DE-LNCDE) of head sizes by. For fixed d,, choosing smaller blocks yields greater speed; choosing larger blocks yields greater expressivity. Under fixed compute budget, there trade-off between expressivity and hidden dimension, and this explored empirically in Appendix D.2
4.4
Sparse SLiCEs
A sparse SLiCE (S-SLiCE) takes each A be sparse matrix with Odd non-zero entries, selected at random according to a Bernoulli distribution. This reduces the parameter count and computational cost of calculating a hidden state update from dd) to dd Furthermore, it does restrict the expressivity.
Let
Theorem 4.2. Sparse SLiCEs have maximal probabilistic expressivity.
In theory, S-SLiCEs have faster training and inference times than DE-LNCDEs. practice, current deep-lcarning frameworks (e.g. JAX [10]. Py PoTch (75) are optimised for unstructured sparsity, so practical speed-ups are not observed in our implementations. Nonetheless, we anticipate that ongoing work sparse matrices will enable future gains in efficiency.
4.5 Walsh-Hadamard SLiCEs
Hadamard matrix of order n is an n X matrix H. with entries +11 whose rows (and columns) are mutually orthogonal, H H. nI,, where 7, is the n x identity matrix. When 2 for m E N, we can construct these matrices iteratively using the Sylvester construction [33. Commonly, these matrices applied via the Walsh-Hadamart transform (WHT), which admits an On logn)
5

---

algorithm [86]. Many scientific computing libraries include efficient CPU and GPU kernels for performing the Walsh-Hadamard transform (97 T In practice, normalisation factor of n 11/2 can
be applied to ensure the matrix is orthonormal 95
Walsh-Hadamard SLiCEs (WH-SLiCEs) replace each dense matrix Ab by the product
A HD
(12)
where H is a normalised Hadamard matrix, and Dj is a diagonal matrix. This parameterisation reduces the number of trainable parameters from (dd,d))) to (dd Summing the diagonal matrices across the channels and then applying the fast Walsh-Hadamase transform, the computational cost is O (max((d,dd dı log(d))). This is substantially cheaper than the (dw, for dense LNCDEs. Furthermore, this modification does not restrict the expressivity.
Theorem 4.3. Walsh- Hadamadd SLiCEs have maximal probabilistic expressivity.
4.6 Parallel computation
The recurrent cost of SLiCE is based solely on the cost of calculating single hidden state update, whereas the calculation when using an associative scan is repeatedly composing the flow
exp Σ (wt wo,) ((,)
~ I+
( W -w )4a-
(13)
where the first-order approximation of the exponential sometimes used in practice. When the A are diagonal or block diagona,, the composition of (T3) preserves the structure, as these classes of matrices are closed under multiplication. Therefore, using a parallel associative scan reduces the scan depth from to log(n), whilst having computational cost per composition of O(d,) ) or (dg Σ, by), respectively. However, for DPLR, sparse, and WH SLiCEs, the structured matrices are not closed under multiplication, which means that the limiting computational cost per composition is the same as a DE- (df)). Table [ summarises the differences in parameter count, computational cost, existence of an efficient implementation, and expressivity of all the SLiCEs considered in this paper, where for simplicity we have taken da = da.
For large models, parallel associative scans result in high I/O costs, each state-transition matrix must be materialised in GPU memory [100]. A possible approach to mitigating I/O costs for SLiCEs is combining them with the Log-ODE method. By approximating the solution over intervals, this method avoids explicitly materialising intermediate state-transition matrices. However, it does require computing the log-signature of the input path and iterated Lie brackets of the vector fields (98). detailed description of this approach is given in Appendix C and Table I quantifies the impact of the Log-ODE method computational cost. In Section 5.3 implement a hybrid strategy: the Log-ODE method is applied to small intervals, and the resulting outputs are then processed using a parallel associative scan. Yang al. (100) introduced alternative approach for DeltaNet, where a chunk-wise algorithm specifically tailored for -aagonaalplu--rann-ooee state-transition matrices is used bypass the need materialise every intermediate matrix, significantly cutting down I/O costs [100]. Independently, Cirone and Salvi [18] and Siems et al. [88] extended this approach to higher rank matrices. These approaches can also be applied to diagonal state-transition matrices. Therefore, a block-diagonal SLiCE with a large diagonal portion (b: 1 for = 1...., k 1) followed by small dense block emerges as an attractive solution. The large diagonal section can efficiently utilise the chunk-witg algorithm and the smaller dense section can be processed using parallel associative scans without incurring significant I/O costs. We refer to this structure as diagonal SLiCE (D-DE-SLiCE),
5 Experiments
5.1 The Ag benchmark
The As benchmark tests models on their ability to state-track [66]. Each sequence in the dataset consists of a series of permutations from the group of even permutations on five elements, denoted Ag. The task is to compose the permutations, which requires state-tracking. Following Merrill et al.

---

Table I: Comparison of SLiCEs on parameter count, computational cost, the existence of an efficient implementation, and expressivity. Here, d,, is the hidden dimension, n is the sequence length, by BD's block-sizes, is DPLR's 's rank, is S's sparsity, and for simplicity we have taken d_ = d2. Parallel cost is measured as (scan depth, cost per composition) when applying a parallel associative scan. Log-X-SLiCE corresponds applying the Log-ODE method with fixed-size intervals containing & samples and a truncation depth of N, where X is a specific SLiCE structure with (Pxx) parameters, ((Rx) recurrent cost, and O(Cx) cost per composition.
Efficient Impl.
Maximally Expressive
Model
Parameters
Recurrent Cost
Parallel Cost
DE-LNCDES
((d))
(ndd))
(log(n))
Yes
Yes
D-SLiCEs
(df))
(ndr)
(log(n), d
Yes
No
DPLR-SLiCES
(rdR)
O (nrdh)
(log((), d..
Yes
Yes
S-SLiCEs
(ddtt9)
O (ndfttt)
(log((), d).
No
Yes
WH-SLiCEs
O(dh)
O(ndk)
Ollog(n), di)
Yes
Yes
BD-SLiCEs
(n Σ, O(Px)
ndn E, b2)
llg(()) da Σ, of O (log (?), Cxxd
Yes
Yes
Log X-SLiCEs
Ra di
666, train and evaluate models sequences ranging from length 3 to 20 and determine how many stacked layers each model needs to achieve a validation accuracy greater than 90%.
This benchmark serves as empirical validation of our theoretical results; D-SLiCEs are less expressive than DPLR, sparse, WH, and BD SLiCEs. In addition to the SLiCEs, we consider Mamba [39], LSTM 44), gated DeltaProduct with negative eigenvalues [IOT] 88) and the two components of XLSTM [6] (mLSTM and sLSTM) on this benchmark. All basclines use a hidden dimension of 1024 and all SLiCEs use 1024 parameters per state-transition matrix. Full experimental details are given in Appendix D.I
Figure a shows that the diagonal transttransiton matrices of Mamba, mLSTM, and D-SLiCE mean that an increasing number of stacked layers are needed the sequence length grows. Interestingly, Gated DeltaProduct with negative eigenvalues, which uses DPLR structure, and the D-DE-SLiCE also need a growing number of stacked layers. However, DPLR and BD SLiCE both need one layer for all sequence lengths, suggesting this is not an inherent limitation of theses structures. Similarly, sparse, Hadamaad, and dense SLiCES, as well as the two recurrent baselines L.STM and SLSTM, all need only one layer for all sequence lengths.
To assess length generalisation, we select the models that achieve least 90% validation accuracy on sequences of length 20 and retrain them on sequences ranging from 3 to 40. Early stopping is performed using a validation set with sequence lengths from 40 128. The mLSTM excluded because it requires fixed-length inputs. Figure [b reports test accuracy for lengths from 20 to 5120. The recurrent LSTM and sLSTM generalise well, maintaining high test accuracy beyond both the training and validation ranges. Among the parallel-in-ting WH-SLiCE and Mamba do not attain high accuracy even at training lengths; (ii) DeltaProduct and D-DE-SLiCE generalise to approximately 2x the training length but not beyond the validation range; and (iii) DE-LNCDE, DPLR-SLiCK S-SLiCE, and BD-SLiCE sustain high accuracy on sequences at least 8x the training length, exceeding the maximum validation length.
models, three patterns emerge: (i)
5.2 Regular language tasks
The formal language benchmark is collection of language style tasks split into categories using the Chomsky hierarchy 3T]. Here, we use the regular tasks, which can be solved by processing inputs sequentially with fixed set of internal states and no external memory, i.e. state-tracking. On this benchmark, the models are challenged to generalise to longer sequences, by training on sequences from length 3 to 40 and evaluating on sequences from length 40 to 256. Details on the individual tasks can be found Appendix D.2 A wide range of existing sequence model architectures are used as baselines, including LSTM 44 xI STM and its two components MLSTM and sLSTM 6 four variations of DeltaNet 85
88
38), RWKV-7 (76). a Transformer (96), S4D (4).
and Mamba 39]. All models use two stacked layers. For each dataset and baseline model,
7

---

Diagonal
Minimum Number Of Layers Required
Mamba
DeltaProduct
LSTM
DPURSSWW//DVDEE
10 11
12
13
14
15
16
17
18
19
20
Sequence Length
(a) Sequence length against the minimum number of stacked layers required achieve greater than 90% validation accuracy. Each shaded region indicates an equivalent number of stacked layers.
Token accuracy
0.6
1STM
SLSTM
DPLR-SUCE
0.4
BD-SLICE
DE-LNCDE
0.2
D-DE-SLICE
WHSUIEE
DeltaProduct
20
10
80
160
320
640
1280
2560
5120
Sequence length
(b) Test set token accuracy (%) against sequence length.
Figure 1: Results for Aş Benchmark and Aş length generalisation task. Models evaluated are: Mamba, LSTM, mLSTM, sLSTM, Gated DeltaProduct with negative eigenvalues, and the SLiCEs.
selected the hidden dimension that yields a higher validation accuracy from two choices. The choices were 128 and 512 for all models aside from Mamba, and 256 and 512 for Mamba, as does not support a hidden size of 128. All SLiCEs use two stacked layers and 512 non-zero parameters per state-transition matrix, except for the diagonal and Hadsm-Had which also consider 1 128, with the better performing choice reported for each dataset. For the DPLR, biock-diagonal, and diagonal-denss SLiCEs, we consider multiple choices of rank and block-size, respectively, and present the best performing models. A thorough investigation of the effect of block-size and rank given in Appendix [D.2 We do not consider S-SLiCE on this benchmark, due to the lack of an efficient implementation.
Table presents the results. As expected, the recurrent LSTM generalises almost perfectly on all four tasks, Amongst the parallel models, DeltaNet with negative eigenvalues and Gated DeltaProduct with negative eigenvalues are best performing baselines, aligning with the expectation that increased complexity in the state-transition matrix improves state-tracking performance. Similarly, D-SLiCE outperforms Mamba, aligning with the results Grazzi al. (38) that expanding the eigenvalue range of the state-transitin matrix improves state-tres sate-tracking performance. Similarly to the As length

---

Table Results for formal language tasks. Average and standard deviation of validation accuracy over five runs for a range of recurrent and parallel models.
Mod Arith.
Model
Cycle Nav.
Even Pairs
Parity
Average
No Brack.
Recurrent
LSTM
100.0 0.0
100.0 0.0
99.9
100.0 1: 0.0
100
sLSTM
32.5 0.4
100.0 + 0.0
27.7 + 0.6
100.0 + 0.0
65.1
XLSTM[I:1]
53.5 + 5.6
99.0 + 1.9
29.3 = 1.6
100.0 + 0.0
70.5
Parallel
DeltaNet
49.8 + 4.7
1.0
42.2 4.8
57.8
62.5
DeltaNet [
46.7 6.1
100.0
66.4888
97.7 2.0
77.7
Gated DeltaNet
8.8
100.0
42.8182
56.5 1.9
63.3
Gated DeltaProduut(-L.I
46.3
100.0
10.9
1.
80.7
RWKV-7
3.0
84.1
3.1
51.1
54.1
mLSTM
12.4+10.5
9.1
28.8
53.0
58.5
Transformer
0.5
90.4
23.6
5.4
47.7
Mamba
2.2
1.0.0
6.6
5..2
58.9
1.1
6.77
2..7
1.0
41.3
D-SLiCE
69.5 6.3
100.0
2.1
100
72.6
WH-SLiCE
8.87
93.1
23.8
11.41299
64.5
BD-SLiCE,, -22,66m4
9.2
81.3
54.0
9.9
83.8
D-DE-SL ADE-SLiCE,, 227
73.3 29.4
8.5
98.4
83.8 11.
85.1
DPLR-SIIiCE -S,CCid,, =5.
81.1
16.6
100.0 0.0
68.3 + 19.3
91.0 + 18.0
85.1
Random
20.0
50.0
20.0
50.0
35.0
generalisation task, the WH-SLiCE underperforms other SLiCE structures. However, unlike in the As length generalisation task, the D-DE-SLiC SLiCE achieves the joint highest average validation accuracy among the parallelisable models, alongside DPLR-SLiCE
5.3 UEA multivariate time-series classification archive
Since SLiCEs are descendants of NCDEs, they inherit a number of the desirable properties which arise from having a natural continuous-time formulation. These include robustness to irregular sampling rates and decoupling the number of recurrent steps from the number of observations in the time-series [53] 98. Furthermore, SLiCEs have the same theoretical expressivity as NCDEs, whilst being parallel-in-time, making them an attractive alternative for real-world time-series modelling.
As demonstration of the practical benefits, consider six datasets from the UEA Multivariate Time- Series Classification Archive (UEA-MTSCA), a collection of time-series classification tasks, ranging from classifying worms into species based movement to classifying alcohol by concentration using vibrational spectroscopy [4] Walker et al. (98) showed that Log-NCDEs outperform the
linear recurrent unit (LRU) 773. S5 [9), S6 39 and Mamba [39] on average test accuracy over the six longest datasets with at least 200 observations [98]. However, the per-step training time is significantly higher for Log-NCDEs than the baseline methods.
Keeping all other hyperparameters the same, Table 3 the impact of replacing the non-linear vector field of Log-NCDE with structured linear vector field. The GPU memory and time per 1000 training steps were recalculated for all models on an NVIDIA H100. BD-SLiCE achieves similar performance to the Log-NCDE, whilst reducing the average per-step training time by factor of nearly 20 and increasing the average GPU memory usage by only 8%. Appendix D.3 presents the results for individual datasets and analyses the impact of the Log-ODE method and parallel associative scan on run-time and GPU memory.
Limitations and Future Work
To reduce the computational burden of SLiCEs, our implementation approximates the matrix expo- nential when computing the flow via [3] However, even with this adjustment, scaling SLiCEs to

---

Table 3: Average test accuracy, rank, training time per 1,000 steps, and GPU memory usage across six datasets from the UEA-MTSCA. All SLiCE variants use parallel associative scan during training. Therefore, the Walsh-Hadammrd, DPLR, and sparse SLiCEs are treated as dense LNCDEs (see Section 4.6, and their timing and GPU memory results are omitted.
Model
Av. Acc
Av. Rank
Av. Time / 1k Steps (s)
Av. GPU mem (MB)
BD-SLiCE
64.0
3.2
68.1
2344
Log-NCDE
64.3
4.0
1321.7
2177
D-DE-SLiCE
63.0
5.7
66.7
2302
WH-SLiCE
62.5
6.7
DPLR-SLiCE
62.0
7.0
D-SLiCE
61.7
7.2
11.0
1875
LRU
61.7
7.3
26.9
4308
S6
62.0
7.7
20.1
2938
S5
61.8
8.0
21.9
3327
S-SLiCE
61.8
8.2
DE-LNCDE
61.6
8.3
77.2
12756
NCDE
60.2
8.8
6923
1962
NRDE
60.6
10.3
3431
2858
Mamba
58.6
10.8
60.0
4535
the multi-billion parameter regime remains challenging. A key technical goal is the development of efficient GPU kernels for the matrix exponential and parallel associative scans, particularly when handling many small independent systems, such for BD-SLiCES. Alternatively, building the work of Yang et al. [000 and Cirone and Salvi [18] fast chunk-wise methods for broader class of structured matrices may offer viable path forward.
Alternative SLiCE architectures may achieve maximal expressivity and improved empirical perfor- mance. A theoretical characterisation of the conditions that SLiCE's structured matrix needs to satisfy to achieve maximal probabilistic expressivity would aid the search for additional structures. Moreover, although establishing maximal probabilistic expressivity is significant step towards deeper theoretical understanding of structured sate-transition matrices, expressivity at finite hidden dimensions remains open challenge.
Finally, unlike NCDEs, SLiCES are seuuence-to-sequeeee models that update their state with each input sample. Therefore, similarly to other discrete sequence models, SL.CES are susceptible to over-sampled data. Combining SLiCEs with the Log-ODE method enables path-based inputs by operating with flows over intervals, rather than individual samples. However, the Log-SLiCE outputs a sequence whose elements correspond to the boundary values the output path for each interval the Log-ODE method was applied to. Therefore, you cannot stack two Log-SLiCEs, as the first level has produced a sequence, whereas the second level consumes a path. A natural direction for future work developing true path-to-pa --th model.
7 Conclusion
This paper introduced SLiCEs, a unifying framework for sequence-to-sequene layers that are maximally expressive, computationally efficient, and allow for parallel-in-time computation. We explored four specific instances, diagonal, analysing their theoretical properties and empirical performance. Theorems 4.1 4.2 and 4.3 established that block-diagonal, sparse, and Hadamadd SI achieve maximal probabilistic expressivity. Furthermore. all SLiCE structures demonstrated single-layer state-tracking on the As benchmark. unlike the other parallelisable layers considered: diagonal SLiCEs, mLSTM, Mamba, and DeltaProduct. Among the SLiCES, block-diagonal stands out as the only maximally expressive variant that strictly reduces parameter count, recurrent cost, and parallel cost compared to dense LNCDEs. Additionally, variant of the block -dikoona SLiCE achieved the joint highest average validation accuracy among parallel models on the regular language tasks from the formal language benchmark. Finally, practical speed-ups for real-world time series modelling were demonstrated on six multivariate time-series classification datasets, where replacing the non-linear vector field of Log-NCDE with block-diagonal linear vector field reduced the average time per training step by factor of twenty, without impacting the model's overall performance.
sparse, Walsh-Hadammard, and block-
10

---

Acknowledgements
We thank Joël Mouterde, Jéröme Tomezyk, Sam Morley, and Alexandre Bloch for engaging and insightful discussions on the design and training of linear neural controlled differential equations. We thank Merrill al. [66], Delétang et al. [1,, and Bagnall et UEA datasets respectively.
4 for the As, formal language, and
Benjamin Walker funded by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA). Lingyi Yang is supported by EPSRC (EP/S026347/1] and the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA). Nicola Muca Cirone is supported by the EPSRC Centre for Doctoral Training in Mathematics of Random Systems: Analysis. Modelling and Simulation [EP/S023925/I] Terry Lyons was supported part by the UKRI EPSRC through the Programme Grants High order mathematical and computational infrastructure for streamed data that enhance contemporary generative and large language models (UKRI1010) and Unparameterised multi-model data, high order signatures and the mathematics of data science (EP/S026347/1) and UKRI AI for Science award UKRI2385; he was supported by The Alan Turing Institute under the EPSRC Grant EP/N510129/1, the Defence and Security Programme (funded by the UK Government), and through CIMDA Oxford, part of the AIR @ Technology Commission, HKSAR Government. The authors would like to acknowledge the use of the University of Oxford Advanced Research Computing (ARC) facility in carrying out this work: http://dx. www. /og. 10..28 wzw22o0 .25588. For the purpose of Open Access, the author has applied CC BY public copyright licence to any Author Accepted Manuscript (AAM) version arising from this submission.
InnoHK initiative funded by the Innovation and

---

References
[1] Krish Agarwal, Rishi Astra, Adnan Hoque, Mudhakar Srivatsa, Raghu Ganti, Less Wright, and Sijia Chen. HadaCore: Tensor core accelerated hadamard transform kernel. arXiv preprint arXiv:2412.08832, 2024.
[2] Imanol Perez, Arribas, Cristopher Salvi, and Lukasz Szpruch. Sig-SDEs model for quantitative finance. In ACM International Conference on Al in Finance, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 0005000 2016.
[4] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018. arXiv preprint arXiv: 0ww.1881100007. 1.00075, 2018.
[5] Barbora Barancikova, Zhuoyue Huang, and Cristopher Salvi. Sigdiffusions: Score-based diffu- sion models for long time series via log-signature embeddings. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
[6] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Ex- tended long short-term memory. In Proceedings of the 38th Conference Neural Information Processing Systems (NeurIPS), 2024.
[7] Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.0066 2024.
[8] Torben Berndt, Benjamin Walker, Tiexin Qin, Jan Stühmer, and Andrey Kormilitzin. Permu- tation equivariant neural controlled differential equations for dynamic graph representation learning. arXiv preprint arXiv:2506.203:: 2025.
[9] Guy E. Blelloch. Prefix sums and their applications. (CMU-CS-90-190), 1990.
[10] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Mille and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[11] Thomas Cass and Cristopher Salvi. Lecture notes on rough paths and applications to machine learning. arXiv preprint 2.Xx:2240400683, 2024.
[12] Fabienne Castell and Jessica Gaines. An efficient approximation method for stochastic differential equations by means the exponential Lie series. Mathematics and Computers in Simulation, 38(1):13-19, 1995. ISSN 0378-4754.
[13] Kuo-Tsai Chen. Iterated integrals and exponential homomorphisms. Proceedings of the London Mathematical Sociery, 3(1):502-555 512, 1954.
[14] Kuo-Tsai Chen. Integration of paths, geometric invariants and generalized Lokee-HHauddoff formula. Annals of Mathematics, pages 163-178, 1957.
[15) Ilya Chevyrev and Andrey Kormilitzin. A primer the signature method in machine learning. 2025.
[16] Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.1 10509, 2019.
[17] Noam Chomsky. Three models for the description of language. IRE Transactions on informa- tion theory, 2(3):113-122 124, 1956.
[18] Nicola Muca Cirone and Cristopher Salvi. ParallelFlow: Parallelizing linear transformers via flow discretization. arXiv preprint arXiv:2504. 00492, 2025.
[19] Nicola Muca Cirone and Cristopher Salvi.
Rough kernel hedging. arXiv preprint
7XXi:25501009883 2025.
12

---

[20] Nicola Muca Cirone, Maud Lemercier, and Cristopher Salvi. Neural signature kernels as miinite-width-depee depth limits of controlled resnets. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.
[21] Nicola Muca Cirone, Jad Hamdan, and Cristopher Salvi. Graph expansions of deep neural networks and their universal scaling limits. arXiv preprini arXiv:2407.08459 2024.
[22] Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, and Terry Lyons. Theoretical foundations of deep selective state-space models. Advances in Neural Information Processing Systems, 2024.
[23] Thomas Cochrane, Peter Foster, Varun Chhabra, Maud Lemercier, Terry Lyons, and Cristopher Salvi. SK-Tree: a systematic malware detection algorithm on streaming trees via the signature kernel. In 2021 IEEE International Conference on Cyber Security and Resilience (CSR), pages 35-40. IEEE, 2021.
[24] Samuel N. Cohen, Silvia Lui, Will Malpass, Giulia Mantoan, Lars Nesheim, Áureo de Paula, Andrew Reeves, Craig Scott, Emma Small, and Lingyi Yang. Nowcasting with signature methods. arXiv preprini urXiv:2305. 10256, 2023.
[25] Samuel N Cohen, James Foster, Peter Foster, Hang Lou, Terry Lyons, Sam Morley, James Morrill, Hao Ni, Edward Palmer, Bo Wang, Yue Wu, Lingyi Yang, and Weixin Yang. Subtle variations in sepsis-III definitions markedly affect predictive performance. Nature Scientific Reports, 2024.
[26] Christa Cuchiero, Lukas Gonon, Lyudmila Grigoryeva, Juan-Pablo Ortega, and Josef Teich- mann. Expressive power of randomized signature. The Symbiosis of Deep Learning and Differential Equations, 2021.
[27] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, ([4):303-314 1989.
[28] Tri Dao and Albert Gu. Transformers are SSMs: generalized models and efficient algorithms through structured state space duality. Proceedings of the 41 International Conference on Machine Learning, ICML'24. JMLR.org. 2024.
[29] Tri Dao, Beidi Chen, Nimit Sharad Sohoni, Arjun D Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré. Monarch: Expressive structured matrices for efficient and accurate training. In International Conference on Machine Learning. 2022.
[30] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning, pages
PMLR, 2017.
[31] Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A. Ortega. Neural networks and the Chomsky hierarchy. In /1th International Conference on Learning Representations, 2023.
[32] Terry Lyons al. CoRoPa computational rough paths (software library). 2010.
[33]
Ting-Han Fan, Ta-Chung Chi, and Alexander Rudnicky. Advancing regular language reasoning in linear recurrent neural networks. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 45-53, Mexico City, Mexico, June 2024, Association for Computational Linguistics.
[34] Adeline Fermanian, Terry Lyons, James Morrill, and Cristopher Salvi. New directions in the applications of rough path theory. IEEE BITS the Information Theory Magazine, 2023.
[35] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
13

---

[36] Xavier Gonzalez, Andrew Warrington, Jimmy T. H. Smith, and Scott W. Linderman. Towards scalable and stable parallelization of nonlinear RNNs. In Advances Neural Information Processing Systems (NeurIPS), 2024.
[37] Benjamin Graham. Sparse arrays of signatures for online character recognition. arXiv preprint arXiv: /308.0371, 2013.
[38] Riccardo Grazzi, Julien Siems, Jörg K. H. Franke, Arber Zela, Frank Hutter, and Massimiliano Pontil. Unlocking stack-t in linear RNNs through negative eigenvalues. arXiv preprint arXiv:2411.12537 2024.
[39] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00755. 2023.
[40] Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In Advances in Neural Information Processing Systems (NeurIPS) 2022, 2022.
[41] Marshall Hall. A basis for free Lie rings and higher commutators in free groups In Proceedings of the American Mathematical Society, volume 1, pages 575-581, 1950.
[42] Hambly and Lyons. Uniqueness for the signature of a path of bounded variation and the reduced path group. Annals of Mathematics,
[43] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In The IEFE International Conference on Computer Vision (ICCV), Oct 2017.
[44] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 98): 1735-1780, nov 1997. ISSN 0899-7667.
[45] Melker Hoglund, Emilio Ferrucci, Camilo Hernández, Aitor Muguruza Gonzalez, Cristopher Salvi, Leandro Sanchez- Baacche--Beancouurt and Yufei Zhang. neural RDE approach for continuous- time non-Markovian stochastic control problems. arXiv preprint arXiv:2306. 14258 2023.
[46] Christian Holberg and Cristopher Salvi. Exact gradients for stochastic spiking neural networks driven by rough signals. arXiv preprint arXiv:2405. 13587, 2024.
[47] Kurt Homikk Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251-257, 1991.
[48] Blanka Horvath, Maud Lemercier, Chong Liu, Terry Lyons, and Cristopher Salvi. Opti- mal stopping via distribution regression: higher rank signature approach. arXiv preprint arXiv:2304.01479 2023.
[49] Zacharia Issa, Blanka Horvath, Maud Lemercier, and Cristopher Salvi. Non-adversarial training of neural SDEs with signature kernel scores. Advances in Neural Information Processing Systems, 36, 2024.
[50] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. Proceedings of the 37th International Conference on Machine Learning, 2020.
[51] Patrick Kidger. On neural differential equations. arXiv preprint arXiv:2202.002435, 2022.
[52] Patrick Kidger, Patric Bonnier, Imanol Perez Arribas, Cristopher Salvi, and Terry Lyons. Deep signature transforms. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[53] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. In Advances in Neural Information Processing Systems, 2020.
[54] Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv: 1412.6980 2017.
14

---

[55] Franz J Király and Harald Oberhauser. Kernels for sequentially ordered data. Journal of Machine Learning Research, 20(31):1-45, 2019.
[56] Maud Lemercier, Cristopher Salvi, Thomas Cass, Edwin V Bonilla, Theodoros Damoulas, and Terry Lyons. SigGPDE: Scaling sparse gaussian processes on sequential data. In International Conference on Machine Learning. PMLR, 2021.
[57] Maud Lemercier, Cristopher Salvi, Theodoros Damoulas, Edwin Bonilla, and Terry Lyons. Dis- tribution regression for sequential data. In International Conference on Artificial Intelligence and Statistics, pages 3754-3762 PMLR, 2021.
[58] Daniel Levin, Terry Lyons, and Hao Ni. Learning from the past, predicting the statistics for the future, learning an evolving system. arXiv preprini arXiv: 1300902000 2016.
[59] Yi Heng Lim, Qi Zhu, Joshua Selfridge, and Muhammad Firmansyah Kasim. Parallelizing non- linear sequential models over the sequence length. In International Conference on Learning Representations, 2024.
[60] Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Trans- formers learn shorteuts to automata. In International Conference on Learning Representations, 2023.
[61] Terry J Lyons. Differential equations driven by rough signals. Revista Matemática Iberoameri- cana, 14(2):215310, 1998.
[62] T.J. Lyons, M. Caruana, and T. Lévy. Differential Equations Driven by Rough Paths: École D' étéé de Probabilités de Saint-Flour XXXIV-2004. Springer, 2007.
[63] Georg Manten, Cecilia Casolo, Emilio Ferrucci, Søren Wengel Mogensen, Cristopher Salvi, and Niki Kilbertus. Signature kernel conditional independence tests in causal discovery for stochastic processes. arXiv preprint arXiv:2402. S04i::2002.18777,, 2024.
[64] Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In International Conference on Learning Representations, 2018.
[65] William Merrill and Ashish Sabharwal. The parallelism tradeoff: Limitations of log- log-precision transformers. Transactions of the Association for Computational Linguistics, 2023.
[66] William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.
[67] Asit K. Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2/04.08377 2021.
[68] Sam Morley and Terry Lyons RoughPy. In Proceedings of the 23rd Python in Science Conference, 2024.
[69] James Morrill, Andrey Kormilitzin, Alejo Nevado Hollado, Sumanth Swaminathan, Sam Howison, and Terry Lyons. The signature-based model for early detection of sepsis from electronic health records the intensive care unit. 2019 Computing in Cardiology (CinC), pages Page P-Page 4, 2019.
[70] James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
[71] James Morrill, Patrick Kidger, Lingyi Yang, and Terry Lyons. On the choice of interpolation scheme for neural CDEs. Transactions of Machine Learning Research, 2022.
[72] Sajad Movahedi, Felix Sartthein Nicola Muca Cirone, and Antonio Orvieto. Fixed-point RNNs: From diagonal to dense in a few iterations. arXiv preprint arXiv:2503.10799 2025.
15

---

[73] Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.
[74] Alexandre Pannier and Cristopher Salvi. A path-dependent PDE solver based on signature kernels. arXiv preprint arXiv:2403.
2024.
[75] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-pent seghoperformence deep learning library. arXiv preprint arXiv: 1912.01 2019.
[76] Bo Peng. Ruichong Zhang, Daniel Goldstein, Eric Alcaide. Xingjian Du Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng, RWKV-7 "Goose" with expressive dynamic state evolution. arXiv preprint 2tw::2553..444555 2025.
[77] Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. Random feature attention. In International Conference on Learning Representations (ICLR), 2021.
[78] Tiexin Qin, Benjamin Walker, Terry Lyons, Hong Yan, and Haoliang Li. Learning dynamic graph embeddings with neural controlled differential equations. IEEE Transactions on Pultern Analysis and Machine Intelligence, October 2025. Online ahead of print.
[79] Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong, HGRN2: Gated linear RNNs with state expansion. arXiv preprint arXiv:2404.07900 2024.
[80] C. Reutenauer. Free Lie Algebras. LMS monographs. Clarendon Press, 1993.
[81] Cristopher Salvi. Rough paths, kernels, differential equations and an algebra of functions streams. PhD thesis, University of Oxford, 2021.
[82] Cristopher Salvi, Thomas Cass, James Foster, Terry Lyons, and Weixin Yang. The signature kernel is the solution of Goursat PDE. SIAM Journal on Mathematics of Data Science, 3(3):
973-899, 2021.
[83] Cristopher Salvi, Maud Lemercier, Chong Liu, Blanka Horvath, Theodoros Damoulas, and Terry Lyons. Higher order kernel mean embeddings to capture filtrations of stochastic processes. Advances in Neural Information Processing Systems, 14:16635-166644 2021.
[84] Cristopher Salvi, Joscha Dichl, Terry Lyons, Rosa Preiss, and Jeremy Reizenstein. structure theorem for streamed information. Journal of Algebra, 634:91 2023.
[85] Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Proceedings of the International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research (PMLR), 2021.
[86] John L Shanks. Computation of the fast Walsh-Fourier transform. IEEE Transactions on Computers, 100(5):457-459, 1969.
[87] Daniil Shmelev and Cristopher Salvi. Sparse signature coefficient recovery via kernels. arXiv preprint arXiv:2412.088777 2024.
[88] Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. DeltaProduct: Improving state-tracking in linear RNNs via Householder products. arXiv preprint arXiv:2502. 10297, 2025.
[89] Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling. In International Conference Learning Representations, 2023.
[90] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1958 2014.
16

---

[91] Nikko Ström. Sparse connection and pruning in large dynamic artificial neural networks. In European Conference on Speech Communication and Technology (EUROSPEECH), 1997.
[92] Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xin- lei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (leam at test time): RNNs with expressive hidden states. arXiv preprint arXiv:2407.0462) 2025.
[93] James Joseph Sylvester. Thoughts on inverse orthogonal matrices, simultaneous signsuc- cessions, and tessellated pavements in two or more colours, with applications to Newton's 'ss rule, ornamental tile-work, and the theory of numbers. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 34(232):461447) 1867.
[94] Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6377-6389. Curran Associates, Inc., 2020.
[95] Giacomo Torlai, Christopher Wood, Atithi Acharya, Giuseppe Carleo, Juan Carrasquilla, and Leandro Aolita. Quantum process tomography with unsupervised learning and tensor networks. Nature Communications, 14(1):2858, 2023.
[96] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[97] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weekesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing Python. Nature Methods, 17:261-272. 2020.
[98] Benjamin Walker, Andrew D. McLeod, Tiexin Qin, Yichuan Cheng, Haoliang Li, and Terry Lyons. Log neural controlled differential equations: The Lie brackets make difference. International Conference on Machine Learning, 2024.
[99] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2024.
[100] Songlin Yang, Bailin Wang, Yu Zhang. Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[101] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving Mamba2 with Delta Rule. arXiv preprint arXiv:2412.066666 2025.
[102) Yuxuan Zhang, Shiliang Yang, Rong Zhu, Yichong Zhang, Lei Cui, Yongjing Wang, Bin Wang, Feng Shi, Bing Wang, Wei Bi, Ping Zhou, and Guoxin Fu. Gated slot attention for efficient. linear-time sequence modeling. In Proceedings of the Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurlPS), 2024.
17

---

A Linear Neural Controlled Differential Equations
A.1 The Connection to Structured State-Space Models
Viewing SSMs as LNCDEs provides theoretical framework for comparing architectures and reasoning about their expressivity. Following the approach of Cirone et al. [22]. this appendix shows how S6, the recurrent core of Mamba, can be recast LNCDE. Additionally, this chapter highlights the limits of diagonal state- transition matrices, details the connection between our work and matrix-valued hidden states, and provides further implementation details for our SLiCE models.
S6 is defined by
W Car, + Dy(,,))
(14)
where denotes the input channel,
(wr.) - exp(A((rt))
(z(,)))) (A' (zt, (oa) (eep(()))) 1) (x Doors
(15)
2 A (st,)) Dyst::
with trainable parameters Dg E Rd xd. and a diagonal state-transition matrix Co Rd and
2 (mr.) = softplus(a)) Ft, + +1))
(16)
39 with trainable parameters og R and By E
Equation (1) can be considered zero-order
hold discretisation of
dh CoA DoX,
(17)
where X, is an interpolation of {xr, in As shown by Cirone et al. [22], these equations can be stacked and rewritten as an affine LNCDE,
de
Aphiadw Xii + f Bodex"
h4 = heo +
(18)
1=0
where
w/ ( softplus(a' Xx, + 3's
X,softplus(a" X, + B
(19)
<x /''
ds,
Xysoftplus(a dx X, + gdx )xdx
and
A/ diag (0. ... . 0, Co. .......)) 0) E Rd xdad.
(20)
By diag Dg..... Do) E Rdd
with the non-zero diagonal element of Ab in the 2thh position. Neglecting the bias term gives a specific instance of an LNCDE,
h4 heg +
Agh,dw
(21)
The two core differences between the LNCDEs used in this paper and Mamba are:
1. More general forms for the driving path wixK For example, the LNCDEs in this paper use
wx (4+ - Wi t,+1 to
(1,
(22)
on the As and formal language benchmarks and wix = Xx on the UEA benchmarks.
2. Mamba uses diagonal At. whereas LNCDE uses dense AV.
18

---

Cirone et al. [22] use the LNCDE perspective to provide full theoretical analysis on the impact of using diagonal matrices instead dense matrices. Here, we give a simplistic example demonstrating the difference in expressivity between diagonal and dense state-transition matrices.
Consider stream of bits
T1 T2
In E {0, 1].
(23)
where we want to predict the parity label defined by
Pa Sn mod 2 E [0,1], S, Σ wk.
(24)
Whenever a new bit is the label flips; if the bit is 0 the label stays the same. Taking diagonal LNCDE with hidden dimension of 2 and w wx I then
1n+1 exp (1a
bn: (2)
Tn+1
(25)
and
h4 = hig exp(a;Sn),
i = 1,2.
(26)
With linear read-out sigmoid):
(2), followed by monotone activation (such as tanh, ReLU,
Pr: oo oecicars + + = (f((s))
C r:hi
(27)
Since f(S) has at most one turning point, and á is monotone, P. can cross any chosen threshold at most twice. However, the true label pr, flips every time S, S, + 1. Hence, no diagonal 2
LNCDE can realise parity on arbitrarily long input. Similarly, for a hidden dimension of n. f(S) can have at most n 1 turning points, so no diagonal LNCDE with a fixed hidden dimension can realise parity on arbitrarily long input. If you replace A with
π
(28)
0
then
exp(AIn+1)= = ()) 9).
(29)
when Tn+11 = 0 and
-1 0
exp(AAnnt)) =
-1)
(30)
when Un+1 = 1. Thus
hnt1 hn
(31)
and
hn = (-1)s ha-
(32)
Taking r (1,0)T,48' = 1, and d(s) = (1 -iign (s) /2,
Pn 1 - --sign( sign((11))) ((-1) Sn mod Pn-
(33)
2
Therefore, a dense LNCDE can solve parity exactly with a hidden dimension 2.
This example aligns with the results of Cirone et al. P2] diagonal Aú, are fundamentally less expressive than dense AG- However, as noted in Section 2.2 the additional computational cost makes dense transitonnt matrices infeasible in practice. This trade-off between efficiency and expressivity is the motivation for the structured Aij in the SLiCEs introduced in this paper.
19

---

A.2 Matrix-valueed Hidden States
When rewriting Mamba as an affine LNCDE 18). the hidden states for each channel of the input are stacked vertically. An alternative approach is to view the hidden states for each channel as the columns of matrix, producing LRNN with a matrix-valued hidden state.
Matrix-valued I LRNNS originated as an alternative viewpoint on linear Transformers, where softmax is replaced with kernel admitting finite-dimen minee-dimensional feature map, s(g, k) ((g) ( ((k))) for Φ I + R Letting W gd, W Rd,xdd, and Wy E Rduxda be learnable weights, then causal linear attention is defined by
Σj- ((W o ((W Wa
Oc;
(34)
Σj=1 (W..r.) o @W 21,) As shown by Katharopoulos et al. [50], this admits an RNN formulation,
Sr. St, + (W
24. 2,,1 + 2 (W (.)
(35)
o (W ) T Se.
On, xW
21:
where S; E Rdd with So = 0, 2 E Red with 20 = 0, and o; E Rd,, Ignoring the normalisation, the core recurrence is matrix-valued LRNN.
Matrix-valued LRNNs are a useful framework for understanding several recent sequence models, as highlighted by Yang al. [T00. Many of these models share the general form:
S = St. Mr. v(2r,, (kxx,)
(36)
where denotes an associative operator and and k are potentially non-linear functions of ar. For example, Mamba's 's recurrence (14) can be rewritten as
S = St. O Mr + ((,,,))((,,))
(37)
where @ refers to the Hadamard (element-wise) product and S E Rd, 110]. The Hadamard product is a direct consequence of Mamba's 's diagonal state-transition matrix, which inherently prevents interaction between individual elements of the hidden state. This framework gives clear interpretation for a key modification to the recurrence introduced by Mamba-2 (28)
S = (s,,)S. + v(z,))((,,)
(38)
where y is real-valued function. Additionally, it highlights the structural similarity between Mamba and linear Transformers.
Replacing the Hadamard product by matrix product allows modelling richer interactions between the hidden states,
S Sc, + v(z,, (((,,
(39)
However, similarly to using dense matrices in an LNCDE, the cost makes models of this type intractable in larger models. DeltaNet uses a diagonal-plus plas-rr--o structure,
S ST (2,, T) +
(40)
where 3 is real-valued function [85 [00]. DeltaProduct later generalised DeltaNet to DPLR matrices (88). Beyond reducing parameter count and recurrent computational cost, the DPLR structure of DeltaNet also facilitates efficient chunk-wise algorithm that can outperform parallel associative scans for large hidden dimensions, as outlined in 100] Section 3.2].
Many recent sequence models can also be viewed as matrix-valued LRNNs. These include, but are not limited to, Gated DeltaNet [OT], RWKV-7 (76), HGRN-2 77),, mL.TMM 6), Gated Linear Attention [99), Gated Random Feature Attention (77), Gated Slot Attention [122, TTT-Linear [92]
and Titans 77. A detailed comparison of the specific form of (36) for many of these models
provided in Table 2 of [100]. Beck al. (6 introduced mLSTM alongside non-linear recurrent
20

---

model sLSTM, which together form their sequence model xLSTM. These components are designed to play different roles, with mLSTM acting the memory and sLSTM performing the reasoning. Section 5 Juses mL.TM, sLSTM, and xLSTT as baseline methods to highlight the different roles the components play.
Matrix-valued LRNNS can be converted into vector-valued LRNNS by returning to the stacking approach of Section A.1 Let vec R + 1 be the column-major vectorisation operator, which transforms a matrix into a vector by stacking its columns. Letting @ denote the Kronecker product, the Hadamard product LRNN 37 can be rewritten as
vec(St.,,) = diag (M,)) ve,(St,) + k(z,,) @ e(zz.),
(41)
where the state-transition matrix is diagonal, consistent with Mamba's 's design. Noting that vex(AA (BT @ A) vec(X),
(42)
then the LRNN with matrix product 39 can be written as
vec(S,, ,) = (M, Ia,,) veo(S.;) + k(xx,, @ v(x:;).
(43)
In both cases, a matrix-valued recurrence is equivalent to a vector-valued recurrence on a flattened hidden state, where the transitoanitonn matrix is constrained have specifie structure. For the Hadamard product, this gives diagonal matrix, while the matrix product leads to a Kronecker product structure.
This work is focused on the vector-valued case. The insights gained from this analysis naturally extend to the matrix-vall valued setting. For instance, the limitations in expressivity of diagonal matrices directly translate to the Hadamard product formulation. Similarly, the Kronecker product structure arising from vectorising matrix-valued recurrence clearly illustrates how the properties of the constituent matrices determine the properties of the overall state transition.
LNCDEs can be generalised to matrix-valued hidden states by reversing the above vectorisation. Altematively, one can run m copies of the same LNCDE via
H1 11.0 [/(
Σ Ab xi H1,
(44)
where Hg E Rd4xm. Clearly, copies of vector-valued LNCDE SLiCE match the expressivity of a single copy, so all of our theoretical results naturally carry over to this setting. However, the columns only differ due to their initial conditions. To introduce meaningful column-specific dynamics, you can include a column-specific bias term in the vector field. A natural way to incorporate such biases into matrix-valued LNCDE is to consider an affine LNCDE,
H, = H1o + L'
Σ
Al dw X. H, + Bg diag(dg.),
(45)
where Bg E Rd xm and ex [0. T] + Rm is a path, dependent on the input X. Letting hy for 1 I k S denote the columns of H,, then
h = h + S.
(/ Σ 4b duu ) h + BBdgxx.
(46)
Approximating w and & with linear interpolation the grid to < <t T yields
ik exp
1441
~
Σ
A
x, - w h + B (E
X.M - 522 X.K.
(47)
where we have used the approximation
d
X
X,+
W 11 wi,
exp
dr ~ (1z+1 - 1z)
(48)
tj+1 tj
for small increments. The outputs htj remain computable in Ollo((n) parallel steps using parallel associative scan [9].
21

---

A.3 Implementation Details
Algorithm provides pseudo-code implementation for the forward pass of SLiCE. The approach demonstrated for dense state-transition matrices Aÿ. with comments highlighting where a SLiCE's structure be used to speed up computation reduce memory footprint. In this paper, each SLiCE recurrence is embedded in a simple block structure, combining linear layer to mix the channels, tanh activation function, layer normalisation, and a skip connection. Inspired by the Lip(7) regularisation of Log-NCDEs, all SLiCES employ weight regularisation on their state transition matrices.
Algorithm 1 Structured Linear CDE: The algorithm is presented for dense ttate-trnnsiton matrix Ag. Comments indicate where the structure of Ag can be used to reduce memory footprint and speed-up computation.
Input: w (B, L, d
Output: h (B, L, dn)
1: bo (B, d₂) t- (w(
2: wine (B, I 1,,1,,0 d t diff(w)
3: Ag (dw, dn, dn)) Parameter
P Exploit structure of Aa
4: (ds, dn) + dp da identity matrix.
5: if mode -aarallll then
6: FB
du, d,) + I einsum(bli, (bli, ijk + bljk, wit AA) Broadcast exploit
structure of A
7:
F <- pscan| F))
Parallel associative scan, exploit structure of F
8:
h1:--1 einsum (bjk,, bk: -bjj, Foomp, ho)
9:
h [bo, h1.L-1
10:
else
Recurrent pass
11:
for
/ + 0 L L- 2 do F:: (B, du. dn. - I einsum h ijk + bjk, inn Aa) h4+1 einsum (bjk, i, bbobbbbbg bj, Ft, h,)
12:
b Exploit structure of As Exploit structure of F
13:
14: end for
15: h [bo.-. ... h
Stack along length axis
16: end if
17: return h
B Expressivity
B.1
Introduction
provides mathematical framework for analysing continuous-tine paths X: [0, T Rd. Because it is formulated in continuous time, it can be applied to time series observed irregular sampling times, common situation in real-world applications. A central object the path signature, defined in (51). which is a graded sequence iterated integrals that characterises the path. The signature encodes geometric features such increments and areas, it determines the path up to tree-like equivalence [42). and linear maps of signatures are maximally expressive suitable compact sets of paths in the sense of Definition 3.1 The combination of maximal expressivity and natural grading makes signature coefficients particularly well-suited as feature representations for machine learning tasks involving sequential data (58)
Rough path theory 13
61
34
[[]. These techniques
have experienced substantial growth in popularity and have been successfull fully implemented across diverse domains, with applications spanning deep learning (37 52
70
71
20
45
22
08
49]
5
78 8),
kernel methods [55] 82 81
57
56 63), and quantitative finance 2
83
48
Additionally, cybersecurity
24
74
19
signature methods have proven valuable in information theory
87
23 sepsis
detection (69 225. and computational neuroscience [46], demonstrating their versatility across scientific disciplines. Signature methods are practical thanks to efficient, well-developed packages for computing them 332 68 Some introductory texts to signatures and rough path theory are [15] and [22] This section utilises the tools of rough path theory to characterise the expressivity of the structured linear controlled differential equations introduced in this paper.
22

---

The core of our models is the linear controlled differential equation
d/i Σ A'h,dwy ho E Rdh
(49)
By 22 Proposition B.4], this can be written terms of the signature as
b((A), hos wi) == he
Σ
(A' ho) S' (w))),
(50)
IEWAW
where Wa, the of words in the alphabet [[dw]] := fI...., , d (i.e. Wa =n22olddw1*
and for given word = ii...in in. St (o) referrs to the /th component of the signature tensor (o.)..
(') =
dw ..w...
(51)
211 11---C C [1.
is evident from (50) that any linear readout of h2 expressed as a series in signature terms. Conse- quently, such systems are inherently limited to learning functions that are close to these (uniformly convergent) series. Maximal expressivity is thus achieved when any finite linear combination in signature terms can be approximated by linear readout on h, through appropriate choices of the matrices A
Definition B.1. Fix set of paths X C 1]; RR). We say that a sequence (AN, Hn ) NEN, where H S RN and AN C RRNN, achieves maximal expressivity for X whenever for any positive tolerance O and any finite linear combination coefficients a T(RA), there exists a choice of parameters v, (A'), ho in some R.,, AN, HN in the sequence, such that v h((A'), ho, w). is uniformly close to (ax, ((w))o,))) up to an error of
VE > 0, Va E T(R"), EN 2 O, 3(v, (A*), ho) ERR x A x HN s.t.
Ka, ((w) v h(((A)), ho, <c. (xx1)exX(00.)
sup
If we are given sequence of probabilitiess PN on A x HN such that > 0. Va E T(R"), it holds that
Ka, ((w)) v. =Th((A'), hos w)el < <c==1 =1, (52)
lim PN N00
3u E
sup (..))
then we say that (AN, Hv. w. Px) Nen achieves maximal probabilistic expressivity for X.
deterministic argument by demonstrates the existence of specific choice of AN, Hy that mimics the algebraic structure of tensors and provides maximal expressivity for compact sets of paths. Furthermore, Cirone et 222 Theorem 1.13] established that matrices (almost) replicating algebraic structure of tensors are, in fact, abundant, They showed that the triplet (R.x R.V IPx),
where x a Gaussian measure achieves maximal probabilistic expressivity for compact sets.
The result in [22) implies that for dense matrices A', if the hidden dimension N is sufficiently large, there is significant abundance of parameters capable of achieving uniformly and arbitrarily low error rates. These parameters should therefore be readily discoverable through standard optimisation methods. Unfortunately, discussed Section 2.2 using dense matrices is infeasible in practice due to computational constraints. In this section, we present three alternative choices of parameters that lead maximal probabilistic expressivity for compact sets. These alternatives offer better computational properties compared to the naive use of dense matrices.
B.2 Sparse Matrices
Proposition B.2. The sequence of triplets NNN JRN, RN,FP) where IN is such that
* the initial value has independent standard Gaussian entries [hola ~ N/ 1),
23

---

the weight matrices are distributed as A lid W ④ with W and B independent
VNDN
matrices having entries [W],,8 lid (0,01) 1) and [B] B 3
lid Ber(pN),
the sparsity parameter pn satisfies NpN
N co.
achieves maximal probabilistic expressivity for compact sets.
Proof. Following Cirone et al. 22 Section B.3.5), we only need to prove bound of type
1 N
(Arho, Asho) 81,J (P((N)
2 (x(1Z++ WJD)!! O
(53)
as in the dense Gaussian case. That such sparse matrices present the same bounds as dense Gaussian ones follows from (21 Section 6.2], where is shown that the bounds only differ by correction
term of order 1,,J where the constants are bounded by the number of pairings of I U J U/U J.
Remark B.3, Following Cirone al. 21 Section 6.1], it is possible to prove that W can be taken as having i..dd entries from a centred, symmetric but heavy tailed distribution given finiteness of even moments.
B.3 Walsh-Hadamard Matrices
Proposition B.4. The sequence of triplets (AN, RR,, FN) where AN and PN are such that
AN [Wdiag( A) AERY, W WERNNN WWT IN).
the initial value has independent standard Gaussian entries [hola idd N 1).
the weight matrices are distributed as A ld 1 (diig(A) for a fixed H E RNN satisfying HHT NN and having entries bounded uniformly in N by a constant C, and RERN having entries [A]a iid 1),
VN
achieves maximal probabilistic expressivity for compact sets. In particular one can choose H to be Halls-Haddmmad matrix of order N for computational efficiency.
Proof. Following Cirone al.
22 Section B.3.5), we only need to prove bound of type
N (Ayho, Asho)ay 81,5
S (*([Z++1))))
(54)
VN
12(PN)
We will place ourselves in the graphical setting of 21 and leverage the fact that ((21) Section 7.1)) their results and techniques hold even when the vertices are fixed to random vectors.
The first step is to notice that for R_ one has the equivalence W Wdiag(A) () W (A Ox)
which can be represented graphically as in Figure
diag(a))
Figure Graphical representations of the matrix W Wdiag(A) A). Here edges correspond to identity matrices.
This leads to the product graph representation Gr,, for 1 (Ayho, Asho)gx where /= 1 ... in and N ji Im. given in Figure which allows to use [2 Proposition 1] obtain the bounds.

---

Aii
A
N (Ayhi Ayho)
N ho
11
H
H
Figure The product Y (Arho, Asho)RN as a product graph G.
In the present setting, the vertices labelled ho must be identified, and the remaining decorated vertices must be paired such that A. is identified with As only if a b. Each of these admissible pairings é produces a graph (GI,s) in which each vertex is assigned the vector of ones. Note that since |o < C, one has ok < C The procedure is shown in Figure for all pairings when =1.
ho
11
ho
h0
ho
ho
kp
hd
HI
1I
hg
1g
11
11
11 ho
Ag ho
Figure 4: Construction of (G,,J) )r Here, we display all the pairings, represented by the red dashed lines, for along with their intermediate stages. For simplicity, we omit the H labels from edges with arrows.
Under these boundedness assumptions, note how leading-order pairings of Gy.J are the ones having the maximum number surviving vertices, meaning having |++|J| + vertices. This can happen iff I J. and even in this case, there exists only one pairing (the middle one in Figure 4 for which holds that (GI,)) & N aud /1/1+1 1. To see this, note that only the "middle" vertex of G1,J is
1++|11
all other vertices have to be identified in couples. This implies that the left and right adjacent vertices (call them and vy) must be paired together: in fact the sub-graph comprising the middle vertex and the two adjacent edges corresponds the matrix H NIN. hence we can remove this sub-graph, identify the adjacent vertices and take the factor N in front without changing the value of the graph; but then if t and U, were paired with other vertices we would identify at least vertices. Hence i has to be equal to ýi and i and v, must be identified. Proceeding by induction, from the middle out, we see that only the identity pairing has non-vanishing value.
paired, to get
To conclude we left prove that any pairing is of GI,J GI,J not inducing the identity one on any of the two copies produces (Gr,J Gr,)) 1 S A. since the number of these pairings less than the total number of pairings the A-labelled vertices of GJ,J LGG,J, which is (2( + !J])!! Then the inequality would hold with x 2 since from the definition of product-graph, we see that the bound
[GI,J UGI,))
N ((()++))
(55)
N2 N2 N
25

---

must hold, where Vi is the number of vertices in (GI,J LI.))) Thus it suffices to show Vi. 1
[[] +
To see this, notice that any case V₂ - + and that V = + []] + 2 iff & identifies the random vertices in pairs. Once again there are special vertices which are not paired, the "middle" ones, sub-graphs containing them correspond to the matrix НH
NIn and can thus be removed by identifying the adjacent vertices and taking the N factor out, these vertices must be paired between themselves, and so on. This shows that v has to separately pair both copies of GI.J in G1,J LJ G1,J with identity pairings, but then such v would not be an atom-free pairing! Hence such a & cannot exist and V I | + |J|
always holds.
Remark B.5. Just as in the sparse case, and following 21 Section 6.1], it possible to prove that the A matrices can be taken to have i.i.d. entries drawn from centered, symmetric, but possibly heavy-tailed distribution, provided that the even moments are finite. This distributional adjustment is useful technique for controlling the cigenvalue distribution of Hiigg A), ensuring it has favourable computational properties while providing theoretical guarantee of preserving expressive power.
B.4 Block Diagonal
Proposition B.6. The sequence of triplets (An, RRN, PN) where An and PN are such that
Av == (BlockDiag(AR B1, B2,..., BEx) bn = [log(N)], = [N/bx]; Br. b xbN
the initial value has independent standard Gaussian entries [hola lid NO 1),
lid ~ the weight matrices are distributed as [Blaa,
Vox" 1N(0, 1),
achieves maximal probabilistic expressivity for compact sets.
Proof. Following Cirone et al. Section B.3.5], we only need to prove bound type
1
N (Aiho, Asho)gN 81,1
$ ( o
(56)
VN
1.(PN)
suffices to notice that
1 ky
(Atho, Asho)RN
Σ bn
(B/ hod Bi hoa)
N
AN
where hog == (1 11b.. since we then know that
(Bi
ho.,,
B hoa) R.N - 61 (P(PN)
I {(<(7)+1)))
(57)
bw
VbN
From the sum and the initial factor EN we obtain
1
1 VbN
(Atho, Ayho)an - 61,5
S + O
(58)
N
L2(PN)
C Log Linear Neural Controlled Differential Equations
Combining NCDEs with the Log-ODE method has been shown to produce state-of-the-art perfor-
mance on range of multivariate time series modelling benchmarks 12 70 98 The same approach
can be applied to Linear NCDEs (LNCDEs), and we refer to this approach as Log-L.CDES. For full introduction to the Log-NCDE method see [ Section 3.2.2] and 98] Here, we briefly outline
26

---

the application of the Log-ODE method to an LNCDE, assuming familiarity with the tensor product C and tensor algebra.
Recall the LNCDE model (4)
du Aghydwss
h4
(59)
where Ag is the linear vector field for each channel and w are the channels of our control path. The log-signature of w on [s, t is
(-1) x on E 2((R5),
(s))s)) = log(1 + xx)=
(60)
where x = (..)) S² (w))s.))..... .). St (w.) is the collection of all signature components
with words of length and L ((RR)) is the free Lie algebra generated by R 14
80] From now, we
consider the log-signatury truncated at level N, which lives in the truncated free Lie algebra _x (IR ) basis for the truncated free Lie algebra is the Hall basis, which consists of up to the (N - 1)th
iterated Lie brackets of the basis of RRg denoted { where the product is the tensor product
41 Let fêx T== N)) denote the Hall basis of the truncated free Lie algebra, where B(dw. N) is
the dimension of the truncated log- signnaatt,ee, and let 0 be the corresponding components of the log-signature.
Since & (Rd) the truncated free Lie algebra, the linear map from the increments of the control to the linear ODE in (59) defined by
1
(61)
extends to a Lie algebra homomorphism acting the log-signature of w defined by
((d,,
A Ag
(62)
with
A A
(63)
for 1 S k: S dw and
1 = A/A A3
(64)
when the basis element és corresponds to the lie bracket of é, and ég 80 The Lie bracket of A and Al carries this sign as we are considering them as vector fields on Rd. Similarly to Log-NCDE, the Log-ODE method is applied to (59) over intervals to = ro <... .m = bn with m < n, such that
p ((dw..)
dh =
Σ
AB
(65)
for s [ri, rt41), where
log(S (w) (w)
B(dw.N)
Seen.
(66)
T1+1 T1
1=1
The approximate solution is given by
B(d
hmys
exp Σ
A
hr
(67)
and applying an associative scan has a reduced I/O cost as only m < n state-transition matrices need
to be materialised in GPU memory, as discussed in Section 4.6
When applying the Log-ODE method to an LNCDE, the Lie brackets are calculated using the prod- uets of the linear vector fields for each channel, as opposed to the forward-mode auto-differentiated Jacobian-vector products of the vector field for Log-NCDEs. This significantly reduces the compu- tational cost. For SLiCEs, the Log-ODE method is most beneficial when the linear ODE produced has the same vector field structure as the original vector fields. This is true for both diagonal and block- Miickadiagonal SLiCEs.
27

---

D Additional experimental details and results
Single runs for all experiments be completed 24GB NVIDIA RTX 4090 GPU in less than 24 hours. We use the following publicly available datasets, libraries, and baseline models:
As Benchmark 666 License: MIT.
URL:
Formal Language Benchmark [3] License: CC-BY-4.0.
URL: httpps://arxiv..rrg///ee/ abs 2200 02098
UEA Multivariate Time Series Classification Archive 4 License: GPL-3.0.
URL: http:a://ww..iineerreseeeaesasse
htt.iiiiiiiiciomomoo...... https://githur com/
S4D 440 License: Apache 2.0.
URL: http://aa..ao..cooo/aaaeaeaaaaas
Mamba (3) License: Apache 2.0.
URL:
URL: DewtaNee,, Delaarouuc 88 100 101 88)
DeltaNet, Gated
License: MIT.
XLSTM http:/SSSMMM/SSSS MLSTM 6 License: Apache 2.0.
URL:
RWKV-7 76 License: Apache 2.0.
URL: URL_ LLL
License: CC-BY-4.0 Log-NCDE [98]
URL: htttps:://a.cuub.com//BBB8
JAX 10
License: Apache 2.0.
URL: https: cot.ou///ooogoog/ejjrr..
PyTorch [73] License: BSD Style, see here http://ppp////tttb....o/pttrr///
tabeLLiense-11or-ffl.
URL: http://pg///tthb...oo/rrrrrr/rra
D.1 The Aş benchmark
The Aş benchmark examines a model's state-tracking ability by asking the model to compose a sequence of even permutations five elements [66]. There are 60 elements in the group, and the task to compose between 3 and 20 elements. Our experiments follow the approach of Merrill et al. [66]. Models are trained using token-tagging loss for 100,000 steps with batch size of 256. For all sequence lengths, a small batch of sequences of length 2 are included at each training step to aid convergence. All models use Adam
[54] with weight decay as their optimiser, and linear warm-up followed by cosine annealing with minimum learning rate of 10 and a maximum learning rate of 10 Additionally, all models use dropout [90] at rate of 0.1 and trainable embedding layer.
The baseline models (Mamba, LSTM, mLSTM, sLSTM. and DeltaProduct) all use a hidden dimension of 1024. LSTM uses direct stacking whereas the other baseline models stacked blocks consisting of a sequence model, a GLU layer (30, and layer normalisation
3 DeltaProduct uses both gating
and negative eigenvalues [101 38 All of the SLiCEs use
wxX wi
(1,
(68)
f,+1 - t;
and 1024 non-zer zero parameters for each A For the diagonal and Walsh- SLiCEs, this corresponds a hidden dimension of 1024, and for the dense SLiCE, this corresponds a hidden dimension of 32. The DPLR SLiCE uses rank of 2, giving a hidden dimension of 205, the block- diagonal SLiCE uses b, 4, giving a hidden dimension of 256, and the diagonal-dense SLiCE uses dense block size of 23, giving hidden dimension of 518. The sparse SLiCE uses hidden dimension of 128 and a sparsity followed by tanh activation function, layer normalisation, and weight regularisation. Due to issues with convergence, Halh-aaaddmad'sss diagonal matrix Dj is parametrised take values between and 1.
$ All SLiCES use stacked blocks consisting of a SLiCE, linear layer
28

---

D.2 Regular language tasks
Delétang et al.
31 introduced four regular language tasks as part of the formal language benchmark:
1. Cycle navigation. Infer the final position of a walk on a cycle starting at the origin. Actions are randomly sampled from "go forward one step". "stay in the same place", and "go backward one step". We use a cycle of length 5, therefore a random guesser will achieve an accuracy of 20%.
2. Even pairs. There are two states in the system and the goal is to determine if there is an equal ("even"" number of transitions between the two states. As each sequence is either "even" or not, random guesser will achieve aceuracy of 50%.
3. Modular arithmetic (no brackets). Performs modular arithmetic without any brackets, i.e. only handling addition and multiplication. We use mod 5 and hence a random guesser will achieve 20%
4. Parity. There are two elements in the system. To determine the parity, the number of the second element is counted to determine if it is even or odd. This can be viewed modular summation with mod 2. A random guesser achieves 50%.
This benchmark tests the ability of models to length generalise on state-tracking tasks, by training models on sequences from length to 40 and evaluating models on sequences from length 40 to 256. Following Beck et al. [6]. all baseline models use two stacked layers. In addition to the hidden dimension of 512 used by Beck et al. [6], we also train the baseline models with a hidden dimension of 128, selecting the value that yields the highest average validation accuracy for each model on each task. For Mamba, which does not support a hidden dimension of 128, we instead choose between 256 and 512 based validation performance. All models are trained using loken-tagging loss for 100,000 steps with a batch size of 256. All models use Adam [54] with weight decay as their optimiser, and linear warm-up followed by cosine annealing with minimum learning rate of 10 and a maximum learning rate of 2 x 10-3, Additionally, all models use dropout [90] with rate of 0.01 and a trainable embedding layer.
Similarly to the AG benchmark, LSTM uses direct stacking whereas the other baseline models use stacked blocks consisting of sequence model, a GLU layer (30], and layer normalisation [3]. The baseline models considered are vanilla DeltaNet [85] 99]. DeltaNet with negative eigenvalues
(DeltaNet[-1,I)) 38 Gated DeltaNet [00,, Gated Protuctoduct with negative eigenvalues and
rank of 2, sLSTM mLSTM 6 XLSTM Mamba (39]
6 RWKV-7 76.. a Transformer [96). S4D S4D(40) and
We consider all SLiCEs on this benehmark except sparse due to the lack of an efficient implementation. All SLiCEs use
Why+: Wi, f5+1 tj
X,,),
(69)
and two stacked blocks consisting of the sequence layer. linear layer followed by a tanh activation function, and layer normalisation. For the diagonal and
SLiCE, we consider hidden dimensions of 128 and 512, corresponding to 128 and 512 non-zero parameters per ttae-transition matrix, respectively. For all other SLiCEs, the number of non-zero parameters in the transttran matrix is fixed at 512. For DPLR we consider ranks of r 1,2, 4, 8, and for Block-diagonn we consider two variants, b. b for all with b = 2, 4, 8, 16, and b, = 1 for i 1, .,.... k 1, and then final dense block byg = for b = 2, 4, 8, 16, referred to as diagonal-dense SLiCE (D-DE-SLiCE). Due to issues with convergence, between -1 and 1.
diagonal matrix D2 is parametrised to take values
Table 4 reports the average validation accuracy for all SLiCES. Although replacing the diagonal SLiCE with Walsh-Hadamard SLiCE improves performance on the As benchmark, it decreases the average validation accuracy on regular language tasks. We hypothesise that this degradation arises from the same factors responsible for the Walsh Hadamard SLiCE's poor performance on SLiCE with a fixed block size, any b; > 1
the Aş length generalisation task. For block-diagone leads to higher average validation accuracy than the diagonal SLiCE, with performance peaking at b; = 4. This provides empirical evidence that, under fixed computational budget (or fixed number of non-zero tate-transition parameters), there exists a trade-off between expressivity and hidden dimension. OPLR-SLiCE cxhibits similar pattern, achieving its highest accuracy at r = 4. Using
29

---

Table 4: Results SLiCEs on formal language tasks. Average and standard deviation of validation accuracy over five runs on the regular language tasks for SLiCEs with diagonal, Haadamard (WH), block-diagonal (BD), diagonal-deme (D-DE), and DPLR structures.
Mod Arith.
Model
Cycle Nav.
Even Pairs
Parity
Average
No Brack.
Diagonald; 122
69.5 = 6.3
100.0 0.0
20.8 + 0.2
89.12 = 18.6
69.9
Diagonala, 42 o012
59.7 = 5.1
100.0
20.9
100.0 0.0
70.2
WHA, 1228
69.7 = 8.8
93.1
20.5 0.3
=0.7=003
58.5
WHd,
35.5 = 1.8
2.5
71.4 12.9
47.3
BDa,,=266 2=2
92.5 = 14.0
7.0
37.7
75.6
=122 b=4
99.8 + 0.2
11.
95.3
83.8
BD 66,, km8
99.9
9.33663
21.4
79.0
BD 1616
97.6 3.5
9..7
22.6
30.800.3
79.9
D DEd,-
61.9 20.4
91.
2.2
97.8 2.9
67.9
D D-DEd, 5500 b-g
81.6 15.0
18.3+1800
15.4
8..7
70.0
D-DEd, 4458, bmm
90.6
90.7
4.2
3.5
73.1
D-DEd 227 16
T3.3 29.4
84.8
0.74
=1.3
85.1
DPLRds 11
46.5 = 26.3
91.1444
25.8
87.8 + 9.5
62.8
DPLR dj r=
53.1 = 14.7
961
9.0
79.7 = 14.4
68.4
DPLRd, r
81.1 16.6
1.0
6
91.0 18.0
85.1
DPLRd, -30
90.1 = 10.2
100.0
60.3 - 19.7
50.7 0.3
75.3
Random
20.0
50.0
20.0
50.0
35.0
diagonal-dense SLiCE enables larger hidden dimensions while maintaining dense block connections. The largest dense block configuration, 5 = 16, attains the joint highest average accuracy along with the r = 4 OPLR-SLiCH The strong performance of OPLR-SLiCE is consistent with that of Gated DeltaProduct model with negative eigenvalues, which serves the gest-performing baseline.
D.3 UEA multivariate time series classification archive
The experiments the UEA multivariate time series classification archive follow the approach of Walker et al. 198, using the same data splits. To mitigate convergence issues, the time series were scaled to the range [-1, 1]. The log signaattssses were scaled down by factor of ten on Heartbeat for all structures except DPLR, and by a factor of one hundred on Heartbeat and Motorlmagery for DPLR. All SLiCE models use the same hyperparameters the Log-NCDE, differing only in that the non-linear vector field is replaced by their respective structured linear vector fields. The block-diagonal structure uses = 4, the diagonal-dense structure uses dense block of dimension 16. the DPLR structure uses rank of 4, and the sparse structure uses a sparsity of 0.1. The Log-ODE method is applied over the same intervals as the Log-NCDE. The flows are composed using an associative parallel scan applied to chunks of size 128, with each chunk processed recurrently. All SLiCEs take w X,.
Table 5 presents a breakdown of the average test accuracies from Table across individual datasets. Figure provides visual representation of the performance of the baseline models, dense LNCDE, diagonal SLiCE, and block-diagonal SLiCE on the UEA-MTSCA benchmark. Although Log-NCDEs improve the average time per training step compared with NCDEs and NRDEs, substantial gap remains relative to S5, S6, LRU, and Mamba. Replacing the non-linear vector field with a block- diagonal linear vector field reduces the average time per training step by factor of 20, bringing within the same order of magnitude as the SSM baselines without degrading the average test accuracy, Using a diagonal linear vector field further decreases the training time to below that of the SSM baselines while maintaining comparable accuracy to the baselines. However, the average test accuracy lower than that of the block-diagonal variant, consistent with its reduced expressivity. Employing dense linear vector field slightly increases run-time and substantially raises GPU memory usage compared with the block-diagonal case. also reduces the average test accuracy, which may indicate over-fitting due to the large number of parameters per state transition. This result suggests that using a diagonal linear vector field may have benefits beyond simply reducing the computational cost of the model.
30

---

Table 5: UEA test accuracy (%) for all models across six datasets. The best-performing model in each column is highlighted in bold, and the second-best is underlined. Lower is better for Average Rank.
Model
EW
EC
HB
MI
SCP1
SCP2
Av. Acc
Av. Rank
BD-SLiCE
86 +
29 +
77 +
53 3
85 +
54 + 8
64.0
3.2
Log-NCDE
+
+
54 6
+
+
1.0
D-DE-SLiCE
86 + 6
27 + 7
7144
55 44
85 44
52 =
63.0
5.7
WH-SLICE
85 6
30 =
7
4
82 8
52 6
62.5
6.7
DPLR-SLiCE
84
28 1
74 6
52 6 6
84
51
62.0
7.0
D-SLiCE
6
27
7
54
84
7.2
LRU
88+3
22
78 +
4
83 83
51
61.7
7.3
S6
17
2
77
+ 9
51
83
50 10
62.0
7.7
S5
81
24
78
+ 6
4
90
51 =
61.8
8.0
S-SLiCE
8
30
73
1 6
48 3
83 2
49 1
61.8
8.2
DE-LNCDE
26 5
50
16
1
50
8.3
NCDE
7514
30
7
1
50
33
80 6
53 1
60.2
8.8
NRDE
84
25 2
73 +
4 6
81
54
60.6
10.3
Mamba
71 16
28 =
7
48 48
81 2
48
58.6
10.8
Accuracy, Speed, and Memory Footprint the UEA-MTSCA
65
BD-SLiCE
64
Log NCDE
Average test accuracy (%)
63
S6
DE LNCDE
62
D-SLICE
LRU
Model family
NRDE
Non-linear CDE
Linear CDE
60
SSM
NCDE
RNN
MAMBA
58
101
102
103
Time per 1000 training steps
Figure 5: Average per-step training time versus average validation accuracy across six multi- variate time-series classification datasets from the UEA-MTSCA. Each point represents model, with circle area proportional average GPU memory usage. We compare four families of models: recurrent neural network (LRU), SSMs (S5, S6, and Mamba), non-linear NCDEs (NCDE, NRDE, and Log-NCDE), and linear NCDEs (Diagonal SLiCE, Block-Diagonal SLiCE, and Dense LNCDE). All test accuracy results except linear NCDEs are from Walker et al. 198. All timing and GPU memory results were re- re-performed on an NVIDIA H100 GPU.
31

---

Table 6: Training time for Worms using a parallel associative scan without applying Log-ODE method. Time per 1,000 training steps (s) for diagonal SLiCE, block-diagona SLiCE, and dense LNCDE on Eigen Worms when parallel associative sean is applied with various chunk sizes. Experiments were performed on an NVIDIA H100, and the batch size is 1.
Parallel Steps
D-SLiCE
BD-SLiCE
DE-LNCDE
None
311.0
374.89
444.68
134.2
326.75
439.51
16
57.50
161.74
257.56
64
31.01
68.59
126.01
256
21.10
30.53
71.54
Table 7: Comparison of training time and GPU memory for Eigen WirenWorms using a parallel associative scan and applying the Log-ODE method. Experiments were performed on an NVIDIA H100 with batch size of 4 for diagonal SLiCE, block-diagonal SLiCE, and dense LNCDE.
Log-ODE Interval
Parallel Steps
Model
Metric
None
128
317.49
23.51
Time 1k steps [s]
12
33.03
10.96
D-SLiCE
1
2.69
2.69
GPU Memory [GB]
2.69
2.69
378.20
48.74
Time / 1 steps [s]
12
46.98
13.24
BD-SLiCE
1
2.69
4.73
GPU Memory [GB]
2.69
2.69
465.94
167.86
Time 1k steps [s]
12
47.95
29.37
DE-LNCDE
35.45
51.84
GPU Memory [GB]
2.69
6.79
To conclude, evaluate how the Log-ODE method and parallel associative influence time per training step and GPU memory for the diagonal SLiCE, dlock-diagonal SLiCE, and dense LNCDE. The SigenWorms dataset is chosen for this comparison, it contains approximately 18,000 observations per time series. Table 6 summarises the effect of applying a parallel associative scan with varying chunk sizes on time per 1,000 training steps without the Log-ODE method. For all three models, increasing the number of parallel steps yields strong reductions in time per training step. The impact of the high I/O costs associated with associative is evident from the diminishing benefit of small number of parallel steps you move from diagonal, through block-diagonal, to dense matrices.
Table 7 compares the time per 1,000 training steps and GPU memory for diagonal SLiCE, block- diagonal SLiCE, and dense LNCDE EigenWorms when using a parallel associative scan and the Log-ODE method. As expected, both GPU memory and run-time increase monotonically for every combination of associative
and Log-ODE method as the model structure transitions from diagonal, through block-diagonal, to dense matrices. The consistent 2.69 GB floor across several configurations likely reflects peak memory usage from fixed operations outside the recurrence. Without the Log-ODE method, the dense LNCDE exhibits high GPU memory consumption, which constrained experiments
batch size of 4. When both the Log-ODE method and parallel associative scan are applied, the diagonal and block-diagonal SLiCE achieve comparable times per training step, indicating that the recurrence contributes less overall computation under these parameter settings. Overall, combining the Log-ODE method with a parallel associative scan reduces the time per training step by 30x for the diagonal and dock-diagonal SLiCE without affecting GPU memory, and by ~ 16x for the dense LNCDE while lowering GPU memory usage by over 5xx.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduced the SLiCEs framework and proved they are theoretically expressive in Section 3 and Appendix [ Section provides a range of empirical results
confirming expressivity and demonstrating the superior performance of SLiCEs relative to number of baseline models. Thorough timing experiments are included in Appendix Do support claims made about efficient computation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Table contains thorough comparison of the various proposed methods, with the balance between efficiency and expressiveness discussed throughout. Section 4.6 also discusses the possible downsides of using parallel associative scans. Finally, Section details the limitation and future work. This highlights that although the theoretical results presented in this paper important steps, they do not represent full theoretical picture of structured, input-dependent state-transition matrices.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
33

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
Formal proof' for all theoretical results are given Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Appendix Contaiins full details on all empirical experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
34

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code to reproduce all experiments can be found at https:://gttuu..... com/ Ben amnjamin-o www.ee//stttuttueanraaaaa linear cdes (PyTorch) and https: com/
Ben Seriin-Waaker/Dee neural cdes (JAX).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
sublic/guides CoodeSubmissionool...
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 //nips /nips.cccpublll. wwwpcppwlcgggp...eeeeeeeeeeoeeee hwttic//guies//eeee http:///eeeaiiiiinoooooooooee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Appendix connnainss full details on all empirical experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Where direct comparisons on accuracy are made, empirical experiments have been repeated using five random seeds and mean and standard deviation reported.
Guidelines:
35

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: A contains full details all empirical experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wurrppp.cccppppc/p/////a
Answer: [Yes]
Justification:
We reviewed the NeurIPS Code of Ethics and complied with what is written
there.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [No]
36

---

Justification: This paper presents work whose goal is to advance the development of ex- pressive and computationally efficient peateencett-sequeeee layers. The proposed SLiCE models contribute to improving the theoretical understanding and practical implementa- tion of state-tracking architectures, which may impact various applications in time-series modelling, large language models, and related areas. While our contributions primarily focus on theoretical advancements and algorithmic efficiency, potential societal implications will depend how these models are applied. However, we do not identify any immediate ethical concerns or societal consequences that require specific discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appendix Iopennixxeoontainn contains full details all empirical experiments.
37

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comdsatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Documentation are provided in the associated Github repositories at https
//github.comm haaaa//aaaaaaaaaeeeettuuuuuttrta -inerer-dd-ieeeree-ee- cdes (PyTorch and https //github. com amin -aaeeee/ooooo---errrll
(JAX).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
38

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
39


Reviews:
[{"review_id": "SdvL1zV71f", "reviewer": "Reviewer_7XXo", "summary": "This paper proposes SLiCEs, a framework that unifies sequence-to-sequence layers that are maximally expressive, computationally efficient, and allow for parallel-in-time computation. This framework ties together existing architectures, such as input-dependent block-diagonal linear RNNs and diagonal-plus-low-rank structure, and new variants based on sparsity and the Walsh-Hadamard transform. The authors show that these structured sequence models have maximal probabilistic expressivity and test them on various tasks, including the A5 and regular language tasks along with datasets from the UEA multivariate time-series classification archive.", "questions": "The paper notes that DPLR, sparse, and WH-LNCDEs are not closed under multiplication, which means that the limiting computational cost per composition is the same as a dense LNCDE. If I understand correctly, this is one of the reasons why diagonal SSMs have been used. In practice, how do the runtimes of diagonal, DPLR, sparse, BD, dense, WH-LNCDEs compare? (e.g., the average time / 1k steps (s) as reported in Table 3).\n\nRelated to the question above, why does log-NCDE take orders of magnitude longer in terms of average time / 1k steps (s) in Table 3?\n\nIn Table 3, I suppose that it was infeasible to train the sparse model due to the lack of an efficient implementation. How does the WH-LNCDE model perform on this set of datasets?\n\nOn lines 219-220, the paper introduces a hybrid strategy, where the Log-ODE method is applied to small intervals, and the resulting outputs are then processed using a parallel associative scan. In practice, how does the choice of interval affect the runtime and memory cost?\n\nRelated to line 298, have you tried applying various regularization techniques to DE-LNCDE? It does seem odd that DE-LNCDE performs worse than BD-LNCDE, almost being on par with D-LNCDE.\n\nFor the formal language tasks, why does WH-LNCDE underperform other models that equally have maximal probabilistic expressivity? Do all models have the same number of parameters? Would increasing the state size of WH-LNCDE help its performance match other models? If so, by how much?\n\nOn lines 274-275, the paper notes that diagonal LNCDE outperforms Mamba since it includes a wider range of eigenvalues. However, the result on Mod Arith. No Brack shows the opposite. Do you have an intuition on why that’s the case?\n\nI wonder how different LNCDE models do on generalizing to longer sequences for the A5 task, as in the regular language task. This experiment is of interest in the community, as shown in [1].\n\n[1] Grazzi, Riccardo, et al. \"Unlocking state-tracking in linear rnns through negative eigenvalues.\" arXiv preprint arXiv:2411.12537 (2024).", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: The submission is technically sound. Under the SLiCEs framework, the authors rigorously show that various existing and novel architectures have maximal probabilistic expressivity. The methods are tested on various state tracking tasks, which are great benchmark datasets to test the expressivity of stateful sequence models. However, there are several points that I think would be helpful if the authors could address (in the Questions section)\n\nClarity: Overall, the paper does a great job introducing relevant concepts and technical details to the readers. The experimental details are clear, with additional details written in the appendix. However, I believe that the abstract and the introduction could be improved to better convey the overall message of the paper. For example, just by looking at the abstract, it is unclear whether SLiCEs is an existing or novel framework. In addition, I think adding a couple of more sentences in the introduction to describe LNCDEs and how exactly SLiCEs extend LNCDEs would make it flow better.\n\nSignificance: Designing expressive stateful recurrent models is of great interest to the ML community. Thus, the paper’s framework that ties together various architectures, along with a couple of novel structures, will greatly help the community. The paper’s empirical results on the superior performance of block-diagonal models over others are also interesting.\n\nOriginality: The framework proposed by the paper deepens our understanding of the existing structured recurrent models. In addition, to the best of my knowledge, utilizing the Walsh-Hadamard structure on state transitions is novel and worth further investigation.", "was_revised": true, "final_justification": "Through the rebuttal process, the authors addressed my main concerns. I thus decided to increase the score accordingly.", "rebuttal": "Thank you very much for your thoughtful review.\n\n## Clarity\n\nWe have redrafted the abstract and introduction to improve the clarity. Key edits include:\n\n1. **Line 1**: \"This work introduces Structured Linear Controlled Differential Equations (SLiCEs), a unifying framework\"\n2. **After Line 19**: \"Linear Neural Controlled Differential Equations (LNCDEs) are a continuous-in-time sequence model where the state-transition matrix, or vector field, depends linearly on the current input. This allows for the multiplicative interactions between the input and hidden state necessary for gating.\"\n3. **After Line 28**: \"SLiCEs replace the dense state-transition matrix of an LNCDE with an efficient structured variant, such as block-diagonal, sparse, Walsh–Hadamard, or DPLR, which maintain the maximal expressivity of dense matrices whilst significantly reducing both the parameter count and computational cost.\"\n4. **Line 30**: \"We introduce Structured Linear Controlled Differential Equations (SLiCEs)\"\n\n## Questions\n\n### Questions 1, 3, and 4\n\nWe now evaluate all SLiCEs on the UEA benchmark. The D-DE and DPLR structure use a dense block of dimension $16$ and a rank of $4$, respectively, as these were the best-performing choices on the formal-language benchmarks. BD-LNCDE is the best-performing, followed by Log-NCDE, DE-LNCDE, and WH-LNCDE. We find that D-DE, S, and DPLR LNCDEs underperform S5, S6, LRU, and a D-LNCDE. Together with the formal-language tasks, these findings indicate that BD provides practical benefits relative to the other SLiCE structures considered. In particular, we believe the connection with multi-head attention (Lines 169-170) is important, and have redrafted our paper to expand this discussion.\n\n| Model  | EW   | EC | HB  | MI  | SCP1  | SCP2  | Av. Acc  | Av. Rank |\n| ---------- | -------------- | -------------- | -------------- | -------------- | -------------- | -------------- | -------- | -------- |\n| BD-LNCDE | 86.7 ± 4.1     | 28.6 ± 6.4     | 75.2 ± 6.0     | **58.3 ± 4.1** | 84.9 ± 1.9   | 53.3 ± 7.5   | **64.5** | **3.5**  |\n| Log-NCDE | 85.6 ± 5.1     | **34.4 ± 6.4** | 75.2 ± 4.6     | 53.7 ± 5.3     | 83.1 ± 2.8     | **53.7 ± 4.1** | 64.3   | 3.8    |\n| DE-LNCDE | 87.8 ± 5.8   | 25.6 ± 3.7     | 76.1 ± 3.1     | 54.0 ± 5.3   | 83.5 ± 6.6     | 46.3 ± 3.6     | 62.2     | 6.0      |\n| WH-LNCDE | 83.9 ± 7.3     | 29.4 ± 4.0     | 75.2 ± 5.7     | 50.5 ± 4.2     | 82.4 ± 2.1     | 51.2 ± 4.7     | 62.1     | 6.5      |\n| S6  | 85.0 ± 16.1    | 26.4 ± 6.4     | 76.5 ± 8.3     | 51.3 ± 4.7     | 82.8 ± 2.7     | 49.9 ± 9.5     | 62.0     | 7.0      |\n| LRU | 87.8 ± 2.8   | 21.5 ± 2.1     | **78.4 ± 6.7** | 48.4 ± 5.0     | 82.6 ± 3.4     | 51.2 ± 3.6     | 61.7     | 7.0      |\n| D-LNCDE | 80.0 ± 5.4     | 25.8 ± 4.0     | 72.9 ± 5.0     | 54.0 ± 7.3   | 83.5 ± 2.1     | 53.0 ± 5.8     | 61.6     | 7.2      |\n| S5  | 81.1 ± 3.7     | 24.1 ± 4.3     | 77.7 ± 5.5   | 47.7 ± 5.5     | **89.9 ± 4.6** | 50.5 ± 2.6     | 61.8     | 7.8      |\n| D-DE-LNCDE | **88.9 ± 6.1** | 27.6 ± 2.9     | 74.8 ± 3.3     | 47.7 ± 2.8     | 81.6 ± 4.9     | 50.9 ± 4.6     | 61.9     | 8.0      |\n| S-LNCDE | 82.2 ± 2.8     | 29.1 ± 6.6     | 73.2 ± 4.7     | 50.5 ± 3.5     | 81.4 ± 0.9     | 52.3 ± 4.1     | 61.5     | 8.3      |\n| DPLR-LNCDE | 77.8 ± 6.8     | 28.1 ± 3.7     | 73.9 ± 5.1     | 54.0 ± 6.4   | 82.8 ± 4.6     | 45.6 ± 5.0     | 60.4     | 8.3      |\n| NCDE | 75.0 ± 3.9     | 29.9 ± 6.5   | 73.9 ± 2.6     | 49.5 ± 2.8     | 79.8 ± 5.6     | 53.0 ± 2.8     | 60.2     | 8.7      |\n| NRDE | 83.9 ± 7.3     | 25.3 ± 1.8     | 72.9 ± 4.8     | 47.0 ± 5.7     | 80.9 ± 2.5     | **53.7 ± 6.9** | 60.6     | 9.8      |\n| MAMBA | 70.9 ± 15.8    | 27.9 ± 4.5     | 76.2 ± 3.8     | 47.7 ± 4.5     | 80.7 ± 1.4     | 48.2 ± 3.9     | 58.6     | 10.5     |\n*UEA test accuracy*\n\nAdditionally, we have run a number of experiments comparing the time per $1000$ training steps for the different SLiCE structures on EigenWorms ($\\approx 18{,}000$ observations). When computing recurrently, going from DE to D, BD, or WH reduces the run-time. Note DPLR actually has a longer runtime than DE. This is likely due to our implementation incurring the computational overhead of three separate operations, which outweighs the benefit of the reduced number of FLOPS. For the D and BD LNCDE, applying the parallel associative scan strictly decreases run time, as expected. Although the same is theoretically true for DE-LNCDE, the I/O costs are magnified for DE matrices, making the associative scan slower than recurrent computation when using a small number of parallel steps. Since WH and DPLR are treated as DE when applying a parallel associative scan, they follow the same behaviour. We did not include S in the comparison as it is always treated as DE.\n\n| Parallel Steps | D-LNCDE | BD-LNCDE | WH-LNCDE | DPLR-LNCDE | DE-LNCDE |\n| -------------: | ------: | -------: | -------: | ---------: | -------: |\n|   None |  213.45 |   226.59 |   242.49 |     298.67 |   279.31 |\n|      4 |   92.87 |   179.94 |   396.25 |     399.98 |   406.56 |\n|     16 |   38.56 |    91.15 |   245.04 |     245.55 |   244.60 |\n|     64 |   19.48 |    39.53 |   124.75 |     123.84 |   123.73 |\n|    256 |   11.15 |    19.98 |    95.86 |      94.66 |    90.46 |\n\n*Training time per 1 k steps (s)*\n\nWe also compare runtime and GPU memory on EigenWorms when using the Log-ODE method. See our response to reviewer EATe for a theoretical computational cost analysis. The additional memory for BD-LNCDE is negligible. However, for WH-LNCDE, where the matrices are treated as dense, the Log-ODE method incurs significant additional memory. For both models, the Log-ODE method leads to a significant decrease in runtime. A parallel associative scan leads to further runtime reductions for the BD-LNCDE. However, for the WH-LNCDE, the parallel associative scan incurs too high a GPU memory cost when not using the Log-ODE method, and is slower than recurrent calculation when using the Log-ODE method. This is due to the high I/O costs when using dense matrices.\n\n| Metric                    | Log-ODE Interval | Recurrent | Parallel steps 128 |\n| ------------------------- | ---------------- | ---------: | -----------: |\n| **Time / 1 k steps [s]** | 1                |     227.26 |        52.58 |\n|                           | 12               |      24.66 |        13.40 |\n| **GPU Memory [GB]**      | 1                |       3.48 |         7.59 |\n|                           | 12               |       3.48 |         3.49 |\n\n*BD-LNCDE*\n\n| Metric                    | Log-ODE Interval | Recurrent | Parallel steps 128 |\n| ------------------------- | ---------------- | ---------: | -----------: |\n| **Time / 1 k steps [s]** | 1                |     450.41 |          OOM |\n|                           | 12               |      55.46 |       135.52 |\n| **GPU Memory [GB]**      | 1                |       3.48 |          OOM |\n|                           | 12               |      13.72 |        18.58 |\n\n*WH-LNCDE*\n\n### Question 2\n\nNon-linear NCDEs numerically solve a non-linear differential equation, which is costly and recurrent.\n\n### Question 5\n\nAll SLiCEs use weight regularisation on their state-transition matrices. The paper did not make this explicit and we apologise for this oversight; the main body and Appendix now discuss it. In addition, due to training-stability issues, we constrained the entries of the Walsh–Hadamard LNCDE’s diagonal matrix to be in the range $[-1,1]$ (Lines 808-809, 856-857), restricting the eigenvalues of the state-transition matrix to modulus $\\leq 1$. A broader investigation of regularisation techniques is important future work but sits outside the scope of this paper.\n\n### Question 6\n\nAll SLiCEs were initially evaluated on the formal-language tasks using the same number of non-zero parameters ($512$) in their state-transition matrices (Line 264). This yields a wide range of total parameter counts, owing to linear layers between SLiCE layers and varying hidden dimension with varying block sizes/ranks. To mitigate hidden-dimension effects, the WH, D-LNCDE, and all baselines were also run with hidden dimension $128$, with the best choice selected for each dataset (Lines 259, 265, 835, 851). We have clarified this motivation. In practice, WH-LNCDE performed better on some tasks with hidden dimension $128$ (cycle navigation, even pairs) and others with $512$ (modular arithmetic without brackets, parity). Overall average validation accuracies were $44.9$ (dim 128) and $45.2$ (dim 512).\n\n### Question 7\n\nWe have added a detailed comparison between D-LNCDEs and Mamba. See our response to reviewer oajU under the heading \"Clarity\" for full details. In brief, Mamba combines the S6 recurrence, a convolution, and a gated MLP, whereas LNCDEs are a recurrence followed by a non-linear channel mixing. We added S6 as a baseline on the formal-language tasks; it fails to solve modular arithmetic without brackets, suggesting that Mamba’s additional components are what enable performance above random guessing.\n\n| Model | Cycle Nav. |  Even Pairs | Mod Arith.  |     Parity | Average |\n| ----- | ---------: | ----------: | ---------------------: | ---------: | ------: |\n| S6    | 33.9 ± 1.0 | 100.0 ± 0.0 |  21.4 ± 1.2 | 50.8 ± 0.4 |    51.5 |\n\n### Question 8\n\nWe are adding a length-generalisation experiment on the A5 benchmark. Models are trained on sequences of lengths 3–40, with early stopping on a validation set of sequences of lengths 40–128. Preliminary results show nearly 90 % token accuracy for sequences up to 16 × longer than those seen in training.\n\n| Model    |     40 |     80 |   160 |   320 |   640 |  1280 |  2560 |\n| -------- | -----: | -----: | ----: | ----: | ----: | ----: | ----: |\n| S-LNCDE  | 100.00 | 100.00 | 99.95 | 99.06 | 94.33 | 78.32 | 47.60 |\n| DE-LNCDE |  99.99 |  99.98 | 99.99 | 99.99 | 91.86 | 53.52 | 29.27 |\n| BD-LNCDE | 100.00 | 100.00 | 99.49 | 95.95 | 89.41 | 70.07 | 39.06 |"}, {"review_id": "N6otTY5aNz", "reviewer": "Reviewer_g3ZB", "summary": "Technically dense paper. The denseness of the paper makes it slightly unapproachable for a non-expert. The paper introduces SLiCEs which are “Structured Linear Controlled Differential Equations”. The paper theoretically and experimentally shows the block-diagonal, sparse, and Walsh-hadamard SLiCEs are maximally probabilistically expressive. Empirically the paper validates many of its theoretical claims.", "questions": "1.How does performance degrade when mapping non-power-of-two dimensions before and after the transform? WHT requires a dimension which is a power-of-two – but curious to see performance outside of that. Please include results on one formal-language and one UEA dataset. \n2. Could you evaluate BD-LNCDE and Log-NCDE on a synthetically irregularly subsampled version of one UEA dataset? Report accuracy and runtime (this would increase eval)\n3. I fear many readers might find this paper technically dense, could you add a toy example to help readers get through the denser portions of the paper?", "limitations": "Yes albeit scattered throughout the paper. Would always appreciate a dedicated section.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: The paper is theoretically rigorous and then provides decently robust empirical analysis. \n\nClarity: The paper did a good job of framing the role of SLiCEs as well as doing a good job with their figures. However the paper’s main weakness is the approachability. The paper could seriously benefit from some running toy examples to help cut through all the formalism.\n\nSignificance: The paper seems reasonably significant, both their theoretical results and practical results are good (20x speed up on per step time).\n\nOriginality: The sparse and Walsh SLiCEs and taxonomy seem quite novel.", "was_revised": true, "final_justification": "The authors did a great job assessing any concerns I had. I am still not extremely confident in the material - so I would be open to changing my score if a technical rebuttal was raised -- but for now I am happy with the paper.", "rebuttal": "Thank you very much for your detailed review.\n\n## Clarity\n\nWe have redrafted the paper to improve the clarity. In particular, we have included the following two new lines in the introduction:\n\n1. **After Line 19**: \"Linear Neural Controlled Differential Equations (LNCDEs) are a continuous-in-time sequence model where the state-transition matrix, or vector field, depends linearly on the current input. This allows for the multiplicative interactions between the input and hidden state necessary for gating.\"\n2. **After Line 28**: \"SLiCEs replace the dense state-transition matrix of an LNCDE with an efficient structured variant, such as block-diagonal, sparse, Walsh–Hadamard, or DPLR, which maintain the maximal expressivity of dense matrices whilst significantly reducing both the parameter count and computational cost.\"\n\nAdditionally, we have introduced a new chapter in the Appendix, which serves as a broad introduction to neural controlled differential equations. Topics covered include:\n\n* **Paths**, including bounded variation and discretisation/interpolation,\n* **Solving controlled differential equations**, including an expanded discussion on the discretised solve used in equation (2) (Lines 67-68),\n* **The signature and log-signature of a path**, which are crucial tools in the proofs of Theorems 4.1, 4.2, and 4.3 and the Log-ODE method.\n\nFurthermore, we have expanded our comparison to S6 and Mamba, by adding the following paragraph as Section 4.7 in the paper.\n\n### Comparison to Existing Architectures\n\nSLiCEs are closely related to S6, the recurrent component of Mamba. In particular, if you remove the bias term (i.e. $B = 0$), then the continuous form of S6 is given by a LNCDE,\n\n$$\ndh_t = Dh_t \\mathrm{d}\\omega^{X}_t,\n$$\n\nwhere\n\n$$\n\\omega^{X}_t = \\int_0^t \\sigma\\bigl(\\alpha \\cdot X_s + \\beta\\bigr)\\mathrm{d}s,\n$$\n\nand the trainable parameters are $\\alpha \\in \\mathbb{R}^d$, $\\beta \\in \\mathbb{R}$, and diagonal $D \\in \\mathbb{R}^{d_h \\times d_h}$ [1]. It is possible to include the bias term in the Linear NCDE formulation, as shown in [1], but S6’s expressivity still remains limited. Furthermore, the block-diagonal, DPLR, Walsh–Hadamard, and Sparse SLiCEs are maximally expressive without a bias term, so it is not included in the SLiCE architecture. The other key difference with the S6 architecture is considering more general forms for the driving term $\\omega^{X}_t$. In particular, all the experiments in this paper use\n\n$$\n\\frac{\\omega^X_{t_{j+1}}-\\omega^X_{t_j}}{t_{j+1}-t_j} = (1, X_{t_{j}}).\n$$\n\nIn contrast to models such as Mamba, which combines the S6 recurrence with a convolution and gated MLP, each SLiCE recurrence is combined with only:\n\n* a linear layer to mix the channels,\n* a tanh activation function,\n* layer normalisation, and\n* a skip connection.\n\n## Toy Example\n\nWe appreciate your recommendation for a toy example, and believe this will significantly benefit the approachability of the paper. Therefore, we have added the following motivating example as a demonstration of the difference in expressivity between models which do not mix their hidden-state channels (i.e. models with diagonal state-transition matrices) and models which do mix their hidden-state channels.\n\nGiven a stream of bits\n\n$$\nx_1, x_2, \\dots , x_n \\in \\\\{0,1\\\\},\n$$\n\nwe want to predict the parity label defined by\n\n$$\np_n = S_n \\bmod 2 \\in \\\\{0,1\\\\},\n$$\n\nwhere\n\n$$\nS_n = \\sum_{k=1}^n x_k .\n$$\n\nWhenever a new bit is \\$1\\$ the label flips; if the bit is \\$0\\$ the label stays the same.\nIf we consider a diagonal LNCDE with a hidden dimension of \\$2\\$ and \\$\\omega^x\\_{k+1}-\\omega^x\\_k = x\\_{k+1}\\$, then\n\n$$\nh_{n+1}\n  = \\exp\\left(\n      \\begin{bmatrix}\n        a_1 & 0 \\\\\\\\\n        0   & a_2\n      \\end{bmatrix}\n      x_{n+1}\n    \\right)h_n,\n$$\n\n\nand\n\n$$\nh_n^{i} = h_0^{i}\\exp\\bigl(a_i S_n\\bigr), \\qquad i = 1,2 .\n$$\n\nWith a linear read-out \\$r=(r\\_1,r\\_2)^\\top\\$ followed by a monotone activation \\$\\phi\\$ (such as \\$\\tanh\\$, ReLU, sigmoid):\n\n$$\n\\hat p_n\n  = \\phi\\Bigl(r^\\top h_n\\Bigr)\n  = \\phi\\Bigl(c_1 e^{a_1 S_n} + c_2 e^{a_2 S_n}\\Bigr)\n  = \\phi\\Bigl(f(S_n)\\Bigr),\\qquad\n  c_i = r_i h_0^{\\,i}.\n$$\n\nSince \\$f(S)\\$ has at most one turning point and \\$\\phi\\$ is monotone, \\$\\hat{p}\\_n\\$ can cross any chosen threshold at most twice. However, the true label \\$p\\_n\\$ flips every time \\$S\\_n \\mapsto S\\_n + 1\\$. Hence, no diagonal \\$2 \\times 2\\$ LNCDE can realise parity on arbitrarily long input.\n\n---\n\nHowever, if you replace \\$A\\$ with\n\n$$\nA =\n\\begin{pmatrix}\n0 &  \\pi \\\\\\\\\n-\\pi & 0\n\\end{pmatrix},\n$$\n\nthen\n\n$$\n\\exp(A x_{n+1})\n  = \\begin{cases}\n      \\begin{pmatrix}\n        1 & 0 \\\\\\\\\n        0 & 1\n      \\end{pmatrix}, & \\text{if } x_{n+1}=0, \\\\\\\\\n      \\begin{pmatrix}\n       -1 & 0 \\\\\\\\\n        0 & -1\n      \\end{pmatrix}, & \\text{if } x_{n+1}=1.\n    \\end{cases}\n$$\n\nThus\n\n$$\nh_{n+1} = (-1)^{x_{n+1}} h_n\n\\quad\\Longrightarrow\\quad\nh_n = (-1)^{S_n} h_0 .\n$$\n\nTaking \\$r=(1,0)^\\top\\$, \\$h\\_0^{(1)}=1\\$, and \\$\\phi(s)=\\bigl(1-\\operatorname{sign}(s)\\bigr)/2\\$,\n\n$$\n\\hat p_n\n  = \\frac{1 - \\operatorname{sign}\\bigl((-1)^{S_n}\\bigr)}{2}\n  = S_n \\bmod 2\n  = p_n.\n$$\n\nHence, a dense \\$2 \\times 2\\$ LNCDE can solve parity exactly with a hidden dimension of 2.\n\n---\n\n## Questions\n\n1. Both the DPLR and diagonal-dense LNCDE use exclusively non-power-of-two dimensions for their hidden state on the regular-language tasks, and we do not see a major degradation in performance. In particular, the diagonal-dense LNCDE with a hidden dimension of \\$272\\$ achieves the highest average validation accuracy on the formal-language benchmark. For the UEA benchmarks, we specifically chose to match the hidden dimensions selected via the hyper-parameter grid search in \\[1] in order to isolate the impact of the SLiCE structure from alternative hyper-parameter choices.\n\n2. Results for S6, Log-NCDE, and BD-LNCDE on a synthetically irregularly subsampled version of EigenWorms (70 % of the samples dropped):\n\n   | Model    | EigenWorms (original) | EigenWorms (70 % dropped) |\n   | -------- | --------------------: | ------------------------: |\n   | S6       |     \\$85.0 \\pm 16.1\\$ |         \\$76.3 \\pm 11.8\\$ |\n   | Log-NCDE |      \\$85.6 \\pm 5.1\\$ |          \\$68.9 \\pm 6.4\\$ |\n   | BD-LNCDE |      \\$86.7 \\pm 4.1\\$ |          \\$85.0 \\pm 2.8\\$ |\n\n   *Table 1: Comparison of S6, Log-NCDE, and BD-LNCDE on irregularly subsampled EigenWorms.*\n\n   In addition to these results, we have significantly expanded our analysis of SLiCEs on the UEA datasets, including runtime comparisons. See our response to reviewer 7XXo for more details.\n\n3. We have added the toy example outlined in the previous section.\n\n---\n\n## Limitations\n\nWe have added a dedicated Limitations section to bring together our discussion of the constraints of this work in one place.\n\n---\n\n[1] Benjamin Walker, Andrew D. McLeod, Tiexin Qin, Yichuan Cheng, Haoliang Li, and Terry Lyons. *Log Neural Controlled Differential Equations: The Lie Brackets Make a Difference.* International Conference on Machine Learning, 2024."}, {"review_id": "KjmiWET6tB", "reviewer": "Reviewer_oajU", "summary": "This paper introduces the concept of linear neural control differential equations (LNCDE), which have input-dependent state transition matrices. The paper explores a theoretical framework of many variants of these with varying structures on their transition matrices, including diagonal, diagonal plus low rank (DPLR), block diagonal, and Hadamard, discussing their expressivity properties and efficiency characteristics. These methods are empirically validated on difficult synthetic tasks such as state tracking on the A5 permutation group.", "questions": "Several of my questions are listed above in the weaknesses section.\n\nI think the paper would be more valuable to a broader ML audience if there were more intuitive comparisons to existing methods. For example, a direct contrast between the diagonal LNCDE with Mamba (showing specific equations and computation method, discussing all axes in which they differ, etc)\n\nAdditional minor comment: In several places, it is stated that S4 is a diagonal SSM (e.g. line 148). This is not entirely accurate as the original S4 is DPLR, but the more popularly used S4D is indeed diagonal. They should perhaps be clarified where appropriate.", "limitations": "The work does not discuss limitations too much. It would be useful to contrast the proposed LNCDE methods more carefully to popular models in literature.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper introduces a useful theoretical framework on the expressivity of various sequence models, which is a very timely topic.\n- The paper is technically sound, although it could benefit from unpacking some background and notation more.\n- It introduces some novel families of methods, such as the Walsh-Hadamard variant.\n- The methods have concrete empirical benefits, such as being able to solve difficult state tracking tasks, verifying their increased expressivity over popular recurrent models in the literature.\n\nWeaknesses\n- A number of implementation details are missing. For example, I found it hard to understand how the proposed L and C/D/E layers are incorporated into a neural architecture block outside of the core recurrence; most of the baseline architectures (e.g. Mamba, DeltaNet, Transformer etc.) have slight variations in their structure such as the dimensions, whether there is a multiplicative gate, the head dimensions and sharing structure, and so on.\n- Similarly, I feel that details were missing from the implementation of the core layer. Figure 5 is surprising to me that Diagonal LNCDE is much faster than Mamba and block diagonal/dense are comparable to Mamba. Shouldn't diagonal be the same speed as Mamba since they are the same structure? What is the difference between Mamba and S6 here? Where is the Walsh-Hadamard method, is that the same speed as the Dense? I think the exact nomenclature differences between all these methods and how they relate to the surrounding neural network block structure and the exact dimensions and so on are not clear throughout the manuscript.\n- Empirically, the experiments test sequential tasks that are not the most popular or impactful ones that the related work (such as Mamba, DeltaNet, xLSTM, etc.) care about. In particular, most of those methods were designed for language modeling, which was not evaluated in this paper.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough review.\n\n## Clarity\n\nWe have redrafted the paper to improve the clarity. Changes to the abstract and introduction are detailed in our response to reviewer 7XXo, and details about a new introductory chapter in the Appendix are given in our response to reviewer g3ZB. Here, we focus on the changes we have made to improve the clarity of our model architecture and how it compares to existing models in the literature. We have added the following paragraph as Section 4.7 in the paper.\n\n### Comparison to Existing Architectures\n\nSLiCEs are closely related to S6, the recurrent component of Mamba. In particular, if you remove the bias term (i.e. $B = 0$), then the continuous form of S6 is given by a LNCDE,\n\n$$\ndh_t = Dh_t \\mathrm{d}\\omega^{X}_t,\n$$\n\nwhere\n\n$$\n\\omega^{X}_t = \\int_0^t \\sigma\\bigl(\\alpha \\cdot X_s + \\beta\\bigr)\\mathrm{d}s,\n$$\n\nand the trainable parameters are $\\alpha \\in \\mathbb{R}^d$, $\\beta \\in \\mathbb{R}$, and diagonal $D \\in \\mathbb{R}^{d_h \\times d_h}$ [1]. It is possible to include the bias term in the Linear NCDE formulation, as shown in [1], but S6’s expressivity still remains limited. Furthermore, the block-diagonal, DPLR, Walsh–Hadamard, and Sparse SLiCEs are maximally expressive without a bias term, so it is not included in the SLiCE architecture. The other key difference with the S6 architecture is considering more general forms for the driving term $\\omega^{X}_t$. In particular, all the experiments in this paper use\n\n$$\n\\frac{\\omega^X_{t_{j+1}}-\\omega^X_{t_j}}{t_{j+1}-t_j} = (1, X_{t_{j}}).\n$$\n\nIn contrast to models such as Mamba, which combines the S6 recurrence with a convolution and gated MLP, each SLiCE recurrence is combined with only:\n\n* a linear layer to mix the channels,\n* a tanh activation function,\n* layer normalisation, and\n* a skip connection.\n\n---\n\nIn addition to the above section, we have also included an algorithm outlining how to implement a structured linear CDE in practice.\n\n**Algorithm — Structured Linear CDE:**\n*The algorithm is presented for a dense state-transition matrix $A_{\\theta}$. Comments indicate where the structure of $A_{\\theta}$ can be used to reduce memory footprint and speed-up computation.*\n\n**Input:**  $\\boldsymbol{\\omega}$ : $(B, L, d_\\omega)$  \n**Output:** $\\mathbf{h}$ : $(B, L, d_h)$\n\n1. $\\mathbf{h_0}$ : $(B, d_h)$ $\\leftarrow$ $\\xi_\\phi(\\boldsymbol{\\omega_0})$  \n2. $\\boldsymbol{\\omega}^{\\text{inc}}$ : $(B, L−1, d_\\omega) \\leftarrow \\text{diff}(\\boldsymbol{\\omega})$  \n3. $A_{\\theta}$ : $(d_\\omega, d_h, d_h) \\leftarrow$ *Parameter*  $\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad$ # Exploit structure of $A_\\theta$  \n4. $I$ : $(d_h, d_h) \\leftarrow d_h\\times d_h$ identity matrix  \n\n5. **if** *mode* = parallel **then**  \n6. $\\quad F$ : $(B, L−1, d_h, d_h) \\leftarrow I + \\text{einsum}(bli, ijk \\rightarrow bljk, \\boldsymbol{\\omega}^{\\text{inc}}, A_\\theta)$  # Broadcast $I$, exploit structure of $A_\\theta$  \n7. $\\quad F^{\\text{comp}}$ : $(B, L−1, d_h, d_h) \\leftarrow \\text{pscan}(F)$ $\\qquad\\qquad\\qquad\\qquad\\quad\\ \\ $ # Parallel associative scan, exploit structure of $F$  \n8. $\\quad \\mathbf{h_{1:L-1}}$ : $(B, L-1, d_h) \\leftarrow \\text{einsum}(bljk, bk → blj, F^{\\text{comp}}, \\mathbf{h_0})$  \n9. $\\quad \\mathbf{h}$ : $(B, L, d_h) \\leftarrow [\\mathbf{h_0}, \\mathbf{h_{1:L-1}}]$\n\n10. **else**  $\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\ \\ \\ $ # Recurrent pass  \n11. $\\quad$**for** $t = 0$ **to** $L−2$:  \n12. $\\quad \\quad F_t$ : $(B, d_h, d_h) \\leftarrow I + \\text{einsum}(bi, ijk \\rightarrow bjk,  \\boldsymbol{\\omega_t}^{\\text{inc}}, A_\\theta)$  $\\qquad$ # Exploit structure of $A_\\theta$  \n13. $ \\quad \\quad \\mathbf{h_{t+1}}$ : $(B, d_h) \\leftarrow \\text{einsum}(bjk, bk \\rightarrow bj, F_t, \\mathbf{h_t})$ $\\qquad\\qquad\\quad\\ \\ $ # Exploit structure of $F$  \n14. $\\quad$**end for**  \n15. $\\quad \\mathbf{h}$ : $(B, L, d_h) \\leftarrow [\\mathbf{h_0}, …, \\mathbf{h_{L−1}}]$ $\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\ \\ \\ $ # Stack along length axis  \n16. **end if**\n17. **return** $\\mathbf{h}$\n\n---\n\n## Weaknesses\n\n1. We believe that the comment regarding missing implementation details stem from a misunderstanding of the simplicity of our proposed models. In particular, there are no multiplicative gates, head dimensions, or sharing structure to provide details on. We hope that our proposed additional section clarifies this, and that you now feel the implementation details for our models are sufficiently provided in Appendix D.\n2. We hope that the proposed additional section has also clarified several of your questions here. The speed difference between the diagonal LNCDE and Mamba is partly due to the additional structure surrounding the S6 recurrence, but also due to different hyper-parameter choices, which were selected via a grid search. We have also expanded the results on the UEA dataset to include all of the SLiCE architectures, as detailed in our response to reviewer 7XXo under the heading \"Questions 1, 3, and 4\".\n\nThank you for highlighting the distinction between S4 and S4D. We have updated our paper accordingly. Additionally, we have added a dedicated Limitations section to consolidate our discussion of constraints in one place.\n\n---\n\n[1] Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, and Terry Lyons. *Theoretical foundations of deep selective state-space models.* Advances in Neural Information Processing Systems, 2024."}, {"review_id": "bCtvbWjnD8", "reviewer": "Reviewer_EATe", "summary": "The paper introduces Structured Linear CDEs (SLiCEs), a unified framework that captures and extends various structured state-transition matrices (block-diagonal, sparse, Walsh-Hadamard, etc.) under linear Linear Neural Controlled Differential Equations (Linear NCDEs or LNCDEs), providing asymptotic expressivity guarantees in the infinite-width limit. The authors prove that diagonal LNCDEs are not universal, whereas sparse, block-diagonal and Walsh-Hadamard structures are. Empirically, SLiCEs match dense NCDEs on synthetic benchmarks, confirming the predictions by their theory.", "questions": "- Does the first-order approximation of the exponential in Eq. 13 not compromise numerical stability or accuracy on long intervals compared to higher-order Log-ODEs? The text just refers to \"common practice\" but no further explanation or justification is given for this approximation.\n- Are the S-LNCDEs the only model variant that doesn't have an efficient implementation (and would therefore benefit from dedicated hardware), or do you see potential for further improvement in other variants, too? \n- There is no evaluation of eigenvalue distributions or stability of structured $A_i$ over long horizons, though state-tracking depends on avoiding blow-up or vanishing modes. How is this handled by your theory and in your experiments?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- Introduces SLiCEs as a unified framework to subsume and extend a wide range of structured, input-dependent state-transition matrices (block-diagonal, sparse, Walsh–Hadamard, DPLR, diagonal) under a common LNCDE lens, clarifying connections across prior work and novel variants. \n- Proves maximal probabilistic expressivity for block-diagonal (Thm 4.1), sparse (Thm 4.2), and Walsh–Hadamard (Thm 4.3) structures, matching dense matrices in the limit. This is an interesting and insightful finding for the community to build upon.\n- Evaluates on (1) $A_5$ state-tracking (which clearly delineates expressive and non-expressive SLICEs as predicted by the theory), (2) formal-language length's generalisation, and (3) various multivariate tasks (showing block-diagonal SLiCE matches Log-NCDE accuracy while cutting per-step time by ~20x).\n- Offers a promising route for block-diagonal SLICEs. \n- While the theory and experiments are limited to vector-valued states, a derivation for matrix-valued states is provided. \n\n**Weaknesses**:\n- Theoretical results are only in the limit of infinite hidden dimension, which does not clearly translate to practical implementations with finite hidden dimensions. Besides a computational complexity analysis and results on toy tasks (which are convincing, but still synthetic), there is little guidance (or predictions) for practical applications of these models. \n- Results for the sparse LNCDE variant are not very convincing, as they are only evaluated on the A5 benchmark and not on the other two benchmarks. Moreover, it was only evaluated for a fixed sparsity factor of 3/7 which was chosen arbitrarily (Appendix D.1, no further explanation). There is no discussion on how the sparsity factor influences the expressivity of the resulting S-LNCDE model. A similar analysis to that presented in Appendix D.2 would strengthen these results. It would further be interesting to connect these results to existing work on sparsity in LNCDEs such as [1] on sparse inputs with SSMs and [2] on unstructured sparsity in S5, which relates closely to the proposed S-LNCDE and presents hardware that appears to accelerate unstructured sparsity.\n- For the BD-LNCDE, block size $b$ and rank $r$ critically affect performance (e.g. large $b$ yields poor `Parity` accuracy), yet the analysis is confined to appendix, and no clear practical guidelines are given.\n- In the matrix-valued state derivation, since each column $h^k_t$ follows its own copy of the same LNCDE (plus bias), there is no mechanism for $H$’s columns to interact—mixing between columns is prohibited unless one uses a full $d_hn\\times d_hn$ coupling, which would blow up parameters.\n\nMinor\n- relation of this work to sparsity through pruning and the lottery ticket hypothesis is not quite clear from the related work section. \n- Figure 1's caption refers to a \"gray region\" that is not gray, perhaps better to just call it the \"shaded region\"\n- is Appendix B referenced in the main text?\n- it would be helpful for the reader to also have the Log-LNCDE added to Table 1, especially for the memory-compute tradeoff that is mentioned in Section 5.3.\n\n[1] [Schöne et al, 2024](https://arxiv.org/abs/2411.12603)\n\n[2] [Pierro et al, 2025](https://openreview.net/forum?id=UNrfYfbLZ3)", "was_revised": true, "final_justification": "The additional experimental results are convincing and all my comments and questions have been fully addressed", "rebuttal": "Thank you very much for your comprehensive review.\n\n## Weaknesses\n\n1. We have expanded our experiments on the UEA benchmark, which consists of real-world time series datasets, to include all of the SLiCE structures. Furthermore, we have performed empirical runtime and GPU memory experiments on EigenWorms, the longest dataset considered in this paper, to provide additional guidance when applying these models in practice. For more details, please see the section of our response to reviewer 7XXo titled \"Questions 1, 3, and 4\". We hope these additions partly address your concerns, although we acknowledge that there remains open work on understanding the behaviour of these models at finite hidden dimension and how to choose their hyperparameters.\n2. On the A5 benchmark, all of the SLiCE architectures were chosen to have \\$1024\\$ non-zero parameters per state-transition matrix (Lines 241, 802). We were computationally limited to a hidden dimension of \\$128\\$ for dense matrices, and so chose a hidden dimension of \\$128\\$ and a sparsity factor of \\$3/7\\$ for S-LNCDE, giving \\$1024\\$ non-zero parameters. We have amended the paper to make this decision clearer. We have also added an additional experiment on the A5 benchmark to investigate the impact of sparsity. We trained the S-LNCDE on the length \\$20\\$ A5 dataset and varied the sparsity factor \\$\\epsilon\\$. We find that as \\$\\epsilon\\$ decreases, the number of training steps taken to converge increases, but the model still converges to $100\\\\%$ validation accuracy with only $1\\\\%$ of the parameters being non-zero. Thank you for bringing these two sparse papers to our attention, we will incorporate them into our discussion.\n3. We haved moved part of the discussion on the impact of block size and rank to the main body of the text.\n4. We note that many existing sequence-model architectures that use matrix-valued hidden states, including Gated Linear Attention, RetNet, RWKV6, mLSTM, and Mamba, do not have a method for column interaction in the recurrence. Furthermore, if desired, column mixing can be achieved by right-multiplying by an input-dependent \\$n \\times n\\$ matrix, as is done in DeltaNet, without requiring a full \\$d\\_h n \\times d\\_h n\\$ coupling. We have expanded Appendix C to broaden the comparison with existing matrix-valued hidden-state models.\n\n## Minor\n\n1. We have modified Lines 56–60 to the following (citations removed for brevity):\n“Utilising structured matrices to reduce the computational burden of neural networks extends beyond sequence models. The lottery-ticket hypothesis argues that dense networks contain sparse sub-networks that, when trained in isolation, can match the accuracy of the full model. Such sub-networks have been uncovered by pruning before, during, and after training. SLiCEs differ from pruning by imposing structured sparsity at initialisation and training the resulting sparse model directly. Other structured-matrix approaches include sparse Transformers, 2:4 sparsity in linear layers, and Monarch layers, which factorise weight matrices into two block-diagonal components.”\n\n2. We have amended the caption to “shaded region”.\n\n3. Yes, Appendix B is referenced on Line 218.\n\n4. We have added the following line to **Table 1**:\n\n   | **Model**   | **Parameters**        | **Recurrent Cost**                                     | **Parallel Cost**                                              | \n   | ----------- | --------------------- | ------------------------------------------------------ | -------------------------------------------------------------- | \n   | Log-X-LNCDE | \\$\\mathcal{O}(P\\_X)\\$ | \\$\\mathcal{O}\\left(\\tfrac{R\\_X}{s}d\\_h^{N-1}\\right)\\$ | \\$\\mathcal{O}\\left(\\log\\tfrac{n}{s},C\\_X d\\_h^{N-1}\\right)\\$ |\n\n   *Log-X-LNCDE corresponds to applying the Log-ODE method with intervals containing \\$s\\$ samples and a truncation depth of \\$N\\$, where X corresponds to a specific SLiCE structure (DE, D, DPLR, S, WH, or BD), \\$P\\_X\\$ is the order of parameters for X, \\$R\\_X\\$ is the order of the recurrent cost for X, and \\$C\\_X\\$ is the order of the cost per composition for X.*\n\n   We have also expanded the discussion in Section 5.3 to contrast the theoretical computational cost of the Log-ODE method with the reduction in I/O costs. Empirically, we show that the Log-ODE method leads to practical speed-ups for the BD-LNCDE on the longest dataset we consider, the EigenWorms dataset, which has almost \\$18{,}000\\$ observations per sequence. See the table below for details.\n\n   | Metric                    | Log-ODE Interval | Recurrent | Parallel steps 128 |\n   | ------------------------- | ---------------- | ---------: | -----------: |\n   | **Time / 1 k steps [s]** | 1                |     227.26 |        52.58 |\n   |                           | 12               |      24.66 |        13.40 |\n   | **GPU Memory [GB]**      | 1                |       3.48 |         7.59 |\n   |                           | 12               |       3.48 |         3.49 |\n\n## Questions\n\n1. Yes, although we did not find that first-order approximation compromised training stability in practice. We have expanded the discussion to mention that this is done as a computational cost-saving measure, and we did not find training stability impacted by this choice.\n2. The fast Walsh–Hadamard transform, although supported, would likely benefit from improved software and dedicated hardware. For example, the fastest inference method we are aware of (PyTorch’s Hadacore) does not support back-propagation.\n3. Inspired by the \\$\\mathrm{Lip}(\\gamma)\\$ regularisation introduced in [1], all SLiCEs use weight regularisation on their state-transition matrices. The paper does not make this explicit and we apologise for this oversight. The main body and Appendix now discuss this. The Walsh–Hadamard LNCDE’s training was still sometimes unstable. Therefore, we constrained the range of the diagonal weights to $\\[-1,1]\\$, which means that the eigenvalues of the state-transition matrix have modulus \\$\\leq 1\\$. This was briefly discussed in Appendix D, but we have now included an expanded discussion in Section 4.5.\n\n[1] Benjamin Walker, Andrew D. McLeod, Tiexin Qin, Yichuan Cheng, Haoliang Li, and Terry Lyons. *Log neural controlled differential equations: The lie brackets make a difference.* International Conference on Machine Learning, 2024."}]

Author rebuttal:

Meta-review: All reviewers valued the technical contributions of the SLiCEs framework, how it unifies several existing deep SSM architectures, and the avenues it opens for new designs. While the manuscript is technically challenging for non-experts, the authors have taken steps to make it more approachable. I congratulate the authors on their nice work — it is a valuable and timely contribution to the field.
Decision: Accept (spotlight)