SpectraLDS: Provable Distillation for Linear Dynamical Systems
Authors: Devan Shah
Shlomo Fortgang
Sofiia Druchyna
Elad Hazan
Venue: NeurIPS 2025 poster ()
Keywords: Linear Dynamical Systems
Sequence-to-Sequence Neural Networks
State-Space Models
Primary area: general_machine_learning
TL;DR: distillation of high dimensional LDS to low dimension

Abstract: We present the first provable method for identifying symmetric linear dynamical systems (LDS) with accuracy guarantees that are independent of the system’s state dimension or effective memory. Our approach builds upon recent work that represents symmetric LDSs as convolutions learnable via fixed spectral transformations. We show how to invert this representation—recovering an LDS model from its spectral transform—yielding an end-to-end convex optimization procedure. This distillation preserves predictive accuracy while enabling constant-time and constant-space inference per token, independent of sequence length. We evaluate our method, SpectraLDS, as a component in sequence prediction architectures and demonstrate that accuracy is preserved while inference efficiency is improved on tasks such as language modeling.

SpectraLDS: Provable Distillation for Linear Dynamical Systems
Devan Shah
Shlomo Fortgang'
Sofiia Druchyna'
Elad Hazan 1.2
Computer Science Department, Princeton University Google DeepMind Princeton
Abstract
We present the first provable method for identifying symmetric linear dynamical systems (LDS) with accuracy guarantees that are independent of the systems' state dimension effective memory. Our approach builds upon recent work that repre- sents symmetric LDSs as convolutions Iearnable via fixed spectral transformations. We show how invert this representation, thereby recovering an LDS model from its spectral transform and yielding an end-to-end convex optimization procedure. This distillation preserves predictive accuracy while enabling constant- time and constant- space inference per token, independent of sequence length. We evaluate our method, SpectraL.DS, as component in sequence prediction architectures and demonstrate that accuracy is preserved while inference efficiency is improved on tasks such as language modeling.
Introduction
The emergence attention-barn transformer architectures has revolutionized sequence modeling tasks, particularly in natural language processing [37 and large-scale sequence-to-seqqeece learning These transformer models rely on the self-attention mechanism, which allows each token in sequence attend to every other token, enabling strong contextual understanding. However, this approach suffers from quadratic complexity in sequence length, making it computationally expensive for longer sequences. Recent research has thus explored alternative, more efficient architectures that preserve expressiveness while reducing computational costs for long sequences. Among these are attention-free approaches such as convolution-bass or state-space models (SSMs) 9 22 2 which can offer sub-quadratic or even near-linear time generation.
16 29
The most basic SSM, which is the starting point for all aforementioned models, linear dynamical systems (LDS), a foundational framework for modeling sequential dependencies in control theory, signal processing, and learning theory 20 13]
Ar Bu,
Ú1 = CC, t + Duy-
Here de represents the input sequence, 2; encodes the past state information, and ýr approximates the target sequence. By maintaining fixed latent state, LDSs enable efficient inference and allow for efficient memory utilization. However, gradient-based approaches for learning LDSs suffer from exploding vanishing gradients, particularly when modeling systems with long-term memory large hidden state dimension [14].
To address these limitations, Agarwal et al. 2 leveraged the spectral filtering method [4] and introduced the Spectral Transform Unit (STU), convex relaxation that shifts from learning the hidden
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

transition matrix A directly to reparameterization of the LDS impulse response that learns how input signals convolve with fixed spectral filters. This approach provably preserves the expressiveness of an LDS with symmetric A while making the training problem far more tractable. In practice, it has proven robust for systems requiring long-range memory and empirical results show that hybrid STU models-witt alternating attention and STU layers- match or even surpass purely attention- based architectures, as well as other popular hybrid state-space models, on long-context tasks such as language modeling and time-series prediction [22]. Furthermore, unlike self-attention, which requires O(L") operations per L tokens during training and identically for inference with KV-caching, the STU operates in OL log L) operations per tokens during training and O log" L) operations per L tokens during inference through algorithms based on the Fast Fourier Transform (T (2)..
However, although STUs capture LDS-like dynamics using spectral filtering, there has been no straightforward way to convert (or "distill") a trained STU layer back into explicit LDS form. Moreover, it has remained unclear whether every STU layer can be represented as LDS with provably small error. Such distillation from an STU layer to LDS layer with hidden-dimension h would permit recurrent inference in place of convolution operations, reducing the cost of generating L tokens during inference to O(h L). while maintaining the training robustness guarantees of the STU.
Such a distillation is especially desirable in the light of the resurgence of SSMs as scalable alternatives to Transformers long-context tasks I0 34 Thus, our work allows one retain the STU's performance on long sequences while enabling logarithmic-time per-token generation-provvee an appealing alternative to both long convolutions and Transformer-basee self-attention caching.
1.1 Our Contribution
We present novel technique for distilling STU filters into LDS form, achieving a substantial reduction in operations- frrti Ollog" L) to Ollog L) per toker_ during generation-wiee preserving the STU's expressivity and performance. Moreover, due the training stability guarantees of the STU architecture, even when learning marginally stable symmetric LDS or an LDS with large hidden dimension, this distillation procedure provides the first provable method to directly learn the parameters of symmetric LDS of arbitrarily high effective memory and with bounded noise. Specifically:
In Algorithm show how convert learned STU into explicit system matrix parame- ters whose recurrence can be computed in logarithmic time.
We provide a theoretical analysis of this distillation in Theorem | and empirically demon- strate that the new LDS form incurs negligible degradation in modeling quality while improving the autoregressive generation speed.
We demonstrate in Section 6 that by applying SpectraLDDS to trained STUs, we maintain the same accuracy and attain Ölog I.) per-token computational generation cost.
As a consequence of our theoretical analysis, we show in Section 5.4 how we can efficiently convert a symmetric LDS of arbitrary high dimension, via an intermediate STU learning step, to one of Ollog L) dimension with minimal error.
We open-source the SpectraLDS code at https: //g httpt//...commeeaaa///pepttaaaaa
2 Related Work
State-space models have long been cornerstone of control theory and signal processing. The simplest variants, linear dynamical systems, provide succinct way to capture temporal dependencies via a hidden state that evolves with linear transitions. Classical methods include the Kalman filter 20 which remains widely used due its robust thcoretical properties and computational efficiency.
In recent years, a new wave of SSMs have emerged as efficient alternatives to attention- methods for long-sequence tasks, promising sub-quadratic or even near-linear complexity without
We expect the actual reduction to be much larger: from VI, which is a practical method for generation, as opposed to log L, which is complex algorithm, especially to implement efficiently on GPU.
2

---

model (e.g.. Hyena (4 ) compromising expressive power. Models like S4 [TO] and its diagonal variants [ exploit structured state matrices to learn long-range dependencies, while works such as Hyena, Mega, and Mamba incorporate gating or convolution-based parameterizations to compete against (and sometimes outperform) transformers in language modeling, time-series analysis, and other long- context applications. Their growing popularity and the challenges of training SSMs underscore the need for methods with greater training robustness, efficiency, and performance guarantees. The Spectral Transform Unit (STU) (2) lies squarely in this tradition, offering a powerful convex relaxation for training LDS-like systems that achieves impressive empirical results on long-context tasks. Our work builds directly this line of research, introducing the first method to distill a learned STU layer into an explicit LDS with provable guarantees, thereby unifying the convex training advantages of spectral filtering with the real-time inference benefits of recurrent LDS.
34
24 9
Our contributions also align with the long tradition of system identification for L.DSs, where the aim is to learn the hidden transition and output matrices (A, B, C, D) from observed sequences. Early influential approaches- -uch as the Ho- Kalman algorithm [S]. Kung's method [21], and the
Eigensystem Realization Algorithm (ERA) [8 -rely on linear-algebraic decompositions (e.g., SVD) of Hankel matrices consistent with observations. Modern variants allow for single-trajectory identification [3], and subsequent refinements like MOESP and N4SID [[7] added stochastic noise modeling, while prediction-errol and muximum-likellhoo accuracy and statistical efficiency. More recent lines of work incorporate regularization and spectral methods (e.g., stable-spline kernels, sparse identification) to yield more robust or interpretable LDS representations. In the context of distillation, these methods still require matrix decompositions that scale super-linearly in the problem size.
methods improved estimation
In contrast to many previous approaches, since the STU's parameterization avoids direct recon- struction of the system matrix A, distillation from a STU layer into an LDS remains agnostic to hidden dimension. Moreover, our method uses fixed and abstract Hankel matrix-rather than having to construct anew from observed data- we can perform a significant part of the distillation computation offline.
Finally, we highlight the recent closely related work of (28), which distills state-space-like recurrences from convolution- based sequence models. Their Laughing Hyena method accelerates long convolu- tional filters by approximating them with a diagonal LDS,, thus allowing constant-tine generation at inference. While this approach generalizes to any convolution-base
does not provide formal guarantees on distillation quality. In contrast, we focus on the STU's spectral filters, which have expressive power comparable to symmetric LDS with real eigenvalues, and present the first theoretical framework to convert the filters into such an LDS with provable bounds on distillation quality. By leveraging the STU's fixed bank of spectral filters, our method pre- serves mengeee expressiveness while achieving a symmetric LDS realization with a guaranteed approximation error (see Section 5).
3 Token Generation and Complexity for Language Modeling
In this section, we summarize the autoregressive generation costs for three model classes, Transform- ers, Convolutional Models, and RNNs, considering a prompt length of I and the generation of K tokens by each model, with L the length of the convolutional filters (i.c., the maximum sequence length). We show the runtimes and memory requirements for each of the listed models in Table
Attention.
Processing prompt requires ( time. However, token-by-token generation can be accelerated to O + K) for each generated token via key-value caching, with a total of O + ( + K)) = 0 + K2) operations and requiring O(T K) space (37 28
Convolutional Model. A Convolutional Model with k convolutional filters will require O(kN) operations to autoregressively generate new output given N inputs, and thus a naive autoregressive convolutional implementation will require Okk- K (T + K)) operations to generate K tokens. A more refined "Epoched Future Fill" algorithm with prompt prefilling can reduce this to O(k (T log T + K3/2 V Tog K)) to generate K tokens. The "Continuous Future Fill" algorithm has theoretical guarantees of O(k- (Tlog T + K log" K)) operations, although it suffers from numerical instability and has not been implemented or used in practice (0. For the guarantees of the STU architecture, we require k O(log L) and in practice we choose k 24 22
3

---

RNN (LDS). For an RNN with state dimension h, autoregressive generation requires (h) operations per token generated and (hh) memory, allowing generation of K tokens with O (T + K)) operations. As we will prove, for an LDS with representation capacity comparable to an STU with k filters, we require h = Okk) = O(log L.)
Method
Prefill + Generation Runtime
Cache Size
Runtime with K, K,T = O())
Standard Conv
(TK T TlogT + K2)k T K²
T + K
L log L
Standard Attn.
T T+L K
L L (log L) L log"
EpochedFF
(Tlog T + 3/2 Vlg K)k
K
ContinuousFF
(T log T + K log" K)k
K
SpectraLDDS (ours)
(T + K'h
h
I. Log L
Table 1: Comparison of architecture runtime and memory requirements for generating K tokens from a length prompt with k, (log L), where O+)) is omitted for brevity.
4 Problem Background
In this section, we survey the fundamentals relevant to our approach. First, we discuss linear dynamical systems and the inherent challenges of training them directly on tasks requiring long memory. We then present the main theoretical results of spectral filtering and outline how the Spectral Transform Unit (STU) leverages fixed spectral filters model linear recurrences without explicitly learning the transition matrix A. Finally, we set the stage for our method of distilling STUs back into linear dynamical systems.
4.1 Linear Dynamical Systems
Linear dynamical systems (LDS) have been widely used in control theory to represent time-dependent processes, forming the basis of classical state-space formulations and optimal control methods 20
23
3
6 Concretely, we consider an input sequence "1 u2,...,, Me E Ik", and the
corresponding output sequence 11,32:-- 12. .. E Ihm. The hidden state He E IR summarizes the
system's 's memory of past inputs, with the evolution of the system being represented as
Jt Ar + Bus, Yt Cr. Duc.
where E Rdxdd E Rdxx, E Rmxd and E Rmxxn We omit dynamics and observation noise terms for simplicity with this derivation, although we test our methods on signals with noise.
Expanding the LDS to the Convolutional Form. In a noiseless environment with xo = Ō, we can expand the LDS equations follows:
y: Cxx + Du:- C(A + Bui) Du:
CA DWr.
Note that if any eigenvalue (A:(A) 1, the system becomes unstable and ye may tend to infinity in magnitude. Even for (A)) < 1, systems with ||A|| = 1 are prone to failure due to large powers in backpropagation, as noisy algorithm may approximate with spectral radius greater than during training. If [A((A) < 1 - for some spectral gap 5 > 0, then
yt CCABuL + ST. ||e|| S E,
where r = 0 log !) and the effective memory is thus on the order 5
As 8 0, learning A directly becomes unstable for large contexts [5] 32 30), highlighting the need for methods such as spectral filtering. Since the matrix serves as a skip connection that does not affect the representation capacity, we fix it to a 0-matrix and omit its consideration for the remainder of this paper. We will sometimes use the shorthand LDS(C, (C, A, B) to refer to linear dynamical system with those parameters and D 0.

---

Additionally, for the remainder of this paper, we restrict our attention to systems where A is symmetric real matrix. An LDS with symmetric A can be diagonalized over the real numbers, making it equivalent to an LDS with a diagonal A. Without loss of generality, we therefore assume A is diagonal with eigenvalues (),....;6
Spectral Filtering. With initial state zo 0. defining (aaa = (1, a²...... and indexing the columns of C' as Ce and the rows of B as be, we can extend the convolutional representation:
t-1 V/ = Σ -C' 1=0
2-1 Eo Eajler aflee ses) Bu- = Σ (ce @ be) Σ p(ae)(i) W t=1
d
c
==0
To circumvent the non-convex optimization problem of finding that best fit an LDS, [14] propose the spectral filtering algorithm, which learns an approximation of p(a) in convex manner. They prove that given cigenvalue-eigenveed pairs Øū of the Hankel matrix Z.
7 S( p(a) do, Za (1+ (i+
2
any p(ar) with a S can be approximated by the top eigenvectors (d),.... 02 of with an exponentially decreasing error in k. Thus, if y is generated by PSD linear dynamical system, we have the following result:
V1 Σ 1=1
ce & be)
Σ ((a))( 11-+11
M
(i) W6-6t1)
j=1
where we define ji(a)) : Ex= (u(a), Φ and learn suitable parameters My E R m Rather than depending on powers of A. learning an LDS with this parameterization remains convex in {My}. since eigenvectors {øy} from the matrix Z are computed offline.
The Spectral Transform Unit. To account for learning negative eigenvalues of A, the spectral filtering construction can be adapted by introducing positive and negative sets of feature maps. If {a,, Φ3} ) are the digenvalue-cigenveetor pairs of Z, then for each time and each respective filter Φj. we define the projections of the inputs onto the spectral basis:
U
Σ 41-+1 O1(1,,
U Σ - (-1)"- (y()).
One then forms the output by learning linear combinations of both U and U and optional autoregressive term
112 2]:
h M UT + +
3
Σ
M + Ut +
Σ
M u
(3)
=1
j=1
2=1
Specirul Fillering component
component
Without the autoregressive component we compute y5th = - M Uit + My U, The above expression is considered the Spectral Transform Unit, where (M,) is the set of parameters to be learned using differentiable algorithm. Following [22), we consider the STU without the autoregressive component. Empirical evidence [22] shows that hybrid STU models can compete with even outperform purely ditention-basee architectures.
Error Bounds for Spectral Approximation. We repeat the result from 14 (see also 2
26 27
stating that given any LDS parameterized by A, B, C' where A is a symmetric matrix with ||||| S 1,
there exist matrices M M M ...., M such that for all L and for all input sequences M1, ur, with [[w/]| < 1, the following holds for all E [L]:
|/ I.D.
SF
5

---

where & is the number of spectral filters, y! is the sequence generated by the LDS, and yr" is the sequence generated by spectral filtering.
Therefore, one can approximate any LDS that meets these specifications up to error e by selecting k = (log L log(2) ) spectral filters. Thus, the STU capture LDS dynamics with only logarithmic number of filters k, providing a compact and stable representation even for systems with high effective memory (||A|| x 1).
4.2 Distilling STU into an LDS
While the STU avoids direct learning of A, still implements LDS- dynamics via spectral filtering. natural question is whether we can recover explicit parameters (A, B, C)) from the learned STU, enabling equivalent recurrence rather than O(L) costs when convolving spectral filters with input sequences. This can be achieved by approximating the convolution kernel of the STU by the implicit convolution kernel of an LDS. Such distillation would bridge the gap between the stable convex training of the STU and the fast inference of a recurrent LDS.
Algorithm and Main Result
We now present our main theoretical result and accompanying algorithm, which shows how to recover an accurate LDS representation from the learned spectral filters. Concretely, we demonstrate that each STU filter Φ, can be approximated by linear combination of geometrically decaying LDS impulse response filters (i.e., the LDS implicit convolutional kernel).
5.1 A General Transformation from Spectral Filters to LDS
Our main result is given in Algorithm response filter by
2 For the dynamics LDS(1 - a, at 1), we denote the impulse
(a) (1 (1-
a²
at-1).
Note that μr. (a) W..t. Et= (1- a)aaa U1 LDS(1 a, a, 1)(...)), and thus inference with the LDS is the same convolution with its impulse response filter. Let 01,-.., du RR be the spectral filters of length L. We write the first & filters in matrix form as Pisk E TR such that the i-th row is d;-
first observation that we can write any LDS impulse filter approximately the spectral basis. This is direct consequence of the spectral filtering methodology for learning LDS [14.
Algorithm FindSpectralReprreetttinn
Input: Scalar LDS parameter a representation size k.
2: Output: Spectral parameters m e R²
3: Construct the impulse response vector (a E R as (z( = (1 a) a; ,,,
4:
return best spectral fit of the system over random inputs E R using gradient descent
[mn T Pi (r(()) ud?)
m = arg min E NERL
mER
As consequence of results from spectral filtering for learning LDSs, we show in Appendix A.4 that the procedure FindSpecuralRepreeeenttion returns a vector for which, for some constant >>>
m T Φ1*k
(z(a)) < e m
We proceed to use this subroutine to find a distillation to the spectral filters.
Our main performance guarantee is given in the following theorem.
Theorem 1. As long h k. Algorithm returns w.h.p. matrix M such that
L M M1 (at.......aa)) < chuuuh e ing
where Amax is the largest eigenvalue of the Penrose-Moore pseudo inverse of the matrix M.

---

Algorithm 2 Spectral Filters to LDS Filters
Input: The first k spectral filters matrix Pi:k E R&XL, and parameter k.
2: Output: The transformation matrix M.
3: Sample h randomly chosen independent scalars oi,,,., Oi with each E [0,
4: Construct the matrix az (a1:)) E Rx with ith row μu (a.) (1- as)ll, "...,.. af
5: For each scalar impulse response, find the spectral representation by
m, FindSpecrtalRpeesennaainoioa/
6: Let M be the h x matrix whose rows are m,-
7: return M = M-1 ER
The significance of Theorem iis that it allows us to translate between the representation of spectral filters and linear dynamical systems.
We note that it is not immediate to upper bound Amax Indeed, for h ~ k, this can be exponentially large in k, as corresponds to the condition number Vandermonde matrix [4] However, we note experimentally, under the distribution described in the practical algorithm below, that as h grows, Amax quickly becomes smaller. This an overparamerrization effect, which we show experimentally in Appendix A.3
We provide analysis of Theorem 1 in Appendix A.4
5.2 Practical Algorithm
For improved practical performance we adopt the following procedure, Fix H > h. As above, let pr. pr((rr (0).)) R H denote H impulse responses and let PISA E R denote the first spectral filters. To obtain a representation of size J, we select a small index set S C {1,.... H). IS| = << H, by running multi-target Orthogonal Matching Pursuit (OMP) 25
36
on the regression
Fr. (o1.8) X (P1sk)
OMP greedily sets one coefficient per step from to reduce the regression loss, with the selected coefficients after h steps providing the indices for S. Given S, we solve the unregularized least- squares problem
X arg min (as) X
to yield Pisk Ri M pr.. (on..)) where M (X*)T.
Another change is the distribution from which the a, are drawn. We employ symmetric, near- unit-radius sampling to emphasize long-memory behavior: for each i, draw U, Unif(0, and S; ~ Unif([-1, 1)), and set
a, = S.(1-
Equivalently, |as| _ UH with an independent random sign. This deliberately skews mass toward lasl ~= 1, improving reconstruction at better h to H ratio. We ablate this choice in Appendix A.9 For numerical stability we use float64 arithmetic end-to-end and scale 1 by Spum > 0 so that OMP, we normalize the
(M1:H) M| R [|mmgg P1 r. To mitigate the impact of magnitude columns of pz (QI:H) We later undo the scaling. In Figure we report the reconstruction error of & against LDS of hidden dimension h produced by this practical method.
5.3 Converting M into an LDS.
As a result of Theorem D with = biag(a)....... an)) := Diag(1 a1,... ; an) and = [LI an), we can replace the costly STU convolutions:
7

---

Reconstructio Error vs
10-7
L Revenstracion Ervee
MSE (log seale)
10-11
10
10-19
10
25
50
100
125
150
175
200
Hidden Dircensoo A
Figure 1: Reconstruction error of spectral filters as function of LDS state dimension. For this experiment, H 10000, but results are similar for H 1000 to 20000 (see Appendix A.8)
U1-++1 ()(
MdHH (i) 11-i+1 M, Σ
TA 11'1,
LDS A,1,),
1=1
1=1
1=0
Σ UgJ 0t-141 25(1) Mj
A(-A)$ In 1h LL, AN
This provides the basis of our autoregressive inference advantage: rather than computing Uiyy and UnJ pure convolutions,
maintain the hidden state for LDS A, and LD,,,,, AA I) for the O(h) computation during inference. For practical efficiency, can compute all U ......... Uitx U,, ... UT, simultaneously with a single LDS by leveraging the similari- ties in the state updates (see Appendix A.7
5.4 LDS to LDS Distillation
direct consequence of our approach is that we can distill any sigh-dimensional symmetric LDS into a low-dimensional LDS with bounded error. For LDS with input dimension d,, and output dimension dout, spectral filtering provides an e-approximation with only Odin dout log L Iolog log(?)) parameters regardless of the hidden dimension.
Accepting that Ammaz 0(1) for h > k, as justified Appendix A.3 we can then convert this spectral representation by application of our distillation procedure into an LDS with state dimension din Jt, where with = O(k) = O(log L log (t)) we maintain error. Thus, the distilled LDS has Oddin dout log L log(?) total parameters. In other words, the combination of spectral filtering for LDS learning and the following distillation step yields practical method reduce the state dimension while preserving the system's 's dynamics within tight error bounds. Empirically, for tasks difficult modeling language, we only require & 24 filters for strong performance, and the state dimension 2 80 of the distilled LDS suffices to retain performance.
Experiments
To demonstrate the effectiveness of our distillation algorithm, we begin by illustrating that low- dimensional LDS can effectively approximate the spectral filters Figure and further examine the eigenvalues of the resulting system in Appendix A.7 Building on this, to further quantify the effectiveness of Algorithm in fitting spectral filters with practically efficient LDSs, we plot reconstruction error of the spectral filters across different LDS state dimensions in Figure I These results show that our algorithm, guided by Theorem Tmmply reduces the approximation error to sufficiently small level without excessive increase in state dimension.

---

Online Learning of Symmetric LDS
Online Learning of Symmetric LDS with Noise
600
SpectraLDD
Attestion
S4
LDS
300
Mambo2
He- Kalnuc
MSB Lees
300
MSE Lees
4000
200
300
100
200
5000
100
2000
3000
Step
Step
Figure 2: Comparison of Spectral and other methods learning an arbitrary symmetric LDS with and without noise. The shaded region shows the 95% confidence interval over 8 runs. Each model leveraged default configurations except the LDS, which required a lower learning rate to converge. More details are available in Appendix A.5
Spectral Filters
Selected LDS Impulse Rosponses
Reconstructed Filters
10
96
0.4
0.2
2
0.0
0
1
-2
-0.2
-2
12K22
Figure 3: Fit of Spectral Filters by an LDS state dimension 160, where x-axis represents the time domain. 80 dimensions are for the spectral filters (U,,2 and 80 for the negative features (U,, not shown). Filters are normalized to be comparable. The shading on the middle figure represents filter quickly alternating (negative a). complete comparison for 24 without normalization provided in Appendix A.6 These filters have MSE error 7.689 x 10
To validate our method quantitatively, we perform synthetic experiments comparing SpectraLDS against other benchmarks learning high-memor LDSs (see Appendix [0. We find that our method significantly outperforms strong baselines in both sample efficiency and reconstruction accu- racy, confirming that our approach is well suited for learning systems with long-range dependencies.
Turning to the large-scale evaluation, we distill 340M-parameter FlashSTU model into an LDS-based architecture and compare its performance across suite of language benchmarks. From the results Table 2 we point out that despite the change from convolution-basee spectral filters to an explicit LDS representation for the STU layers, the performance remains identical across all tasks. This observation supports our claim that the STU can closely approximated by low-dimensional LDS without compromising predictive accuracy. We provide details of the experimental setup and hyperparameters for the models used in Appendix A.13
Model
MMLU
Hella,
PIQA
BoolQ
Wino.
CSQA
OBQA
ARC-e
ARC-c
Average
Flush STU 340M
26.58
30.46
65.34
60.12
51.85
20.48
20.60
54.08
23.29
39.20
Spectral 340M
26.58
30.45
65.29
60.12
50.99
20.15
20.20
54.17
23.29
39.03
Flush Err.
0.37
0.47
1.40
1.16
2.06
1.02
SpectraLDS Std. Emr.
0.37
0.46
1.11
0.86
1.40
1.15
2.07
1.02
1.24
Transformer 340M
26.81
30.41
64.64
61.10
51.62
19.98
18.80
55.47
21.84
38.96
Table 2: Evaluation 340M parammeee FlashSTU model and its distilled representation, replacing each convolution with LDS of state dim. 160, language benchmarks. Despite converting convolutions into an explicit LDS formulation, performance remains statistically equivalent.

---

Runtime for STU-Only Models
Runtime for Hyhrid Modela
Runtine for Convolution Only
1000
0000
1250
SpectraLDD 900)
SpectraLDD 900)
SpectralD (SD
Rantine (seronds)
Bescline Natee Comr
Beselin: Natve Com
1000
Spectraliee (SD 100)
Epoctaod
Ejpcciood
Baoeiie Nacv Corv
20000
3000
Epoched Future Fil
1000
1003
200-
4006
s02
32703
13t022
20244
606
atg
12u02
38244
306
696
60kk
3006
Sequence Length (tokens)
Sequeace Length (tokens)
Sequence Length (tokens)
Figure 4: Runtime for generating sequences of increasing length across STU implementations. The naive convolution approach exhibits quadratic growth, the FutureFill variants show logarithmic growth, and the distilled STU-to-LDS layers achieve linear growth. The STU-Only Epoched Future Fill OOMs for the largest sequence length. As shown in the rightmost figure, the SpectraL.DS models have nearly identical runtime despite varied state dimension. Further results are in Appendix A.
Finally, we present various measures of inference speed to illustrate the constant-time per token generation provided by the distilled STU. In Figure we compare the inference speed of distilled STU model against naive convolutional approach and numerically stable FutureFill variant M. In a 12 STU layer model, the naive convolution exhibits quadratic runtime growth with sequence length; the FutureFill variants achieve logarithmic growth; and the distilled STU-to-LDS model demonstrates the best performance with linear growth.
In the hybrid model with 6 attention and 6 STU layers, we find the distilled LDS implementation still provides a significant performance increase. Additionally, we note increases in the LDS state dimension have little impact the overall runtime, indicating that LDS operations are not compute bottleneck (see Appendix I.. For both the hybrid and STU-only models up to 16, 384 tokens, the distilled LDS, naive convolution. and Epoched Future Fill all have similar runtimes. All performance benchmarks were conducted on a single H100 GPU, with each generation process evaluated separately to ensure consistent measurements. We provide full experiment details in Appendix [A.12
Looking ahead, further investigation is warranted to better understand how convolution, LDS, and attention layers interact at the hardware level, and to optimize their coordination for even greater speedup. Additionally, further work is required to determine if the LDS layer stable below float64.
To conclude, since optimized transformer implementations suffer from KV-Cache memory bottlenecks rather than compute bottlenecks [9] and the LDS layers have drastically lower memory requirements, we anticipate that, with appropriate optimization, the inference speed of hybrid attention-STU architectures will be independent of the amount of LDS layers.
7 Conclusion and Discussion
We have provided the first provable technique for learning the parameters of a symmetric LDS with arbitrarily high effective memory. By leveraging their convex-leaming approach, we show how spectral filters can be distilled into an explicit LDS representation, enabling the construction of a state-space model with logarithmic-time inference and theoretical guarantees on the loss bounds.
We have provided a lower bound on how increasing state dimension affects reconstruction loss for a linear dynamical system, and we envision SpectraLDS drop-in replacement for autoregressive convolutional layers certain sequence-to-See to-suueeee tasks.
8 Acknowledgments
EH gratefully acknowledges support from the Office of Naval Research and Open Philanthropy.
10

---

References
[1] Naman Agarwal, Xinyi Chen, Evan Dogariu, Devan Shah, Hubert Strauss, Vlad Feinberg. Daniel Suo, Peter Bartlett, and Elad Hazan. Futurefill: Fast generation from convolutional sequence models, 2025.
[2] Naman Agarwal, Daniel Suo, Xinyi Chen, and Elad Hazan. Spectral state space models, 2024.
[3] Brian D.O. Anderson and John Moore. Linear Optimal Control. Prentice-Hall, 1971.
[4] Bernhard Beckermann. The condition number of real vandermonde, krylov and positive definite
hankel matrices. Numerische Mathematik, 85(4):553-577, 2000.
[5] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent difficult. IEEE Transactions on Neural Networks, 5(2):157-166, 1994.
[6] Dimitri Bertsekas. Dynamic Programming and Optimal Control, volume 1. Athena Scientific, 2005.
[7] Tri Dao. F Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
[8] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré, FlashAttention: Fast and memory-efficient exact attention with 10-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[9] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024.
[10] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2022.
[11] Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models, 2022.
[12] E. Hazan, H. Lee, K. Singh, C. Zhang, and Y. Zhang, Spectral filtering for general linear dynamical systems, 2018
[13] Elad Hazan and Karan Singh. Introduction to online nonstochastic control. arXiv preprint arXiv:2211.09611 2022.
[14] Elad Hazan, Karan Singh, and Cyril Zhang. Learning linear dynamical systems via spectral filtering, 2017.
[15] B. L. Ho and R. E. Kalman. Effective construction of linear state-variable models from input/output functions. Automatica, 3(1):49-59, 1966.
[16] A. Huang, S. Subramanian, J. Sum, K. Almubarak, S. Biderman, and S. Rush. The annotated transformer, 2022.
[17] W. Jamaludin, N. A. Wahab, N. S. Khalid, S Sahlan, Z. Ibrahim, and F. Rahmat. N4sid and moesp subspace identification methods. In 20/3 IEEE 9th International Colloquium on Signal Processing and its Applications, pages 140-145, 2013.
[18) Jer-Nan Juang and Richard Pappa. An eigensystem realization algorithm for modal parameter identification and model reduction. Journal of Guidance, Control, and Dynamics, 8(5):620-627, 1985.
[19] Thomas Kailath. Linear Systems. Prentice-Hall, 1980.
[20] Rudolph Emil Kalman. new approach to linear filtering and prediction problems. Journal of Basic Engineering, 12.1:35-45, 1960.
[21] S. Y. Kung. A new identification and model reduction algorithm via singular value decom- positions. In Proc. 12th Asilomar Conf. Cireuits, Systems and Computers, pages 705-714, 1978.
11

---

[22] I. Liu, W. Nguyen, Y. Devre, E. Dogariu, A. Majumdar, and E. Hazan. Flash stu: Fast spectral transform units, 2024.
[23] David Luenberger. An introduction to observers. IEEE Transactions on Automatic Control, 16(6):596-66 602, 1971.
[24] Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention, 2023.
[25] S.G. Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. FEEE Transactions on Signal Processing, (1133333977334 1993.
[26] Annie Marsden, Evan Dogariu, Naman Agarwal, Xinyi Chen, Daniel Suo, and Elad Hazan. Provable length generalization in sequence prediction via spectral filtering, 2024.
[27] Annie Marsden and Elad Hazan. Universal sequence preconditioning. arXiv preprint arXiv:2502.06545, 2025.
[28] Stefano Massaroli, Michael Poli, Daniel Y. Fu, Hermann Kumbong, Rom N. Parnichkun, Aman Timalsina, David W. Romero, Quinn McIntyre, Beidi Chen, Atri Rudra, Ce Zhang, Christopher Ré, Stefano Ermon, and Yoshua Bengio. Laughing hyena distillery: Extracting compact recurrences from convolutions, 2023.
[29] OpenAl. Gpt-4 technical report, 2024.
[30] A. Orvieto, S. L. Smith. A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and De. Resurrecting recurrent neural networks for long sequences, 2023.
[31] Samet Oymak and Necmiye Ozay. Non-asymptotic identification of Iti systems from a single trajectory. In 2019 American control conference (ACC), pages 5655-5661. IEEE, 2019.
[32] R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks, 2013.
[33] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[34] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models, 2023.
[35] Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022.
[36] Ron Rubinstein, Michael Zibulevsky, and Michael Blad, Efficient implementation of the k-svd algorithm using batch orthogonal matching pursuit. CS Technion, 40, 01 2008.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
12

---

A Appendix
A.1
Notation throughout the paper
24 E Rd
state time
E I
control (input) at time
V
E
TR
sequence model outputs time system matrices for linear dynamical system Number of used in final LDS representation Number of random samples of a Number of spectral filters
A, B. D,
H
k
K
Length of generated sequence
Length of prefill sequence
L
Length of sequence (or of filters)
A.2
Experimental details and notation
All experiments were performed on Nvidia H100-80GB GPUs in PyTTrrch [33] All layers except the LDS leverage 6 precision, whereas the LDS layers require float64 precision. All tests, unless otherwise stated, use k = 24 spectral filters and replace the k filters with LDS with state-dimension h = 80. To fit the additional alternating filters of the negative component (i.e., Φj- where o, [i] = (-1)--1 (ai))=1o0) Φy [i]. is needed compute U(, purely convolutionally), expand to an LDS with state-dimension h 160.
For the layer-level inference speed benchmarks, we leverage the STU with tensor-dot approximation (STU-T) 22 the variant used in the FlashSTU language model 22 It worth noting that the STU-to-LDS distillation leads to LDS layers with comparable speed regardless of whether the tensor-dot approximation is employed, and thus this approximation only accelerates the STU layers in benchmarks. A formal definition of this approximation is as follows:
2 introduced an important optimization, the tensor-dot approximation, wherein each tensor M X2exooox representing concatenation of the tensors M ...., M MM-- ..... M is learned as M() M(2 for M() R4 x28 and (2) E [dduxdow with input 21, 22,,,,,,,,, we can compute y² (o M(2)) @ Mittes[j). where Miters = [d1..... Φk: oT ........ .T......7! IF (() E Lxd and @ refers to the Schur product (i.c. (xr @ y) = x, Vj). This allows for only dout convolutions with the tensor-dot approximation, as opposed to k dm convolutions without it. Although this method can reduce expressivity and does not inherit the same marginal-stability guarantees, it maintains competitive empirical performance while yielding significant improvements in efficiency.
STU Tensor-Dot Approximation:
This approximation allows for reduction in convolutions as.
Additionally, we frequently refer to the impulse response the LDS and STU models. The impulse response of linear sequence model R² + Fº is the vector or convolutional kernel IR that is equivalent to f (i.e. f().....)) = to Tp. xi).....)) and thus:
[tt = f(10 ... 01)
position 1+1
We only consider the impulse response of f R& - Rd. but the impulse response is closely related to the derivative with respect to the inputs and is generalized identically. For an STU model
with d,n dout 1 and parameters M ..... M.. M .M M R. the impulse response v/s is thus described by vss[(]] = Σ M [b E M 2z Similarly, for an LDS with di dant = and parameters a, bec E IR, the impulse response vI.ss described by duDss(]] = -16
13

---

A.3 Experimental Results on the Condition Number of M, the Spectral Coefficients Matrix
In Figure we present experimental results on the condition number of M as defined in Section Recall that . E A is the spectral coefficients matrix produced by Algorithm 2 its i-th row m; stores the coefficients that express the LDS impulse response pz (aa) (with geometric decay factor o;)) in the spectral basis I Starting at h k with k 24, we repeatedly add additional independent vectors O; and measure the maximum singular value of the constructed M- The blue central line shows the mean maximum singular value across 5 experiments for a given value of h, with the shaded region showing the maximum and minimum of the largest singular values across experiments. We draw independently from the distribution in Figure To justify our statement section 5.2 that Amaz h can be considered O(1), we additionally plot Amoz h under the same experimental setup in Figure 6
Singulur Value vs. Sample Size
10
10
Largest Singular Value (log
10
10a
10a
10a
0
250
500
750
1000
1250
1500
1750
2000
Sample Size
Figure 5: Largest Singular Value as we increase h.
Amahh vs. Sample Size
10ts
101a
Anch (log scale)
1041
10
107
10a
250
500
750
1000
1250
15000
1750
2000
Sample Size
Figure 6: Amaz h as we increase h.
A.4 Analysis of Theorem
Proof of Theorem 4
Let
pr(a) = =1 (1-a) a² at-1] R
14

---

Histogram of a
10a
Frequency (log scale)
10"
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Value
Figure 7: Distribution of (H 10, 000 samples)
be the L-impulse response of the one dimensional linear dynamical system with parameters a,b c 1 a. Lemma 11.3 from [13] asserts that for any E [0, 1], there exist real coefficients m....... m& E R$ and constant
0, such that any sequence of inputs u1 E R with
| < 1,
k
Σ
mi (0r, una) (u)(()) mE.1) S ce To
The RHS term represents the evolution of the LDS, with system matrices b, that are assumed to be identity, and (a) the system evolution.
As the above is true for all W1:1, for each Oy,, there exist my such that,
Vj
mm
Pisk (z(aa) < ce toxz,
Moreover, as the optimization problem is convex in my and Lipschitz continuous, FindSpectralRep- resentation will return such my.
Let M E Rixk be the matrix whose rows are m² E :
111
171 2
M
m
Let = =MM4 11.k pr(an...... ax). By the triangle inequality and 44)
||E|| S
m disk (I(a))) S ch the mrz
(5)
Thus, assuming that M full rank, multiplying E by the Penrose-Moore pseudo-inverse of M and using Holder's inequality, get
P - ( an)l |M | - Amax che lng E
It remains to argue that M is full rank. This follows since P is orthogonal basis, and the matrix Pz (01:)) a Vandermonde matrix. Thus, both matrices are full rank.
A.5 Learning Symmetric LDS with and without noise
Figure in the main paper compares SpectraI.DS to other system-identification methods on the task of learning arbitrary symmetric LDS with and without noise. The LDS signal has hidden
15

---

dimension 256, input and output dimension 5, and maximum eigenvalue magnitude 0.99 with Gaussian initialization. Each step provides Gaussian input sequence of length 100 with variance 1/din and the final output. For the learning with noise experiment, Gaussian noise with 0.5 variance was added the hidden state at each step, and Gaussian noise with 5 variance was added to the output. Each architecture was trained with the default Py Torcch RMSProp optimizer configuration, except for the LDS, which required lower learning rate to converge. The Ho- Kalman matrices are recomputed every 20 steps for computational ease, while the other methods were updated every step, and the Ho-Kalman parameters are the maximum for an input of length 100 (that is T T_horizon = 99, TI 49, T2 49, state dim est 48). The shaded region in the figure shows the 95% confidence interval over 8 runs.
16

---

A.6 Fit of the Spectral Filters with Linear Dynamical System (State Dimension 160)
P3
24
0.0000
Origiaal Approdluand
0.0031
0.0011
0.0033
000010
0.0001
000005
00000
110000
0.0000
000001
-0.0001
-0.0001
-0.0002
-0.0005
-0.0002
-0.0004
100
150
200
50
150
200
50
100
150
200
100
150
200
0.0004
0.00044
0.000500
0.0010
0.0007
0,0002
0.000225
000005
0.00000
0.000000
000002
00,03025
0.0000
010002
-0,00050
-0.0001
000000
-1.001777
-0.0005
50
100
150
200
50
100
130
200
50
100
150
200
50
100
150
200
ch11
<hy2
0.0015
00,0015
(100150
0.000
0.0000
0,0010
0.0010
0,002
0.0005
0.0005
0.0000
0.001
0.0000
0,0000
000003
-000005
0.000
-00000
-020120
-0.0055
0,001
-0.0030
0.0020
010022
1000
150
2010
50
100
130
2000
50
100
150
200
10.0
200
1
111
6h15
C/16
0.006
0.004
0.002
0.004
0.000
0.000
00.001
-00002
0,002
00,004
00,002
-0.000
00,008
100
200
2117
19
(
0.0075
0.0090
0.010
0.01
00.00
0.0025
0.005
01:02
0.0000
-110015
-0.005
0001
-0.0050
0..000
-002
0.01
-0.0100)
-0.105
0.02
-0003
(221
0/22
<224
0005
0.04
0.080
0.00
0.12
-0.025
Duo0
-0005
0.5
6.4
-0.10
00.00
0,177
-0.15
-0.01
-0.100
-0.20
0U25
100
150
200
50
100
150
200
50
100
150
200
160 150
200
Figure 8: Visualization of the positive spectral filters and their approximation using an LDS with state dimension 160, as obtained via the practical algorithm. As described previously, only the first 80 dimensions of the state are used for approximate the Spectral Filters, with the remaining used for the alternating filters (negative filter maps), as described in Appendix A.7 The figure illustrates that even with a low-dii dimensionn state, the LDS can accurately fit the spectral filters, confirming the efficacy of our distillation process.
17

---

A.7 State Dynamics of the Linear Dynamical System
We train linear dynamical system with state dimension 160 to fit the Spectral Filters and alternating filters using Algorithm 2 and examine the eigenvalues of the resulting system in Figure We only plot the 80 eigenvalues corresponding to the Spectral Filters, as the remaining 80 corresponding to the alternating filters are identical except multiplied by -1 (see below the plot). The reconstruction error of the spectral filters for the trained LDS is 7.69 10 We transform the eigenvalues to visualize the full range the values as they approach and differentiate the negative and positive eigenvalues blue and red (e.g, 0.97 and -0.97 would both map to & 3.5, although the first would join a red column and the second blue).
Distribution of Transformed Eigenvalues f(x) = In(1 [x|))
Posisive Negative
3-
Count
5.0
10.0 (0. (0.990955)
12.5
(0. 00000)
(0.91792)
(0.99326)
(0.90945)
(0.909996)
Traneformee Eigenvvaiee (Origimal)
Figure 9: Distributions of Eigenvalues for LDS corresponding to the first 24 Spectral Filters.
For practical efficiency, we compute all U
U.
UT .......,,, UT simultaneously with single
t,k
LDS parameterized by {
[
A
0
12h This LDS has state dimension of 2hm,
MI
0
A
0
where each of the m input dimensions have a hidden state updated by
The hidden state
0 -A
dimension differs from 2h as we treat m as a batch dimension, resulting in Bu, E 2222 rather than 222 This mimics an LDS acting independently across each of the dimensions, equivalent treating m as a batch-axis, and this is the analog to how each of the spectral filters convolve along each input dimension independently. The above Figure showcases the eigenvalues of matrix A,
Snuure/ssowerr
whereas for practical deployment, the state matrix is A
A.8 Reconstruction Error of the Spectral Filters varying LDS State Dimension and
We demonstrate the efficacy the practical algorithm in fitting the Spectral Filters with a low initial LDS State Dimension. In Figure TO we ablate the choices of H and h for the practical algorithm. We do not fully understand why larger H leads to worse fits for small state dimension. Notably, the error quickly becomes no longer human perceptible. In this procedure we only aim to fit the positive spectral filters. To fit both can achieve the same error with double the state dimension, as described in Appendix A.7 With our implementation sparsity, resulting hidden dimension will near but not exactly that requested, which is why each line has different reconstruction dimensions.
positive spectral filters and k negative spectral filters, we
18

---

Reconstruction Error vs h
10-7
H 1000
H 2000
10-9
- H 5000
И = 10000
II 20000
10
MSE (log scale)
10-13
10 15
10
10-19 19
10
10 28
50
100
150
200
Hidden Dimension h
Figure 10: Reconstruction Error varying H and LDS state dimension h.
A.9 Reconstruction Error of the Spectral Filters with the Practical Algorithm and Uniform Distribution
In Figure 11 we illustrate that rather than choosing (Jaul - U+) with U Unif 1] with an
independent random sign, instead choose a; 1], we require larger H achieve similar performance. At sufficiently large H, uniform sampling will have sufficient coverage of a near and 11 Thus, we instead prefer distribution with natural bias towards these extreme points, as E [-0.9, 0.9] decay towards 0 quickly.
A.10 Experiments on Synthetic Tasks
We evaluate SpectraL.DS (state dimension 160) against strong baseline synthetic task, reporting results in Table 3 and Figure [2 A symmetric linear dynamical system with input dimension di and state dimension d₁ is initialized by drawing all entries from standard normal distribution, after which the update matrix A is rescaled so its largest eigenvalue has magnitude 1 5. We then distill SpectraLDS model from STU model trained with AdaGrad (learning rate 1.0) for 2000 steps; each step minimizes the mean-squane llaan--qared-erre (MSE) between the STU output and the ground-truth LDS randomly initialized
output over batches 32 sequences length seq_ For the baseline, symmetric LDS with AdaGrad (learning rate 0.0001) under the same loss. To keep training times practical, most baseline runs use shorter sequence length, and several runs were stopped early due to computational limits, marked with an asterisk. When the state dimension satisfies d,, > 1000, baseline training with learning rate of 0.01 becomes unstable and often diverges, whereas the STU remains stable even with a learning rate of 1.0.
19

---

Reconstruction Error under Uniform a Distribution
10-7
1000
H 2000
10 9
H 5000
II 10000
10-11
H = 20000
MSE (log scale)
10
10
10-17
10 19
10 10-21
10
50
100
150
200
Hidden Dimension h
Figure 11:
Reconstruction Error varying H and LDS state dimension when choosing a
Unif[-1,
Performance Results
Type
Delta
Ang MSE
Step 10 Loes
Step
Loss
Rune
Time (0)
Std (s)
8192
10
2:15
1.32
2153.12
14.16
2.53
3.18
4.53
1.80
1DS GD
8192
x 10
100
3.03
10
6.00 x 10
5,99 x 10
2.29 x 10
34
2150.75
28.05
108 GD
1024
10
100
9.54
10
0.00 10a
1.18 10
0.72
10
248
0.00
10
10-5
252.24
108
1024
7.34
1.02
1.06 x
4.97
248.68
0.13
1n8 09
1074
10-3
10
1000
2.40
10-5
1:71 x 10
4.00 x 10-4
3.71 x 10
250.44
0.7
10S GD
1021
100
7.72
2.32
2
248.32
0.38
105 GD
1024
10
100
6.77
10
6.43
1.01 x 10
4.84 10
25080
0.17
LDS GD
1024
10
1000
1.04
10
4.92 10
4.79 x 10
3.39 10-5
251.51
0.67
SpectnLDS
10
51.85
SpecmLDS
2.27
7,99
Spectui.DS
8192
10
100
3.78
10
1.57 x
9.82 x
4.12 10
63.92
24.06
Spectri.DS
8192
10
1000
1.78
10
2.27
10
5.06 x 10
5.49 10
$1.93
1.55
Spatni.DS
10
$2.23
SpectaLDS
Sperul.nS ..n
8102
10
100
2.84
10
8.78 x
1.18 10
3:30 x 10
$1.60
10.37
SpateL.DS
8192
10
1000
1.66
4.47
9.00 x 10
5,27 10
$1.15
0.07
SpeamL.D
SpecenLDD
8192
10-5
1000
1.96
10
1.27 x
8.55 x
5.63
51.95
1,63
SpeemLLD
8192
10
10
100
4.50
1.73
10
9.63 10
4,87
10
45,51
0.06
SpectnLDS
10
10
Table 3: SpectraL.DS performance learning synthetic linear dynamical systems with maximum eigenvalue against strong baseline.
20

---

Loss Comparison on learning Symmetric LDSs
10"
LDS 8192
LDS 1024
SpectraLDS
10-1
Loss (log scale)
10-2
10
10-4 Steps
10
100
2000
10
100
2000
10
100
2000
Delta
= 10
= 10
8 10
Elseration
LJS (2156)) 1022 (24%))
SpertzaLUSS (Mi)
Figure 12: Comparison of learning performance synthetic LDS tasks between SpectraLDDS and a peadiien-descent-upe varying 6, input/output dimensions, and sequence lengths. SpectraLDS consistently achieves lower loss with significantly reduced runtime, especially at larger output dimensions. Full experimental details are provided in Table
LDS baseline. The plot shows the loss at different training steps under
A.11 Layer Speeds of SpectraLDS and STU
benchmark the inference speed of a single SpectraLDS layer across several state dimensions and compare it with two accelerated schemes for computing the STU convolution: Epoched Future Fill [ and an STU that employs the tensor-dot approximation [22]. The SpectraL.DS layer itself produced by applying
-TU-to-LDS distillation to the tensor-dot STU. For the timings reported in Table each model is evaluated on L (Seq. Len.) autoregressive convolutions on inputs of dimension 128, and the mean and standard deviation over five runs recorded. All STU variants use filter length 1,048,576 to accommodate the longest sequences. Although every architecture begins the linear-scaling regime, only SpectraL continues scale favorably the sequence length increases. The resulting layer-speed comparison is visualized in Figure [3] benchmarks of SpectraLDD layers embedded in large language- moddel architectures appear Appendix A.12
Inference Time Performance (ms)
Seq. Len.
STU Future Fill
STU Tensor-Dot
STU
Spectral (SD 100)
Spectral. (SD 800)
Meun
Std
Mcan
Std
Meun
Std
Meun
Mean
Std
30.86
1.20
31.87
6.43
134.96
22.13
21.27
3.86
18.98
0.73
36.55 52.52
0.72
28.90
0.42
123.
2.54
22.40
0.14
22.27
0.29
135.94
256
75.73
0.35
45.98
1.12
162. 74
2.90
43.54
0.27
43.62
0.17
512
136.67
7.90
70.24
0.61
66
2.99
71.40
0.56
73.01
0.88
1024
234.10
6.99
122.89
1.19
317.91
4.79
127.52
1.25
128.98
0.97
2048
441 (09
3.69
226. 47
8.02
525 65
5,09
241.16
4.45
252.58
8.78
8192
1856.63
191.17
837.46
2,03
1772 30
20.46
921 55
10.25
929.06
9.76
16384
1846.04
32768
7139.35
515.40
1286.86
24.74
6721
19.65
3620.84
43.71
3686.90
26.45
65536
13485 25
234.22
11614,43
28.04
13478 53
89.72
7181.13
35.96
7362.73
78.96
323.97
14649.56
262144
63437 26
123.25
117796.55 38
268.48
5S775 49
275.16
28573 69
119.09
291556.42
113.55
524288
177502.85
651.96
142896
621.72
57003.23
404.91
58027.04
276.39
1048576
576129.57
1425.11
1145862.83
1805.50
451607.11
721.73
114044.79
1215.07
115999.77
670.78
Table 4: Autoregressive Inference Time (ms) across model architectures (5 runs).
21

---

Layer Speed Comparison: SpectraLDS
10
STU
SpectraLDS (SD 100)
SpectraLDS (SD 000)
Time (m03)
104
109
101
22
128
312
2K
8K
32K
128K
512K
Sequence Length
Figure 13: Autoregressive Inference Time (ms) across model architectures.
A.12 FlashSTU Ablations
A.12.1
Implementation Architecture Details
To perform the FlashSTU performance evaluations with STU-T, we employ the architecture depicted in Figure [4] We test token generation efficiency with a hybrid architecture, alternating between layers using STU-T and sliding window attention (SWA), and we additionally using STU-T only. Each layer consists of RMS-Norm, followed by STU-T or SWA, followed by RMS-Norm, and then followed by an MLP. Inputs are tokenized by the o200k base tokenizer and the FlashSTU model begins with an embedding layer, which is weight-tyed to the output unembedding layer. To start generation, we add special tokens such as /ndoofeext > and endofprompt
with each layer
The sliding window attention layers leverage Flash Attention v2 g 7 and ALiBi position embeddings 335. The tested model has input dimension 896 and 12 layers, which has 550.31 million parameters for the hybrid model, and 535.99 million parameters for the STU-only model. All layers are run in bfloat 16 except for the LDS distilled layers, which require float64 precision. The Flash STU-T leverages the STU with tensor-dot [22] approximation rather than the base STU layer for faster inference, and thus we perform the STU-to-LDS distillation on the STU with tensor-dot approximation. For tests with generation length up to 131072, we leverage STU filter length of 131072. For the generation length of 262144, we leverage an STU filter length of 262144. For each runtime measurement, we first run a warmup generation, before reporting the mean two generations of that length. All benchmarks include only inference time and not model setup or filter computation costs. Additionally, each MIP layer has hidden dimension 12x the input dimension (MI. expansion factor). Other configuration details are identical to those in Table 7
22

---

Input
RMMNorrm
Output
Figure 14: FlashSTU architecture 22
A.12.2 Implementation Efficiency
Using the setup of Fig. 14 we time autoregressive generation for sequence lengths 4,096-262, tokens under two architectures- (i) hybrid network that interleaves STU-T and wiiddoww attention layers and (ii) an STU-only network in which every layer STU-T. Tables 5 report mean runtimes (over two runs after warmup) for SpectraLDS with state dimensions 100-800 alongside the STU-T with naive convolutions (Base STU) and the Epoched FutureFill method. We note three main results. First, SpectraLDS runtimes grow nearly linearly with sequence length and are virtually independent of the chosen state dimension. Second, while the STU-T with naive convolutions is competitive at 4-8 k tokens, it becomes progressively slower, that SpectraLDS is x= 2x faster by 65 k tokens and over 4x faster at 262 k tokens the STU-only setting (and 2x faster in the hybrid setting). Third, Epoched FutureFill narrows the gap at medium lengths but is still outpaced by SpectraL.DS beyond 131 k tokens and, in the STU-Only architecture, exhausts memory (OOM) at 262 k tokens, whereas SpectraLDDS completes the run. Together these results demonstrate that SpectraLDS delivers the most favorable long-context scaling and remains robust across model hyper-parameters.
Seq. Len.
LDS SD
LDS 200
LDS SD
LDS 800
Base STU
FutureFill
4096
20.34
20.71
20.40
20.40
18.91
19.92
8192
40.48
41.32
40.69
40.82
38.09
38.39
16384
80.78
82.90
80.95
81.63
76.49
74.89
32768
161.43
163.92
162.13
163.30
164.70
141.16
65536
323.40
327.31
323.90
325.92
389.72
290.68
131072
646.66
653.96
648.99
651.35
1014.67
666.24
262144
1588.81
1639.24
1591.50
1536.20
3113.45
2498.89
Table 5: Hybrid Model Runtime (seconds) for generation aeross Spectral.DS LDS with different state dimensions and Baseline and Epoched FutureFill implementations
23

---

Seq. Len.
LDS SD
LDS 200
LDS SD
LDS 800
Base STU
FutureFill
4096
14.04
13.87
13.87
13.74
12.14
19.35
8192
28.15
27.74
27.67
27.42
24.84
35.86
16384
55.71
55.51
55.34
54.73
57.32
67.96
32768
110.85
110.88
110.57
108.68
149.12
135.75
65536
220.11
221.55
221.13
215.00
429.20
329.91
131072
439.87
442.65
442.04
428.36
1352.40
952.70
262144
1104.97
1101.76
1132.18
1107.49
4672.60
OOM
Table 6: STU-Only Runtime (seconds) for generation across SpectraLDS with different state dimen- sions and Baseline and Epoched FutureFill implementations
A.13 Details of STU Model In Experiments
We summarize in Table 7 all relevant details for the FlashSTU model used the language evaluations in Table The distilled LDS layer used in the language benchmarking experiments obtained by A [ has a state dimension of 160, incorporating both positive and negative spectral components. The weights for the distilled model were directly transferred from the FlashSTU model described below. The FlashSTU architecture is further described in Appendix A.12 and graphically shown in Figure 14
Benchmarks were conducted with task-specific few-shot configurations. For each task, if the few-shot value is not set -1, that number of examples is provided during evaluation; otherwise, the few-shot setting is left unchanged. Specifically, the few-shot settings are as follows: HellaSwag: shots, MMLU: shots, PIQA: 0 shots, BoolQ: 0 shots. Winogrande: -1, Commonsense QA: 7 shots, OpenBookQA: -1, ARC-Fasy: -1, ARC-Cha IRC-Challenge: -1. The model employs 6 precision except for the LDS computations, where it requires float64 precision, and global batch size of 3 examples per task. All computations were performed on single H100 GPU.
A.14 Example Generation
The following is example completion from the FlashSTU 340M model after LDS distillation with temperature 1.0:
Prompt: After school each day, Deidre Smith would look over the essays her students submitted at the end of class and, without fail, each day Alex would surprise her. Today, he wrote about
After school each day, Deidre Smith would look over the essays her students submitted at the end of class and, without fail, each day Alex would surprise her. Today, he wrote about 40 essays from his freshman class, all of which were collected
Generated Text:
the past three years for The Book of Sentiments: What Students Should Do When Reading Literature - First published in 1962, this book was very well received. What people who read this book would not immediately accept or appreciate the author's 's thought-pree prouoii- message of what it means to read literature.
What exactly you doing with your newfound ability to read? You reading? What emotions do you feel at the time? What do you do with your newfound ability to read? These are just a few examples of how much reading students think they can gain from the experience. It is accident, then, that each and every level of student will have difficulty with the text in this book. Students are given a choice: choose an assignment that you comfortable with and one that you would like to read more deeply and critically. As the task to read becomes more difficult and the pressure for it is much greater, students will struggle to put their best minds together when reading literature. To help students overcome this hurdle, have devised a book that makes the task a bit easier by providing all the information they need from a reader. If reader is too afraid read, have designed page that provides guidance from the textbook that will...
24

---

Model Architecture
Description
Flash STU
Parameter Count
Total number of parameters
340M
Embedding Dimension
Dimensionality emhedding space
1024
Number of Heads
Attention heads
Number of Layers
Transformer + STU layers
12
Attention
Attention scores modification using linear
RoPE Theta
RoPE scaling lactor for rotary embeddings
10,000
Sliding Window Size
Sliding window attention context lookhack size Input sequence length during training
512
Sequence Length (Training)
4,096
Sequenee Length (Inference)
Input sequence length during inference via position interpolation
131,072
Vocabulary Size
Size of the model's 's vocabulary
200,064
MLP Expansion Factor
Expansion factor MLP layers
Blas
Use bias
linear layers
No
Dropout
Dropout rate Number filters
0.0
Number Filters
24
Training and Optimization
Epochs
Number training epochs
Global Batch Size
Number tokens processed per step
524,288
Miero Batch Size
Batch size per GPU
8
Gradient Accumulation Steps
Number sleps before performing gradient update Number warmup steps
Warmup Steps
1,907
Evaluation Period
Evaluation frequency (steps)
50
Max Grad Norm
Maximum gradient norm for clipping
1.0
Optimizer Configuration
Optimizer
Optimizer type
AdamW
Learning Rate Schedule
I.R scheduling strategy
Linear decay with warmup
Max Learning Rate
Maximum learning
4.0 10-4
Min Learning Rate
Minimum learning rate
4.0 10
Torch Dtype
Data type for PoTorch tensors
biloat16 (0.9, 0.999)
Optimizer
Epsilon
Optimizer epsilon
10-8
Weight Decay
Weight decay
10
AMSGrad
Use AMSGrad variant
No
Fused
Use fused optimizer
Yes
Optimization Techniques
Activation Checkpointing
Enable activation checkpointing Enable Flash FFT
Yos
Use Flash FFT
No
Use Tensordot Approx.
Fnable tensordot approximation
Yes
Use Attention
Enable attention mechanism
Yes
Softcup
Softcap threshold
50.0
Torch Compile
Fiaale Torch compile optimization
Yes
Table 7: Model and training configuration details for the 340M Flash STU model.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that we have identified a method to distill spectral filters to lin- ear dynamical system form and demonstrate in the experiments section that method successfully works.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: At the end of the experiments section, we mention the current limitations of our distillation. Additionally, throughout we demonstrate understanding of the limitations of our approach, such as restricting our results linear dynamical systems with symmetric system matrices.
3. Theory assumptions and proofss
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our main theoretical result is Theorem | in which mention complete assumptions made and demonstrate when and why they would hold both theoretically and experimentally in the main paper and in the Appendix.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
Justification: Yes, we describe our algorithm completely in and 2 and elaborate on the practical implementation in Section 5.2. Our code repository is also open sourced.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: A link to the code is provided.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
many other instances throughout the paper. Justification: Experiment details are expansively explained in 6
A.13
A.3
A.I0
and in
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: and other graphs where relevant include error bars and statistical signifi- cance.
26

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: In 6 A.13, and in all places where relevant, the paper describes completely the computer resources needed and used. Furthermore, since many of results relate to inference speed, we explain the hardware used and time of execution/memory extensively.
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://uwwwp.ccccpooa.c/caaataaa
Answer: [Yes]
Justification: The paper conforms simply in every respect with the NeurlPS Code of Ethics as explained.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All previous work is fully credited and cited throughout.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our new assets are well documented and the documentation is provided.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
27

---

Answer: [NA]
Justification: [NA]
16. Declaration LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
28


Reviews:
[{"review_id": "RucBAAp1Qh", "reviewer": "Reviewer_kMfJ", "summary": "This paper proposes SpectraLDS, a provably correct method for distilling Spectral Transform Units (STUs) into linear dynamical systems (LDSs). Since training LDSs directly is often challenging, the authors propose to train the computationally heavier STU instead and then distill it into an LDS to reduce inference cost. Concretely, the goal is to recover LDS parameters $A, B, C$ from the trained STU such that the output of the resulting LDS closely approximates that of the original STU. Theorem 1 shows that the required hidden dimension after distillation is $O(\\log L)$, and that the per-token inference cost is reduced from $O(\\sqrt{L})$ to $O(\\log L)$.", "questions": "**(Q1)** In line 54, it is stated that the per-token computation cost is reduced to $O(\\log L)$. Does this account for the increased hidden dimension? This seems inconsistent with the explanations in lines 47 and 66, making it difficult to interpret.\n\n**(Q2)** Is the $M_k$ that appears in line 172 different from the matrix $M$ in Algorithm 2? Is my understanding correct that the former corresponds to spectral filters and the latter to LDS filters?\n\n**(Q3)** In line 232, what does the statement \"A direct consequence of our approach is that we can distill any high-dimensional symmetric LDS into a low-dimensional LDS with bounded error\" mean? Does it imply that the STU corresponds to a high-dimensional LDS?\n\n**(Q4)** In line 254, the phrase “greedily adding 1D-LDS increments” is used. Does this mean that the matrix $\\tilde{M}$ is constructed by incrementally adding the vectors $m_i$ obtained from Algorithm 2 one by one?\n\n**(Q5)** Figure 2 shows a visual comparison of filters before and after distillation. Can you also report a quantitative error metric?\n\n**(Q6)** Does the proposed distillation algorithm require access to new data, or is the trained model alone sufficient? Also, how much computational time is needed to run the algorithm?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n**(S1)** The paper provides a clear summary of the computational costs between existing methods such as Attention, STU, and LDS.\n\n**(S2)** The proposed method is grounded in a solid theoretical framework.\n\n**(S3)** The experiments cover multiple aspects, including comparisons between filters before and after the distillation, inference speed, and downstream language task performance.\n\n**Weaknesses:**\n\n**(W1)** The explanation of Algorithm 3 is insufficient. It is only referenced in lines 224 and 250, and there is almost no description of what the algorithm actually does.\n\n**(W2)** Similarly, Figure 1 is shown in the paper with very little explanation or commentary in the main text.\n\n**(W3)** The numerical experiments are limited to a model size of 340M, which is relatively small in scale.", "was_revised": true, "final_justification": "This paper introduces SpectraLDS, a theoretically grounded method that provably distills Spectral Transform Units (STUs) into linear dynamical systems (LDSs). While STUs are computationally heavier but easier to train, LDSs offer low inference cost yet are harder to train. The proposed distillation is therefore practically important. The paper presents not only Algorithm 2, which has a strong theoretical foundation, but also a more practice-oriented Algorithm 3.\n\nAlthough the contribution is important and potentially impactful, the submitted version lacks sufficient explanation of Algorithm 3 and of the experiments (especially Figure 1) which hurts clarity. The authors addressed these issues in the rebuttal; accordingly, I assign a borderline accept rating.", "rebuttal": "Dear Reviewer,\n\nThank you for your time and for your feedback, and we appreciate your recognition of the significance of our work. We address each point below:\n\nAs Algorithm 3 has many moving parts, we felt that a full explanation of it in the main paper would be confusing to the reader, so we moved the algorithm to appendix A.4 and a full explanation to section A.5.  Based on your feedback, we believe the best approach is somewhere in the middle, where we will instead move section 5.2 to the appendix and introduce a new section 5.2 where we explain how algorithm 3 differs from the main algorithm. We included a sample of what that section may look like below at the end of our response.\n\nFor Figure 1, we have a comprehensive description of the experiments performed in Appendix A.7, and in our forthcoming open-source release, we will include all the setup necessary to replicate this experiment. We will move more of the core details from Appendix A.7 to the figure caption and main text so the paper is easier to go through.\n\nAlthough 340M is not large in scale, for custom model architectures, such as the FlashSTU, evaluating distillation techniques on models of this size is generally common practice (I.e., see [26], which evaluates performance after Laughing Hyena Distillery on models of size 153M and 355M). We tested our technique on all public pretrained FlashSTU [26] models, which included earlier checkpoints of a 550M parameter model as well. We achieved successful distillation without performance drop on that model and have included those results here:\n\n| Model                | MMLU  | Hella. | PIQA  | BoolQ | Wino. | CSQA  | OBQA  | ARC-e | ARC-c | Average |\n|----------------------|-------|--------|-------|-------|-------|-------|-------|--------|--------|---------|\n| Flash STU 550M       | 22.99 | 26.51  | 58.00 | 39.30 | 52.01 | 19.49 | 29.80 | 39.10  | 23.12  | 34.48   |\n| SpectraLDS           | 22.98 | 26.55  | 58.22 | 39.33 | 52.01 | 19.49 | 29.40 | 39.02  | 23.04  | 34.34   |\n| Flash STU Std. Err.  | 0.35  | 0.44   | 1.15  | 0.85  | 1.40  | 1.13  | 2.05  | 1.00   | 1.23   | --      |\n| SpectraLDS Std. Err. | 0.35  | 0.44   | 1.15  | 0.85  | 1.40  | 1.13  | 2.04  | 1.00   | 1.23   | --      |\n\n\n\n\nHowever as this base model had worse evaluations, we only included our distillation on the 340M model in the final paper. The 550M model was published on 1/28/25, and the 340M model on 4/29/25, so we suspect the greater performance is due to better training choices.\n\nNotably, our technique has shown effectiveness on a large range of evaluations, including modeling large linear dynamical systems with noise and with high effective memory, which we include in Appendix A.11. Although we agree with the reviewer that testing on larger FlashSTU language models would be ideal, it is not feasible for us to train larger FlashSTU language models, and we have tested distillation, and shown strong performance, to the maximal extent we are able to. \n\n**(Q1):** Thank you for this observation. Line 66 is inaccurate and per-token generation costs are O(log L), and we will correct this in the revision. \n\n**(Q2):** We agree this is confusing. The connection is that $[M_1, M_2, … M_K]$ from line 172 is equal to $m_i$ from algorithm 2, where the $M_k$’s were distilled from the LDS defined by the convolutional kernel $\\mu_L(\\alpha_i)$. In our revision, we will explicitly state that connection in Algorithm 1 and switch Algorithm 2 to refer to the matrix as Q. \n\n**(Q3):** Yes, we can use the STU to learn a given LDS S and we can then distill the STU into another LDS S’. As the STU can learn S with error $\\epsilon$ with $k = O(\\log L \\log \\left( \\frac{1}{\\epsilon} \\right))$ filters from line 190, where $L$ is the maximum sequence length, we thus have that we can achieve an $\\epsilon$ approximation S’ of S with $O(d_{in} \\cdot d_{out} \\cdot \\log L \\cdot \\log\\frac{1}{\\epsilon})$ parameters. Notably, S’ has size independent of the state-dimension of S, and thus for a high-dimensional LDS S, we can distill it into a relatively low-dimensional LDS S’ with bounded error. We agree that the term “high-dimensional” was not clear in this case, and we will clean up this exposition to make clear that our method for LDS-to-LDS distillation has $O(1)$ dependence on the state-dimension of the initial LDS.\n\n**(Q4):** Yes precisely. For Algorithm 3, the matrix M is constructed in part by greedily choosing pairs $(\\alpha_i, m_i)$ from a precomputed set so that $|| \\Phi_{1:k} - M^{\\dagger}\\mu_L(\\alpha_1, \\dots, \\alpha_i) || $ is minimal. We will include this description in the revised section 5.2, where we discuss the differences between the improved but more complicated Algorithm 3, and the main algorithm we present. \n\n**(Q5):** For the filters in figure 2, these are the same filters as in Appendix A.8 and A.9, and have reconstruction error of $1.23 \\times 10^{-12}$.  For a sense of scale, we show the relationship between LDS dimension and filter reconstruction in Appendix Figure 9. We will add the error metric to the figure caption.\n\n**(Q6):** The proposed distillation algorithm does not require access to new data. In fact, once we have fit the spectral filters with Algorithm 2 or 3, we can immediately transfer the weights of any STU model that uses those filters to a SpectraLDS model. With the 340M FlashSTU model, this process was completed within 150ms. As all FlashSTU models are trained with the same set of 24 spectral filters, we can perform this immediate parameter transfer for any model in the FlashSTU series. The distillation algorithm provides an LDS of a given state dimension per set of spectral filters, and so generally will be rerun only when a different state-dimension is desired, either for more accurate reconstruction or for faster inference. However, it is also worth noting that after the initial step of collecting a set of pairs $(\\alpha_i, m_i)$, the remainder of both Algorithm 2 and Algorithm 3 can be computed speedily. In our open-source release, we will provide the set of 61,784 pairs we have computed. \n\nThank you for your detailed feedback, and we will work on improving the clarity of the paper in the ways you suggested and fixing the minor mistakes. We hope that these fixes largely address your concerns, and we would be happy for you to elaborate on any concerns that you believe still remain in the paper. We also are happy to note that recent work [1*] has extended the STU to linear dynamical systems with asymmetric transition matrices, with bounds based on the maximum imaginary component of the eigenvalues, so we are enthusiastic about further applications of our distillation beyond symmetric LDSs.\n\nThank you for your time and feedback,\n\nAuthors\n\n[1*] Marsden, A., & Hazan, E. (2025). Universal Sequence Preconditioning. arXiv. \n\n\nSAMPLE SECTION 5.2\n\nWhile Algorithm 2 provides the theoretical foundation for our distillation approach, we developed Algorithm 3 (detailed in Appendix A.4 and A.5) with several practical improvements that significantly enhance performance in applications. Rather than sampling eigenvalues $\\alpha$ uniformly in $[0,1]$, Algorithm 3 samples in the range $[-1, 1]$ and deliberately oversamples eigenvalues near $\\pm 1$ to better capture high effective memory (see Figure 6), which we use to assemble a large set of pairs $\\mathcal{P} = \\{(\\alpha_i, m_i)\\}_i$.\n\nInstead of using a fixed random sample of $h$ eigenvalues, we employ a greedy selection strategy. We start with an initial subset of size $h_{\\text{start}}$ chosen to minimize the reconstruction error $\\| M^\\dagger_{\\text{sub}} \\Psi_{\\text{sub}} - \\Phi_{1:k}\\|^2_F$, where $\\Psi_{\\text{sub}}$ is a concatenation of the LDS impulse responses $\\mu_{L}(\\alpha_i)$ of that subset and $\\Phi_{\\text{sub}}$ contains the corresponding STU weights, where this subset is chosen through sampling over $\\mathcal{P}$. We then incrementally add $h - h_{\\text{start}}$ pairs $(\\alpha_i, m_i) \\in \\mathcal{P}$ that most reduce the reconstruction error. After obtaining the initial transformation matrix $\\tilde{M}$, Algorithm 3 refines it using gradient descent on the least-squares objective $\\|\\Phi_{1:k} - \\tilde{M}\\Psi_{\\text{sub}}\\|^2_F$, which typically yields approximately $1.4\\times$ improvement in reconstruction accuracy."}, {"review_id": "eMHUwPVTxw", "reviewer": "Reviewer_ZCK4", "summary": "The paper builds off the work of agarwal et al.’s Spectral Transform Unit. The paper identifies the problem that there is now way to convert a trained STU layer into an explicitly LDS layer. The paper identifies this as being important as the industry seeks a new efficient way of handling particularly long sequences. The paper presents a novel technique for converting STU filters into LDS form. The paper provides the “first provable method to directly learn the parameters of a symmetric LDS of arbitrarily high effective memory and with bounded noise”. The paper’s main result is in algorithm 2, which then offers performance guarantees and empirical verifications for.", "questions": "1.Why does epoched future fill in figure three only go until 131072 tokens? (decrease) \n2.Can you provide some analysis in the h-vs-accuracy trade off? (increase)\n3. Is there a clear way or some promising directions to be able to achieve similar results without the symmetric assumption? (Neutral)", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Quality: Good theory paper with a nice result, theorem 1 wasn’t terribly difficult to follow. Would like more analysis of pseudo-inverse conditions. \n\nClarity: Figure 1 is barely readable, please make the ‘keys’ better so those with poor eye sight are able to better distinguish the lines. Other than that well organized\n\nSignificance: Feels quite significant. The biggest issue is that real-world systems may require non-symmetric dynamics. \n\nOriginality: Explicitly novel algorithm and work to go along-side it.", "was_revised": true, "final_justification": "The Authors have adequately addressed my concerns. While my concern regarding Q3 prevents me from increasing my score past a 5 (IE I think the limitation of symmetrical systems is still too great) I do still regard this as a technically solid paper. Thus I will not be changing my score.", "rebuttal": "Dear Reviewer,\n\nThank you for your time and constructive feedback on our paper. We appreciate your positive assessment of the theoretical contributions and novelty of our work. We address each point below:\n\n\n**[Pseudo-inverse Analysis]:** We agree that more analysis of the pseudo-inverse conditions would strengthen the paper. In Theorem 1, we show that the error bound depends on $\\lambda_{max}$, the largest eigenvalue of the Penrose-Moore pseudo inverse of matrix M. While for $h \\ll k$, this can be exponentially large in k, we provide experimental evidence in Appendix A.2 showing that as h grows, $\\lambda_{max}$ decays doubly exponentially.\n\n\n**[Figure 1 Readability]:** We will improve Figure 1's accessibility by using more line styles (dashed, dotted, solid) and colors, increasing line thickness, and improving the legend font.\n\n**[Q1 - Epoched Future Fill Memory Limitation]:** The Epoched Future Fill algorithm encounters a CUDA Out of Memory error on our H100-80GB setup at 262,144 tokens. While the theoretical memory complexity is reasonable, the implementation requires intermediate buffers for FFT operations and caches. The naive convolution approaches this limit but remains within bounds, whereas EFF's additional caching and overhead pushes it beyond the threshold. In contrast, SpectraLDS maintains constant memory requirements per token (O(h)) regardless of sequence length.\n\n**[Q2 - h vs. Accuracy Trade-off]:** We provide detailed analysis of this trade-off in Figure 9 (Appendix A.10), which shows reconstruction error as a function of LDS state dimension. For our language modeling experiments, benchmark accuracy remains nearly identical for $h \\geq 60$ but degrades rapidly below this threshold. We will expand Table 2 to include performance comparisons at various state dimensions to better illustrate this trade-off.\n\n**[Q3 - Beyond Symmetric Systems]:** We are encouraged by recent developments in this direction. Recently, there has been promising work [1*] extending STUs to broader classes of transition matrices, with error bounds dependent on the maximum magnitude of the imaginary components of eigenvalues. We expect our distillation techniques to apply to these expanded filter sets. In our discussion section, we will add additional discussion in this direction.\n\n\nWe believe these revisions will address your concerns while improving the quality of the paper. We would be more than happy to elaborate on any additional concerns you have with the paper, and we appreciate your time and effort.\n\nThank you, \n\nAuthors\n\n[1*] Marsden, A., & Hazan, E. (2025). Universal Sequence Preconditioning. arXiv."}, {"review_id": "BSsL4tRyA2", "reviewer": "Reviewer_ssg6", "summary": "The paper introduces an algorithm to distill a neural network module Spectral Transform Unit (STU) into a standard state space model (or control-affine linear system).  STU implicitly models state space models with real eigenvalues by learning a basis of spectral filters, which in turn define the coefficients of linear combinations of the input tokens, thus mimicking a linear attention mechanism. The objective is to improve the token-generation computational complexity of STUs without losing their expressivity. \n\nThe STU hinges on the fundamental assumption that the linear dynamics matrix of a control-affine linear system is symmetric and hence constrained to be diagonalizable and to have real eigenvalues. This assumption is required to satisfy the strong decay of the magnitude of eigenvalues of the Hankel matrix characterising the time evolution of the system’s linear dynamics matrix, which is the main argument to motivate the approximation of the most relevant spectral filters—the top eigenvectors of this Hankel matrix.\n\nThe proposed methodology takes as an input an STU with learned top spectral filters (eigenvectors of the Hankel matrix) and tries to estimate the parameters of a control-affine linear system that would lead to such spectral filters. The authors claim that this distillation process can be achieved rigorously, selecting the distilled state dimension as a function fo the approximation error of the STU.", "questions": "- The main suggestion is to enlarge the presentation and details of the proposed distillation approach in the main text. Specifically, I am concerned on the reproducibility of this distillation process by others, and the number of hyperparameters and intermediate steps omitted to be discussed in the main text. I would suggest to have a detailed version of the algorithm (like Algorithm 3) in the main text. \n\n- Furthermore, I would like authors to clarify in detail in the main text:\n    - How the dataset of 1D-LDS and corresponding STUs spectral filters, and spectral filter expansion coefficients of the LDS time evolution, is generated and used. Is this process completedly done offline and is it independent on the STU we are trying to distill?\n    - How many iterations of process of sampling a subset of this dataset are used in practice to determine the initial subset of the dataset used?.\n    - How to determine when to stop the “greedy” expansion of this dataset? Is this process determining the final dimension of the distilled LDS?\n    - How is the final state dimension of the distilled LDS chosen?.", "limitations": "Limitations were not discussed at all. While other aspects should be addressed, too, I believe it is important to elaborate on how prohibitive  the assumption of real-eigenvalues (symmetric LDSs) can be in practice. Personally, I am unaware if this assumption is reasonable / prohibiting in the context of language modelling. Authors could provide a brief statement on the implications of this assumption, or to references that discuss this topic.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### __Strengths__\n\n- The general motivation of the paper is clear and well-articulated once the reader is familiar with STUs. The goal is to improve computational complexity of token generation in generative sequential models for language modeling.\n- The different types of generative sequential model architectures and their respective computational complexity are well presented and explained, with Table 1 providing easy access to this information.\n- Related work comparison is well structured and complete. I specially appreciated the references to standard literature on system identification.\n- Experimental evaluations appear to be comprehensive and support the main thesis of the paper—that a distillation of an STU into a LDS($C$, $A$,$B$) with diagonal $A$,$C$, and $B$ can be achieved without losing precision while gaining substantial improvements in inference computational complexity.\n- Once the reader is familiar with the concept of an STU, the paper's core idea and motivation are clearly explained and well-justified.\n\n### __Weaknesses__\n\n- __[Major]__ The main contribution of the paper is a procedure to distill a STU into a linear control affine form. Hence to comprehend and understand the motivation of the paper, it is crucial that the reader is aware of what an STU is, or that a good intuitive idea of an STU is presented to the reader early on. Unfourtanetly, as a reader unfamiliar with spectral attention methods, I was forced to reading references [2] and [14] to get a good intuitive and technical idea of an STU. \n\nWhile producing self-contained theoretical research papers is challenging, I believe the authors could better introduce STUs and their relationship to SSMs in the introduction. This would help the average ML reader better understand and appreciate the work. Note that the terminology \"spectral transform unit,\" \"spectral filtering method,\" \"spectral filter,\" and \"impulse response\" in the introduction provide little-to-no intuition to the average reader, as these terms aren't widely known in the community—or are used in non-standard ways. \n\nAlthough STUs and spectral filtering methods are difficult to summarize intuitively in few words, I believe that the authors should make an effort to present these concepts clearly. Moving Fig. 2 to the introduction might help by showing both a standard evolution of an eigenfunction (what you call \"impulse response\") and the basis of spectral filters used to express the impulse response as a linear combination. This closely relates to the paper's objective. An early introduction of the transition from spectral filters to eigenvalues—with supporting visualisations—would dramatically improve the paper's accessibility.\n\n- __[Major]__ The core contribution of the paper, explained in Section 5, is restricted to a single page, leaving several of the most important details of the proposed methodology unclear. For instance I was confused with:\n    - Algorithm 1. appears to propose to solve a linear regression of the time evolution of an eigenfunction of $A$ (called ”impulse response” in the paper) in the basis of the spectral filters. This represents simply a linear transformation on the time (sequence-length) dimension, changing basis from time (/token order) to a spectral basis, in the same spirit as in a Fourier transform. It is unclear to me why there is a need for generating a random input vector, as an intermediate step to solve this process.\n    - Algorithm 2. Line 3. States that to learn the basis expansion coefficients a sample of $h$ random eigenvalues $A$ is needed. However in the appendix it is explicitly stated that the STU→LDS distillation hinges on a dataset of $h$ synthetic 1-dimensional LDS systems with parameters ($\\alpha_i$,, $b_i$, $c_i$), where the three scalar parameters are randomly sampled. It is unclear why algorithm 2 does not mention the generation of this datasets of 1-dimensional LDSs or why the sampling of the $B$ and $C$ parameters is ignored.\n    - Algorithm 3 and appendix A.5 mention a \"suitably chosen subset\" of 1D-LDS systems (and their STU's representation) used to train the distillation process. This selection process of subsets and \"greedy expansion of the subset\" appears to be a hand-engineered step of the algorithm that is not well explained in Algorithm 3/appendix A.5, and is crucially omitted entirely from the main body.\n    - Appendix A.5.4 mentions a fine-tunning of the approximated matrix M using gradient descent. This subprocess of the algorithm is again completedly omited in the main text.\n- __[Minor]__ The expansion after line 150 assumes the state and noise a time zero are zero vectors, but this is never stated.\n- __[Minor]__ In the equation after line 207, what was called previously an “impulse response”  $(\\mu_{L}(\\alpha)$ is now called a “single spectral filter” and conflated with the previously defined spectral filters $\\phi_k$.\n- __[Minor]__ Confusing nomenclature.\n    - The term “impulse response” in control theory is associated with the response of a controlled dynamical system to an step function on its control signal. Given that there is an overlap between control and machine learning nomenclature, the use of the term “impulse response” to denote the vector of discrete-time evolution of an eigefunction of a dynamical system is confising.\n    - Linear Dynamical Systems (LDS). For machine learning and control theoretic communities, LDS hints at a uncontrolled linear dynamical system, not a control-affine linear system, or a state space model.\n    - It is unclear what the readers imply by “LDS filters” (line 205)\n    - Line 3 of algorithm 2 says: “Sample h randomly chosen independent vectors $\\alpha_1$…”, but \\alpha is used to denote (scalar) eigenvalues.\n- __[Minor]__ Typo or wrongly format sentence in line 57. Probably need to remove “this”.", "was_revised": true, "final_justification": "Based on the authors' rebuttal, I believe that they can successfully revise manuscript to integrate all the remarks from myself and other reviewers and meet the acceptance bar.  Therefore, I rise my score. That said, I find that revision is very much needed to improve clarity, especially issues of limiting to symmetric LDS.", "rebuttal": "Dear Reviewer,\n\nThank you for your thorough and constructive feedback, and we appreciate the abundance of detail! Below, we will address each point.\n\n\n\n.\n\n\n**[Major]** As a reader unfamiliar with spectral attention methods, I was forced to read references [2] and [14] to get a good intuitive and technical idea of an STU.\n\nWe appreciate the extra time you spent on understanding our work, and we agree this certainly should not be necessary. Since submission, we have tried hard to improve accessibility and clarify core intuition, as well as reduce jargon in places we can. This has involved paper edits, but also the preparation of accompanying blog posts and content. For the submitted paper, we had added a section on notation in Appendix A.1, where we introduce the “impulse response” notation, among others, but in our revision we plan to add a formal glossary so the interested reader can accelerate through the introduction, and if they miss a definition, can easily glance to check. \n\nWe agree largely with the reviewer that more visuals would be helpful, and we plan to move Figure 2 to the introduction as you suggested, and we also plan to add Fig. 2 from [2], which is a convenient schematic for showing how the STU operates by decomposing an input filter into the spectral basis.\n\n.\n\n\n\n**[Major]** The core contribution of the paper, explained in Section 5, is restricted to a single page, leaving several of the most important details of the proposed methodology unclear.\n\n**Algorithm 1:** Your assessment is correct, the random input vector is not necessary as an intermediate and the convolution suffices as well. We chose to introduce Algorithm 1 in this manner as we thought it was simpler to digest, and it has the same performance both theoretically and in practice, and both approaches are exceptionally quick. \n\n**Algorithm 2:** For ease of reading, we have switched all notation to the first version of sampling $h$ random eigenvalues of $A$. However, both sections are accurate as the b and c coefficients can be incorporated into the matrix M, and training with them randomly sampled or with b = c = 1 are both very effective.\n\n**Algorithm 3 and appendix A.5:** Thank you for the observation, we will be adding a table with all the parameters used for this part of the algorithm. In our forthcoming open-source release, we will provide code and all files needed for a complete reproduction, including all parameters for our results. We want to ensure that all aspects of this work are reproducible and publicly accessible to build on, and we thank the review for noticing areas we fell short on. We will additionally publish all model weights, infrastructure, and our set of 61,784 LDS-STU pairs used in Algorithm 3.\n\n**Appendix A.5.4:** After the matrix inversion, we ran 50,000 steps of gradient descent to improve the fit of the coefficients, although this was a relatively minor improvement compared to the rest of the algorithm. We omitted this in the main text for clarity, although it will be included in our open-source filter generation code and in our revised section 5.2.\n\nGenerally, we agree the main section was not as clear as we would like, and we plan to address this by clarifying the above in the main paper, and by including the revised section 5.2, which we will discuss below.\n\n\n.\n\n\n\n**[Minor]**\nThank you for catching the errors on lines 57, 150, 207, and in algorithm 2. We will fix them in the revision.\n\nThank you for the suggestion with the “impulse response.” We will add a clear definition and glossary reference early on to avoid any misconception.\n\nThe LDS filters refer to the convolutional kernel corresponding to the LDS, but we will define this more formally earlier.\n\nThank you for the tip on the LDS definition. We have found the definition remarkably inconsistent across SSM papers, and we will be more formal in our earlier definitions to make this point clear.\n\n\n.\n\n\n**[Questions]**\n\nWe certainly agree that it is important we improve the presentation and build greater STU intuition at the start. The additions of the new figures earlier on should hopefully help align the reader better with the STU and prepare them for our discussion.\n\nAdditionally, we plan to improve exposition of our main algorithm and clarify the additional steps taken in the practical algorithm. In earlier drafts of the paper, we had Algorithm 3 in the main paper. However, this made the exposition substantially more challenging and we felt the core ideas were less accessible. We plan to move section 5.2 to the appendix and instead replace it with a short text description of the improvements in Algorithm 3, leaving the full description of Algorithm 3, including all of our hyperparameters, to the appendix. We have included an example of the revised section 5.2 at the end of our response.\n\nGenerating the dataset of 1D-LDSs and corresponding filters is done offline and only needs to be done once as long as the STUs use the same set of spectral filters. We had fit 61,784 pairs offline, as described in section A.10, but the algorithm still has strong performance well below 10,000 pairs. In our forthcoming open-source release, we will provide the pair set we have computed. We will add full details for all the parameter choices in Algorithm 3 elsewhere in the appendix, and we will move a few of the core details, such as improving distribution election for $\\alpha$, the gradient descent improvement, and the techniques to optimally select the eigenvalues of A to the main paper. Our hope is to strike a balance between being complete in the main paper while avoiding cluttering the main section and losing readability. \n\nThe final state dimension was chosen by examining filter plots, such as Figure 7, and reconstruction losses, such as Figure 9, and finding that the sharpest loss drop is in the first 60 state dimensions. The state dimension of 80 was validated in our testing, but it is possible larger models may need up to 160 state-dimension and smaller models can use much less.\n\n.\n\n\n**[Limitations]**\n\nWe agree that the restriction to symmetric LDSs is important to understand. This aspect was largely studied in earlier work on the STU [2], with the general results that although asymmetric LDSs have greater modeling capacity, they are practically bottlenecked on high-effective memory tasks by the challenges of training. In the S4 paper [10], which studied an expanded class of LDSs modeled by a normal plus low-rank matrix, the authors found that the LDSs they learned are nearly symmetric.\n\nRecent work [1*] has extended the STU to a larger class of LDSs, with performance dependent on the maximal imaginary component of the eigenvalues of the transition matrix. In this case, our work may have applications to a much larger class of LDSs than we had initially anticipated.\n\n\n.\n\n**[Closing]**\n\nWe would be more than happy to elaborate on any lingering concerns or issues we have not yet suitably addressed. As a larger comment on the concerns of missing certain parameters, we care deeply about making this work something that people can replicate and build on, and we appreciate your feedback on cases where we fall slightly short. We have a full open-source release prepared, and we will be releasing all code, checkpoints, and data that could be helpful to end users.\n\nThank you for your time and feedback,\n\nAuthors\n\n[1*] Marsden, A., & Hazan, E. (2025). Universal Sequence Preconditioning. arXiv. \n\n\n. \n\n\n**[SAMPLE SECTION 5.2]**\n\nWhile Algorithm 2 provides the theoretical foundation for our distillation approach, we developed Algorithm 3 (detailed in Appendix A.4 and A.5) with several practical improvements that significantly enhance performance in applications. Rather than sampling eigenvalues $\\alpha$ uniformly in $[0,1]$, Algorithm 3 samples in the range $[-1, 1]$ and deliberately oversamples eigenvalues near $\\pm 1$ to better capture high effective memory (see Figure 6), which we use to assemble a large set of pairs $\\mathcal{P} = \\{(\\alpha_i, m_i)\\}_i$.\n\nInstead of using a fixed random sample of $h$ eigenvalues, we employ a greedy selection strategy. We start with an initial subset of size $h_{\\text{start}}$ chosen to minimize the reconstruction error $\\| M^\\dagger_{\\text{sub}} \\Psi_{\\text{sub}} - \\Phi_{1:k}\\|^2_F$, where $\\Psi_{\\text{sub}}$ is a concatenation of the LDS impulse responses $\\mu_{L}(\\alpha_i)$ of that subset and $\\Phi_{\\text{sub}}$ contains the corresponding STU weights, where this subset is chosen through sampling over $\\mathcal{P}$. We then incrementally add $h - h_{\\text{start}}$ pairs $(\\alpha_i, m_i) \\in \\mathcal{P}$ that most reduce the reconstruction error. After obtaining the initial transformation matrix $\\tilde{M}$, Algorithm 3 refines it using gradient descent on the least-squares objective $\\|\\Phi_{1:k} - \\tilde{M}\\Psi_{\\text{sub}}\\|^2_F$, which typically yields approximately $1.4\\times$ improvement in reconstruction accuracy."}, {"review_id": "BCUwJwr0z6", "reviewer": "Reviewer_TKYC", "summary": "The authors present a method to distill spectral filter-based models (STUs) into state-space model-based (linear dynamical systems, LDS) representations. They provide a provable technique for this distillation that guarantees accuracy independent of the system’s state dimension or effective memory. Empirical evidence is presented demonstrating that their approach maintains predictive accuracy while substantially improving inference efficiency.", "questions": "Can the authors clarify or better motivate the practical significance or broader implications of this distillation technique?", "limitations": "yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "**Strengths**:\n1. Well-written: The paper clearly explains it's contribution, methorodlogy, and experiments.\n2. Thoroughness: The authors include extensive theoretical analysis, proofs, and multiple experimental validations, complemented by numerous detailed expansions in the appendix.\n3. Convincing experiments: The experiments robustly validate their claims, showing that the distillation effectively maintains accuracy across multiple benchmarks.\n\n**Weaknesses**:\n1. Although somewhat subjective, the result appears to be somewhat niche and not strongly motivated. Specifically, STUs are not currently predominant in sequence modeling, and it is unclear whether they significantly outperform existing SSM-based models, many of which already achieve constant-time and constant-space inference per token. Even if in certain settings STUs do outperform other approaches, writing an entire paper solely about distilling such models into LDS form seems like too narrow of a contribution.\n2. Again, subjective, but I find that both the proposed algorithm and associated theoretical proofs are rather simple (and not surprising), raising the question of whether the overall contribution is substantial enough to warrant acceptance. Specifically, the main result (Algorithm 2) simply randomly samples a diagonal LDS and linearly fits its parameters to match the spectral filters; this seems to me more like a straightforward linear approximation of spectral filters using randomly chosen exponential bases rather than a deep or novel theoretical insight.", "was_revised": true, "final_justification": "I appreciate the authors' detailed rebuttal and engagement.\n\nThe authors acknowledge the significant restriction of their approach to symmetric transition matrices. In my view, this restriction considerably weakens the contribution, as filters from symmetric LDSs are essentially limited to linear combinations of exponential decays, forming a low-dimensional representation space. Consequently, I find it unsurprising that such filters can be effectively approximated by randomly sampling diagonal real LDS elements and linearly combining them.\n\nNevertheless, I acknowledge that within the broader context of literature addressing the challenge of distilling diverse filter representations into LDS frameworks, this result holds some value. Given this recognition, I have revised my score upward to a 3.", "rebuttal": "Dear Reviewer, \n\nThank you for your time and your feedback. We appreciate your concerns about the scope of our results, especially as the STU is not as popular as other SSM architectures. However, our work carries large implications for the LDS, a common primitive forming the basis of most SSM architectures, and whose pitfalls provide intuition for many popular SSMs, such as S4 [10], Hyena [32], and Mamba [9]. \n\nRegarding your concern about simplicity: we respectfully disagree. We argue that simplicity is an advantage. Our method, while straightforward, is surprising, and circumvents the long-held understanding of parameter fitting of an LDS. Decades of work has culminated in methods that work in special cases (no noise or normally distributed noise), or agnostic improper learning (online) via convex relaxation. The new development of spectral filtering allows our proper learning of an LDS via learning the filters, rather than directly, which we consider unexpected. It is also worth noting that, although in our main result we present the more straightforward algorithm, this algorithm is far from the most practically performant, and we introduce more complex algorithms with substantially better performance in the appendix. \n\nAs far as the broader impact of our work, historically, despite their theoretical maturity, there have been no efficient and provably accurate learning methods for fitting LDS parameters to a signal with guarantees independent of the effective-memory of the system. This challenge has been a popular area of research over the last few years, with the Legendre Memory Unit [1*] and HiPPO initialization [2*] being significant results in this line of work. Although our result is currently restricted to fitting LDSs with symmetric transition matrices, we provide what is, to our knowledge, the strongest result to date for learning LDSs with high effective-memory, with performance independent of the effective-memory the signal and with strong convergence bounds.\n\nBelow, we will mention a few other examples of the practical significance of our work:\n\nThe Laughing Hyena Distillery (LHD) [26] technique has facilitated convolutional models 10x faster than Transformers (at equal performance) for language modeling. LHD transforms convolutional layers into complex symmetric LDS layers with gradient methods, offering no guarantees but having strong practical performance. With SpectraLDS, we can now convert convolutional layers into a real symmetric LDS without attempting non-convex optimization by projecting onto the spectral basis. Moreover, we also now have an understanding of which filters can be well represented, and the SpectraLDS layer serves as a drop-in method for learning ‘fast’ convolutional layers. As prior work has shown with the STU [2], the layer has high learning capacity on sequence-to-sequence tasks, as it is well suited for oscillating or decaying dependence. \n\nFor a signal produced by a symmetric LDS, we have now provided a strong bound for how increasing LDS parameters corresponds to reducing error. As a corollary of our main result, we have that, with a bounded input sequence, loss is bounded above by $O(e^{-p})$ when fitting an LDS with SpectraLDS, where p is the number of parameters and the constant is well understood. To our knowledge, this is the strongest scaling law for an LDS. As prior work and industry practice has shown success in modeling audio, among other signals, with increasingly large LDS models, the knowledge of scaling remains important. To be clear, our bound is still weaker than what is practically observed, but we still believe this is an important step in the right direction.\n\nMoreover, recent work [3*] has extended the STU to a larger class of LDSs, with performance dependent on the maximal imaginary component of the eigenvalue of the transition matrix. In this case, our work may have applications to a much larger class of LDSs than we had initially anticipated, and provide the basis for general LDS training and understanding.\n\nWe appreciate your feedback and thank you for sharing your honest opinions. We hope we have largely addressed your concerns, and we would be happy for you to elaborate on any concerns that you believe still remain in the paper. In our revision, we will improve the discussion section to make the impact more visible, and we will include some of the points we have raised here.\n\nThank you for your time and feedback,\n\nAuthors\n\n[1*] Voelker, A., Kajić, I., & Eliasmith, C. (2019). Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 32). \n\n[2*] Gu, A., Dao, T., Ermon, S., Rudra, A., & Ré, C. (2020). HiPPO: Recurrent Memory with Optimal Polynomial Projections. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.), Advances in Neural Information Processing Systems (Vol. 33, pp. 1474–1487). \n\n[3*] Marsden, A., & Hazan, E. (2025). Universal Sequence Preconditioning. arXiv."}]

Author rebuttal:

Meta-review: Most of the reviewers recommend acceptance, with one recommendation on borderline rejection. The main holdout argument was that the restriction to symmetric transition matrices greatly limits applicability of the algorithm. This was echoed by two other reviewers who gave a 5 rating. The reviewers did agree that while this was a serious limitation, they also all acknowledged that it would be a valuable contribution. The presented evidence suggests that the method is still applicable to real problems despite this limitation. All reviewers also thought that the core part of the algorithm required more explanation, but that could be achieved in revisions. Therefore, the AC believes that the reasons to accept outweigh the holdout reason to reject, and recommends accepting.
Decision: Accept (poster)