Title: SpectraLDS: Provable Distillation for Linear Dynamical Systems
Venue: NeurIPS 2025
Accepted: yes
Keywords: Linear Dynamical Systems

Sequence-to-Sequence Neural Networks

State-Space Models
Abstract: We present the first provable method for identifying symmetric linear dynamical systems (LDS) with accuracy guarantees that are independent of the system’s state dimension or effective memory. Our approach builds upon recent work that represents symmetric LDSs as convolutions learnable via fixed spectral transformations. We show how to invert this representation—recovering an LDS model from its spectral transform—yielding an end-to-end convex optimization procedure. This distillation preserves predictive accuracy while enabling constant-time and constant-space inference per token, independent of sequence length. We evaluate our method, SpectraLDS, as a component in sequence prediction architectures and demonstrate that accuracy is preserved while inference efficiency is improved on tasks such as language modeling.

SpectraLDS: Provable Distillation for Linear Dynamical Systems

Abstract

We present the first provable method for identifying symmetric linear dynamical systems (LDS) with accuracy guarantees that are independent of the systems' state dimension or effective memory. Our approach builds upon recent work that represents symmetric LDSs as convolutions learnable via fixed spectral transformations. We show how to invert this representation, thereby recovering an LDS model from its spectral transform and yielding an end-to-end convex optimization procedure. This distillation preserves predictive accuracy while enabling constant-time and constant-space inference per token, independent of sequence length. We evaluate our method, SpectraLDS, as a component in sequence prediction architectures and demonstrate that accuracy is preserved while inference efficiency is improved on tasks such as language modeling.

1 Introduction

The emergence of attention-based transformer architectures has revolutionized sequence modeling tasks, particularly in natural language processing [37] and large-scale sequence-to-sequence learning [16, 29]. These transformer models rely on the self-attention mechanism, which allows each token in a sequence to attend to every other token, enabling strong contextual understanding. However, this approach suffers from quadratic complexity in sequence length, making it computationally expensive for longer sequences. Recent research has thus explored alternative, more efficient architectures that preserve expressiveness while reducing computational costs for long sequences. Among these are attention-free approaches such as convolution-based or state-space models (SSMs) [9, 2, 22, 28], which can offer sub-quadratic or even near-linear time generation.

The most basic SSM, which is the starting point for all aforementioned models, are linear dynamical systems (LDS), a foundational framework for modeling sequential dependencies in control theory, signal processing, and learning theory [20, 13]:

Here u t represents the input sequence, x t encodes the past state information, and ˆ y t approximates the target sequence. By maintaining a fixed latent state, LDSs enable efficient inference and allow for efficient memory utilization. However, gradient-based approaches for learning LDSs suffer from exploding or vanishing gradients, particularly when modeling systems with long-term memory or large hidden state dimension [14].

To address these limitations, Agarwal et al. [2] leveraged the spectral filtering method [14] and introduced the Spectral Transform Unit (STU), a convex relaxation that shifts from learning the hidden

transition matrix A directly to a reparameterization of the LDS impulse response that learns how input signals convolve with fixed spectral filters. This approach provably preserves the expressiveness of an LDS with symmetric A while making the training problem far more tractable. In practice, it has proven robust for systems requiring long-range memory and empirical results show that hybrid STU models-with alternating attention and STU layers-can match or even surpass purely attentionbased architectures, as well as other popular hybrid state-space models, on long-context tasks such as language modeling and time-series prediction [22]. Furthermore, unlike self-attention, which requires O ( L 2 ) operations per L tokens during training and identically for inference with KV-caching, the STU operates in O ( L log L ) operations per L tokens during training and O ( L log 3 L ) operations per L tokens during inference through algorithms based on the Fast Fourier Transform [1, 22].

However, although STUs capture LDS-like dynamics using spectral filtering, there has been no straightforward way to convert (or 'distill') a trained STU layer back into explicit LDS form. Moreover, it has remained unclear whether every STU layer can be represented as a LDS with provably small error. Such a distillation from an STU layer to an LDS layer with hidden-dimension h would permit recurrent inference in place of convolution operations, reducing the cost of generating L tokens during inference to O ( h · L ) , while maintaining the training robustness guarantees of the STU.

Such a distillation is especially desirable in the light of the resurgence of SSMs as scalable alternatives to Transformers in long-context tasks [9, 10, 34]. Thus, our work allows one to retain the STU's performance on long sequences while enabling logarithmic-time per-token generation -providing an appealing alternative to both long convolutions and Transformer-based self-attention caching.

1.1 Our Contribution

Wepresent a novel technique for distilling STU filters into LDS form, achieving a substantial reduction in operations-from O (log 3 L ) to O (log L ) per token 1 during generation-while preserving the STU's expressivity and performance. Moreover, due to the training stability guarantees of the STU architecture, even when learning a marginally stable symmetric LDS or an LDS with large hidden dimension, this distillation procedure provides the first provable method to directly learn the parameters of a symmetric LDS of arbitrarily high effective memory and with bounded noise . Specifically:

In Algorithm 2, we show how to convert a learned STU into explicit system matrix parameters whose recurrence can be computed in logarithmic time.

We provide a theoretical analysis of this distillation in Theorem 1, and empirically demonstrate that the new LDS form incurs negligible degradation in modeling quality while improving the autoregressive generation speed.

We demonstrate in Section 6 that by applying SpectraLDS to trained STUs, we maintain the same accuracy and attain O (log L ) per-token computational generation cost.

As a consequence of our theoretical analysis, we show in Section 5.4 how we can efficiently convert a symmetric LDS of arbitrary high dimension , via an intermediate STU learning step, to one of O (log L ) dimension with minimal error.

We open-source the SpectraLDS code at https://github.com/dshah02/SpectraLDS .

2 Related Work

State-space models have long been a cornerstone of control theory and signal processing. The simplest variants, linear dynamical systems, provide a succinct way to capture temporal dependencies via a hidden state that evolves with linear transitions. Classical methods include the Kalman filter [20], which remains widely used due to its robust theoretical properties and computational efficiency.

In recent years, a new wave of SSMs have emerged as efficient alternatives to attention-based methods for long-sequence tasks, promising sub-quadratic or even near-linear complexity without

√

compromising expressive power. Models like S4 [10] and its diagonal variants [11] exploit structured state matrices to learn long-range dependencies, while works such as Hyena, Mega, and Mamba [34, 24, 9] incorporate gating or convolution-based parameterizations to compete against (and sometimes outperform) transformers in language modeling, time-series analysis, and other longcontext applications. Their growing popularity and the challenges of training SSMs underscore the need for methods with greater training robustness, efficiency, and performance guarantees. The Spectral Transform Unit (STU) [2] lies squarely in this tradition, offering a powerful convex relaxation for training LDS-like systems that achieves impressive empirical results on long-context tasks. Our work builds directly on this line of research, introducing the first method to distill a learned STU layer into an explicit LDS with provable guarantees, thereby unifying the convex training advantages of spectral filtering with the real-time inference benefits of a recurrent LDS.

Our contributions also align with the long tradition of system identification for LDSs, where the aim is to learn the hidden transition and output matrices ( A,B,C,D ) from observed sequences. Early influential approaches-such as the Ho-Kalman algorithm [15], Kung's method [21], and the Eigensystem Realization Algorithm (ERA) [18] -rely on linear-algebraic decompositions (e.g., SVD) of Hankel matrices consistent with observations. Modern variants allow for single-trajectory identification [31], and subsequent refinements like MOESP and N4SID [17] added stochastic noise modeling, while prediction-error and maximum-likelihood methods improved estimation accuracy and statistical efficiency. More recent lines of work incorporate regularization and spectral methods (e.g., stable-spline kernels, sparse identification) to yield more robust or interpretable LDS representations. In the context of distillation, these methods still require matrix decompositions that scale super-linearly in the problem size.

In contrast to many previous approaches, since the STU's parameterization avoids direct reconstruction of the system matrix A , distillation from a STU layer into an LDS remains agnostic to hidden dimension. Moreover, as our method uses a fixed and abstract Hankel matrix-rather than having to construct it anew from observed data- we can perform a significant part of the distillation computation offline.

Finally, we highlight the recent closely related work of [28], which distills state-space-like recurrences from convolution-based sequence models. Their Laughing Hyena method accelerates long convolutional filters by approximating them with a diagonal LDS, thus allowing constant-time generation at inference. While this approach generalizes to any convolution-based model (e.g., Hyena [34]), it does not provide formal guarantees on distillation quality. In contrast, we focus on the STU's spectral filters, which have expressive power comparable to a symmetric LDS with real eigenvalues, and present the first theoretical framework to convert the filters into such an LDS with provable bounds on distillation quality. By leveraging the STU's fixed bank of spectral filters, our method preserves long-sequence expressiveness while achieving a symmetric LDS realization with a guaranteed approximation error (see Section 5).

3 Token Generation and Complexity for Language Modeling

In this section, we summarize the autoregressive generation costs for three model classes, Transformers, Convolutional Models, and RNNs, considering a prompt length of T and the generation of K tokens by each model, with L the length of the convolutional filters (i.e., the maximum sequence length). We show the runtimes and memory requirements for each of the listed models in Table 1.

Attention. Processing a prompt requires O ( T 2 ) time. However, token-by-token generation can be accelerated to O ( T + K ) for each generated token via key-value caching, with a total of O ( T 2 + K ( T + K )) = O ( T 2 + K 2 ) operations and requiring O ( T + K ) space [37, 28].

Convolutional Model. A Convolutional Model with k convolutional filters will require O ( kN ) operations to autoregressively generate a new output given N inputs, and thus a naive autoregressive convolutional implementation will require O ( k · K · ( T + K )) operations to generate K tokens. A more refined 'Epoched Future Fill' algorithm with prompt prefilling can reduce this to O ( k · ( T log T + K 3 / 2 √ log K )) to generate K tokens. The "Continuous Future Fill" algorithm has theoretical guarantees of O ( k · ( T log T + K log 2 K )) operations, although it suffers from numerical instability and has not been implemented or used in practice [1]. For the guarantees of the STU architecture, we require k = O (log L ) and in practice we choose k = 24 [22].

4 Problem Background

In this section, we survey the fundamentals relevant to our approach. First, we discuss linear dynamical systems and the inherent challenges of training them directly on tasks requiring long memory. We then present the main theoretical results of spectral filtering and outline how the Spectral Transform Unit (STU) leverages fixed spectral filters to model linear recurrences without explicitly learning the transition matrix A . Finally, we set the stage for our method of distilling STUs back into linear dynamical systems.

4.1 Linear Dynamical Systems

Linear dynamical systems (LDS) have been widely used in control theory to represent time-dependent processes, forming the basis of classical state-space formulations and optimal control methods [20, 23, 3, 19, 6]. Concretely, we consider an input sequence u 1 , u 2 , . . . , u t ∈ R n , and the corresponding output sequence y 1 , y 2 , . . . , y t ∈ R m . The hidden state x t ∈ R d summarizes the system's memory of past inputs, with the evolution of the system being represented as

where A ∈ R d × d , B ∈ R d × n , C ∈ R m × d , and D ∈ R m × n . We omit dynamics and observation noise terms for simplicity with this derivation, although we test our methods on signals with noise.

Expanding the LDS to the Convolutional Form. In a noiseless environment with x 0 = ⃗ 0 , we can expand the LDS equations as follows:

Note that if any eigenvalue | λ i ( A ) | > 1 , the system becomes unstable and y t may tend to infinity in magnitude. Even for | λ i ( A ) | < 1 , systems with ∥ A ∥ ≈ 1 are prone to failure due to large A i powers in backpropagation, as a noisy algorithm may approximate A with spectral radius greater than 1 during training. If | λ i ( A ) | < 1 -δ for some spectral gap δ > 0 , then

where τ = O ( 1 δ log 1 ε ) and the effective memory is thus on the order 1 δ [2]. As δ → 0 , learning A directly becomes unstable for large contexts [5, 32, 30], highlighting the need for methods such as spectral filtering. Since the D matrix serves as a skip connection that does not affect the representation capacity, we fix it to a 0 -matrix and omit its consideration for the remainder of this paper. We will sometimes use the shorthand LDS ( C, A, B ) to refer to a linear dynamical system with those parameters and D = 0 .

Additionally, for the remainder of this paper, we restrict our attention to systems where A is a symmetric real matrix. An LDS with symmetric A can be diagonalized over the real numbers, making it equivalent to an LDS with a diagonal A . Without loss of generality, we therefore assume A is diagonal with eigenvalues α 1 , . . . , α d .

Spectral Filtering. With initial state x 0 = ⃗ 0 , defining µ ( α ) = (1 , α, α 2 , . . . ) , and indexing the columns of C as c ℓ and the rows of B as b ℓ , we can extend the convolutional representation:

To circumvent the non-convex optimization problem of finding α that best fit an LDS, [14] propose the spectral filtering algorithm, which learns an approximation of µ ( α ) in a convex manner. They prove that given eigenvalue-eigenvector pairs { σ j , ϕ j } j of the Hankel matrix Z ,

any µ ( α ) with 0 ≤ α ≤ 1 can be approximated by the top k eigenvectors { ϕ 1 , . . . , ϕ k } of Z with an exponentially decreasing error in k . Thus, if y t is generated by a PSD linear dynamical system, we have the following result:

where we define ˜ µ ( α ) := ∑ k i =1 ⟨ µ ( α ) , ϕ i ⟩ ϕ i and learn suitable parameters M j ∈ R m × n . Rather than depending on powers of A , learning an LDS with this parameterization remains convex in { M j } , since eigenvectors { ϕ j } from the matrix Z are computed offline.

The Spectral Transform Unit. To account for learning negative eigenvalues of A , the spectral filtering construction can be adapted by introducing positive and negative sets of feature maps. If { σ j , ϕ j } are the eigenvalue-eigenvector pairs of Z , then for each time t and each respective filter ϕ j , we define the projections of the inputs onto the spectral basis:

One then forms the output by learning linear combinations of both U + t,k and U -t,k and an optional autoregressive term [12, 2]:

Without the autoregressive component we compute y SF t = ∑ k j =1 M ϕ + j U + t,j + ∑ k j =1 M ϕ -j U -t,j . The above expression is considered the Spectral Transform Unit , where { M j } is the set of parameters to be learned using a differentiable algorithm. Following [22], we consider the STU without the autoregressive component. Empirical evidence [22] shows that hybrid STU models can compete with or even outperform purely attention-based architectures.

Error Bounds for Spectral Approximation. We repeat the result from [14] (see also [2, 26, 27]) stating that given any LDS parameterized by A,B,C where A is a symmetric matrix with ∥ A ∥ ≤ 1 , there exist matrices M ϕ + 1 , . . . , M ϕ + k , M ϕ -1 , . . . , M ϕ -k such that for all L and for all input sequences u 1 , . . . , u L , with ∥ u t ∥ ≤ 1 , the following holds for all t ∈ [ L ] :

where k is the number of spectral filters, y LDS t is the sequence generated by the LDS, and y SF t is the sequence generated by spectral filtering.

Therefore, one can approximate any LDS that meets these specifications up to error ε by selecting k = O ( log L log ( 1 ε )) spectral filters. Thus, the STU can capture LDS dynamics with only a logarithmic number of filters k , providing a compact and stable representation even for systems with high effective memory ( ∥ A ∥ ≈ 1 ).

4.2 Distilling STU into an LDS

While the STU avoids direct learning of A , it still implements LDS-like dynamics via spectral filtering. A natural question is whether we can recover explicit parameters ( ˜ A, ˜ B, ˜ C ) from the learned STU, enabling an equivalent recurrence rather than O ( L ) costs when convolving spectral filters with input sequences. This can be achieved by approximating the convolution kernel of the STU by the implicit convolution kernel of an LDS. Such a distillation would bridge the gap between the stable convex training of the STU and the fast inference of a recurrent LDS.

5 Algorithm and Main Result

We now present our main theoretical result and accompanying algorithm, which shows how to recover an accurate LDS representation from the learned spectral filters. Concretely, we demonstrate that each STU filter ϕ j can be approximated by a linear combination of geometrically decaying LDS impulse response filters (i.e., the LDS implicit convolutional kernel).

5.1 A General Transformation from Spectral Filters to LDS

Our main result is given in Algorithm 2. For the dynamics LDS (1 -α, α, 1) , we denote the impulse response filter by

Note that µ L ( α ) ∗ u 1: L = ∑ L i =1 (1 -α ) α i -1 · u L -1 = LDS (1 -α, α, 1)( u 1: L ) , and thus inference with the LDS is the same as convolution with its impulse response filter. Let ϕ 1 , . . . , ϕ k ∈ R L be the spectral filters of length L . We write the first k filters in matrix form as Φ 1: k ∈ R k × L , such that the i -th row is ϕ i .

A first observation is that we can write any LDS impulse filter approximately in the spectral basis. This is a direct consequence of the spectral filtering methodology for learning an LDS [14].

Algorithm 1 FindSpectralRepresentation

1: Input: Scalar LDS parameter α ∈ R , representation size k .

2: Output: Spectral parameters m ∈ R k .

3: Construct the impulse response vector µ L ( α ) ∈ R L as µ L ( α ) = (1 -α )[1 , α i , α 2 i , . . . , α L -1 i ] .

4: return best spectral fit of the system over random inputs u ∈ R L using gradient descent

As a consequence of results from spectral filtering for learning LDSs, we show in Appendix A.4 that the procedure FindSpectralRepresentation returns a vector m for which, for some constant c > 0 ,

We proceed to use this subroutine to find a distillation to the spectral filters.

Our main performance guarantee is given in the following theorem.

Theorem 1. As long as h ≥ k , Algorithm 2 returns w.h.p. a matrix ˜ M such that

where λ max is the largest eigenvalue of the Penrose-Moore pseudo inverse of the matrix M .

Algorithm 2 Spectral Filters to LDS Filters

1: Input: The first k spectral filters matrix Φ 1: k ∈ R k × L ; and parameter h > k .

2: Output: The transformation matrix ˜ M .

3: Sample h randomly chosen independent scalars α 1 , ..., α h with each α ∈ [0 , 1] .

4: Construct the matrix µ L ( α 1: h ) ∈ R h × L with i th row µ L ( α i ) = (1 -α i )[1 , α i , α 2 i , . . . , α L -1 i ] .

5: For each scalar impulse response, find the spectral representation by

6: Let M be the h × k matrix whose rows are m i .

7: return ˜ M := M -1 ∈ R k × h

The significance of Theorem 1 is that it allows us to translate between the representation of spectral filters and linear dynamical systems.

We note that it is not immediate to upper bound λ max . Indeed, for h ∼ k , this can be exponentially large in k , as it corresponds to the condition number of a Vandermonde matrix [4]. However, we note experimentally, under the distribution described in the practical algorithm below, that as h grows, λ max quickly becomes smaller. This is an overparametrization effect, which we show experimentally in Appendix A.3. We provide analysis of Theorem 1 in Appendix A.4.

5.2 Practical Algorithm

For improved practical performance we adopt the following procedure. Fix H ≫ h . As above, let µ L ( α 1: H ) ∈ R H × L denote H impulse responses and let Φ 1: k ∈ R k × L denote the first k spectral filters. To obtain a representation of size h , we select a small index set S ⊆ { 1 , . . . , H } , | S | = h ≪ H , by running multi-target Orthogonal Matching Pursuit (OMP) [25, 36] on the regression

OMP greedily sets one coefficient per step from 0 to reduce the regression loss, with the selected coefficients after h steps providing the indices for S . Given S , we solve the unregularized leastsquares problem

to yield Φ 1: k ≈ ˜ Mµ L ( α 1: H ) , where ˜ M = ( X ∗ ) ⊤ .

Another change is the distribution from which the α i are drawn. We employ symmetric, nearunit-radius sampling to emphasize long-memory behavior: for each i , draw U i ∼ Unif[0 , 1] and S i ∼ Unif ( {-1 , 1 } ) , and set

Equivalently, | α i | d = 1 -U 4 i with an independent random sign. This deliberately skews mass toward | α i | ≈ 1 , improving reconstruction at a better h to H ratio. We ablate this choice in Appendix A.9. For numerical stability we use float64 arithmetic end-to-end and scale Φ 1: k by s num > 0 so that ∥ µ L ( α 1: H ) ∥ F ≈ ∥ s num Φ 1: k ∥ F . To mitigate the impact of magnitude on OMP, we normalize the columns of µ L ( α 1: H ) ⊤ . We later undo the scaling. In Figure 1 we report the reconstruction error of Φ 1: k against an LDS of hidden dimension h produced by this practical method.

5.3 Converting ˜ M into an LDS.

As a result of Theorem 1, with A := Diag ( α 1 , . . . , α h ) , Γ := Diag (1 -α 1 , . . . , 1 -α h ) and µ := µ L ( α 1 , . . . , α h ) , we can replace the costly STU convolutions:

Reconstruction Error vs h

10-7

Φ1:24Reconstruction Error

scale)

10-11

(log

10-15

MSH

10-19

10-23

0

25

50

75

100

125

150

175

200

Hidden Dimension h

This provides the basis of our autoregressive inference advantage: rather than computing U + t,j and U -t,j as pure convolutions, we can maintain the hidden state for LDS ( ˜ M j Γ , A, ⃗ 1) and LDS ( ˜ M j Γ , -A, ⃗ 1) for the O ( h ) computation during inference. For practical efficiency, we can compute all U + t, 1 , . . . , U + t,k , U -t, 1 . . . U -t,k simultaneously with a single LDS by leveraging the similarities in the state updates (see Appendix A.7).

5.4 LDS to LDS Distillation

A direct consequence of our approach is that we can distill any high-dimensional symmetric LDS into a low-dimensional LDS with a bounded error. For an LDS with input dimension d in and output dimension d out , spectral filtering provides an ε -approximation with only O ( d in · d out · log L · log( 1 ε )) parameters regardless of the hidden dimension.

Accepting that hλ max is O (1) for h ≫ k , as justified in Appendix A.3, we can then convert this spectral representation by application of our distillation procedure into an LDS with state dimension d in · h , where with h = O ( k ) = O (log L log ( 1 ε ) ) we maintain ε error. Thus, the distilled LDS has O ( d in · d out · log L · log( 1 ε )) total parameters. In other words, the combination of spectral filtering for LDS learning and the following distillation step yields a practical method to reduce the state dimension while preserving the system's dynamics within tight error bounds. Empirically, for tasks as difficult as modeling language, we only require k ≤ 24 filters for strong performance, and the state dimension h ≥ 80 of the distilled LDS suffices to retain performance.

6 Experiments

To demonstrate the effectiveness of our distillation algorithm, we begin by illustrating that a lowdimensional LDS can effectively approximate the spectral filters in Figure 3 and further examine the eigenvalues of the resulting system in Appendix A.7. Building on this, to further quantify the effectiveness of Algorithm 2 in fitting the spectral filters with practically efficient LDSs, we plot the reconstruction error of the spectral filters across different LDS state dimensions in Figure 1. These results show that our algorithm, guided by Theorem 1 amply reduces the approximation error to a sufficiently small level without an excessive increase in state dimension.

500

600

SpectraLDS

Attention

LDS

400

500

Mamba2

SSOT

300

SSOT

400

MSE

MSE

200

300

100

200

0

0

1000

2000

3000

4000

5000

100

0

1000

2000

3000

4000

5000

Step

Step

SpectralFilters

Selected LDSImpulse Responses

ReconstructedFilters

×10-5

×10-5

P5

Φ15

0.4

Φ3

Φ10

20

2

0.2

2

0

0.0

0

LDS 12

LDS 21

0.2

LDS 27

LDS 32

LDS 51

LDS 91

0.4

LDS 109

LDS 137

0

00000090009000000000070001

8000

0

000800020009000900000000007000

0

1000

000000090009000000080007

8000

To validate our method quantitatively, we perform synthetic experiments comparing SpectraLDS against other benchmarks on learning high-memory LDSs (see Appendix A.10). We find that our method significantly outperforms strong baselines in both sample efficiency and reconstruction accuracy, confirming that our approach is well suited for learning systems with long-range dependencies.

Turning to the large-scale evaluation, we distill a 340M-parameter FlashSTU model [22] into an LDS-based architecture and compare its performance across a suite of language benchmarks. From the results in Table 2, we point out that despite the change from convolution-based spectral filters to an explicit LDS representation for the STU layers, the performance remains identical across all tasks. This observation supports our claim that the STU can be closely approximated by a low-dimensional LDS without compromising predictive accuracy. We provide details of the experimental setup and hyperparameters for the models used in Appendix A.13.

Runtime for STU-Only Models

Runtime for Hybrid Models

Runtime for Convolution Only

5000

5000

1200

SpectraLDS (SD 800)

SpectraLDS (SD 800)

SpectraLDS (SD 800)

(seconds)

4000

Baseline Naive Conv

4000

Baseline Naive Conv

1000·

SpectraLDS (SD 100)

Epoched Future Fill

Epoched Future Fill

Baseline Naive Conv

3000

3000

800

Epoched Future Fill

Runtime

·009

2000

2000

400

1000

1000

200

0

8192

16384

32768

65536

131072

262144

8192

16384

32768

65536

131072

262144

32K°

65K

131K

262K

524K·

1048K·

Finally, we present various measures of inference speed to illustrate the constant-time per token generation provided by the distilled STU. In Figure 4, we compare the inference speed of a distilled STU model against a naive convolutional approach and a numerically stable FutureFill variant [1]. In a 12 STU layer model, the naive convolution exhibits quadratic runtime growth with sequence length; the FutureFill variants achieve logarithmic growth; and the distilled STU-to-LDS model demonstrates the best performance with linear growth.

In the hybrid model with 6 attention and 6 STU layers, we find the distilled LDS implementation still provides a significant performance increase. Additionally, we note increases in the LDS state dimension have little impact on the overall runtime, indicating that LDS operations are not a compute bottleneck (see Appendix A.11). For both the hybrid and STU-only models up to 16 , 384 tokens, the distilled LDS, naive convolution, and Epoched Future Fill all have similar runtimes. All performance benchmarks were conducted on a single H100 GPU, with each generation process evaluated separately to ensure consistent measurements. We provide full experiment details in Appendix A.12.

Looking ahead, further investigation is warranted to better understand how convolution, LDS, and attention layers interact at the hardware level, and to optimize their coordination for even greater speedup. Additionally, further work is required to determine if the LDS layer is stable below float64.

To conclude, since optimized transformer implementations suffer from KV-Cache memory bottlenecks rather than compute bottlenecks [9], and the LDS layers have drastically lower memory requirements, we anticipate that, with appropriate optimization, the inference speed of hybrid attention-STU architectures will be independent of the amount of LDS layers.

7 Conclusion and Discussion

We have provided the first provable technique for learning the parameters of a symmetric LDS with arbitrarily high effective memory. By leveraging their convex-learning approach, we show how spectral filters can be distilled into an explicit LDS representation, enabling the construction of a state-space model with logarithmic-time inference and theoretical guarantees on the loss bounds.

We have provided a lower bound on how increasing state dimension affects reconstruction loss for a linear dynamical system, and we envision SpectraLDS as a drop-in replacement for autoregressive convolutional layers in certain sequence-to-sequence tasks.

8 Acknowledgments

EH gratefully acknowledges support from the Office of Naval Research and Open Philanthropy.

References

Naman Agarwal, Xinyi Chen, Evan Dogariu, Devan Shah, Hubert Strauss, Vlad Feinberg, Daniel Suo, Peter Bartlett, and Elad Hazan. Futurefill: Fast generation from convolutional sequence models, 2025.

Naman Agarwal, Daniel Suo, Xinyi Chen, and Elad Hazan. Spectral state space models, 2024.

Brian D.O. Anderson and John B. Moore. Linear Optimal Control . Prentice-Hall, 1971.

Bernhard Beckermann. The condition number of real vandermonde, krylov and positive definite hankel matrices. Numerische Mathematik , 85(4):553-577, 2000.

Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks , 5(2):157-166, 1994.

Dimitri P Bertsekas. Dynamic Programming and Optimal Control , volume 1. Athena Scientific, 2005.

Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR) , 2024.

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024.

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2022.

Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models, 2022.

E. Hazan, H. Lee, K. Singh, C. Zhang, and Y. Zhang. Spectral filtering for general linear dynamical systems, 2018.

Elad Hazan and Karan Singh. Introduction to online nonstochastic control. arXiv preprint arXiv:2211.09619 , 2022.

Elad Hazan, Karan Singh, and Cyril Zhang. Learning linear dynamical systems via spectral filtering, 2017.

B. L. Ho and R. E. Kalman. Effective construction of linear state-variable models from input/output functions. Automatica , 3(1):49-59, 1966.

A. Huang, S. Subramanian, J. Sum, K. Almubarak, S. Biderman, and S. Rush. The annotated transformer, 2022.

I. W. Jamaludin, N. A. Wahab, N. S. Khalid, S Sahlan, Z. Ibrahim, and M F. Rahmat. N4sid and moesp subspace identification methods. In 2013 IEEE 9th International Colloquium on Signal Processing and its Applications , pages 140-145, 2013.

Jer-Nan Juang and Richard S. Pappa. An eigensystem realization algorithm for modal parameter identification and model reduction. Journal of Guidance, Control, and Dynamics , 8(5):620-627, 1985.

Thomas Kailath. Linear Systems . Prentice-Hall, 1980.

Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Journal of Basic Engineering , 82.1:35-45, 1960.

S. Y. Kung. A new identification and model reduction algorithm via singular value decompositions. In Proc. 12th Asilomar Conf. Circuits, Systems and Computers , pages 705-714, 1978.

Y. I. Liu, W. Nguyen, Y. Devre, E. Dogariu, A. Majumdar, and E. Hazan. Flash stu: Fast spectral transform units, 2024.

David Luenberger. An introduction to observers. IEEE Transactions on Automatic Control , 16(6):596-602, 1971.

Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention, 2023.

S.G. Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions on Signal Processing , 41(12):3397-3415, 1993.

Annie Marsden, Evan Dogariu, Naman Agarwal, Xinyi Chen, Daniel Suo, and Elad Hazan. Provable length generalization in sequence prediction via spectral filtering, 2024.

Annie Marsden and Elad Hazan. Universal sequence preconditioning. arXiv preprint arXiv:2502.06545 , 2025.

Stefano Massaroli, Michael Poli, Daniel Y. Fu, Hermann Kumbong, Rom N. Parnichkun, Aman Timalsina, David W. Romero, Quinn McIntyre, Beidi Chen, Atri Rudra, Ce Zhang, Christopher Ré, Stefano Ermon, and Yoshua Bengio. Laughing hyena distillery: Extracting compact recurrences from convolutions, 2023.

OpenAI. Gpt-4 technical report, 2024.

A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resurrecting recurrent neural networks for long sequences, 2023.

Samet Oymak and Necmiye Ozay. Non-asymptotic identification of lti systems from a single trajectory. In 2019 American control conference (ACC) , pages 5655-5661. IEEE, 2019.

R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks, 2013.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models, 2023.

Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022.

Ron Rubinstein, Michael Zibulevsky, and Michael Elad. Efficient implementation of the k-svd algorithm using batch orthogonal matching pursuit. CS Technion , 40, 01 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.

A Appendix

A.1 Notation throughout the paper

A.2 Experimental details and notation

All experiments were performed on Nvidia H100-80GB GPUs in PyTorch [33]. All layers except the LDS leverage bfloat16 precision, whereas the LDS layers require float64 precision. All tests, unless otherwise stated, use k = 24 spectral filters and replace the k filters with an LDS with state-dimension h = 80 . To fit the additional 24 alternating filters of the negative component (i.e., ϕ -j , where ϕ -j [ i ] = ( -1) i -1 ϕ j [ i ] , as is needed to compute U -t,j purely convolutionally), we expand to an LDS with state-dimension h = 160 .

For the layer-level inference speed benchmarks, we leverage the STU with tensor-dot approximation (STU-T) [2], the same variant used in the FlashSTU language model [22]. It is worth noting that the STU-to-LDS distillation leads to LDS layers with comparable speed regardless of whether the tensor-dot approximation is employed, and thus this approximation only accelerates the STU layers in benchmarks. A formal definition of this approximation is as follows:

STU Tensor-Dot Approximation: [2] introduced an important optimization, the tensor-dot approximation, wherein each tensor M ∈ R d in × 2 k × d out , representing a concatenation of the tensors M ϕ + 1 , . . . , M ϕ + k , M ϕ -1 , . . . , M ϕ -k , is learned as M (1) × M (2) for M (1) ∈ R d in × 2 k and M (2) ∈ R d in × d out . This approximation allows for a reduction in convolutions as, with input x 1 , x 2 , . . . , x ℓ , we can compute y SF ℓ ≈ ∑ ℓ i =1 ( x ⊤ ℓ -i +1 M (2) ) ⊙ M filters [ i ] , where M filters = [ ϕ 1 , . . . , ϕ k , ϕ -1 , . . . , ϕ -k ] ⊤ M (1) ∈ R L × d out and ⊙ refers to the Schur product (i.e. ( x ⊙ y ) j = x j · y j ). This allows for only d out convolutions with the tensor-dot approximation, as opposed to k · d in convolutions without it. Although this method can reduce expressivity and does not inherit the same marginal-stability guarantees, it maintains competitive empirical performance while yielding significant improvements in efficiency.

Additionally, we frequently refer to the impulse response of the LDS and STU models. The impulse response of a linear sequence model f : R L → R 1 is the vector or convolutional kernel ψ ∈ R L that is equivalent to f (i.e. f ( x [1 ,...L ] ) = ψ ∗ x [1 ,...L ] ), and thus:

We only consider the impulse response of f : R L → R 1 , but the impulse response is closely related to the derivative with respect to the inputs and is generalized identically. For an STU model with d in = d out = 1 and parameters M ϕ + 1 , . . . , M ϕ + k , M ϕ -1 , . . . , M ϕ -k ∈ R , the impulse response ψ SF is thus described by ψ SF [ t ] = ∑ k j =1 M ϕ + j ϕ j [ t ] + ( -1) t -1 ∑ k j =1 M ϕ -j ϕ j [ t ] . Similarly, for an LDS with d in = d out = 1 and parameters a, b, c ∈ R , the impulse response ψ LDS is described by ψ LDS [ t ] = ca t -1 b

A.3 Experimental Results on the Condition Number of M , the Spectral Coefficients Matrix

In Figure 5, we present experimental results on the condition number of M as defined in Section 5. Recall that M ∈ R h × k is the spectral coefficients matrix produced by Algorithm 2; its i -th row m ⊤ i stores the coefficients that express the LDS impulse response µ L ( α i ) (with geometric decay factor α i ) in the spectral basis Φ 1: k . Starting at h = k with k = 24 , we repeatedly add additional independent vectors α i and measure the maximum singular value of the constructed M -1 . The blue central line shows the mean maximum singular value across 5 experiments for a given value of h , with the shaded region showing the maximum and minimum of the largest singular values across experiments. We draw α independently from the distribution in Figure 7. To justify our statement in section 5 . 2 that λ max · h can be considered O (1) , we additionally plot λ max · h under the same experimental setup in Figure 6.

Singular Value vs. Sample Size

1012

Largest Singular Value (log scale)

1010

108

106

104

102

0

250

500

750

1000

1250

1500

1750

2000

Sample Size

1015

1013

Amaxzh (log scale)

1011

109

107

105

250

500

750

1000

1250

1500

1750

2000

Sample Size

A.4 Analysis of Theorem 1

Proof of Theorem 1. Let

Histogram of α

103

Frequency (log scale)

102

101

100

-1.00-0.75-0.50-0.25

0.00

0.25

0.50

0.75

1.00

Value

be the L -impulse response of the one dimensional linear dynamical system with parameters α, b = 1 , c = 1 -α . Lemma 11.3 from [13] asserts that for any α ∈ [0 , 1] , there exist real coefficients m 1 , . . . , m k ∈ R k and constant c > 0 , such that any sequence of inputs u L :1 ∈ R L with ∥ u L :1 ∥ ∞ ≤ 1 ,

The RHS term represents the evolution of the LDS, with system matrices b, c that are assumed to be identity, and µ L ( α ) is the system evolution.

As the above is true for all u L :1 , for each α j , there exist m j such that,

Moreover, as the optimization problem is convex in m j and Lipschitz continuous, FindSpectralRepresentation will return such m j .

Let M ∈ R h × k be the matrix whose rows are m j ∈ R k :

Let E = M Φ 1: k -µ L ( α 1 , . . . , α h ) . By the triangle inequality and (4),

Thus, assuming that M is full rank, multiplying E by the Penrose-Moore pseudo-inverse of M and using Holder's inequality, we get

It remains to argue that M is full rank. This follows since Φ 1: k is an orthogonal basis, and the matrix µ L ( α 1: h ) is a Vandermonde matrix. Thus, both matrices are full rank.

A.5 Learning a Symmetric LDS with and without noise

Figure 2 in the main paper compares SpectraLDS to other system-identification methods on the task of learning an arbitrary symmetric LDS with and without noise. The LDS signal has hidden

dimension 256, input and output dimension 5, and maximum eigenvalue magnitude 0.99 with Gaussian initialization. Each step provides a Gaussian input sequence of length 100 with variance 1 /d in and the final output. For the learning with noise experiment, Gaussian noise with 0.5 variance was added to the hidden state at each step, and Gaussian noise with 5 variance was added to the output. Each architecture was trained with the default PyTorch RMSProp optimizer configuration, except for the LDS, which required a lower learning rate to converge. The Ho-Kalman matrices are recomputed every 20 steps for computational ease, while the other methods were updated every step, and the Ho-Kalman parameters are the maximum for an input of length 100 (that is T_horizon = 99, T1 = 49, T2 = 49, state_dim_est = 48). The shaded region in the figure shows the 95% confidence interval over 8 runs.

A.6 Fit of the Spectral Filters with a Linear Dynamical System (State Dimension 160)

?4

0.00020

Original

0.0003

0.0004

0.00015

Approximated

0.0002

0.0003

0.0002

0.0002

0.00010

0.00005

0.0001

0.0001

0.0001

0.0000

0.00000

0.0000

0.0000

0.0001

0.00005

0.0001

0.0001

0.0002

0.00010

0.0002

0.0003

0.00015

0.0002

0.0004

0

50

100150200

0

50

100150200

0

50100150200

0

50100150200

P5

P6

P8

0.00075

0.0004

0.0004

0.00050

0.0010

0.0002

0.0002

0.00025

0.0005

0.0000

0.0000

0.00000

0.0002

0.00025

0.0000

0.0002

0.0004

-0.00050

0.0004

-0.0006

-0.00075

-0.0005

0.00100

0

50

100150200

0

50100150200

0

50100150200

0

50100150200

Φ10

Φ11

Φ12

0.0015

0.0015

0.0020

0.003

0.0015

0.0010

0.0010

0.0010

0.002

0.0005

0.0005

0.0005

0.001

0.0000

0.0000

0.0000

-0.0005

-0.0005

0.000

0.0010

0.0005

-0.0010

0.001

0.0015

0.0010

0.0015

0.0020

0.002

0

50

100150200

0

50

100150200

0

50100150200

0

50100150200

Φ13

P14

Φ15

P16

0.003

0.004

0.006

0.002

0.004

0.002

0.004

0.001

0.002

0.000

0.002

0.000

0.000

0.001

0.000

0.002

0.002

0.002

0.004

0.004

0.003

0.002

0.006

0.006

0.004

0.008

0

50

100150200

0

50100150200

0

50100150200

0

50100150200

Φ17

Φ18

P19

Φ20

0.0100

0.015

0.041

0.0075

0.010

0.0050

0.01

0.03

0.005

0.0025

0.02

0.00

0.0000

0.000

0.01

0.0025

0.005

0.01

0.00

0.0075

0.0050

-0.010

0.02

0.01

-0.0100

0.015

0.02

0

50100150200

0

50100150200

0.03

0

50100150200

0

50

100150200

P21

Φ23

Φ24

0.075

0.10

0.7

0.04

0.050

0.05

0.6

0.025

0.02

0.00

0.5

0.000

0.00

0.025

0.05

0.4

0.050

0.10

0.3

0.02

0.075

0.15

0.2

0.04

0.100

0.20

0.1

0.125

0.25

0.0

0

50

100150200

0

50

100150200

0

50

100150200

0

50

100150200

A.7 State Dynamics of the Linear Dynamical System

We train a linear dynamical system with state dimension 160 to fit the Spectral Filters and alternating filters using Algorithm 2 and examine the eigenvalues of the resulting system in Figure 9. We only plot the 80 eigenvalues corresponding to the Spectral Filters, as the remaining 80 corresponding to the alternating filters are identical except multiplied by -1 (see below the plot). The reconstruction error of the spectral filters for the trained LDS is 7 . 69 × 10 -19 . We transform the eigenvalues to visualize the full range of the values as they approach 1 and differentiate the negative and positive eigenvalues as blue and red (e.g, 0 . 97 and -0 . 97 would both map to ≈ 3 . 5 , although the first would join a red column and the second blue).

Distribution of Transformed Eigenvalues f(α) = - In(1 - |xl)

5

Positive

Negative

3

Count

2

0

0.0

2.5

5.0

7.5

10.0

12.5

(0.00000)

(0.91792)

(0.99326)

(0.99945)

(0.999955)

(0.999996)

Transformed Eigenvalue (Original)

For practical efficiency, we can compute all U t, 1 , . . . , U t,k , U t, 1 . . . U t,k simultaneously with a single LDS parameterized by {[ ˜ M Γ 0 0 M Γ ] , [ A 0 0 -A ] , 1 2 h } . This LDS has a state dimension of 2 hm where each of the m input dimensions have a hidden state updated by [ A 0 0 -A ]

dimension differs from 2 h as we treat m as a batch dimension, resulting in Bu t ∈ R 2 h × m rather than R 2 h . This mimics an LDS acting independently across each of the m dimensions, equivalent to treating m as a batch-axis, and this is the analog to how each of the k spectral filters convolve along each input dimension independently. The above Figure 9 showcases the eigenvalues of matrix A

whereas for practical deployment, the state matrix is

+ + --˜ , . The hidden state , .

[ A 0 0 -A ]

A.8 Reconstruction Error of the Spectral Filters varying LDS State Dimension and H

We demonstrate the efficacy of the practical algorithm in fitting the Spectral Filters with a low initial LDS State Dimension. In Figure 10, we ablate the choices of H and h for the practical algorithm. We do not fully understand why larger H leads to worse fits for small state dimension. Notably, the error quickly becomes no longer human perceptible. In this procedure we only aim to fit the k positive spectral filters. To fit both the k positive spectral filters and k negative spectral filters, we can achieve the same error with double the state dimension, as described in Appendix A.7. With our implementation of sparsity, the resulting hidden dimension will be near but not exactly that requested, which is why each line has different reconstruction dimensions.

Reconstruction Error vs h

10-7

H =1000

H=2000

H =5000

10-9

H = 10000

H =20000

10-11

scal

10-13

(log

10-15

MSE

10-17

10-19

10-21

10-23

0

50

100

150

200

Hidden Dimension h

A.9 Reconstruction Error of the Spectral Filters with the Practical Algorithm and Uniform α Distribution

In Figure 11, we illustrate that if rather than choosing ( | α i | d = 1 -U 4 ) with U ∼ Unif [0 , 1] with an independent random sign, we instead choose α i ∈ [ -1 , 1] , we require larger H to achieve similar performance. At sufficiently large H , uniform sampling will have sufficient coverage of α near 1 and -1 . Thus, we instead prefer a distribution with a natural bias towards these extreme points, as α ∈ [ -0 . 9 , 0 . 9] decay towards 0 quickly.

A.10 Experiments on Synthetic Tasks

We evaluate SpectraLDS (state dimension 160 ) against a strong baseline on a synthetic task, reporting results in Table 3 and Figure 12. A symmetric linear dynamical system with input dimension d in and state dimension d h is initialized by drawing all entries from a standard normal distribution, after which the update matrix A is rescaled so its largest eigenvalue has magnitude 1 -δ . We then distill a SpectraLDS model from an STU model trained with AdaGrad (learning rate 1 . 0 ) for 2000 steps; each step minimizes the mean-squared-error (MSE) between the STU output and the ground-truth LDS output over batches of 32 sequences of length seq_len . For the baseline, we fit a randomly initialized symmetric LDS with AdaGrad (learning rate 0 . 0001 ) under the same loss. To keep training times practical, most baseline runs use a shorter sequence length, and several runs were stopped early due to computational limits, marked with an asterisk. When the state dimension satisfies d h ≥ 1000 , baseline training with a learning rate of 0 . 01 becomes unstable and often diverges, whereas the STU remains stable even with a learning rate of 1 . 0 .

Reconstruction Error under Uniform α Distribution

10-7

H=1000

H=2000

10-9

H=5000

H=20000

10-11

cale)

10-13

SC

10-15

E

MSI

10-17

10-19

10-21

10-23

0

50

100

150

200

Hidden Dimension h

Loss Comparison on learning Symmetric LDSs

100

LDS 8192

LDS1024

SpectraLDS

10-1

(log scale)

10-2

LOSS

10-3

10-4

Steps

10

1002000

10

1002000

10

1002000

Delta

=10-2

= 10-3

= 10-4

Execution Times:LDS 8192 (2150s)—LDS 1024 (248s)—SpectraLDS (56s)

A.11 Layer Speeds of SpectraLDS and STU

We benchmark the inference speed of a single SpectraLDS layer across several state dimensions and compare it with two accelerated schemes for computing the STU convolution: Epoched Future Fill [1] and an STU that employs the tensor-dot approximation [22]. The SpectraLDS layer itself is produced by applying STU-to-LDS distillation to the tensor-dot STU. For the timings reported in Table 4, each model is evaluated on L (Seq. Len.) autoregressive convolutions on inputs of dimension 128 , and the mean and standard deviation over five runs are recorded. All STU variants use a filter length of 1 , 048 , 576 to accommodate the longest sequences. Although every architecture begins in the linear-scaling regime, only SpectraLDS continues to scale favorably as the sequence length increases. The resulting layer-speed comparison is visualized in Figure 13; benchmarks of SpectraLDS layers embedded in large language-model architectures appear in Appendix A.12.

Layer Speed Comparison: SpectraLDS vs STU

STU Future Fill

106

STU Tensor Dot

STU

SpectraLDS (SD 100)

SpectraLDS (SD 800)

105

(su)

Inference Time

104

103

102

101

32

128

512

2K

8K

32K

128K

512K

SequenceLength

A.12 FlashSTU Ablations

A.12.1 Implementation Architecture Details

To perform the FlashSTU performance evaluations with STU-T, we employ the architecture depicted in Figure 14. We test token generation efficiency with a hybrid architecture, alternating between layers using STU-T and sliding window attention (SWA), and we additionally test with each layer using STU-T only. Each layer consists of RMS-Norm, followed by STU-T or SWA, followed by RMS-Norm, and then followed by an MLP. Inputs are tokenized by the o200k_base tokenizer and the FlashSTU model begins with an embedding layer, which is weight-tyed to the output unembedding layer. To start generation, we add special tokens such as 
 and 
 .

The sliding window attention layers leverage Flash Attention v2 [8, 7] and ALiBi position embeddings [35]. The tested model has input dimension 896 and 12 layers, which has 550 . 31 million parameters for the hybrid model, and 535 . 99 million parameters for the STU-only model. All layers are run in bfloat16 except for the LDS distilled layers, which require float64 precision. The Flash STU-T leverages the STU with tensor-dot [22] approximation rather than the base STU layer for faster inference, and thus we perform the STU-to-LDS distillation on the STU with tensor-dot approximation. For tests with generation length up to 131072 , we leverage STU filter length of 131072 . For the generation length of 262144 , we leverage an STU filter length of 262144 . For each runtime measurement, we first run a warmup generation, before reporting the mean of two generations of that length. All benchmarks include only inference time and not model setup or filter computation costs. Additionally, each MLP layer has hidden dimension 12 × the input dimension (MLP expansion factor). Other configuration details are identical to those in Table 7.

x

Input

RMSNorm

STU-T/SWA

XN

RMSNorm

MLP

A.12.2 Implementation Efficiency

Using the setup of Fig. 14, we time autoregressive generation for sequence lengths 4 , 096 -262 , 144 tokens under two architectures-(i) a hybrid network that interleaves STU-T and sliding-window attention layers and (ii) an STU-only network in which every layer is STU-T. Tables 5-6 report mean runtimes (over two runs after a warmup) for SpectraLDS with state dimensions 100 -800 alongside the STU-T with naive convolutions (Base STU) and the Epoched FutureFill method. We note three main results. First, SpectraLDS runtimes grow nearly linearly with sequence length and are virtually independent of the chosen state dimension. Second, while the STU-T with naive convolutions is competitive at 4 -8 k tokens, it becomes progressively slower, so that SpectraLDS is ≈ 2 × faster by 65 k tokens and over 4 × faster at 262 k tokens in the STU-only setting (and 2 × faster in the hybrid setting). Third, Epoched FutureFill narrows the gap at medium lengths but is still outpaced by SpectraLDS beyond 131 k tokens and, in the STU-Only architecture, exhausts memory (OOM) at 262 k tokens, whereas SpectraLDS completes the run. Together these results demonstrate that SpectraLDS delivers the most favorable long-context scaling and remains robust across model hyper-parameters.

Table 6: STU-Only Runtime (seconds) for generation across SpectraLDS with different state dimensions and Baseline and Epoched FutureFill implementations

A.13 Details of STU Model In Experiments

We summarize in Table 7 all relevant details for the FlashSTU model used in the language evaluations in Table 2. The distilled LDS layer used in the language benchmarking experiments was obtained by Algorithm 2 and has a state dimension of 160, incorporating both positive and negative spectral components. The weights for the distilled model were directly transferred from the FlashSTU model described below. The FlashSTU architecture is further described in Appendix A.12 and graphically shown in Figure 14.

Benchmarks were conducted with task-specific few-shot configurations. For each task, if the few-shot value is not set to -1, that number of examples is provided during evaluation; otherwise, the few-shot setting is left unchanged. Specifically, the few-shot settings are as follows: HellaSwag: 0 shots, MMLU: 5 shots, PIQA: 0 shots, BoolQ: 0 shots, Winogrande: -1, Commonsense_QA: 7 shots, OpenBookQA: -1, ARC-Easy: -1, ARC-Challenge: -1. The model employs bfloat16 precision except for the LDS computations, where it requires float64 precision, and a global batch size of 3 examples per task. All computations were performed on a single H100 GPU.

A.14 Example Generation

The following is an example completion from the FlashSTU 340M model after LDS distillation with temperature 1 . 0 :

Prompt: After school each day, Deidre Smith would look over the essays her students submitted at the end of class and, without fail, each day Alex would surprise her. Today, he wrote about

Generated Text: After school each day, Deidre Smith would look over the essays her students submitted at the end of class and, without fail, each day Alex would surprise her. Today, he wrote about 40 essays from his freshman class, all of which were collected over the past three years for The Book of Sentiments: What Students Should Do When Reading Literature - First published in 1962, this book was very well received. What people who read this book would not immediately accept or appreciate the author's thought-provoking message of what it means to read literature.

What exactly are you doing with your newfound ability to read? You are reading? What emotions do you feel at the time? What do you do with your newfound ability to read? These are just a few examples of how much reading students think they can gain from the experience. It is no accident, then, that each and every level of student will have difficulty with the text in this book. Students are given a choice: choose an assignment that you are comfortable with and one that you would like to read more deeply and critically. As the task to read becomes more difficult and the pressure for it is much greater, students will struggle to put their best minds together when reading literature. To help students overcome this hurdle, I have devised a book that makes the task a bit easier by providing all the information they need from a reader. If a reader is too afraid to read, I have designed a page that provides guidance from the textbook that will...

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim that we have identified a method to distill spectral filters to linear dynamical system form and demonstrate in the experiments section that our method successfully works.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: At the end of the experiments section, we mention the current limitations of our distillation. Additionally, throughout we demonstrate understanding of the limitations of our approach, such as restricting our results to linear dynamical systems with symmetric system matrices.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our main theoretical result is Theorem 1, in which we mention complete assumptions made and demonstrate when and why they would hold both theoretically and experimentally in the main paper and in the Appendix.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, we describe our algorithm completely in 1 and 2, and elaborate on the practical implementation in Section 5.2. Our code repository is also open sourced.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: A link to the code is provided.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experiment details are expansively explained in 6, A.13, A.3, A.10, and in many other instances throughout the paper.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: 3, 2, and other graphs where relevant include error bars and statistical significance.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In 6, A.13, and in all places where relevant, the paper describes completely the computer resources needed and used. Furthermore, since many of our results relate to inference speed, we explain the hardware used and time of execution/memory extensively.

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper conforms simply in every respect with the NeurIPS Code of Ethics as explained.

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: [NA]

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All previous work is fully credited and cited throughout.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our new assets are well documented and the documentation is provided.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a method to distill spectral filter-based models (STUs) into state-space model-based (linear dynamical systems, LDS) representations. They provide a provable technique for this distillation that guarantees accuracy independent of the system’s state dimension or effective memory. Empirical evidence is presented demonstrating that their approach maintains predictive accuracy while substantially improving inference efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n1. Well-written: The paper clearly explains it's contribution, methorodlogy, and experiments.\n2. Thoroughness: The authors include extensive theoretical analysis, proofs, and multiple experimental validations, complemented by numerous detailed expansions in the appendix.\n3. Convincing experiments: The experiments robustly validate their claims, showing that the distillation effectively maintains accuracy across multiple benchmarks.\n\n**Weaknesses**:\n1. Although somewhat subjective, the result appears to be somewhat niche and not strongly motivated. Specifically, STUs are not currently predominant in sequence modeling, and it is unclear whether they significantly outperform existing SSM-based models, many of which already achieve constant-time and constant-space inference per token. Even if in certain settings STUs do outperform other approaches, writing an entire paper solely about distilling such models into LDS form seems like too narrow of a contribution.\n2. Again, subjective, but I find that both the proposed algorithm and associated theoretical proofs are rather simple (and not surprising), raising the question of whether the overall contribution is substantial enough to warrant acceptance. Specifically, the main result (Algorithm 2) simply randomly samples a diagonal LDS and linearly fits its parameters to match the spectral filters; this seems to me more like a straightforward linear approximation of spectral filters using randomly chosen exponential bases rather than a deep or novel theoretical insight.\n\nQUESTIONS:\nCan the authors clarify or better motivate the practical significance or broader implications of this distillation technique?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an algorithm to distill a neural network module Spectral Transform Unit (STU) into a standard state space model (or control-affine linear system).  STU implicitly models state space models with real eigenvalues by learning a basis of spectral filters, which in turn define the coefficients of linear combinations of the input tokens, thus mimicking a linear attention mechanism. The objective is to improve the token-generation computational complexity of STUs without losing their expressivity. \n\nThe STU hinges on the fundamental assumption that the linear dynamics matrix of a control-affine linear system is symmetric and hence constrained to be diagonalizable and to have real eigenvalues. This assumption is required to satisfy the strong decay of the magnitude of eigenvalues of the Hankel matrix characterising the time evolution of the system’s linear dynamics matrix, which is the main argument to motivate the approximation of the most relevant spectral filters—the top eigenvectors of this Hankel matrix.\n\nThe proposed methodology takes as an input an STU with learned top spectral filters (eigenvectors of the Hankel matrix) and tries to estimate the parameters of a control-affine linear system that would lead to such spectral filters. The authors claim that this distillation process can be achieved rigorously, selecting the distilled state dimension as a function fo the approximation error of the STU.\n\nSTRENGTHS AND WEAKNESSES:\n### __Strengths__\n\n- The general motivation of the paper is clear and well-articulated once the reader is familiar with STUs. The goal is to improve computational complexity of token generation in generative sequential models for language modeling.\n- The different types of generative sequential model architectures and their respective computational complexity are well presented and explained, with Table 1 providing easy access to this information.\n- Related work comparison is well structured and complete. I specially appreciated the references to standard literature on system identification.\n- Experimental evaluations appear to be comprehensive and support the main thesis of the paper—that a distillation of an STU into a LDS($C$, $A$,$B$) with diagonal $A$,$C$, and $B$ can be achieved without losing precision while gaining substantial improvements in inference computational complexity.\n- Once the reader is familiar with the concept of an STU, the paper's core idea and motivation are clearly explained and well-justified.\n\n### __Weaknesses__\n\n- __[Major]__ The main contribution of the paper is a procedure to distill a STU into a linear control affine form. Hence to comprehend and understand the motivation of the paper, it is crucial that the reader is aware of what an STU is, or that a good intuitive idea of an STU is presented to the reader early on. Unfourtanetly, as a reader unfamiliar with spectral attention methods, I was forced to reading references [2] and [14] to get a good intuitive and technical idea of an STU. \n\nWhile producing self-contained theoretical research papers is challenging, I believe the authors could better introduce STUs and their relationship to SSMs in the introduction. This would help the average ML reader better understand and appreciate the work. Note that the terminology \"spectral transform unit,\" \"spectral filtering method,\" \"spectral filter,\" and \"impulse response\" in the introduction provide little-to-no intuition to the average reader, as these terms aren't widely known in the community—or are used in non-standard ways. \n\nAlthough STUs and spectral filtering methods are difficult to summarize intuitively in few words, I believe that the authors should make an effort to present these concepts clearly. Moving Fig. 2 to the introduction might help by showing both a standard evolution of an eigenfunction (what you call \"impulse response\") and the basis of spectral filters used to express the impulse response as a linear combination. This closely relates to the paper's objective. An early introduction of the transition from spectral filters to eigenvalues—with supporting visualisations—would dramatically improve the paper's accessibility.\n\n- __[Major]__ The core contribution of the paper, explained in Section 5, is restricted to a single page, leaving several of the most important details of the proposed methodology unclear. For instance I was confused with:\n    - Algorithm 1. appears to propose to solve a linear regression of the time evolution of an eigenfunction of $A$ (called ”impulse response” in the paper) in the basis of the spectral filters. This represents simply a linear transformation on the time (sequence-length) dimension, changing basis from time (/token order) to a spectral basis, in the same spirit as in a Fourier transform. It is unclear to me why there is a need for generating a random input vector, as an intermediate step to solve this process.\n    - Algorithm 2. Line 3. States that to learn the basis expansion coefficients a sample of $h$ random eigenvalues $A$ is needed. However in the appendix it is explicitly stated that the STU→LDS distillation hinges on a dataset of $h$ synthetic 1-dimensional LDS systems with parameters ($\\alpha_i$,, $b_i$, $c_i$), where the three scalar parameters are randomly sampled. It is unclear why algorithm 2 does not mention the generation of this datasets of 1-dimensional LDSs or why the sampling of the $B$ and $C$ parameters is ignored.\n    - Algorithm 3 and appendix A.5 mention a \"suitably chosen subset\" of 1D-LDS systems (and their STU's representation) used to train the distillation process. This selection process of subsets and \"greedy expansion of the subset\" appears to be a hand-engineered step of the algorithm that is not well explained in Algorithm 3/appendix A.5, and is crucially omitted entirely from the main body.\n    - Appendix A.5.4 mentions a fine-tunning of the approximated matrix M using gradient descent. This subprocess of the algorithm is again completedly omited in the main text.\n- __[Minor]__ The expansion after line 150 assumes the state and noise a time zero are zero vectors, but this is never stated.\n- __[Minor]__ In the equation after line 207, what was called previously an “impulse response”  $(\\mu_{L}(\\alpha)$ is now called a “single spectral filter” and conflated with the previously defined spectral filters $\\phi_k$.\n- __[Minor]__ Confusing nomenclature.\n    - The term “impulse response” in control theory is associated with the response of a controlled dynamical system to an step function on its control signal. Given that there is an overlap between control and machine learning nomenclature, the use of the term “impulse response” to denote the vector of discrete-time evolution of an eigefunction of a dynamical system is confising.\n    - Linear Dynamical Systems (LDS). For machine learning and control theoretic communities, LDS hints at a uncontrolled linear dynamical system, not a control-affine linear system, or a state space model.\n    - It is unclear what the readers imply by “LDS filters” (line 205)\n    - Line 3 of algorithm 2 says: “Sample h randomly chosen independent vectors $\\alpha_1$…”, but \\alpha is used to denote (scalar) eigenvalues.\n- __[Minor]__ Typo or wrongly format sentence in line 57. Probably need to remove “this”.\n\nQUESTIONS:\n- The main suggestion is to enlarge the presentation and details of the proposed distillation approach in the main text. Specifically, I am concerned on the reproducibility of this distillation process by others, and the number of hyperparameters and intermediate steps omitted to be discussed in the main text. I would suggest to have a detailed version of the algorithm (like Algorithm 3) in the main text. \n\n- Furthermore, I would like authors to clarify in detail in the main text:\n    - How the dataset of 1D-LDS and corresponding STUs spectral filters, and spectral filter expansion coefficients of the LDS time evolution, is generated and used. Is this process completedly done offline and is it independent on the STU we are trying to distill?\n    - How many iterations of process of sampling a subset of this dataset are used in practice to determine the initial subset of the dataset used?.\n    - How to determine when to stop the “greedy” expansion of this dataset? Is this process determining the final dimension of the distilled LDS?\n    - How is the final state dimension of the distilled LDS chosen?.\n\nLIMITATIONS:\nLimitations were not discussed at all. While other aspects should be addressed, too, I believe it is important to elaborate on how prohibitive  the assumption of real-eigenvalues (symmetric LDSs) can be in practice. Personally, I am unaware if this assumption is reasonable / prohibiting in the context of language modelling. Authors could provide a brief statement on the implications of this assumption, or to references that discuss this topic.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper builds off the work of agarwal et al.’s Spectral Transform Unit. The paper identifies the problem that there is now way to convert a trained STU layer into an explicitly LDS layer. The paper identifies this as being important as the industry seeks a new efficient way of handling particularly long sequences. The paper presents a novel technique for converting STU filters into LDS form. The paper provides the “first provable method to directly learn the parameters of a symmetric LDS of arbitrarily high effective memory and with bounded noise”. The paper’s main result is in algorithm 2, which then offers performance guarantees and empirical verifications for.

STRENGTHS AND WEAKNESSES:
Quality: Good theory paper with a nice result, theorem 1 wasn’t terribly difficult to follow. Would like more analysis of pseudo-inverse conditions. 

Clarity: Figure 1 is barely readable, please make the ‘keys’ better so those with poor eye sight are able to better distinguish the lines. Other than that well organized

Significance: Feels quite significant. The biggest issue is that real-world systems may require non-symmetric dynamics. 

Originality: Explicitly novel algorithm and work to go along-side it.

QUESTIONS:
1.Why does epoched future fill in figure three only go until 131072 tokens? (decrease) 
2.Can you provide some analysis in the h-vs-accuracy trade off? (increase)
3. Is there a clear way or some promising directions to be able to achieve similar results without the symmetric assumption? (Neutral)

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SpectraLDS, a provably correct method for distilling Spectral Transform Units (STUs) into linear dynamical systems (LDSs). Since training LDSs directly is often challenging, the authors propose to train the computationally heavier STU instead and then distill it into an LDS to reduce inference cost. Concretely, the goal is to recover LDS parameters $A, B, C$ from the trained STU such that the output of the resulting LDS closely approximates that of the original STU. Theorem 1 shows that the required hidden dimension after distillation is $O(\\log L)$, and that the per-token inference cost is reduced from $O(\\sqrt{L})$ to $O(\\log L)$.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n**(S1)** The paper provides a clear summary of the computational costs between existing methods such as Attention, STU, and LDS.\n\n**(S2)** The proposed method is grounded in a solid theoretical framework.\n\n**(S3)** The experiments cover multiple aspects, including comparisons between filters before and after the distillation, inference speed, and downstream language task performance.\n\n**Weaknesses:**\n\n**(W1)** The explanation of Algorithm 3 is insufficient. It is only referenced in lines 224 and 250, and there is almost no description of what the algorithm actually does.\n\n**(W2)** Similarly, Figure 1 is shown in the paper with very little explanation or commentary in the main text.\n\n**(W3)** The numerical experiments are limited to a model size of 340M, which is relatively small in scale.\n\nQUESTIONS:\n**(Q1)** In line 54, it is stated that the per-token computation cost is reduced to $O(\\log L)$. Does this account for the increased hidden dimension? This seems inconsistent with the explanations in lines 47 and 66, making it difficult to interpret.\n\n**(Q2)** Is the $M_k$ that appears in line 172 different from the matrix $M$ in Algorithm 2? Is my understanding correct that the former corresponds to spectral filters and the latter to LDS filters?\n\n**(Q3)** In line 232, what does the statement \"A direct consequence of our approach is that we can distill any high-dimensional symmetric LDS into a low-dimensional LDS with bounded error\" mean? Does it imply that the STU corresponds to a high-dimensional LDS?\n\n**(Q4)** In line 254, the phrase “greedily adding 1D-LDS increments” is used. Does this mean that the matrix $\\tilde{M}$ is constructed by incrementally adding the vectors $m_i$ obtained from Algorithm 2 one by one?\n\n**(Q5)** Figure 2 shows a visual comparison of filters before and after distillation. Can you also report a quantitative error metric?\n\n**(Q6)** Does the proposed distillation algorithm require access to new data, or is the trained model alone sufficient? Also, how much computational time is needed to run the algorithm?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}