Uncovering the Spectral Bias in Diagonal State Space Models
Authors: Ruben Solozabal
Velibor Bojkovic
Hilal AlQuabeh
Kentaro Inui
Martin Takáč
Venue: NeurIPS 2025 poster ()
Keywords: SSMs
S4
Diagonal state space models
Primary area: deep_learning

Abstract: Current methods for initializing state space models (SSMs) parameters mainly rely on the \textit{HiPPO framework}, which is based on an online approximation of orthogonal polynomials. Recently, diagonal alternatives have shown to reach a similar level of performance while being significantly more efficient due to the simplification in the kernel computation. However, the \textit{HiPPO framework} does not explicitly study the role of its diagonal variants. In this paper, we take a further step to investigate the role of diagonal SSM initialization schemes from the frequency perspective. Our work seeks to systematically understand how to parameterize these models and uncover the learning biases inherent in such diagonal state-space models. Based on our observations, we propose a diagonal initialization on the discrete Fourier domain \textit{S4D-DFouT}. The insights in the role of pole placing in the initialization enable us to further scale them and achieve state-of-the-art results on the Long Range Arena benchmark, allowing us to train from scratch on very large datasets as PathX-256.

Uncovering the Spectral Bias in Diagonal State Space Models
Ruben Solozabal MBZUAI
Velibor Bojkovic MBZUAI
ruben. ruben.sol www.zoaaaalmmbzaai.......mmaaa.. ac. ae
velibor kovojkbzuaaiiii....ommmmm
Hilal AlQuabeh
Kentaro Inui MBZUAI, RIKEN AIP
MBZUAI, RIKEN AIP
hilal. alaalaaalguaaettmaaaaa...... ac. ac
kentaro. , inui @mbzuai ac ae
Martin Takáč
MBZUAI Takac Maaeeee.@@maa..... com
Abstract
Current methods for initializing state space models (SSMs) parameters mainly rely on the HiPPO framewor,, which is based on an online approximation of orthogonal polynomials. Recently, diagonal alternatives have shown to reach similar level of performance while being significantly more efficient due to the simplification in the kernel computation. However, the HiPPO frumework does not explicitly study the role of its diagonal variants. In this paper, we take further step to investigate the role of diagonal SSM initialization schemes from the frequency perspective. Our work seeks to systematically understand how to parameterize these models and uncover the learning biases inherent in such diagonal state-space models. Based observations,
propose diagonal initialization on the discrete Fourier domain S4D-DFouTT The insights the role of pole placing in the initialization enable us to further scale them and achieve state-of of-e--o--he-rr results the Long Range Arena benchmark, allowing us to train from scratch on very large datasets PathX-256.
Introduction
ICEAR (2640)
State space models (SSMs) have recently emerged as a principled and scalable means modeling long se- quences across diverse domains, including image process-
ing I 3 time-series forecasting 44, and natural lan- guage understanding [5] 6 7) At their core, SSMs perform a sequence-to- -o-suuueeee--- mapping via a long-range convolu- tion kernel that is parameterized under continuous time linear dynamical system. This formulation captures depen- dencies multiple timescales and admits strong stability guarantees. Crucially, the "HiPPO matrix" g provides principled method to structure state transition matrices that performs online compression of the input stream by projecting onto orthogonal polynomial basis. However, naively propagating this dense state through a sequence of
200
Figure 1:
The kernels leameed by S4D eCIFAR present a "local attention" profile. When unrolled, these kemness align with po- sitions corresponding to the vicinities of the pixel being attended.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

length L incurs in L) cost, where N is the state dimension, making direct implementation impractical for very long sequences. To overcome this bottleneck, the Structured State Space Se- quence (S4) [9] model introduces normal plus low-rank decomposition of the transition matrix, which reduces the computational complexity while maintaining the expressive capacity [1]
Building on the foundations of S4, recent diagonal variants M 12 13 [4] have shown that restricting the state matrix be diagonal still preserves the performance. By introducing complex- valued initialization schemes [2], these diagonal SSMs achieve performance comparable to the original S4 architecture while being more computationally efficient, as the computation of the kernel simplifies to Vandermonde matrix multiplication. Nevertheless, while the S4 framework has a mathematical interpretation for addressing long-range dependencies, the efficacy of its diagonal variants remains theoretically unexplained.
In this work, we take a step further by investigating the initialization of diagonal SSMs from frequency perspective [15. We aim to uncover how the initialization of the state matrix, particularly the distribution of the imaginary part of its eigenvalues, directly influences the system's 's ability to capture long-range dependencies. Our analysis reveals that current schemes highly depend on the input sequence exhibiting a well-defined inherent timescale that aligns with the model's discretization step A [0]. Since such alignment is unknown priori, existing models often compensate by spreading the poles across a wide frequency range, resulting in over over-vavameeeettoo peer-aaramttrizztiion [6], non-uniform spectral sensitivity, and aliasing artifacts.
Building this analysis, we introduce S4D-DFou'T, an initialization in the discrete domain that explicitly ensures a uniform coverage of the frequency spectrum independently of the selection of the discretization step A. By placing poles directly in the discrete domain, reduce the sensitivity these models present to the frequency response without compromising its ability to capture both short- and long-range effects.
To summarize, the key contributions of this paper are threefold:
Frequency-aware analysis of Diagonal SSMs: The theoretical understanding of the roles played by the initialization schemes on diagonal SSMs is still lacking. Therefore, we systematically describe diagonal SSMs from frequency perspective. We analyze how different initialization schemes affect the model's ability to represent temporal dependencies (Section 3
Proposal of S4D-DFouT initialization: Our observations indicate that current models are highly sensitive to the discretization. We propose novel approach, domain that explicitly designs the pole distribution to optimize spectral coverage. This strategy eliminates the existing coupling between the decay and frequency selection, resulting in a more robust and easy-to-scale initialization. (Section 4).
directly in the discrete
SSMs initialization synchronization enhances model efficiency: Finally, we propose synchro- nized initialization across all the SSMs in given layer. By coordinating their pole placements, the entire layer can be jointly initialize. Our experiments demonstrate that such frequency- syn- chronization enables diagonal SSMs to scale more effectively and outperform previous methods on Long Range Arena (LRA) 17 enabling to train from scratch on PathX-256 (Section
2 Related literature
Initial approaches to sequence modeling, such as recurrent neural networks (RNNs) and long short- term memory (LSTM) [18] models, have achieved notable success but encountered significant limitations when modeling long-range dependencies, primarily due to vanishing and exploding gradients [99 20 These challenges motivated the search for efficient alternative architectures, such as Legendre Memory Units (LMUs) (21), which introduce
con-trainable continuous-time SSM parameterized with Legendre polynomial bases to encode long-term memory. Building on the LMU, the HiPPO framework provided principled method to construct state matrices with trainable state matrices, leading to a successful family of long-range convolution models 9 14 23).
Diagonal variants of SSMs have recently emerged as a simplification of the DaLRdbased S4 architec- ture [2 13 The Diagonal State Spaces (DSS) [ method empirically demonstrated that simply removing the low-rank term from the HiPPO DPLR representation to form a diagonal HiPPO matrix preserves performance. Building on this insight, the S4D framework 112 systematically explores diagonal initialization. Nevertheless, while the S4 framework has mathematical interpretation

---

for addressing long-range dependencies, the efficacy of its diagonal variants remains theoretically unexplained.
Most recently, [4 provided a data-centric perspective on SSM initialization. Their work emphasizes the importance of aligning model initialization with the autocorrelation structure of the input data. We push beyond this data-driven criterion by investigating the initialization of diagonal SSMs from frequency perspective. We observe that current initialization schemes (whether based on HiPPO inverse-frequency laws, or linear grids) drive the model to learn convolutional kernels whose resonance aligns with the principal frequential components of the data. As illustrated in Fig. [ the kernels learned by SSMs act as localized, image-sized filters, effectively aggregating spatially local features rather than capturing truly global, long-range range interactions. Previous works [25] have leveraged exactly this effect on Transformers [26 27 to perform well on LRA. By combining attention with an exponential moving average, those works inject a built-in position-aware encoding mechanism that facilitates learning such tasks.
Our findings show that current initialization schemes rely heavily on coincidental alignment between the dominant timescales in the input and the poles in the system. Since such alignment is rarely known priori, existing models often compensate by spreading poles across a wide frequency range, resulting in over-paramectrrizatonn. Recent works have shown that large portion of the modes in these models can be pruned with a minimal effect in the model performance [16]
Building on these siquency-domaiin insights, we developed the S4D-DFouT initialization. A universal initialization directly in the discrete domain that decouples the interdependency between the decay rate and the oscillation frequency. This design not only liberates from the sensitivity in the selection of A, but also produces kernels that maintain consistent expressivity across wide range of frequencies. Leveraging DFouT, we demonstrate, for the first time, successful training from scratch on PathX-256, task that previous state-of-the-are methods could only tackle after extensive self-pretraining 288
3 Preliminaries
In this section, we briefly introduce the diagonal SSM and the problem setting we considered throughout this paper. Specifically, we analyze the single-input single-output (SISO) variant of the S4 model, where the state transition matrix is defined over the field of complex numbers C, while the model outputs reside in the real domain R. The system dynamics is described with the following equations:
dt M(t) (1))) + Ba((),
v(t) = (()))(()) 1>0.
Here, E Rz0 is the continuous time variable, and BY(-) represents the real part of the complex entry vector. The input and output signals sr((, v(() R are real valued, while the hidden state M(L) E CN evolves in the complex domain. In particular, we consider the setting where the state matrix is diagonal A diag(Ar Ar, ,2,,..,,N) Aw). Under these settings, the input-output relation in ① is explicitly given by the integral
v(1) / S'C eABB) x0(1 - ds,
(2)
i.c the continuous- time SSM can be represented as continuous convolution y(t) = (z (t) of the input signal with the kernel function K- = RVCT B
3.1 Discretization
To obtain a discrete-time version of the continuous diagonal SSM introduced above, one can apply standard discretization techniques. Common approaches include the forward and backward Fuler schemes, bilinear transform, and zeroth-order hold (ZOH) method. In this work, we adopt the latter, which yields the following discretized parameters from the continuous-time system:
π exp(AA), B (AA) (exp(AA) 1) AB.
Under this discretization, the state update and output equations take on recursive form and become:
A[[] Ah(l 1] Bz|(], vlll R(C h(4),, 1= O...., L 1,
(4)
3

---

1.0
-1A--0
1.0
9021
[H(eP)]
0.5
0.0
225
-1.00
-1.0e
1.0m
9 (rad/sample)
Q (radisample))
An example of pole configuration (i.e. state matrix entries A1....; AN) for two discrete SSM systems of order N = 10 exhibiting conjugate symmetry. Following the angular frequency of the equation (8) plot the corresponding frequeney responses in dependence
Figure 2: Frequency response of diagonal SSM. Left:
corresponding discretized systems with A 0.1 (center and right figures). The system with R(A) = (center) presents a narrower response around the resonant frequencies, whereas in the system with R(A) = 0.5 (right) those are wider.
where the input = (z]0)...... ..L. - 1]) now becomes sequence of numbers of length L. The corresponding convolutional or kernel-based view of the model is:
vI4 = (x K)J4), with K= (C AB, (C.. B).
(5)
3.2 Frequency interpretation of the kernel
To understand the frequency characteristics of the discrete convolution kernel K, useful to decompose it into linear combination of basis kernels, where each basis kernel corresponds to single mode (entry) of the state-space model and captures a distinct frequency component of the system. We define the basis kernels K K,,
K2 [1 = X,,, 1=0....,
(6)
where X exp(-A Oz
wa) is the nihh eigenvalue (entry) of the discretized state matrix X, and B,, the corresponding component of the input projection vector B. The full kernel K is then obtained as linear combination of these basis elements, weighted by the components C₂ of the output projection vector C:
N-1 K Σ C,K Σ C,B,X
(7)
From the expressions above, it follows that each individual basis kernel K,, corresponds to a scaled damped complex exponential with decay rate A a, and discrete angular frequency Q2 A W,, As such, each channel n can be seen as single-pole filter that resonates at frequency Q, with its temporal decay determined by A O'n-
The discrete-time frequency response of the kernel K(] is computed via the discrete-time Fourier transform as
N-1
H(e²0)= = Σ K(ie--00
Σ
e- (R.
120
==
120
(8)
N-1
Cn B
Σ
1 c AecAAan et (2
The last expression shows that each component contributes resonant peak centered on 0 = Ωn, with the amplitude controlled by C, and the bandwidth governed by the decay rate A a,, When A a, small, the resonance is sharp, indicating a narrow- -and frequency response, while when the decay rate is large, the frequency response becomes broader (Fig. 2
3.3 Parameterization and Initialization of Diagonal State Matrices
The trainable parameters the diagonal state-space models are the eigenvalues E CNNN and the input/output projections B, C' e C. with hidden state initialized h(0)) = 0. In practice, B

---

30
A-1223
30
A-1220
180
0
180
P
110
225
3150
225
315
225
2707
270
270
(a) S4D-Lin
(b) S4D-Inv
(c) S4D-DFouT
Figure 3: Visualization of the poles configuration obtained on S4D initializations for a system of order N = 32 upon the selection of different discretization or deeay steps A, {0.001, 0.025, 0.1}. In the S4D-Lin and S4D-Inv schemes, controls both the decay (radius) and resonant frequencies (angle) of the system. By contrast, S4D-DFouT is designed to ensure even spectral coverage at any decay rate.
is often initialized as a vector of ones, while the cigenvalues A are initialized according to different proposed schemes 12
In the original S4 model, initialization is done in the continuous-time domain where a structured
matrix A is selected (e.g. from HiPPO-LegS), then discretized with chosen parameter A, obtain A. Furthermore, several S4D variants provide simplified diagonal initializations [I] 12]
S4D-LegS. Uses only the diagonal part A(D) of the HiPPO-LegS matrix A when the later is put into diagonal plus low-rank form A (D) + PPT, dropping the low-rank correction.
S4D-Inv. Eigenvalues of the state matrix A are initialized following an unverse-frequent law for the imaginary parts, with fixed real parts as in the left side of
S4D-Lin. Eigenvalues of the state matrix are coming from linearly spaced frequency grid, with fixed real parts, in the right side of 9
N
N 2n 2n+1 -1) -1)
S4D-Inv: An = 2 +
S4D-Lin: An = + imn
(9)
π
For the latter two cases, we make a crucial observation: The diagonal state matrix is initialized in the continuous domain, which during the training is discretized with learnable parameter A. In particular, the poles of the discretized diagonal system will highly depend on A. This observation serves starting motivation for our proposed method.
4
Main Results
4.1 Motivation
Entanglement decay and frequency via discretization. Existing initialization schemes, such as the ones described in the previous section, are typically defined in continuous time and then discretized before training. This process introduces fundamental coupling between the decay rate and the oscillation frequency through the discretization parameter A. Specifically, both the real and imaginary part the eigenvalues the discretized system (3) scale linearly with A. Consequently, adjusting the temporal resolution A alters both the decay and resonant frequencies of the system, making the model's behavior sensitive to the discretization grid. This coupling potentially complicates both model tuning and interoperability, as changing the discretization has unintended effects on the frequency response.
Limited control over Spectral Coverage. The frequency components induced by previous initializa- tion strategies often lack direct interpretability or fine-graid fine-grained control. For instance, S4D-Inv spreads frequencies according to an inverse-law, clustering models disproportionately at low frequencies, while S4D-Lin distributes them linearly but with a fixed decay rate across all modes (Fig. 3 (a), (b)). These choices may not align well with the actual spectral content of input signals, especially when dealing with broadband inputs or tasks requiring uniform resolution across frequency bands. Furthermore, the decay rates in these schemes are typically hardcoded indirectly determined via
5

---

the continuous-timg representation, offering little flexibility in shaping the temporal memory profile of the model without extensive hyprrparameter tuning.
In summary, the discretization step entangles decay and frequency components in a way that compli- cates interpretability and control, while also making the initialization sensifive the choice of time step A. Moreover, these schemes often impose rigid spectral structures that may not align with the needs for real-world tasks. These limitations motivate the need for an initialization approach defined directly in the discrete domain.
4.2 S4D-DFouT: A Universal Initialization Scheme of Discrete SSMs
As we argued the previous section, a central challenge in diagonal SSMs is choosing the discretiza- tion step A when the data's intrinsic timescale is unknown. Common practices sample A at random from log-uniform range An
(Aginn Amax), to hedge over possible time resolutions. While this increases the odds that at least some modes align with the true dynamics, it also inflates the model's parameter count and yields uneven coverage of the frequency axis. Furthermore, due to the initialization in the continuous domain, the choice of A becomes critical: if A is too small, all modes collapse into a narrow low- frequencccy band: if it is too large, modes fold onto one another and alias, thus reducing spectral diversity the following Lemma suggests.
Lemma 1. After discretizing with step A, continuous pole x= -a w (a 0) becomes == ==xxp(-
exp(-aa -as + isl) with Q = Aw (mod 2T). Non-aliasing condition states that distinct analogue frequencies Wm. + w., map to distinct digital frequencies Qm + Smn In particular, resonant frequencies would not alias with any other if [Aw] < n, the Nyquist bound. Thus, choosing the imaginary parts so that A (\n)) < π for all n guarantees an alias-free frequency coverage.
To address these problems, we propose new initialization scheme directly in the discrete domain.
S4D-DFouT: Single SSM initialization. We propose a novel initialization scheme, S4D-DFouT, which directly constructs the discrete-time state matrix with diagonal entries
S4D-DFouT: Am exp (- En
(-2
2T N
= 0, 1.......,N--
(10)
where <> 0 is a learnable damping factor. This initialization places all poles uniformly around the unit circle in the complex plane, modulated by a shared exponential decay. The imaginary components
i 2 ensure complete and uniform coverage of the frequency spectrum, while the damping term
governs memory retention by controlling how quickly each mode decays over time.
When & = 0, the resulting state transition matrix becomes unitary, and the associated SSM implements perfect frequency basis, reducing exactly to the Discrete Fourier Transform (DFT). In this limiting case, the model can exactly represent any circular convolutional kernel of length N. Moreover, due to its complete and non-rddundant spectral basis, the S4D-DFouT initialization enables the system to act as a universal approximator.
S4D-DFou':: Layer-wise initialization. An S4 layer consists of multiple single-input, single- output SSMs operating in parallel, one per feature dimension in the embedding space. Suppose the embedding dimension is H, then S4 layer comprises H diagonal SSMs, each with an internal hidden state of size N. To avoid redundant poles and to increase the coverage of angular frequencies of SSMs working together, we synchronize their DFouT initializations by assigning a fixed phase 2 (h-1))
offset to every machine {Φn { = NH
C.
[0, 2T (N). Thus, the poles of the h-th SSM are
An.n exp
exp
(- En 2
+i(
2 N
+ (nn)
exp
&n 2
+i 2T (nH (h 11) -h-11))
(11)
N H
Across the layers, the collection (T } forms a uniform grid of size N H on i0, 2 T)), ensuring every resonance band is represented exactly once and preventing multiple machines from competing at nearly identical frequencies.
S4D-DFouT:: Real vs. Complex Inputs. Ensuring real-valued outputs requires that any complex eigenvalues occur in conjugate pairs. Rather than enforcing this conjugate symmetry directly- an operation complicated in S4's learnable parameterization-Bae the original S4 formulation simply takes

---

copying
10-1
WSE
10--
/10
3200
Figure 4: Continuous copying Delay task. Left: An example input =]4 and corresponding delayed output y||]. Center: Reconstruction MSE for different initializations a function of Right: The S4D-Lin kernel learned at the thcoretically optimal A 0.002 (of similar performance of S4D-DFouT),, and distortion when
A = 0.003.
the real part of the resulting kernel. In our proposal, we adopt a similar strategy. Furthermore, when the input sequence z/|] is real-valued, hence its N -pointt DFT X [[N] exhibits Hermitian symmetry X[N n] X X|n]. for n 1, | all negative-frequency bins are redundant and the spectrum is fully determined by the positive frequencies one can initialize the diagonal SSM with only N+
0, 1, .....: [N/2]. Exploiting this fact, // + poles distributed on the upper disc
{e Q E [0, π]} instead of the full 2n range. This "half-plane" S4D-DFouT variant therefore reduces the state dimension and computational cost by factor of two, yet retains the complete information content of the sliding DFT for real-valued signals (Fig. (c)).
Experiments
Our experimental evaluation proceeds as follows. First, we introduce a motivating example in the Continuous Copying task. Then, utilize sCIFAR to probe the inductive biases that SSMs exhibit when learning on serialized image data. Finally, we demonstrate the benefits of our S4D-DFouTT initialization across the Long Range Arena benchmark [17), and further ablation datasets as the Speech Commands dataset [29 Details of the experimental settings are provided in the Appendix C
5.1 Continuous copying task
To motivate our frequency-based analysis, we first consider the Continuous copying or Delay task, a test of long-term memory where the correct pole placement critical for success. Here, the model must learn to perform a sequence-to-sequeen
mapping R 1000 - IR 1000 where the output is lagged by 1000 steps (Fig 4 Left). The architecture comprises single SSM of order N 1024 followed by a linear output layer, and is trained on white-noise inputs bandlimited to 1000 Hz.
Successfully solving this task requires the SSM to approximate the ideal delay kernel, which is zero everywhere except at lag т. In our experiment, we fix the R(A) = 0 to ensure no decay and center the demonstrate that for previous initializations
experiment the frequency selection. In this setup, under an inappropriate selection A fail to capture the desired dynamics (reconstruction MSE error depicted in Fig 4 Center. In the particular case of S4D-Lin, the solution this task is almost trivial when 2 as stated in the following Proposition.
Proposition 1. Let the S4D-Lin convolutional kernel under ZOH discretization be K[[] N-1 2 MY NN Cn K,y). If we choose A 2. where T Zza, and take C, 1 for all i, then K presents "spike" l T, with KT] < [[]]] (AI + r). Moreover, if we take C be a trainable parameter in general, the Vandermonde matrix (Vi.) (X,))) is well-conditioned and gradient descent for C converges at linear rate.
Moreover, for S4D-Lin any choice of no isolated spike at can be formed in the kerne.. Empirically, this misalignment leads to large reconstruction error as shown 4 In contrast, S4D-DFouT successfully manages to reconstruct the ideal delay kernel regardless of the initialization.
makes the fundamental period to fall below т, and
7

---

Table 1: Accuracy on the Long Range Arena (LRA) benchmark. "X" indicates computationnlly infeasible runs, "["" denotes unreported results, and "I" experimentally reproduced this work. The highest-performing diagonal variant is shown in bold. Hyperparameter settings are detailed in Appendix Cc.8
ListOps (2048)
Text
Retrieval
Inage
Pathfinder
PathX-12 128
PathI
Method
(4096)
(4000)
(1024)
(1024)
(16,784)
(65,536)
Transfrmmrr
36.37
64.27
57.46
42.44
71.40
x
S4 S4-LegS
59.60
86.82
90.90
88.65
94,20
96.35
S4-FouT
57.88
86.34
S5
62.15
89.31
91.40
88.00
95,33
98.58
S4D-Lin
60.52
86.97
(96.021)
S4D-Im Inv
60.18
87.34
91.09
87.83
93.78
92,80
S4D-Lag8
60.47
91.95
S4D-DFouT
61.89
87.41
90.95
88.48
94.30
94,17
87.89
5.2
Pixel-level image classification
As introduced in Fig. D the kernels learned using S4D initialization on SCIFAR present a "local attention profile", exhibiting clear peaks at offsets of 32 and being almost zero beyond the first entries. When unrolling these kernels to the original 32 x 32 image grid, the peaks align with positions corresponding the vicinities of the pixel being attended pixels in the adjacent rows). Therefore, we conclude that the model's receptive field learned is compact band of locally nearby pixels in the image.
certain time step (i.e., local nearby
We compare this behavior to CNNs, where early convolutional layers specialize detecting localized features, while deeper layers gradually integrate these local patterns into more abstract global representation.
To quantify the impact of this effect, we retrain the baseline model pro-
Normalized Namm SSMs across layers
SCIFAR-10 (150 apech)
1.0
gressively shortening the
100
0.8
length of the kernel (Fig. Left). As depicted, the model preserves the orig-
Accuracy
70-
L1
0.6
60
Layers
Layers
12
0.4
Layers
inal accuracy when the kernel is reduced only 32 coefficients, confirm- ing that the model's per- formance relies almost ex- clusively very local context. Beyond that, the degradation gradual, with nontrivial accuracy preserved down to surprisingly small kernel sizes.
Layers
128
10
10
10
Karnel Hength
Apa
Figure 5: Pixel-level image classification on sCIFAR. Left: Ablation of the
accuracy upon a reduced kernel length and number of trainable S4D-Lin layers. Right: Normalized H.s -norm for each individual SSM initialized under S4D- Lin, sorted by its operating A.
Furthermore, we measure how much each SSM actually contributes to the output by computing its norm [16]. The model initialized with S4D-Lin comprises four layers of 128-dimensional embeddings, yielding a total of 512 SSMs, each initialized with a random discretization step sampled from the range A [10-3 10-1) J. Strikingly, only those SSMs whose discretization step falls within a narrow subinterval in the first layer exhibit non-negligible H -norms; the vast majority of them, partieularly in deeper layers, remain effectively inactive (Fig. 5 Right). We confirm this in the following experiment. We further ablate the layer's importance by retraining only with the first k-layer kept active and replacing all subsequent SSM layers with identity mappings (Fig. Left). In this setup, we observe that almost all of the learned sequential information resides in that very first SSM layer, capturing the critical local pixel-row dependencies, while deeper SSM layers only provide incremental refinement.
5.3 Long range arena
The LRA benchmark has been widely adopted to assess a model's capacity for capturing long-range context across diverse sequence lengths and modalities. In this sense, the Fourier-initialzeed variants have repeatedly underperformed on its hardest tasks, failing to solve challenges like PathX-128 that other initializations could handle. We show that this shortfall is not intrinsic limitation of Fourier
8

---

Listpps
Image
Pathlinder
75
S4D-Lin
70-
Accurscy
65-
540 DFouT
Acturary
Accurarc
60-
80-
S4D-Lin
54D-0FouT
54D-DFou!
50
70
70
10'
10
10:7
10"
Areer Eruz
Benn Errx
Deaa Erec
Figure 6: Ablation experiment in LRA. Accuracy upon different initialization schemes for fixed Amins Smiin 10 and variahle A Aaxx Emox. Datasets Image and Pathf Sehfinddrs inder presenting strong frequential component initialization that captures that frequency.
(see Fig. 7 are highly sensitive
initialization, but rather of the difficulty selecting the hyperparameter. The intuition gained on the "local-attention" profile these kernels exhibit on serialized images provided us prior on the selection of to successfully learn this task (marked with "f" in Table These results show that Fourier-based state- space models can equally perform well, provided their discretization aligns with each task's 's intrinsic timescale.
Furthermore, our proposed S4D-DFouT initialization liberates from these difficulties, and we could directly apply successfully to scale to harder tasks such as PathX-255 without any problem-specific tuning. So far, this is the first work to succeed in learning PathX- Path from scratch, as previous approaches relied on self-pretraining (28) to handle sequences of this length.
5.4 Raw speech classification
We also evaluate the method on the Speech Commands, 35-way spoken- spoken-word classification task using raw audio waveforms. Each input is a I-second signal sampled at 16 kHz. While previous studies typically extract spectral features such MFCCs 30), state-space models like S4 have demonstrated the ability to learn directly from time-domain inputs. As shown Table we quantify the performance gains under a zero-shot resampling scenario, where each waveform is uniformly subsampled to 8 kHz without retraining. In the latter, we surpass the previous initializations by points.
5.5 Ablations
In Fig. 6 we perform an ablation study on the effect of upper bounds Amaz and Emaz used in initialization and training for parameters A (in baselines) and & (44D-DFouT), respectively. In particular, the results show that the S4D-DfouT initialization scheme is less sensitive to the parameter compared to the baselines, especially on the datasets that present strong frequential component (Image and Pathf inder).
Next, ablate several discrete-domain ini- tialization variants. First, we assign each pole random phase by sampling its imag- inary component as 22 ~ 20[0, 2T), called Next, we evaluate token synchronization scheme, which aligns pole angles to periodic intervals every k-tokens. Finally, we try a different S4D-DFou'T syn- chronization using batches (details in Ap- pendix C.6).
Table 2:
Accuracy on ablation experiments, averaged over three random seeds (std. in parentheses).
SCIFAR
SC35
Method
16kHz
8kHr
S4-LegS
(0.43)
(0.15)
S4-FouT
91.22 (0.25)
95.27 (0.20)
91,59 (0.23)
S5
90.10 (-)
96.53 (-)
94.53 (-)
S4D-LegS
89.92 (1.69)
95.83 (0.14)
91.08 (0.16)
S4D-Inv
(0.066
(0.27)
(0.24)
S4D-Lin
90.42, (0.03)
96.25 (0.03)
91.58 (0.33)
S4D-DRouT
89.87 (0.07)
96.07 (0.19)
93.85 (0.27)
Conclusion and Limitations
S4D-Token
84,29 (0.14)
94,93 (0.17)
92.92 (0.22)
S4D-Rndlmmgg
(0.27)
(0.26)
(0.47)
Although LRA tasks are considered hard in the literature, especially tasks as PathX that are designed to stress model's long-range reasoning, our analysis reveals that SSMs often succeed by exploiting local biases rather than learning truly global kernels. As a result, even the hardest LRA benchmarks can be solved with receptive fields much shorter than the input length as long as the task contains principal frequency components that could be captured by the SSM's spectral modes. Prior initializations in the continuous domain hedge against unknown timescales by
S4D-Bacched-DFouT
88.67 (0.11)
95.31 (0.09)
0.07 (0.14)

---

sampling A over a wide range. In this work, we show that our initialization in the discrete domain achieves uniform, alias-frre spectral support, eliminating the need for Á tuning. These insights indicate that the intrinsic difficulty of LRA tasks may be overestimated, underscoring the need for new benchmarks that cannot be circumvented through the presented learning biases and that truly challenge models to capture long-range dependencies.
We note that S4D-DFouT initialization did not help in solving a more challenging problem of permuted p&CIFAR, achieving 65.7% accuracy (see Appendix CC7). Furthermore, we did not explore our proposed initialization in the context of textual data and LLMs, which we leave for future study.
References
[1] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In ICML. OpenReview.net, 2024.
[2] Dong Li, Yidi Liu, Xueyang Fu, Senyan Xu, and Zueng-Jun Zha. Fouriermamba: Fourier learning integration with state space models for image deraining. CoRR, abs/2405. 19450, 2024.
[3] Eric Nguyen, Karan Goel, Albert Gu, Gordon W. Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4ND: modeling images and videos multidimensional signals with state spaces. In NeurlPS, 2022.
[4] Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao, Karan Goel, and Christopher Ré. Effectively modeling time series with simple discrete state spaces. In ICLR. OpenReview.ner, 2023.
[5] Daniel Y. Fu, Tri Dao, Khaled Kamal Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. In ICLR. OpenReview.net, 2023.
[6] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, abs/2312.00752 2023.
[7] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In ICML. OpenReview.net, 2024.
[8] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. In NeurlPS, 2020.
[9] Albert Gu, Karan Goel, and Christopher Ré, Efficiently modeling long sequences with structured state spaces. In ICLR. OpenReview.net, 2022.
[10] Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. How to train your HIPPO: state space models with generalized orthogonal basis projections. In ICLR. OpenReview.net, 2023.
[11] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. In NeurlPS, 2022.
[12] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In NeurlPS, 2022.
[13] Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Çaglar Gülçehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 26670-26698 PMLR, 2023.
[14] Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling. In ICLR. OpenReview.net, 2023.
[15] Rom N. Parnichkun, Stefano Massaroli, Alessandro Moro, Jimmy T. H. Smith, Ramin M. Hasani, Mathias Lechner, Qi An, Christopher Ré, Hajime Asama, Stefano Ermon, Taiji Suzuki, Michael Poli, and Atsushi Yamashita. State-free inference of state-space models: The *transfer function approach. In ICML. OpenReview.net, 2024.
10

---

[16] Minseon Gwak, Seongrok Moon, Joohwan Ko, and PooGyeon Park. Layer-adaptive state pruning for deep state space models. In NeurIPS, 2024.
[17] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. Long range arena benchmark for efficient transformers. In ICLR. OpenReview.net, 2021.
[18] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 988)11335-1788 1997.
[19] Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In ICML (3), volume 28 of JMLR Workshop and Conference Proceedings, pages 1310-1318. JMLR.org 2013.
[20] Jonas M. Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with mns. In NeurlPS, 2022.
[21] Aaron Voelker, Ivana Kajie, and Chris Eliasmith. Legendre memory units: Continuous-time representation in recurrent neural networks. In NeurlPS, pages 15544-15555 15553, 2019.
[22] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 28043-28078 PMLR, 2023.
[23] Ramin M. Hasani, Mathias Lechner, Tsun-Hsuar Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. ICLR. OpenReview.net, 2023.
[24] Fusheng Liu and Qianxiao Li. Autocorrelation matters: Understanding the role of initialization schemes for state space models. CoRR, abs/2411.19455. 2024.
[25] Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig. Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. In ICLR. OpenReview.net, 2023.
[26] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pages 5998 9998-000, 2017.
[27] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurlPS, pages 5244 5254, 2019.
[28] Ido Amos, Jonathan Berant, and Ankit Gupta. Never train from scratch: Fair comparison of long-sequence models requires data-driven priors. In ICLR. OpenReview.net, 2024.
[29] Pete Warden. Speech commands: A dataset for vocabul-y speech recognition. CoRR, 2018.
[30] David W. Romero, Anna Kuzina, Erik J. Bekkers, Jakub Mikolaj Tomezak, and Mark Hoogen- doorn. Ckconv: Continuous kernel convolution for sequential data. In ICLR. OpenReview.net, 2022.
[31] Yanfei Qin and Ziwen Sun. Observer-based asynchronous event-triggered robust h adaptive switching control for nonlinear industrial cyber physical systems under data injection attacks. International Journal of Control, Automation and Systems, 2177)2215--118,, 2023.
[32] Chang Wei Tan, Christoph Bergmeir, François Petitjean, and Geoffrey I. Webb. BIDMC blood oxygen saturation dataset (32 seconds window) (version 2). lhttps://dao org/10.0005000 5281/ zenodo. 4001144) June 2020. Accessed YYYY-MM-DD.
[33] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR. OpenReview.net, 2020.

---

[34] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañóón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big Transformers for longer sequences. In NeurIPS, 2020.
bird:
[35] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rmns: Fast autoregressive transformers with linear attention. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 5156-5165. PMLR, 2020.
[36] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In ICLR. OpenReview.net, 2021.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claim reflected in the abstract and introduction are reflected in Sections] 3 and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in the Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

The theoretical results in the main text Section 4 and [ as well as in the appendices provides full set of assumptions and proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section Stogether with Appendix provide full details on the experimental
setting for reproduction of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The data used in the paper are publicly available. The codebase upon which the experimental part was built is also publicly available, while the custom part of our code is added in the supplementary material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
y /nips. /nips.cc//pppp http:/cccccccgguueeeeeseeeeeeeee wwaicc//uuiss//ee htgool/ccguiddes/ddeeuuuuiiiisse
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The full details are available in the Experimental section as well as in the Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report standard statistics (mean and variance) where applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: We provide general framework (GPUs used in the experiments) in Appendix C but we do report the running time and memory consumption during training.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httt:::tttt/uu: http:///arrrppscc//p//c//ccccccc
Answer:
[Yes]
Justification:
We conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The paper has limited socictal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: does not apply.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Original S4 sourcecode from https//giituub.co//stttepppppsee is under Apache-2.0 license.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We do not introduce new assets in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification:
No crowdsourcing and research with human subjects were conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [No]
Justification:
We did not conduct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: LLMs were not used in the core methods this research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neunee http://rrssscc/ooofferrreeee///e cc/Coonferencces/2225//LL for what should or should be described.
19

---

A Theoretical insights
A.1 Transfer Function and Stability Conditions
Consider a continuous-tinn time diagonal state-space system
d
h(()) Ah(l) + Ba((), =(1)=
- C h(), 1> 0,
di
and its corresponding discretized model (for the sake of simplicity, we do not enforce the real output
in (I)
hll + A A14 B xx([],
E:
(12)
y/1 = Ch(])
where
A diag( A1
AN),
A = diag (ogg(A... (A1,.... ,N):
B [
Bx] T
B B1
BN]
C = [C1
CN].
Focusing the discretized model, the transfer function in the Z-domain, by definition, given by
H=)) (=))U((=) C A
Because A is diagonal, this simplifies -iie to a sum of first-order terms
H(z) = Σ Cn By
N
Cn Bn
Σ
Xn Z-1-1"
n=1
The poles of (z) correspond to the cigenvalues of A, = (An The discrete-time SSM is stable if and only if all poles lie inside the unit circle:
[uml 1
N
Direct pole training. is possible for diagonal SSMs models whose poles correspond to trainable parameters control them to satisfy stability conditions directly. If the continuous diagonal system stable, i.e. ((())) < O, then under the zero-order hold discretization with step R the
magnitude of the poles become
A
|e
AR(An) <1
(13)
Thus, the discretized SSM remains stable.
A.2
Hos Score
In robust control [31] the Hox score of a discrete-time LTI system, such as Σ in (12). with the transfer function matrix H, is defined
Hoc = Hox ( = |1 = sup o (H()))))
@c(o.2a]
where (-) denotes the maximum singular value of a matrix. This score captures the worst-case amplification of disturbances across all frequencies. Crucially, also provides an energy bound
< H bxx ||=|
so that the total output energy can never exceed the input energy scaled by Hon
For diagonal discrete SSM in 12 each scalar-state subsystem Eg has transfer function
Hx(z) = Cn Ba
2 Xn
hence closed-form Hos score
C, |Ba 1
Hos (.,)
(1- 12,.)22
highlighting how pole magnitudes limit worst-case gain. This score directly measures each state's worst-case contribution output energy. This score underpins modal-truncation-styy pruning [16].
20

---

B Proofs
Proposition Let the S4D-Lin convolutional kernel under ZOH discretization be K(1] 2 Cn Xn)- If choose A ?, where I Z,ya, and take C2 1 all i, then K presents "spike" at = T, with |K]] ! < [[[r] (Vl + 7). Moreover, if we take be a trainable parameter general, the Vandermonde matrix (Vin) (Xr) is well-conditioned and gradient descent C' converges at linear rate.
Proof. We proceed in five detailed steps:
1. Discrete poles via ZOH. For the sake of simplicity, let's suppose poles of the continuous system have no decay Wn imn; then, ZOH gives
d exp(St) exp(u,, w A) exp(innA).
2. Fundamental discrete period.
The first harmonic n = has frequency Ω = =A. A full 2m T 2/A. Thus cycle of takes precisely T 2/A
rotation requires Ω2 T 2T steps.
3. Matching 7' to T. Set = =, i.e. A 2/T. Then for any
d expl n A) exp 2 n) 1,
so all modes align in phase step |
Kernel evaluation. The SSM kernel is
N-1 Σ
N-1 Cn Cn d',
K
R
- 22 d.). ==0
n=00
At I r, since d" = 1 for all
N-1
K[7] 2 2
Σ
2N,
whereas for * r, the d', are not all unity and partially cancel, yielding |[[]| < K[T].
5. Conditioning and GD. Fix an integer delay > N. Then for 0 < 1 and 0 s n S N 1, the Vandermonde matrix
VG, d² expl 2TT 2mm) 1)
exactly a (r + 1) N partial Fourier matrix. Concretely, Gram matrix
VV
d di dl n.m
Σ
(2(nn
+ 1, m,
0, n n m
so VVV = (T+ 1) IN. where the latter the identity matrix. Hence, the eigenvalues of V*V are equal
Amiu Amin(V') (V*V) Amax (V*) +
and the condition number of V is exactly one.
Now consider fitting the target vector I by minimizing L= | IIVC- yll over C CN
Since the Hessian is V²L(C) VV = (T 1)I, we have that every eigenvalue of it is exactly (r+1). so L (T + 1) -troongy convex. Furthermore, the largest eigenvalue (smoothness constant)
7 + 1, thus the function is also (r + sm-ohhhhh. Consequently, the optimal gradient-descen step-size
7
Amar (V*V) T 1
21

---

reaches the exact solution in one step.
At each iteration i, gradient descent update C(++1) = O( 17 V)((C())) and for the least-squares loss [(C)), the gradient is
i 1 (VC yVC - ac 2
VL(C)
1 and using V*V = (r + 1)1, we have
Therefore, choosing 1 ++1
( ((0) V (VC - v)
((
V (VVC -
T ++1
V
C
++1 V
More generally any 0 7 <
2 ++I
still guarantees linear convergence at rate ||Ct) C*1 S
(1 (1- 77T 1)) C*
This completes the proof.
C
Experimental details
In this section, we provide descriptions of the datasets, task formulations, model architectures, and training protocols used throughout our evaluations.
C.1 Continuous Copying Task
The Continuous Copying (or Delay) task is posed as a sequence-fo-sequence problem in which the model must reproduce the input delayed by r 1000 time steps. Specifically, we utilize white-noise sequences of length 4000 samples, band-limited to 1000 Hz, and train a single linear state-space model with state dimension N 1024 end-to-end for 20 epochs.
C.2 Pixel-level image classification
The serialized CIFAR-10 (sCIFAR) dataset is constructed by flattening the 32 x 32 color images into a one-dimensional sequence of length 1024. All sequences then standardized to zero mean and unit variance. In the permuted serialized (psCIFAR) variant, the same preprocessing is applied, but with a single fixed random permutation of the 1024 pixel positions before flattening, thereby removing any local spatial structure. The permutation vector utilized is included in the supplementary code. Ablation results on PSCIFAR are available Table 3
C.3 Long Range Arena
The Long Range Arena (LRA) 17 benchmark comprises seven diverse sequencceclassiffcation tasks designed to test a model's ability capture long-range dependencies:
ListOps. The dataset consists of mathematical expressions formed by nested operators-such as min and max-applied to integer numbers in the range 0-9, and written in prefix notation with explicit brackets. Each character is represented as a one-hot vector over 17 possible tokens (with all opening brackets and operators grouped into single token). Because sequence lengths vary, shorter sequences are padded with a special padding symbol up to a maximum length of 2048. The task is to classify each expression into of 10 classes, corresponding to its integer result.
Text. The IMDb sentiment classification task is as follows, given movie review represented as sequence of character tokens, the goal is to predict whether the review is positive or negative. Each character is one-hot encoded over 129 possible tokens. Because reviews vary length, sequences are padded to maximum length of 4096.
22

---

Table 3: Results additional datasets. Accuracy for p&CIFAR and RMSE for predicting respiratory rate (RR).
heart rate (HR), and blood oxygen (SpO2).
PSCIFAR
BIDMC-HR
BIDMC-RR
BIDMC-SPD
Method
Acc. (1)
RMSE (4)
RMSE (1)
RMSE (4)
S4-LegS
64.96
0.332
0.247
0.090
S4-FouTT
65.27
0.339
0,301
0.068
S4D-LegS
0.367
0.248
0.102
S4D-Lin
64.51
0.379
0.226
0.114
S4D-Inv
64.88
0.373
0.254
0.110
S4D-DFouTT
65.72
0.415
0.273
0.122
S4D-Token
60.25
0.490
0.440
0.126
S4D-RndImag
60.55
0.437
0.216
0.087
S4D-Batched-DFouT
63.61
0.456
0.344
0.118
Retrieval. The task is based on the ACL Anthology network corpus; given two textual citations encoded as sequences of character tokens, predict whether they refer to the same publication. Each character is one-hot encoded over 97 possible tokens. Sequences vary in length and are padded to maximum of 4000 tokens using a special padding symbol, and a reserved end-of ond-o--se token appended. The model outputs one of two classes: equivalent non-equivalent.
Image. Considers the CIFAR-10 image classification task, but images are first converted to single- channel grayscale intensity map, and pixel values are normalized to zero mean and unit variance across the entire dataset. The resulting 32 32 image flattened into sequence of length 1024. All sequences are of equal length, and the model predicts one of ten classes corresponding to the 10 categories.
Pathfinder. In this task, the images contain two marked points (start and end) represented by small circles, and of dashed line segments. The objective is predict whether there exists continuous dashed-line path connecting the start and end points. Each input is a 32 x 32 grayscale image with pixel values normalized to the range [-1, 1].
PathX. An "extreme" version of the Pathfinder challenge. Instead, the images are 128 128 pixels in the PathX-128, 256 256 in the PathX- -256, resulting in much longer sequences. As denoted in by (t), we manage to successfully train S4D-Lin on PathX by increasing the range from (Amin Amaz) = (0.0001, 0.01) to (Amsns Amaz) (0.0001, 0.1). This is done in order to accommodate the fundamental harmonic within the model frequencies explored at initialization. This requires
A 2 2/128 0.015, which not covered in the initial range proposed in [2
C.4 Speech Commands
We use the full 35-class Speech Commands dataset [29]. which comprises second recorded voice commands sampled at 16 kHz. Additionally, introduced in [12), we also test the zero-shot performance when the signal undersampled 8kHz.
C.5 BIDMC Vital signs
The BIDMC dataset 32 consists of continuous physiological signals from raw EKG and PPG traces: heart rate (HR), respiratory rate (RR), and blood oxygen levels (SpO2) over length-4000 signal,s which are normalized per channel to zero mean/unit variance.
C.6 Ablations details
We compare three variants of pnscrete-domann initialization beyond the main S4D-DFouT scheme. The ablation methods share the real part with DFou',, i.e. exp(-E/25 while the imaginary part is configured as follows:
S4D-Token: Discrete frequencies are chosen to resonate integer periods = 1, .... N, i.e. Q, = 2Tt 2n/n. n.. This assigns one pole per token in the sequence, yielding a coarse grained coverage of the frequency spectrum biased towards low frequencies.
23

---

Lece
Lyst
0.003
Laya
0.2
0.59
0.002
0.005
0.001
0.001
0.04
0.002
0.002
LRA/Listops
clear frequency (b) LRA RRAImagee
LRA/PathX-I22 -28
[H1 (dB)
(Ale (88)
40
60
40
1.000
-0.30m
0.00m
0.50n
000
4.0m
-2.5m
0.0n
0.5m
1.0g
-0.10n
0.50x
0.05m
0.10m
tradhsmmpte!
D trathhmmppl
(reshhample)
(d) LRA/Listops
(0) LRA/Image
(1 LRA/PatX---E 128 (Zoomed)
Figure 7: Subplots (a)-(c) show the absolute value of the learned discrete kernel [[] for each of the six SSM layers on three LRA benchmarks (a) ListOps, (b) Image, and (c)
only the first L coefficients are displayed on the x-axis to facilitate the visualization. Subplots (d)-(e) present the magnitude of the discrete Fourier transform (in dB) for 10000 randomly sampled sequences from the same datasets. These visualizations demonstrate that datasets with strong local structure succh the serialized image benchmarks SCIFAR PathX-128 exhibit clear frequency components corresponding to row-wise correlations. This reflected the learned kernels exploit a "local attention" profile. sCIFAR, a pronounced peak in the kernel occurs at multiples of 32-pixel (the row-stride), which corresponds to clear frequency component (N 2nr /32); whereas PathX-128 shows periodic modes at 128 pixels, i.c., frequencies of f2 2 /228 along with their higher-frqqeent harmonies. By contrast, the ListOps benchmark exhibits such localized frequency peaks, reflecting its lack inherent local structure.
R4D-aadlma:: Each imaginary component Qr is i.i.d. sampled from the uniform distribu- tion 2/(0, 2T).
S4D-Batched-DFouf Synchronization across the H SSMs in each layer is achieved by partitioning the spectrum into H contiguous blocks, each of size N. Therefore, the h-th SSM assigned with batch of adjacent frequencies shifted by a phase On E [0, 2r):
(- En 2 NH
2 2 ( + (n)) where on
2x (h 1)
Batched-DFouT: An. = exp
+ +
(14)
C.7 Extended Results
In Figure 7 we show the kernel visualization together with the discrete Fourier transform on three representative LRA benchmarks. These visualizations demonstrate that datasets with strong local structure such as the scrialized image benchmarks sCIFAR and PathX-cxhin components corresponding to row-wise correlations. SSM models exploit this fact to learn kernels showing a "local attention" profile. The learned kernels "attends" primarily to its nearby inputs, imposing an inductive bias toward locality. By contrast, the ListOps benchmark exhibits no such localized frequency peaks, reflecting its lack of inherent local structure.
RegueeBextends FigurefJfrom the main manuscript to the remaining datasets LRA benchmarks. Here, we perform an ablation study on the effect upper bounds Amaz and Emaz used initialization and training for parameters A (in baselines) and (S4D-DFouT)) respectively. The results show that the S4D-DFouT initialization scheme is less sensitive the E parameter compared to the baselines, especially the dataset which present strong frequential component (Image and Pathfinder).

---

SC35
UstOps
70
90
Accuracy
540 DFOUT
Accurary
80
60
70
D-OFout
50
70
---
540-DfouT
1.0
10'4
10
10"
10
10
deeer frec
Arae Grac
Deaar Ever
Retrieval
Pathvlnddr
1.00
100
100
90
Arcuracy
90
Accuracy
90
80
Lin
80-
S40 Lin
S40 Un
S41-Inv
S4D-ir
540
54D.DFouT
DFouT
10a2
10
10
10
10
10
10-2
10-1
Figure 8: Ablation experiment in LRA. Accuracy upon different initialization schemes for fixed Amiin Emin 10 3 and variable Amar, Emax. Datasets Image and Pathf inder presenting strong frequential component are highly sensitive to an initialization that captures that frequency.
Finally, extended comparison Long Range Arena is presented in Table It spans across all seven tasks, including the most challenging PathX- 256 setting. While prior work denoted as "S4 + Self-Pretrain" (28) (marked f) also succeed on it, it does relying on self aaffprettrainin. Whereas our S4D-DFouT achieves it training from scratch.
25

---

Table 4: Extended comparison LRA. (f) denotes the method requires self-pretraining.
Model
ListOps (2,048)
Retrieval
Inage (1,024)
Pathfinder (1,024)
PathI-122
(4,09))
(4,000)
(16,384)
(65,536)
Transfomer
25
36.37
64.27
57.46
42.44
71.40
Reformer 3
37.27
56.10
33.40
38.07
68. 50
K
BigBard
34
36.05
59.29
40.83
Lincar Trans
16.13
65,90
53.09
42.34
75.30
Performer
136
18.01
65.40
53.82
42.77
77.05
MEGA
25
63.14
90.43
91.25
90.44
96.01
97.98
DSS 1
60.60
84,80
87,80
85.70
84.60
87.80
S4D-LegSS
12
60.47 (0.34)
86.18 (U.43)
89:46 (0.14)
88.19 (0.26)
93.06 (1.24)
91.95
S4D-Inv
60.18 (0.35)
87.34 (0.20)
91.09 (0.01)
87.83 (0.37)
93.78 (0.25)
92.80
S4D-Lin
60.52 (0.51)
86.97 (0.23)
90.96 (0.09)
87.93 (0.34)
93. 96 (0. 60)
S4-FouT
57.88
86.34 (0.31)
89.66 (0.88)
(0.24)
S4-Tag5
59.60 (0.07)
86.82 (0.13)
90.90 (0. 15)
88, 65 (0.23)
94.20 (0.25)
96.35
Liquid-S4 [6]
62.75 (0.20)
89.02 (0.04)
91 (0.01)
89.50 (0.40)
(0.20)
96.66 (0.001)
$5.Inv
14
60.07 (0.26)
87.37 (0 (0.29)
91.26 (0.12)
86.41 (0.17)
93.42 (0.42)
54 (0.74)
S5-Lin
59.98 (0.53)
88.15 (0.24)
91.31 (0.24)
86.05 (0.96)
94.31 (0.36)
65 60 (000)
55 14
62.15 (0.23)
89.31 (0.15)
91 90 (0.05)
88.00 (0.23)
95.33 (0.26)
98.58 (0.17)
LRU 15
60.2(0.8)
89.4(0.1)
89.9(0.1)
89.010.1)
95.1(0.1)
94.2(0.4)
Self
(t)
S4D-DFouT
61.89
87.41
90.95
88.48
94.30
94.17
87.89
26

---

S4D-Lin
Layerw
Layer=1
Layer=2
90
150
160
PUTT
220
390
210
330
210
330
230
270
270
270
210
SnD-Inv
Layem
Layer1
Layer-2
Leyer
120
157
30
150
30
150
30
350
10
10
180
240
240
240
240
270
270
270
S4D-DFouT
Layero
Layer1
Layer
Laver=3
60
120
60
120
60
220
190
120
120
50
210
210
210
230
3.00
2.0
240
240 2.70
240
Figure 9: Polar plots of the learned complex poles on SCIFAR for three initialization schemes- S4D-Lin (top row), S4D-Iav (middle row), and S4D-DFouT (bottom row). Poles are shaded by their H..-norm (warmer tones indicate higher norms), emphasizing the modes that drive the system's worst-case gain. The dashed unit cirele marks the stability boundary. S4D-Lin entanglement between decay and frequency can be observed, resulting spiral-like clustering of poles; SnD-Inv spreads poles more evenly around the circle; and S4D-DFou'T maintains the original half-plane distribution of poles throughout training.
C.8 Hyperparameters
The S4D-DFouT hyperparameter configuration we adopt in the experimentation is provided in T
Table 5: Hyperparameters used for the S4D-DFouT reported results. denotes the number layers: H. the embedding size; N, the hidden dimension; Dropout, the dropout rate; Lr, the global learning rate; Bs, the batch size; Epochs, the maximum number of training epochs; WD, weight decay; and (max) the range of decay rate values.
Task
L
Norm
Pre-nerm
Dropout
Epoch
Wd
(Emin. Emace)
sCIPAR
6
128
64
LN
False
0.1
0.01
64
100
0.05
(0.001, 0.1)
PSCIFAR
(0.001, D.1)
ListOps
6
256
64
BN
False
0.01
40
0.05
(0.001, 0.1)
Text
6
256
64
BN
True
0.01
0.05
(0.001, 0.1)
Retrieval
6
256
64
BN
True
0.01
64
20
0.05
(0.001. 0.1)
Imago
LN
(0.001, 0.1)
Pathfindar
6
256
128
BN
True
0.001
64
200
0.03
(0.001, 0.1)
PathX-128
128
BN
True
0.0005
0.05
(0.0001. 0.1)
PethI-256
6
256
128
BN
True
0.0001
20
0.05
(0.0001. 0.1)
SC35
BN
0.001
0.05
(0.001, D.1)
BIDNC
6
128
256
BN
True
0.01
500
0.05
(0.001, 0.1)
27


Reviews:
[{"review_id": "4GyqHhy9jL", "reviewer": "Reviewer_V8Gu", "summary": "This paper investigates diagonal variants of State Space Models (SSMs), focusing on how initialization affects their ability to capture long-range dependencies. The authors analyze these models through a frequency-domain lens and highlight that existing initialization schemes suffer from spectral aliasing and sensitivity to discretization. To address this, they propose S4D-DFouT, a novel initialization strategy that places poles directly in the discrete Fourier domain, ensuring uniform spectral coverage and robustness. Empirical results on the Long Range Arena benchmark show that this method improves performance and enables training from scratch on previously unsolved tasks like PathX-256.", "questions": "- Can the proposed initialization strategy (S4D-DFouT) be extended to low-rank or more general structured SSMs beyond the diagonal case? If so, what challenges or adjustments would be required?\n\n- What types of tasks or data characteristics might cause S4D-DFouT to underperform? Are there known limitations where this initialization is less effective?\n\n- Could you provide a comparison of computational cost (e.g., training time, memory usage) versus accuracy to better illustrate the efficiency gains of S4D-DFouT relative to other models?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n- Clear motivation and direction: The paper’s focus on model initialization as a path to improving diagonal SSMs is timely and relevant. The emphasis on long-range dependency modeling aligns well with current research trends and opens avenues for future work.\n\n- Well-written and theoretically grounded: The paper is clearly written and includes a solid theoretical foundation, particularly in Section 3, where the authors provide a detailed frequency-domain analysis of diagonal SSMs.\n\n- Novel method (S4D-DFouT): The proposed initialization scheme, S4D-DFouT, is theoretically justified and introduces a principled way to achieve uniform spectral coverage. It is a simple yet impactful contribution.\n\n- Strong empirical results: The method outperforms existing diagonal SSM variants across multiple tasks in the Long Range Arena benchmark (Table 1) and is the first to train PathX-256 from scratch without pretraining.\n\n\nWeaknesses:\n\n- Limited ablation beyond SSMs: While the paper provides thorough comparisons within the diagonal SSM family, it lacks ablations against non-SSM models like Transformers. Such comparisons would help clarify the broader applicability of the proposed method.\n\n- Motivation could better connect to practical use cases: Although the motivation is theoretically compelling, it would benefit from examples or discussion showing potential industrial applications or real-world scenarios where spectral control via initialization is especially valuable especially compared to other models including transformers, MoE.\n\n- Lack of compute resource reporting: The paper does not report training time, memory consumption, or hardware setup. These details are critical for evaluating claims of efficiency and scalability. I could not find these details mentioned in the paper. \n\n- Environmental impact unaddressed: The authors could strengthen the paper by reporting on the carbon footprint or potential energy savings of their method compared to more compute-intensive baselines like Transformers.", "was_revised": true, "final_justification": "I believe the rebuttal was convincing and I look forward to  seeing all the revisions mentioned by the author for the final paper.", "rebuttal": "Dear reviewer thank for your time in assessing our paper, and for the valuable comments.\n> **Reviewer[V8Gu] Point #1** : Limited ablation beyond SSMs: While the paper provides thorough comparisons within the diagonal SSM family, it lacks ablations against non-SSM models like Transformers. Such comparisons would help clarify the broader applicability of the proposed method.\n\nIn fact, in Table S4 in Supplementary Material, we already include a comparison on Long Range Arena tasks with standard Transformer architecture and some of its variants as Performer, Reformer, and MEGA. In the revised version of the paper, we plan to include these variants in the main article, Table 1. \n\n> **Reviewer[V8Gu] Point #2** : Motivation could better connect to practical use cases: Although the motivation is theoretically compelling, it would benefit from examples or discussion showing potential industrial applications or real-world scenarios where spectral control via initialization is especially valuable, especially compared to other models, including transformers, MoE.\n\nWe actually provide two tasks that pertain to industrial applications or real-world scenarios. First, we evaluate on the Speech Commands dataset (SC35) for audio signal classification, where our proposed initialization scheme achieves state-of-the-art in zero-shot resampling (Table 2). Second, we assess physiological signal prediction using the BIDMC dataset, targeting respiratory rate, heart rate, and blood oxygen continuous time-series data. As shown in Appendix Table S3, a variant of our decoupling method also achieves state-of-the-art results in this domain.\n\nPrior work [1] notes that SSMs often underperform in tokenized or language tasks but excel in continuous domains such as signals or time series. Therefore, we are confident that our method has a big potential in industrial applications.\n\n[1] - Dao, Tri, and Albert Gu. \"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality.\" International Conference on Machine Learning. PMLR, 2024.\n\n> **Reviewer[V8Gu] Question #1** :Extension to Low-rank: Can the proposed initialization strategy (S4D-DFouT) be extended to low-rank or more general structured SSMs beyond the diagonal case? If so, what challenges or adjustments would be required?\n\n Introducing off‑diagonal or low‑rank components does make the spectrum, and thus mode behavior, less interpretable than in the purely diagonal case. In structured/low‑rank SSMs as S4, modes become linear combinations of eigenvectors, and low‑rank terms $UV^T$ perturb the poles. As a result, it is difficult to attribute specific frequency behavior to individual parameters—for example, pinpointing which row captures a given frequency or understanding how the amplitude and phase of that mode are governed by $C_n\\bar{B}_n$. So, to answer the question, we believe the extension is possible, but we do not know how to do it in an \"explainable\" way. \n\n> **Reviewer[V8Gu] Question #2** :Discussion on underperformance: What types of tasks or data characteristics might cause S4D-DFouT to underperform? Are there known limitations where this initialization is less effective?\n\nIn Section 6, we briefly outline the current limitations of SSMs—a discussion we will expand in the final draft. We show that many Long Range Arena benchmarks regarding sequential images can be “solved” by exploiting local spectral biases (for example, capturing frequencies corresponding to pixel offsets within image rows), rather than genuine long‑range modeling. In contrast, tasks that span the full frequency spectrum—such as ListOps or permuted sequential images like psCIFAR (see the flattened spectrum in Supplementary Figure S7)—remain far from solved. Although SSMs still outperform other architectures as Transformers on the latter problems, these are far from being considered solved.\n\n> **Reviewer[V8Gu] Question #3**:Computational cost: Could you provide a comparison of computational cost (e.g., training time, memory usage) versus accuracy to better illustrate the efficiency gains of S4D-DFouT relative to other models?\"\n\nA brief mention of the computational cost is done in the introduction, but we would increase the information in the manuscript. Diagonal SSMs present a significant cost reduction w.r.t. non-diagonal variants. As deeply inspected in [2], by materializing the Vandermonde matrix of A, the computation cost of computing the kernel in diagonal SSMs can be obtained in O((N+L) log2(N+L)) arithmetic operations. Our work builds upon these efficient diagonal variants; we do not propose architectural modifications, but rather aim to provide a mechanistic interpretation of those models. As such, the computational/memory cost remains unchanged.\n\n[2] Gu, Albert, et al. \"On the parameterization and initialization of diagonal state space models.\" Advances in Neural Information Processing Systems 35 (2022): 35971-35983.\n\nWe appreciate the reviewer’s comments, which helped us clarify both the motivation and broader implications of our work. We believe these revisions better position the paper’s contributions and respectfully invite reevaluation based on the new context."}, {"review_id": "c8IbhaUo5l", "reviewer": "Reviewer_YT22", "summary": "This paper presents the S4D-DFouT initialization scheme for deep SSM models. Whereas other methods either use initialization schemes that have no consistent frequency-domain implications, or tie the discrete frequency spectrum to the discretization parameter, DFouT directly targets a well-designed discrete frequency spectrum. Experiments are presented and DFouT-initialized methods appear to perform favorably. Some light interpretability inspections are made.", "questions": "1. Doesn’t the locality of the receptive field influence the decay rate as opposed as well the frequency?  Because if I had an energy-preserving eigenvalue then that frequency is repeated indefinitely, meaning its receptive field would become _every_ (e.g.,) 32 pixels?  So why is spacing the frequency components so important, and do you underweight/miss the contribution of amplitude components?", "limitations": "No societal impacts. \n\nNo substantial technical limitations not already discussed in weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Summary\nI think this is fundamentally a solid paper. The contribution is small, but neat and fairly well-motivated. I personally always found the initializion of the timescales and eigenvalues irritating, and so this is a nice exploration of that. The paper itself is well prepared, reads easily with nice figures, and has few typographical errors. I do think the contribution is on the smaller side however. Not insufficient for publication, but nothing to rave over. I err towards acceptance, and am very open to increasing my score should the authors convince me the contribution is larger than I initially give it credit (and answer the few questions I have).   \n# Strengths\n1. The initialization seems to improve performance on real world and simulated tasks.\n2. The little theory-esque blocks are nice, helping to concretely isolate and define anything that is mathematically non-trivial, requires precision, or that is difficult to explain in text.\n3. Some meaningful ablations and further probes were performed which do flesh out the methods reliability and interest. \n4. Something that is not commented on is that S4D-DFouT is a _drop-in replacement and simplification_ into existing codebases and should yield performance improvements across the board.\n5. Performance seems less sensitive to $\\eta$ than it is to $\\Delta$ (making hyperparameter tuning easier).\n6. I like tying the initialization across layers (cf. Equ. (11)).\n\n# Weaknesses\n1. _I stress to the authors I am very open to changing my mind on this point._ A tendency with initialization-focused papers I don’t like is that: you open with saying you’ll advance the understanding of the frequency-domain computation of these models, propose a new initialization, and then quote improved performance metrics on tasks (that you identify yourself as being flawed in key ways) as back-justifying whatever tweak to an initialization scheme you make.  There is a “gap” between the initialization and the final results that is nearly always uncommented on, training, and specifically, training dynamics. It always just leaves me wanting more, especially when presented with snippets like Figure 1 that begin to unpick the explainability and mechanistic interpretation of these models. I would love to see some slightly more incisive experiments (or justification from the authors that this exists) \n    - Mechanistically how the initialization leads to better models through training?  \n    - Can we design a task with known frequencies and show that poles approach known poles more quickly and reliably across pole distributions? \n    - Can we look at raster-scan like tasks (sCIFAR) and preferentially add poles around the image width multiples? \n    - Is there a way we can use the frequencies present in the image to better initialize models or compare learned frequencies to (beyond simply the image width)? \n    - Inspired by the better distribution of frequencies, is there a way to design even better models for multiple sampling frequencies given the Nyquist/Hermitian observations (cf. SC35)? \n    - What is the distribution of learned eigenvalues across layers?  As the computation is fundamentally distributed across layers?\n    - If we were to measure the total change of eigenvalues across training (or limit it with a specific trust region) do you get better performance?  Ie., is it the location of the poles themselves, or something rooted deeper in the dynamics of learning multiple eigenvalues and actually it is just the _spacing_ between eigenvalues (as opposed to their value) that makes the learning dynamics more effective?\n\n    These are the sort of questions that I think fill in some of the gap between initialization and raw results. \n\n    I will highlight you go further than a lot of papers (Figure 1, Figure 5 etc.), but I confess I am still left wanting more. To me, this paper feels like: “here is a way you can initialize that seems to lead to better performance” with some motivational text interspersed throughout. This deficiency makes me slightly less enthusiastic about the “completeness” or “togetherness” of the contribution.  I invite the authors to comment on this because I am very open to removing this from my evaluation. \n\n2. Why is S5 excluded as a diagonal variant in Table 1?  This is a fairly major point because S5 does use a diagonal parameterization, and outperforms DFouT-initialized models in most cases. \n\n# Minor Weaknesses.\n1. Some typos/errata: \n    - Line 216: absis.\n    - Throughout (e.g., Line 337): Some of the references to the appendix are incorrect. \n    - Throughout (e.g., Table 2): Please try to avoid in-lined floats, they are very bad for the readability of the document. \n2. There are a few unsubstantiated claims throughout the paper:\n    - Line 175: What does this have to do with “tuning” and/or “interoperability”; what does interoperability even refer to? \n    - Line 215: why is it uniquely a universal function approximator where other initializations aren’t?\n3. I don’t love punting the one result you apparently do worse on to the supplement, I would like to see results and discussion moved to the main. Especially when psCIFAR probably has the most interesting frequency requirements of the experiments.  It would be a very interesting experiment to use psCIFAR to explore the frequency concentration, because you could use a regular permutation (or even the existing permutation) to explore if the frequencies required for reconstruction are tied to the permutation (because the idea of “locality” has been permuted).\n4. I don’t actually understand why you claim it harms performance, it seems like DFouT does the best in Table 3?  Unless there is something I’m misunderstanding? \n5. I don’t understand the middle panel of Figure 4.  What is the distortion?  I think it has a key meaning that I simply cannot grok from the text.  It seems like smaller timescales with Lin (the default initialization) do better?  What happens if $\\Delta = 10e-4$?", "was_revised": true, "final_justification": "I appreciate the authors comments, particularly in response to W1.  However, the response somewhat validates my core point: it is still essentially an empirical observation that initializing closer to the final distribution yields better performance.  It doesn't then leverage this insight to break new ground or explain computational mechanisms (it just observes this results over and over again).  I think this is fine but it doesn't push me towards campaigning for the acceptance of this paper (essentially leaving my core contention unresolved).  The authors provide some reasonable defense of (although again, not fully resolving) my more minor comments.\n\nPrior to their response I was _absolutely indifferent_ to this paper (genuinely a 3.5).  Now I am happy with very weakly voting for acceptance (4), but I cannot push my score higher because I just do not think are enough contributions in the paper.  The missing comparisons to other baselines/alternatives (also cf. Reviewer eFzT) does not help because I am left wondering how generalizable the insight that is in the paper is.  To drive this paper to the next level I would love to see some more analysis of _why_ and _how_ spectral coverage is key (mechanistically) and the impact on the learning dynamics.  Good luck.", "rebuttal": "Dear reviewer, thank for your time in assessing our paper, and for the valuable comments.\n\n> **Reviewer[YT22] Point #1** : \"Mechanistically how the initialization leads to better models through training?\"\n\nWe understand the concerns of a circular justification—“better initialization implies better performance, therefore the initialization must be inherently superior.” Here are two findings we observed:\n\n**- Preservation of the initial spectral distribution.** We depict in Figure S9 in the Supplementary Material, the final distribution of the eigenvalues of A layer-by-layer after training in sCIFAR. We observe that in all initializations, the angles of the poles (resonant frequencies) remain largely unchanged from their initial values. However, poles exhibit a significant change in the decay, presenting a smaller radius after training. At the end of the training, S4D-Lin still shows the entanglement between decay and frequency,  resulting in spiral-like clustering of the poles; S4D-Inv spreads poles evenly around the circle; and S4D-DFouT keeps the original half-plane distribution of poles throughout training.  This observation indicates that the model preserves its initial spectral basis and primarily refines modal amplitudes during optimization. Consequently, we argue that a comprehensive frequency coverage at initialization contributes ultimately to elevated final performance. This reasoning thereby avoids the circular argument.\n\n**- Enhanced utilization of the spectral domain.** We also observe in Figure S9 that compared to S4D-Lin and S4D-Inv, the poles in DFouT that contribute most significantly (as measured by their $\\mathcal{H}_\\infty$-norm) are distributed more uniformly across both quadrants of the upper half-plane, rather than being concentrated at low frequencies. Consequently, DFouT more effectively leverages high-frequency modes that capture fine-grained input features.\n\nAlthough these observations do not fully explain the learning dynamics, they contribute to a more precise understanding of the role of initialization beyond mere final performance. We appreciate the reviewer’s suggestion to undertake this deeper analysis.\n\n> **Reviewer[YT22] Point #2** : \"Can we design a task with known frequencies and show that poles approach known poles more quickly and reliably across pole distributions?\"\n\nActually, the Continuous Copying task was motivated by a similar reason. In the Continuous Copying, we know a priori the frequencies (fundamental and harmonics) required to generate a _delay_ kernel exhibiting a spike at the desired $\\tau$. \n\nOne insight we have observed during training—as introduced in Point 1—is that the pole angles (resonant frequencies) remain largely unchanged throughout training. In this context, we also note that succeeding in the Continuous Copying task requires spectral coverage of those fundamental and harmonic frequencies from initialization. However, as illustrated in Figure 3, the spectral distribution of previous S4D initializations is strongly influenced by $\\Delta$, making it difficult for this initialization to learn that kernel unless the order $N$ of the model is sufficiently large (we tested up to $N=1024$). In the case of S4D‑Lin, the hyperparameter $\\Delta$ dictates the pole distribution: small $\\Delta$ values cluster poles at low frequencies, while large $\\Delta$ values suffer from aliasing. By contrast, S4D‑DFouT is more robust because it provides a uniform distribution of poles across the spectrum, providing initial coverage of these relevant frequencies otherwise hard to optimize during training. \n\n> **Reviewer[YT22] Point #3-#4** : \"Can we look at raster-scan like tasks (sCIFAR) and preferentially add poles around the image width multiples?\" \n\nThank you for the insightful question. While our primary aim was to design a data‑agnostic initialization that avoids some of the pitfalls of prior schemes, the question of using the model insights to exploit the spectral structure of the data is very on the point.\n\nTo test this hypothesis, we have designed an initialization **S4D-Spectrum** where the real part is preserced from DFouT and the imaginary part matches the discretize spectrum distribution (normalized) of the data. \n\n| Method                 | sCIFAR           | SC35 16kHz         | SC35 8kHz          |\n|------------------------|------------------|--------------------|--------------------|\n| S4D-Spectrum           |      67.17            |        96.10            |          91.31          |\n\n**Results.**  Regarding sCIFAR, its spectrum presents a fundamental frequency $\\Omega=2\\pi/32$ and clear harmonics at multiples of that fundamental (Figure S7 in the Supplementary Material). However, using this distribution S4D-Spectrum results in a noticeably less performant model. The insights we provided in Section 5.2 of the main paper can be useful in explaining such behaviour. In that experiment, we measure how much each SSM actually contributes to the output by computing its H-norm (Figure 5, right). We observe that the SSMs operating in the first layer close to the fundamental frequency, present noticeably larger H-norms. However, in successive layers, the frequencies of importance move toward lower bands.  The signal experiences progressive temporal mixing throughout the layers toward lower frequencies. In this regard, placing poles only at the marked frequencies in -all layers- does not benefit to the model. It seems to us, though, that setting a temperature hyperparameter to smooth that distribution could lead to better results.\n\n> **Reviewer[YT22] Point #5** :  \"Inspired by the better distribution of frequencies, is there a way to design even better models for multiple sampling frequencies given the Nyquist/Hermitian observations (cf. SC35)?\"\n\nWe extended the experiment by training S4D-Spectrum on SC35. In this setting, the spectrum of the data is more evenly distributed, yielding a more uniform pole coverage compared to sCIFAR. As a result, we observe performance on par with other initialization methods. S4D-Spectrum already incorporates Nyquist and Hermitian symmetry. We design its frequencies to span only the positive half of the spectrum—sampling the DFT from \\([0,\\pi]\\)—therefore, it does not suffer from aliasing.\n\n> **Reviewer[YT22] Point #6** :  \"What is the distribution of learned eigenvalues across layers? As the computation is fundamentally distributed across layers.\"\n\nAs previously introduced in Point \\#1, we already depict in Figure S9 in the Supplementary Material the final distribution of the eigenvalues of A layer-by-layer after training in sCIFAR for the different initialization schemes. Our major observation is that in all initializations, the angles of the poles (resonant frequencies) remain largely unchanged from their initial values. However, poles exhibit a significant change in the decay, presenting a smaller radius after training. In the particular task of sCIFAR, we believe that some poles are learned to decay rapidly to fine-control the initial coefficients of the kernel, which, as reported in Figure 5 (left), are particularly relevant for performance in this task (especially the first 32 coefficients). \n\n> **Reviewer[YT22] Point #7** : \"If we were to measure the total change of eigenvalues across training (or limit it with a specific trust region) do you get better performance?\n\nEmpirically —as commented in Point \\#1—we find that the vast majority of change on the poles occurs in the decay (radius); and even due to the time limitation, we could not complete the experiment setting a trust region, we expect no substantial improvement in the accuracy. Our observations suggest that ensuring good spectral coverage upfront is key. \n\n> **Reviewer[YT22] Point #8** : \"Why is S5 excluded as a diagonal variant in Table 1? This is a fairly major point because S5 does use a diagonal parameterization, and outperforms DFouT-initialized models in most cases.\"\n\nWe agree that the composition of Table 1 is not fair to S5 [6], since this method also uses a diagonal state matrix. We will complete Table 1 with S5 results, but also include our proposed initialization with S5 architecture. However, we would like to point an important observation concerning this method that highlights our main findings. Namely, the authors of S5 also reported in Table 7 of their work, results for different initializations as S5-Lin and S5-Inv in addition to S5 (which follows HiPPO-N matrix)—results that we have also collected in our Table S7. Interestingly, S5-Lin again underperforms on tasks as PathX-128, as the authors noted in their Appendix E.3. Therefore, rather than focusing on overall performance, we propose rewriting that table to highlight that previous Fourier-based initializations such as S4-Lin or S5-Lin have repeatedly underperformed in the literature due to some of the limitations we identify in this paper.\n\n> **Reviewer[YT22] Question #1** :   \"Doesn’t the locality of the receptive field influence the decay rate?\n\n We agree that the discussion in the decay is also very relevant for the work. We compiled some insights about the decay in the Supplementary Material (Figure S9, commented in Point \\#6); yet those conclusions should have a space in the main document. \n\nYes, if the eigenvalues responsible for capturing 32-pixel do not decay (i.e., if they preserve energy), then we would also see repeating peaks in the kernel—not only at 32, but at 64, 96... Indeed, this is exactly what we observe in the kernels of Figure S7. However, those later peaks are much attenuated due to the decay of the corresponding poles. We also provide an explanation for that behaviour. Learning a locally constrained receptive field can be beneficial for these models, as it mimics how CNNs operate: where early convolutional layers specialize in detecting localized features, while deeper layers progressively integrate these patterns into more abstract global representations."}, {"review_id": "S1VRUC8hGh", "reviewer": "Reviewer_eFzT", "summary": "This paper investigates new initialization schemes for diagonal state space models (SSMs) to address two issues: first, the existing methods often mix frequency and decay, making the models too sensitive to the chosen time step during discretization; second, the frequency coverage of these methods doesn't usually match the needs of real-world tasks. \nTo overcome these problems, the authors introduce a new initialization scheme. The authors test their method on several benchmarks, including synthetic tasks like the continuous copying task, and real-world task like speech classification, and the Long Range Arena benchmark.", "questions": "While it may be outside the current scope of the paper, I’m curious about how the state matrix weights evolve throughout training. Analyzing this could provide further insight into why the proposed initialization leads to better performance, and whether its benefits persist or diminish as training progresses.", "limitations": "yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is clear, well-written, and easy to follow.\n2. It provides a thorough and insightful ablation study that effectively supports its claims.\n3. A detailed and honest limitation section. \n\n**Weaknesses:**\n1. The paper heavily emphasizes discretization issues in diagonal state space models but does not consider alternative state space methods that avoid discretization entirely, such as LRU (Resurrecting Recurrent Neural Networks for Long Sequences, Orvieto et al.).\n2. Due to point 1, discretization concerns are particularly relevant in selective state space models like Mamba (Mamba: Linear-Time Sequence Modeling with Selective State Spaces, Gu and Dao) where discretization is still used, yet the paper overlooks these types of models.\n\nIn my opinion, these omissions greatly weaken both the scientific framing and empirical contribution of the paper.", "was_revised": true, "final_justification": "Writing the same justification I wrote to the Authors:\n\nWhile this assessment is inherently subjective, I maintain my stance that the scope of the paper is still too narrow for acceptance into the conference. The primary reason is that the subject matter (initialization schemes for discretized state space models (SSMs)) does not hold sufficient significance for two main reasons:\n\n1. As previously highlighted in my original review, simpler models exist today, such as LRU, which are expressively equivalent to S4 but do not require discretization. The authors themselves see in the Appendix that LRU simpler models outperform their newly proposed initialization in approximately 50% of the benchmarks. Specifically, the authors state:\n\n> \"Although we included LRU in the results comparison in LRA (Table S4 in Supplementary Material), this work definitely requires better coverage in the related work.\"\n\nGiven these points, I believe initialization schemes of discretized SSMs are not significant enough to justify inclusion in a top-tier conference.\n\n2. An exception to the above critique would have been possible had the empirical evidence demonstrated clear superiority of their method. However, while the authors do show improvements over other S4 initializations, their approach still underperforms against much simpler alternatives such as LRU or newer models like S5.\n\nWhile I completely understand the difficulties in analyzing selective SSMs like Mamba, due to what is stated above, the importance of the analysis of discretization schemes in non-selective SSMs remains questionable given their complexity and limited performance.\n\nNevertheless, I acknowledge that the authors' analysis of dynamics and the evolution of learned poles provides valuable insights. In their rebuttal, the authors highlight:\n\n> \"We depict in Figure S9 in the Supplementary Material the final distribution of the poles layer-by-layer after training in sCIFAR for the different initialization schemes \\[...]. Our major observation is that in all initializations, the angles of the poles (resonant frequencies) remain largely unchanged from their initial values.\"\n\nThis type of analysis indeed contributes meaningfully to the field. However, if the authors feel that this analytical perspective is their primary contribution, then the paper should be positioned as such, with additional experiments to strengthen and expand these insights. As it currently stands, I do not find the contribution substantial enough to recommend acceptance.\n\n*I want to note that while I am very confident in my view, all my reasons for rejecting are listed above. If the Area Chair, with many more years of experience than myself, believes I am setting the bar for acceptance too high, I encourage them to disregard my opinion.*", "rebuttal": "Dear reviewer thank for your time in assessing our paper, and for the valuable comments.\n\n> **Reviewer[eFzT] Point #1**: Consider alternative SSMs models: The paper heavily emphasizes discretization issues in diagonal state space models but does not consider alternative state space methods that avoid discretization entirely, such as LRU (Resurrecting Recurrent Neural Networks for Long Sequences, Orvieto et al.).\n\nThank you for pointing out the reference LRU [4]. Although we included LRU in the results comparison in LRA (Table S4 in Supplementary Material), this work definitely requires better coverage in the related work. \n\nWe briefly recall the main findings of the LRU paper. As the Reviewer points, \nLRU does not rely on the discretization of a latent continuous-time system but directly optimizes the magnitude and oscillation frequencies independently. LRU method preserves (and in a sense is motivated by) the Glorot initialization from RNN, which in the limit behavior is de-facto uniform spectral initialization in the complex unit disk. \n\nYet our contributions are complementary to this work, both in the findings and in the motivation. We first provide a mechanistic interpretation of S4D models from the frequency perspective. Furthermore, we incur on the analysis on other eigenvalue distributions rather than uniform. We show the importance of the frequential coverage aligned with the spectrum of the data and point out some design flaws of previous S4D initializations as the aliasing and Hermitian symmetry. On the practical side, we provide a layer‑by‑layer analysis of what these models learn (Figure 5), observing that initial layers focus explicitly on capturing the frequency present in the data, revealing emergent “attention” patterns in these initial layers. Leveraging these insights, we propose our initialization.\n\n[4] Orvieto, Antonio, et al. \"Resurrecting recurrent neural networks for long sequences.\" International Conference on Machine Learning. PMLR, 2023.\n\n> **Reviewer[eFzT] Point #2**: Selective State Space Models: Discretization concerns are particularly relevant in selective state space models like Mamba (Mamba: Linear-Time Sequence Modeling with Selective State Spaces, Gu and Dao) where discretization is still used, yet the paper overlooks these types of models.\n\n In Mamba's \"selective\" state‐space formulation, the initialization of the state matrix A plays a much smaller role than in earlier SSMs architectures. While Mamba does discretize a latent continuous‑time state matrix A, its input‑dependent selection operator $\\Delta(x_t)$ dynamically adjusts the effective timestep, making this model less dependent to the initialization scheme. Particularly, in that work [3], authors experiment with three initializations of the SSMs: the real variant is S4D-Real, the complex variant is S4D-Lin, and a random initialization, achieving very similar performance in all cases (see section 4.6.2 of the original work). In Mamba (and more generally in S6 models) the matrix A is not tied to a single fixed discretization step; instead, selectivity produces a context‑dependent state update at every step.\n\nIn that non‑stationary domain, providing model interpretability complicates due to the input‑dependent dynamics that evolve at each timestep. Therefore, we limit our work to time-invariant dynamics.\n\n\n\n[3] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).\n\n> **Reviewer[eFzT] Question #1**: Evolution of poles throughout the training: While it may be outside the current scope of the paper, I’m curious about how the state matrix weights evolve throughout training. Analyzing this could provide further insight into why the proposed initialization leads to better performance, and whether its benefits persist or diminish as training progresses.\n\nThank you for the question. We depict in Figure S9 in the Supplementary Material the final distribution of the poles layer-by-layer after training in sCIFAR for the different initialization schemes (our proposed and the baselines). Our major observation is that in all initializations, the angles of the poles (resonant frequencies) remain largely unchanged from their initial values. However, poles exhibit a significant change in the decay, presenting a smaller radius. In the particular task of sCIFAR, we believe that some poles are learned to decay rapidly to fine-control the initial coefficients of the kernel, which, as reported in Figure 5, are particularly relevant for performance in this task (especially the first 32 coefficients). Furthermore, the initial distribution is preserved: with S4D-Lin showing the entanglement between decay and frequency,  resulting in spiral-like clustering of the poles; S4D-Inv spreading poles evenly around the circle; and S4D-DFouT maintaining the original half-plane distribution of the poles throughout training.\n\nIn the updated manuscript, we will provide an extension of Figure S9, covering the distribution of the poles (with corresponding norms) for different stages throughout the training (current regulations prevent us from sharing figures in the rebuttal). \n\nWe appreciate the reviewer’s comments, which helped us clarify both the motivation and broader implications of our work. We believe these revisions better position the paper’s contributions and respectfully invite reevaluation based on the new context."}, {"review_id": "iFgklrM8yW", "reviewer": "Reviewer_bKH5", "summary": "In this paper, the authors study the initialization of diagonal SSMs from a frequency perspective. They observe that in previous work, the diagonal state matrix is first initialized in the continuous domain and then discretized using a learnable discretization parameter during training. This method has several disadvantages, including the coupling between the decay rate and the oscillation frequency through the discretization parameter, a lack of interpretability or fine-grained control, and misalignment with the needs of real-world tasks. To address these limitations, the authors propose a universal initialization scheme for discrete SSMs from a frequency perspective, where all poles are placed uniformly around the unit circle in the complex plane and modulated by a shared exponential decay. Unlike previous methods, the learnable discretization parameter is removed, thereby eliminating the entanglement between decay and frequency components. Experimental results demonstrate that the proposed initialization scheme enables SSMs to achieve superior performance on the Long Range Arena benchmark. Additionally, it facilitates training SSMs on extremely large datasets, which was previously challenging.", "questions": "1. It seems that there is some typos in Eq. (8). The correct form should be $$\\sum _{n=0}^{N-1} C_n \\overline{B} _n \\sum _{l \\geq 0} e^{-(\\Delta \\alpha _n + i(\\theta - \\Omega_n)l)} = \\sum _{n=0}^{N-1} \\frac{C_n \\overline{B} _n}{1-e^{-\\Delta \\alpha _n} e^{i (\\theta - \\Omega_n)}}.$$\n2. In the proof of Proposition 1, the definition of $d _n$ seems to be confusing. At step 1, $d _n$ is defined as $d _n = e^{-\\frac{\\Delta}{2}} e^{i \\pi n \\Delta}$. Thus, we have $d _n^k = e^{-\\frac{k \\Delta}{2}} e^{i k \\pi n \\Delta}$. Further, if $\\Delta = 2/\\tau$, we can derive that $d _n^k = e^{-\\frac{k}{\\tau}} e^{\\frac{2 i k \\pi n}{\\tau}}$. However, the authors show that $d _n^k = e^{\\frac{2 i k \\pi n}{\\tau}}$ holds in step 5. Can the authors provide more explanations on this?", "limitations": "The study's limitations are explicitly addressed in the final section of the main text.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The motivation of this work is clear and reasonable. From a frequency perspective, the authors propose a more natural and general initialization scheme for discrete SSMs. The experimental results are comprehensive and sufficient to support the effectiveness of the proposed method.\n- From my personal perspective, the proposed initialization scheme is novel. The inspiration comes from the spectral domain, which is different from previous initialization approaches that are designed in the spatial domain.\n\n**Weakness**\n- Theoretical analysis of why the proposed initialization scheme enables SSMs to achieve better performance remains limited. In Proposition 1, the authors demonstrate that a previous method, S4D-Lin, fails to solve the continuous copying task when the discretization parameter is selected inappropriately. However, the authors do not provide theoretical analysis of the learning dynamics of SSMs under their proposed initialization. As a result, it remains unclear why their method enables SSMs to converge to better optima and achieve improved generalization performance.\n- The presentation quality could be enhanced by correcting typographical errors and providing more detailed explanations. Specific suggestions are provided in the Questions section.", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns in the rebuttal. There do exist some typos in Equations or Theorems, yet the authors have fully corrected them in the rebuttal, and I also believe that they do not affect the final results. From my personal perspective, this work is interesting and technical solid, and I think that it meets the conference's acceptance criteria.", "rebuttal": "Dear reviewer thank for your time in assessing our paper, and for the valuable comments.\n\n> **Reviewer[bKH5] Point #1**: \"Theoretical analysis of why the proposed initialization scheme enables SSMs to achieve better performance remains limited. In Proposition 1, the authors demonstrate that a previous method, S4D-Lin, fails to solve the continuous copying task when the discretization parameter is selected inappropriately. However, the authors do not provide a theoretical analysis of the learning dynamics of SSMs under their proposed initialization. As a result, it remains unclear why their method enables SSMs to converge to better optima and achieve improved generalization performance.\"\n\nIn fact, as shown in the center panel of Figure 4, our method actually performs marginally worse than S4D‑Lin under the optimal configuration of $\\Delta$, which, as demonstrated in Proposition 1, converges to the optimal solution for S4D-Lin. The advantage we want to highlight in that motivating experiment on the Continuous Copying task is that our initialization S4D-DFouT does not require tuning of $\\Delta$, achieving competitive results where previous initializations are severely sensitive to the selection of $\\Delta$ hyperparameter. \n\nTo really motivate with a real example the hardness of selecting $\\Delta$, we also discuss in Section 5.3 the following case. In the literature, S4D-Lin has repeatedly underperformed, failing to solve challenges like PathX-128 that other initializations could handle. We show that this shortfall is not an intrinsic limitation of the linear initialization, but rather of the wrong selection of the $\\Delta$ hyperparameter in the literature. To capture a period of 128, according to our insights, we require $\\Delta > 2/128 \\approx 0.015$; however in the literature [2] a range of [0.0001, 0.01] was used, just by increasing the range over the calculated threashold we suceed in learning the task (marked with \"†\" in Table 1).\n\n\n> **Reviewer[bKH5] Question #1**: Typos and erratas. It seems that there is a typo in Eq.(8). In the proof of Proposition 1, the definition of $d_n$ seems to be confusing.\n\n- The typo on the sign on Eq.(8) will be corrected.\n\n- Thank you for pointing this mistake in the Proposition. We corrected the statement to emphasize the assumption that is no decay in the poles. Therefore, the Proposition 1 is as follows: \n\n**Proposition 1.**  \nLet the S4D‑Lin convolutional kernel under zero-order hold (ZOH) discretization be defined as  \n$\\overline{K}[l] = 2\\mathrm{Re} \\left( \\sum_{n=0}^{N-1} C_n \\, \\overline{\\lambda}_n^\\ell \\right)$,  where $\\overline{\\lambda}_n$ are the discrete-time eigenvalues and $C_n \\in \\mathbb{C}$. \n\nIf we choose $\\Delta = \\frac{2}{\\tau}$ and $\\alpha = 0$ (**no decay in the poles**), for some $\\tau \\in \\mathbb{Z}_{>0}$, and set $C_n = 1$ for all $n$,  then $\\overline{K}$ exhibits a spike at $l = \\tau$, satisfying  $|\\overline{K}[l]| < |\\overline{K}[\\tau]|$ for all $l \\ne \\tau$.  \n\nMoreover, if we treat $C = (C_0, \\dots, C_{N-1})$ as trainable parameters,  then the associated Vandermonde matrix defined by $V_{\\ell n} = \\overline{\\lambda}_n^\\ell$  is well-conditioned, and gradient descent on $C$ converges at a linear rate."}]

Author rebuttal:

Meta-review: This paper proposes a novel initialization scheme for diagonal state space models. The reviewers generally found the paper to be well-written, and the contribution appears to be novel, conceptually motivated, and the method empirically outperforms several baselines.

Hopefully the authors can address in their final version some of the concerns about significance. Reviewers were particularly interested in understanding the mechanics of the learning process, both to explain the performance benefits of the proposed method and to potentially extend the impact of the frequency-based analysis to other aspects of the model (like features and/or architectures). It would be great to see some additional baselines in the experiments as well, as suggested by several reviewers.
Decision: Accept (poster)