Tuning Frequency Bias of State Space Models
Authors: Annan Yu
Dongwei Lyu
Soon Hoe Lim
Michael W. Mahoney
N. Benjamin Erichson
Venue: ICLR 2025 Spotlight (2025)
Keywords: state-space models
sequence models
Long-Range Arena
frequency bias
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose two mechanisms to diminish or increase the learning rate of high-frequency components relative to low-frequency ones in a state space model (SSM).

Abstract: State space models (SSMs) leverage linear, time-invariant (LTI) systems to effectively learn sequences with long-range dependencies. By analyzing the transfer functions of LTI systems, we find that SSMs exhibit an implicit bias toward capturing low-frequency components more effectively than high-frequency ones. This behavior aligns with the broader notion of frequency bias in deep learning model training. We show that the initialization of an SSM assigns it an innate frequency bias and that training the model in a conventional way does not alter this bias. Based on our theory, we propose two mechanisms to tune frequency bias: either by scaling the initialization to tune the inborn frequency bias; or by applying a Sobolev-norm-based filter to adjust the sensitivity of the gradients to high-frequency inputs, which allows us to change the frequency bias via training. Using an image-denoising task, we empirically show that we can strengthen, weaken, or even reverse the frequency bias using both mechanisms. By tuning the frequency bias, we can also improve SSMs' performance on learning long-range sequences, averaging an $88.26\\%$ accuracy on the Long-Range Arena (LRA) benchmark tasks.

Published as a conference paper at ICLR 2025
TUNING FREQUENCY BIAS OF STATE SPACE MODELS
Annan Yu Dongwei Lyu² Soon Hoe Lim Michael W. Mahoney N. Benjamin Erichson 5.0
Center for Applied Mathematics, Cornell University, Ithaca, NY 14853, USA
2 Data Science Institute, University of Chicago, Chicago, IL 60637, USA
3 Department of Mathematics, KTH Royal Institute of Technology. Stockholm, Sweden
4 Nordita, KTH Royal Institute of Technology and Stockholm University, Stockholm, Sweden
5 Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA
6 International Computer Science Institute, Berkeley, CA 94704, USA
7 Department of Statistics, University of California at Berkeley, Berkeley, CA 94720, USA ay2620 cornel edu, dw wullyu@cch www.cccccc.o..aaa..oo cago. sh1 im@kth. mmahoney@ stat. berke lekelly..... edu, erichson@icst .es.soonee.s...beeeeellee... edu
ABSTRACT
State space models (SSMs) leverage linear, time-invariant (LTI) systems to ef- fectively learn sequences with long-range dependencies. By analyzing the trans- fer functions of LTI systems, we find that SSMs exhibit an implicit bias toward capturing low-frequency components more effectively than high-frequency ones. This behavior aligns with the broader notion of frequency bias in deep learning model training. We show that the initialization of an SSM assigns it an innate fre- quency bias and that training the model in conventional way does not alter this bias. Based on our theory, we propose two mechanisms to tune frequency bias: either by scaling the initialization tune the inborn frequency bias; by apply- ing a Sobolev- norm- based filter to adjust the sensitivity of the gradients to high- frequency inputs, which allows to change the frequency bias via training. Using an image-denoising task, we empirically show that we can strengthen, weaken, or even reverse the frequency bias using both mechanisms. By tuning the frequency bias, we can also improve SSMs' performance on learning long-range sequences, averaging an 88. 26% accuracy on the Long-Range Arena (LRA) benchmark tasks.
INTRODUCTION
Sequential data are ubiquitous in fields such natural language processing, computer vision, gener- ative modeling, and scientific machine learning. Numerous specialized classes of sequential models have been developed, including recurrent neural networks (RNNs) (Arjovsky et al., 2016; Chang et al,, 2019; Erichson et al., 2021; Rusch & Mishra, 2021; Orvicto et al., 2023), convolutional neural networks (CNNs) (Bai et al., 2018; Romero et al., 2022), continuous-time models (CTMs) (Gu al., 20216; Yildiz et al., 2021), transformers (Katharopoulos et al., 2020; Choromanski et al., 2020; Ki- taev et al., 2020; Zhou et al., 2022; Nic al., 2023), state space models (SSMs) (Gu et al., 2022b;a; Hasani al., 2023; Smith al., 2023), and Mamba (Gu & Dao, 2023; Dao & Gu, 2024). Among these, SSMs stand out for their ability to learn sequences with long-rang dependencies.
Using the continuous-tima linear, invariant (LTI) systems,
x x((t) Ax(t) + Bu(t).
y(t) Cx(t) + Du(t),
where A A Ca BEC CE C and D x SSM computes the output time-series (t) from the input utt) via a latent state vector x(t). Compared to an RNN, a major computational advantage of an SSM is that the LTI system parallelized for long sequences) and numerically robustly (i.e., it does not suffer from vanishing and exploding gradients). An LTT system can be computed in the time domain via convolution:
be trained both efficiently (i.e., the training be
yt) = (h * u (t) { h(t - r)u(T)dr + Du(t),
h(t) = Cexp(tA)B.

---

Published as a conference paper at ICLR 2025
Problem Formulation
Results
d1 cos(3)
Model Input
Model Output
Mcan Eiror
10
d2 (16) AAMANAAA
SSM
10
Og ooe (256x))
-26
Epochs 100
Figure 1: In a synthetic example to illustrate the frequency bias of SSMs, we form the inputs by superposing three waves of low, moderate, and high frequencies, respectively. We train an S4D model to regress the magnitudes of the three waves. We observe that the magnitudes of the low- frequency waves can be approximated much better compared to those of the high-frequency waves. In Figure 10, we show how tune the frequeney bias in this example.
Alternatively, it can be viewed as an action in the frequency domain:
{s) G(is)
G(is) C(isI AB++BDDD S E E.
where i is the imaginary unit and I is the identity matrix. The function G is called the transfer function of the LTI system.
The requenncydomaaie characterization of the LTI systems eq. (2) sets the stage for understanding the so-called frequency bias of an SSM. The term "frequency bias" originated from the study of a general overparameterized multilayer perceptron (MIP) (Rahaman et al., 2019), where it was observed that the now-frequency content was learned much faster than the high-frequency content. It is a form of implicit regularization (Mahoney, 2012). Frequency bias is a double-edged sword: on one hand, it partially explains the good generalization capability of deep learning models, because most figh-frr noises are not learned until the low-frequency components are well-captured; on the other hand, puts a curse on learning the useful high- figh-freuueccy information in the target.
In this paper, we aim to understand the frequency bias of SSMs. In Figure 1, we observe that, sim- ilar to most deep learning models, SSMs are also better at learning the low frequencies than the high ones. To understand that, we develop theory that connects the spectrum of A to the SSM's capability of processing high-frequency signals. Then, based on the spectrum of A, we analyze the frequency bias in two steps. First, we show that the most popular initialization schemes (Gu et al., 2020; 2021b; Yu et al., 2024b) lead to SSMs that have an innate frequency bias. More precisely, they place the spectrum of A, A(A), in the wwwfrrquency region in the s-plane, preventing LTI systems from processing gee-frequency input, regardless of the values of B and C. Second, we consider the training of the SSMs. Using the decay properties of the transfer function, we show that if an eigen- value Oj E A(A) (A) is initialized in the low-frequency region, then its gradient is insensitive to the loss induced by the high-frequent input content. Hence, if an SSM is not initialized with the capability of handling high-frequency inputs, then it will not be trained to do so by conventional training.
The initialization of the LTI systems equips an SSM with a certain frequency bias, but this is not necessarily the appropriate implicit bias for a given task. Depending on whether an SSM needs more expressiveness or generalizability. we may want less or more frequency bias, respectively (see Figure 10). Motivated by our analysis, we propose two ways to tune the frequency bias:
1. Instead of using the HiPPO initializations, the most popular class of initializations used in practice, we scale A(A) to lower or higher-frequency regions initialization tuning strategy" that marks out the regions in the frequency domain that can be learned.
"hard
2. Motivated by the Soboley norm, which applics weights to the Fourier domain, we can apply multiplicative factor of (1 + [s|)* to the transfer function G(is). This is "soft tuning strategy" that reweighs each location in the frequency domain. By selecting a positive or negative B, we make the gradients more or even less sensitive the high-frequency input content, respectively, which changes the frequency bias during training.
One can think of these two mechanisms as ways to tune frequency bias at initialization and dur- ing training, respectively. After rigorously analyzing them, we present an experiment image- denoising with different noise frequencies to demonstrate their effectiveness. We also show that tuning the frequency bias enables better performance on tasks involving long-range sequences. Equipped with our two tuning strategies, a simple S4D model can be trained to average an 88.26% accuracy on the Long-Range Arena (LRA) benchmark tasks (Tay al., 2021).
2

---

Published as a conference paper at ICLR 2025
Contribution. Here are our main contributions: (1) We formalize the notion of frequency bias for SSMs and quantify it using the spectrum of A. We show that a diagonal SSM initialized by HiPPO has innate frequency bias. We are the first to study the training of the state matrix A, and we show that training the SSM does not alter this frequency bias. (2) We propose two ways to tune frequency bias, by scaling the initialization, and by applying Sobolev-norm-based filter the transfer function of the LTI systems. We study the theory of both strategies and provide guidelines for using them in practice. (3) We empirically demonstrate the effectiveness of our tuning strategies using an image-dene denggddeniiiing task. We also show that tuning the frequency bias helps an S4D model to achieve state-of- -otte-o---h-rrrrrree aht performance on the
Arena tasks and provide ablation studies.
To make the presentation cleaner, throughout this paper, we focus on a single dingle-input/sinn output (SISO) L.TI system r= (A, B, C, D) in an SSM, i.e., m = p 1, although all discussions (MIMO) case, as in an S5 model (Smith et al,, 2023). Hence, the transfer funetion G C C is complex-vaie valued. We emphasize that while we focus on single system I', we do not isolate it from a large SSM; in fact, when we study the training of I' in section 4, we backpropagate through the entire SSM.
naturally extend to the multipleinnut/mmuliile..om
Related Work. The frequency bias, also known as the spectral bias, of general neural network (NN) was initially observed and studied in Rahaman et al. (2019); Yang & Salman (2019); Xu (2020). The name spectral bias stemmed from the spectral decomposition of the so-called neural tangent kemels (NTKs) (Jacot et al., 2), which provides a means of approximating the training dynamics of overparameterized NN (Arora et al., 2019; Su & Yang, 2019; Cao et al., 2019). By carefully analyzing the eigenfunctions of the NTKs, Basri et al. (2019); Bietti & Mairal (2019) proved the frequency bias of an overparameteriiee two-layer NN for uniform input data. The case of nonuniform input data was later studied in Basri et al. (2020); Yu et al. (2023). The idea of Sobolev-norm seayy-nom-bbseee training of NNs has been considered in Vlassis & Sun (2021); Yu et al. (2023); Tsay (2021); Son et al. (2021); Czarnecki et al. (2017); Zhu et al. (2021); Son (2023); Liu et al. (2024). The initialization of the L.TI systems in SSMs plays a crucial role, which was first observed in Gu et al. (2020). Empirically successful initialization schemes called "HiPPO" were proposed in Voelker et al. (2019); Gu et al. (2020; 2023). Other efforts in improving the initialization of an SSM were studied in Yu et al. (2024b); Liu & Li (2024a). Later, Orvicto et al. (2023); Yu et al. (2024a) attributed the success of HiPPO to the proximity of the spectrum of A to the imaginary axis (i.e., the real parts of the eigenvalues of A are close zero). This paper considers the imagi- nary parts of the eigenvalues of A, which was also discussed in the context of the approximation- estimation tradeoff in Liu & Li (2024b). The training of SSMs has mainly been considered in Smékal et al. (2024); Liu & Li (2024a), where the matrix A is assumed to be fixed, making the optimization convex. To our knowledge, we are the first to consider the training of A. While we consider the de- cay of the transfer functions of the LTI systems in the frequency domain, there is extensive literature on the decay of the convolutional kernels in the time domain (i.e., the memory) (Hardt et al., 2018; Gu et al., 2020; Wang & Li, 2023; Wang & Xue, 2024; Orvieto et al., 2024; Yu et al., 2024a).
2 WHAT IS THE FREQUENCY BIAS OF AN SSM?
In Figure 1, we see an example where an S4D model is better at predicting the magnitude of a now-frequency component in the input than a high-frequency one. This coincides with our intuitive interpretation of frequency bias: the model is better at "handling" low frequencies than high frequencies. To rigorously analyze this phe- nomenon for SSMs, however, we need to formalize the notion of frequency bias. This is our goal in this sec- tion. One might imagine that an SSM has a frequency bias if, given a time-series input u(t) that has rich high- frequency information, its time-series output y(t) lacks high-frequency content. Unfortunately, this is not the case: an SSM is capable of generating outputs. Indeed, the skip connection D of an LTI system
Transfer Function G
high Erequency repion lows frequdacy region high frequency region
Imaginary atis Eiquenny domain
Obigs Deaph
Vine
Vilie (G)
The frequency bias of an SSM says that the frequency response has more variation in the low-frequency area than the high-frequency one. is an "all-pass" filter, multiplying the whole input u(t) by a factor of D and adding it to the output (t). On the other hand, the secret of successful SSM hides in A, B, and C (Yu et al., 2024a).
Figure 2:
3

---

Published as a conference paper at ICLR 2025
In ablation study, when D is removed, S4D model only loses less than 2% of accuracy the sCIFAR-10 task (Tay et al., 2021; Krizhevsky et al., 2009), whereas the model completely fails when we remove A. B, and C. This can be ascribed to the LTI system'ss ''s power to model complicated behaviors in the frequency domain. That is, each Fourier mode in the input has its own distinct "pass rate" (see Adamyan al. (1971); Sun (2020); Yu & Townsend (2024) for why this is important feature of LTI systems), For example, the task in Figure I can be trivially solved if the LTI system can filter out single mode dn cos(z), 02 cos (166)) or ag cos (256x)) from the superposition of the three: the skip connection D alone is not capable of doing that.
Given that, we can formulate the frequency bias of an LTI system as follows (see Figure 2):
Frequency bias of an SSM means that the frequency responses (i.e., the transfer functions G) of LTI systems have more variation in the low-frequency area than the high-frequency area.
More precisely, given the transfer function G(is), we can study its total variation in a particular interval (a, b] in the Fourier domain defined by
Σ Gi(s)) G(is)-1)]= /"' dG((ss ds
(G)
sup
ds, S a <a<bsoo
NEN
Intuitively, (G) measures the total change of Giss) when s moves from a to b. The larger it is, the better an LTI system is at distinguishing the Fourier modes with frequenciess between and b. Frequency bias thus says that for a fixed-length interval (a, bj. Vd(G is larger when (a, b] is near the origin than when it lies in the high-frequency region, i.e., when it is far from the origin.
3 FREQUENCY BIAS OF AN SSM AT INTTIALIZATION
Our exploration of the frequency bias of an SSM starts with the initialization of SISO LTI system (A, B, C, D), where = diag(a1..... ap. E Caxn is diagonal. The system assumed to be stable, meaning that a = , + iw, for some vy < 0 for all S S where v, and w, are the real and the imaginary parts of aj. respectively. Note that the diagonal structure of A is indeed the most popular choice for SSMs (Gu al., 2022a; Smith al., 2023), in which case it suffices to consider
the Hadamard (i.e., entrywise) product B a CT [c] all 1 S j < Then, the transfer function G is naturally represented in partial fractions:
c.] E C", where ej = E, + iCn for
G (is)
On
D
{ + içi
En iSn
+ D.
is a1
an
U1 + i(s w1
Un i(s w.,)
In most cases, the input u(t) is real-valued. To ensure that output is real-valued, the standard practice take the real part G real-valued transfer function before applying eq. (2):
G(is) == RisG((i)) = Σ ( Egyy +D.
+ + w,)²
We now derive general statement for the total variation of G given the distribution of wy.
Lemma 1. Let G be the transfer function defined in eq. (3). Given any > max, |wyl. we have
c
cy
vーž
(G)
V
W +
W
B
1=1
Given the formula of the transfer function, the proof of Lemma is almost immediate. In this paper, leave all proofs to Appendix B and D. Lemma I illustrates clear and intuitive concept:
If the imaginary parts of aj are distributed in the tow-frequency region, i.e., |wy] are small, the transfer function has small total variation in the high-frequency areas (-o0,- and [B oo) as > oc, inducing a frequency bias of the SSM.
We can now apply Lemma et al., 2020). While there are many variants of HiPPO, we choose the one that is commonly used in practice (Gu al., 2021a). All other variants can be similarly analyzed.
study the innate frequency bias of the HiPPO initialization (Gu
4

---

Published as a conference paper at ICLR 2025
Corollary 1. Assume that a, = -0.5 + ((-1))(1/2]) and Eg. Sy NO, 1) ii.d., where WV 1) is the standard normal distribution. Then, given B > n and >> O, we have
(G) S /2n( Vn + Vln(1/0)) B - n/22
with probability 2 1--6 5.
In particular, Corollary tells us that the HiPPO initialization only captures the frequencies s e [- B3] up to = (n)) because when = (n), we see that V= (G), Vg((G) vanish as n increases. This means that no complicated high-frequency responses can be learned.
4 FREQUENCY BIAS OF AN SSM DURING TRAINING
In section 3, we see that the initialization of the LTI systems equips an SSM with an innate frequency bias. natural question to ask is whether an SSM can be trained to adopt high- geh-frequency responses. Analyzing the training of an SSM (or many other deep learning models) is not easy task, and we lack theoretical characterizations. Two notable exceptions are Liu & Li (2024b); Smékal et al. (2024), where the convergence of a trainable 1.TI system to target L.TI system is analyzed, assum- ing that the state matrix A is fixed to make the optimization problem convex. Unfortunately, this assumption is too strong to be applied for our purpose. Indeed, Lemma I characterizes the frequency bias using the distribution of w/ = Im(ay), making the training dynamics of A a crucial element in our analysis. Even if set aside the issue of A, analyzing an isolated LTI system in SSM remains unrealistic: when an SSM, consisting of hundreds of LTI systems, is trained for a single task, there no clear notion of "ground truth" for each individual LTI system within the model.
To make our discussion truly generic, we assume that there is a loss function ((O) that depends on all parameters + of an SSM. In particular, O contains Uy, wy, Eg. Sy. and D from every LTI system within the SSM, as well as the encoder, decoder, and inter-layer connections. With mild assumptions on the regularity of the loss function L. we provide a quantification of the gradient of L with respect to w that leads to qualitative statement about the frequency bias during training.
Theorem 1. Let L(O) be loss function and (A, B, C, D) be a diagonal LTTI system in an SSM defined in section 3. Let G be its associated real-valued transfer function defined in eq. (3). Suppose the functional derivative of ((O) with respect to (iss) exists and is denoted by (is)G
Then, if (()/a = ([s]p)) for some < 1, we have
ac 0ro
= | (is))
00 Oc Ks(s) ds,
K5($) = Ci((s wzy)² - v²t) 25, w,)
(4)
[03 + (s )
for every S S n. In particular, we have that [55()] = O (15)8-21 + 15, as |s| + 00.
In Theorem sumption that (is)) exists is easily satisfied, and we leave survey of functional derivatives to Appendix C. The assumption that (a/aG(is))
technical tool called the functional derivative (Gelfand et al., 2000). The
grows at most sublinearly is to guarantee the convergence of the integral in eq. (4); it is also easily satisfiable. We will see that the growth/decay rate of (is))((s))) the norbole--oomm-ae based method (see section 5.2). As usual, one can intuitively think of the func- tional derivative (a/ /// [ as measurement of the "sensitivity" of the loss function L to LTI system's is action on a particular frequency s (i.c., G(s)). The fact that is multiplied by a factor of (s) in the computation of the gradient in eq. (4) conveys the following important message:
plays a more important role when we start to tune the frequency bias using
The gradient of L with respect wi highly depends on the part of the loss that has "local" frequencies near s = w,. It is relatively unresponsive to the loss induced by frequencessssss, with decaying factor of (1ss] as the frequency increases, iee, as [s] + 0c.
Hence, the loss landscape of the frequency domain contains many local minima, and LTI system can rarely learn the high frequencies with the usual training. To verify this, we train an S4D model initialized by HiPPO learn the SCIFAR-10 task for 100 epochs. We measure the relative change of each parameter& 8: A( (l((0) - g( () where the superscripts indicate the epoch number. As we will show in section 6, the HiPPO initialization is unable to capture the high fre-
quencies in the CIFAR-10 pictures fully. From Table 1, however, we that Im (n(diag(A) is trained
5

---

Published as a conference paper at ICLR 2025
very little: every w; is only shifted by 1.43% average. This can be explained by Theorem 1: wy is easily trapped by a low-frreug low-frequency local minimum.
Table 1:
The average relative change of each LTI system matrix in an S4D model trained on the SCIFAR- 10 task. We see that the imaginary parts of diag( A) are almost unchanged during training.
Parameter
Re(diag( A))
Im diag A))
C
D
A
1002.705
0.0143
1.1801 0.8913
An Illustrative Example. Our analysis of the training dynamics of w in Theorem is very generic, relying on the notion of the functional derivatives. To make the theorem concrete, we consider a synthetic example (see Figure
3). We fall back to the case of approximating a target function
5
0.2
F = Re is (-1 50i) is (-1 + 50i)
+ 0.01 cos
11 1121,,22x
E R
4
using a trainable G, where 9/4 is chosen to guarantee the continuity of F. We set the number of states to be one, i.e., n 1. For illustration purposes, we fix -1 and ( 0; therefore, we have
5
(iss) = Re is
E IR,
wi)
where our only trainable parameters are w and ç. Our target function contains two modes and some small noises between 2m and 2T,, whereas is unimodal with trainable position and height (see Figure 3 (left)). We apply gradient flow on w and & with respect to the L²-loss in the Fourier domain, in which case the functional derivative (0/0G(is)) simply reduces the residual:
ac -
L= C= [= G(ss)llz..
@G
In Figure 3 (middle), show the training dynamics of (w(r), ((T)), initialized with different val- ues (w)) = 3). where т is the time index the gradient flow. We make two remarkable observations that corroborate our discussion of the frequency bias during training:
1. Depending on the initialization of (0), it has two options of moving left or right. Since we fix = 0. by Theorem I, mode (w, E) (-50, 5) or (50, 0.2) in the residual F - G impacts the gradient (0/0w)C inverse- ayvercepropoortionalle to the cube of the distance between the mode w and the current w()). Since |24.5 50/ /124.5 (-50)) observe that when w0) I 24.5, it tends to move leftward, and rightward otherwise.
0.2/5, we indeed
2. Although the magnitude of the noises in [-2 2n] is only 5% of the smaller mode at W = 50 and 0.2% of the larger mode at = 500 once w(T) of the trainable LTI system enters the noisy region, it gets stuck the two modes of (see Region II Figure 3). This corroborates our discussion that the training dynamics of is sensitive local information and it rarely learns the high frequencies when initialized in the low-frequency region.
local minimum and never converges to one
5 TUNING THE FREQUENCY BIAS OF AN SSM
In section 3 and 4, we analyze the frequency bias of an SSM initialized by HiPPO and trained by a gradient-based algorithm. While we now have a theoretical understanding of the frequency bias, from a practical perspective, we want to be able to tune it. In this section, we design two strategies enhance, reduce, counterbalance, or even reverse the bias of an SSM against the high frequencies. The two strategies are motivated by our discussion of the initialization (see section 3) and training (see section 4) of an SSM, respectively.
5.1 TUNING FREQUENCY BIAS BY SCALING THE INITIALIZATION
Since the initialization assigns SSM some inborn frequency bias, a natural way to tune the fre- quency bias to modify the initialization. Here, we introduce a hyperparameter a >> as a simple way to scale the HiPPO initialization defined in Corollary 1:
aj 005 + i(- 1)d Li/2|ou,
15ism.
(5)

---

Published as a conference paper at ICLR 2025
The L2 Loss Landscape & Trajectories of (w((r),((r)
The H² Loss Landscape & Trajectories of (we(r),(T))
The Target Function
6
Ground Truth
Rogion III
Region III
1500
function value
Trainable LTI
Region
Region
0.4
Region
0.3
1000
2 1 22
0.2
2
500
0.1
-50
50 -50 2 50
-50
50
W
Figure 3: We train an LTI system to learn noisy bimodal target transfer function. The convergence to a local minimum depends on the initial location of the pole. Left: the ground truth contains a large mode and a small mode, plus some small noises. We want to investigate which mode, if any, our trainable LTT system converges to. Middle: we train the LTT system with respect to the 1.2-loss. We show the trajectories of (w(T), {(T)) given different initializations (u(())(()))))) = 3). The two local minima corresponding to the two modes of F are shown in red crosses. The green trajectories (initialized in Region I) converge to the mode at = -5, the magenta trajectories (initialized in Region III) converge to the mode at w 50, and the black ones (initialized Region II) converge to neither. Right: the experiment is repeated with the 1o-loss (see section 5.2).
Compared to the original HiPPO initialization, we scale the imaginary parts of the eigenvalues of A by a factor of a. By making the modification, we lose the "polynomial projection" interpretation of HiPPO that was originally proposed as way of explaining the success of the HiPPO initialization; yet, as shown in Orvicto et al. (2023); Yu et al. (2024a), this mechanism is no longer regarded as the key for good initialization. By setting a < 1, the cigenvalues aj clustered around the origin, enhancing the bias against the high-frequency modes; conversely, choosing a >> allows us to capture more variations in the high-frequency domain, reducing the frequency bias.
So far, our discussion in the paper is from a perspective of the continuous- time LTI systems acting on continuous time-series. For SSMs, however, the inputs come in discrete sequence. Hence, we inevitably have to discretize our LTI systems. To study the scaling laws of a we assume in this work that an LTI system is discretized using the bilinear transform (Glover, 1984; Gu al., 2022b) with a sampling interval At > O. Other discretization choices can be similarly studied. Then, given an input sequence E TR& of length L, the output y can be computed by discretizing eq. (2):
2 exp (i2m(j -
(FF((FFT(U) G())),
8j
At exp (12m 1)/L)
with the same transfer function G in eq. (3). Our goal is to propose general guidelines for an upper bound of a. We leave most technical details Appendix D; but we intuitively explain why a cannot be arbitrarily large for discrete inputs, Given a fixed sampling interval At, there is an upper bound for the frequency, called the Nyquist frequency, above which signal cannot be "seen" by sampling, causing the so so-called aliasing errors (Oppenheim, 1999; Condon & Ransom, 2016; Trefethen, 2019). As straightforward example, one cannot distinguish between cos(5r (5t) and cos(t) from their samples at ( = kn, k E Z. Our next result tells us how to avoid aliasing by constraining the range of a.
Proposition 1. Let = v + iw be given and define G(is) = 1/(is a). Let = G(s) be the vector of length L, where s defined in eq. (6). Then, there exist constants C1, C2 such that
Cillgl1e 5 w At C2
Cillgl1 - 1+
S S |lg|loo.....
(1-((L))))
You may have noticed that in Proposition 1, we study the norm of the complex G instead of its real part restriction G. The reason is that in an LTI system parameterized by complex numbers, we mul- tiply G by a complex number ş + iÇ and then extract its real part. Hence, both Re(G) and Im (G) are important. By noting that maxigjsn wyy = no// / in our scaled initialization in eq. (5), Propo- sition gives two scaling laws of that prevent [g]]2 and ||g]|oo from vanishing. respectively. First, the 2-norm of g measures the average-case contribution of the partial fraction 1/(is - a) to the input-to-output mapping in eq. (6).
7

---

Published as a conference paper at ICLR 2025
Rule I: (Law of Non- Non-vanishing Average Information) For a fixed task, as and At vary, one should scale a = (11(()))) to preserve the LTI system's 's impact on an average input.
Next, the oc-norm of g tells us the maximum extent of the system's 's action any inputs. Therefore, |اg|ا> is too small, then i/ - a) can be dropped without seriously affecting the system at all.
Rule II: (Law of Nonzero Information) Regardless of the task, should never take a > 4 lan ((1-(1-1)/L)) x /nT
avoid a partial fraction that does not contribute to the evaluation of the model.
We reemphasize that our scaling laws provide upper bounds of a. Of course, one can always choose be much smaller to capture the low frequencies better.
5.2 TUNING FREQUENCY BIAS BY SOBOLEV FILTER
In section 5.1, we see that we can scale the HiPPO initialization to redefine the region in the Fourier domain that can be learned by an LTI system. Here, we introduce another way to tune the frequency bias: by applying Soboley-norm-base filter. The two strategies both tune the frequency bias, but by different means: scaling the initialization identifies new set of frequencies that can be learned by the SSM, whereas the filter in this section introduces weights to different frequencies. Our method is rooted in the Sobolev norm, which extends a general 1.2 norm. Imagine that we approximate a ground-truth transfer function F(is) using (is)) We can define the loss to be
FF F-GII := / (1 (1+1s))20 G(s) ds
for some hyperparameter 3 E IR. The scaling factor (1 + |s|) naturally reweighs the Fourier domain. Note that you may have seen other forms of this factor they all lead to the same norm up to norm- corn-equivalency. When 3 = 0, eq. (7) reduces to the standard L loss. The high frequencies become less important when < 0 and more important when 3 O. Unfortunately, as discussed in section there lacks notion of the "ground-truth" F for every single LTI system within an SSM, making eq. (7) uncomputable. To address this issue, instead of using a Sobolev loss function, we apply a babolev-norm-bet filter to the transfer function G to redefine the dynamical system:
((s) = G (P) (P) (is)ū(s),
( = (1 + |s|) Giss.
(8)
This equation can be discretized using the same formula in eq. (6) by replacing G with ((9).
Equation (8) can be alternatively viewed as applying the filter (1 + |s|)@ to the FFT of the input u, which clearly allows to reweigh the frequency components. Surprisingly, there is even more beyond this intuition: applying the filter allows us to modify the training dynamics of wy!
Theorem 2.
Let L(O) be a loss function and r (A, B, C, D) be diagonal LTI system in an SSM defined in section For any 3 E TR, we apply the filter in eq. (8) to I and let (()) be the new transfer function. Suppose the functional derivative of L(O) with respect to ( (is) exists and denoted by (0/aG()) (i.)) Then, (a/0G(()iis)
= O(ss(p) for some < 1 3, we have
OC = Dw; -0 (B)
dL
U, K. (M) (s)ds, kg (1)+(()))))) (5 w, )2- v3) -,,, (s u2
(9)
[0} (s w, 121
for every j S n. In particular, we have that (x) (1(, -2+8 + 3+00) as |s| + co.
Compared to Theorem the gradient of C with respect to wj now depends on the loss frequency by factor of (1ss)-2+0) Thus, the effect of our Sobolev-norm-basee filter is not only rescaling of the inputs in the frequency domain, but it also allows better learning the high frequencies:
The higher the is, the more sensitive wyy is to high-frequency losses. Hence, w, is no longer constrained by the "local-frequency" loss and will activately learn the high frequencies.
The decay constraint that ( (8) (is) = O[spp) ) for some p < 1- 3 is needed to guarantee the convergence of the integral in eq. (9). When it is violated, the theoretical statement breaks, but
8

---

Published as a conference paper at ICLR 2025
could still implement the filter in practice, which is similar to Yu et al. (2023). In Figure (right), we reproduce the illustrative example introduced in Section 4 using the Sobolev-norn norn--aso--- filter in eq. (8) with = 2. This is equivalent to training an ordinary LTI system with respect to the 2 function defined in eq. (7). We find that in this case, the trajectories of (ve(r),((T)) always converge to one of the two modes in F regardless of the initialization, with more of them converging to the high- -egufrrey global minimum on the left. This verifies our theory, because by setting 3 2, we amplify the contribution of the high-freog fighffreuency residuals in the computation of (0/0w)C C. pushing a y out of the noisy region between -2T and 2x. We leave more illustrative experiments to Appendix E, which show the effect of our tuning filter also when B << 0.
EXPERIMENTS AND DISCUSSIONS
(I) SSMs as Denoising Sequential Autoencoders. We now provide example of how our two mechanisms allow us to tune frequency bias. In this example, we train an SSM to denoise an image in the CelebA dataset (Liu et al., 2015). We flatten image into sequence of pixels in the row-major order and feed it into an S4D model. We collect the corresponding output sequence and reshape it into image. Similar to the setting of autoencoder, our objective is to learn the identity map. To make the task non-trivial, we remove the skip connection D from the LTI systems. During inference, we add two different types of noises to the input images: horizontal or vertical stripes (see Figure 4). While the two types of noises may be visually similar to each other, since we flatten the images using the row-major order, the horizontal stripes turn into low-frequency noises while the vertical stripes become high-frequency ones (see Figure 9). In Figure 4, we show the outputs of the models trained with different values of a and /3 as defined in section 5.1 and 5.2, respectively.
Inputs
Outputs of Four Models
a a= 0.1, -1
a=1,3=0
a= a=10, 10, 3=1
a 100, 3=
True Inagge
Low Feg. Nuise
High Freq. Noise
High-Frequency-Per -ass Filter
Figure 4: The outputs of image-denoising S4D models trained with different configurations.
From Figure 4, we can see that as a and [8 increase, our model learns the high frequencies in the input better; consequently, the high frghurreuency noises get preserved in the outputs and the low low-frequency noises are dampened. This corroborates our intuitions from section 5.1 and 5.2. We can further quantify the "pass rates" of the low and high-frequency noises. That is, we compute the percentage of the low and high fighffreueency noises that are preserved in the output. We show in Table 2 the ratio between the low-pass rate and the high-pass rate, which decreases as a and / increase.
(II) Tuning Frequency Bias in the Long-Range Arena. The two tuning strategies section 5.1 and 5.2 are not only good when one needs to deal with a particular high or low frequency, but they also improve the performance of SSM general rangerange tasks. In Table 3, we show that equipped with the two tuning strategies, our SSM achieves state-of otate-of-the-ar performance on the Long-Range Arena (LRA) tasks (Tay al., 2021).

---

Published as a conference paper at ICLR 2025
Table 2: We compute the percentage of the low and high-frequency noises that are preserved the
output of model trained with pair of configurations (a, .9). The table shows the ratio between the low-frequency pass rate and the high-frequency pass rate. The more bluish a cell is, the better our model learns the high frequencies. Circled in red is the S4D default.
10
00.5
0.0
1.0
10*
4,463e-07
2.409++06
1.198--05
4.613e+03
1.738e
10²
4.912e 05
2.124e
1.758e
9.595e
5.730e +01
10
9.654e-04
7.465e+03
6,073e-02
5.699e+01
6.394e+00
10
100
3.243e-00
3.745e-02
e.801e-03
7.299
5.963e-06
10
Table Test accuracies in the Long-Range Arena different variants of SSMs. The bold (resp. underlined) numbers indicate the best (resp. second best) performance on a task. An entry is left blank if no result is found. The row labeled "Ours" stands for the S4D model equipped with our two tuning strategies. Experiments were run with 5 random seeds and the medians and the standard deviations are reported. The S4 and S4D results are from the original papers (Gu et al., 2022b;a). The sizes of our models are the same smaller than the corresponding S4D models.
Model
ListOps
Text
Retrieval
Image
Pathfinder Path-X
Avg.
DSS (Cuutta et al,, 2022)
57.60
76.60
87.60
85,80
84.10
85.00
79.45
S4++ (Qi al., 2024)
57.30
86.28
84.82
82,91
80.24
Reg. S4D (Liu & Li, 2024a)
61.48
88.
91.25
88.12
94.93
95.63
86.60
Spectral SSM (Agarwal et al., 2023)
60.33
89.60
90.00
95.60
90.10
Liquid (Hasuni et al., 2023)
62.75
89.02
91.20
89,50
94.80
96.66
87.32
55 (Smith et al., 2023)
62.15
89.31
91.40
88,00
95.33
98.58
87.46
S4 (Gu al., 2022b)
59.60
86.82
90.90
94.20
96.35
86.09
S4D (Gu al., 2022a)
60.47
86.18
80.46
88.19
93.06
91.95
84.89
Ours
62.75
89.76
92.45
90.89
95.89
97.84
88.26
78
10.16
10.33
10.13
20.91
(III) Ablation Studies. We perform ablation studies of our two tuning strategies by training smaller S4D model to learn the grayscale SCIFAR-10 10 task. From Figure 5, we obtain better per- formance when we slightly increase a or decrease 33 might feel like a contradiction because increasing helps to learn the high frequencies while decreasing 3 downplays their role. It is not: and 3 control two different notions of the bias: scaling a affects "which frequencies we can learn;" scaling 3 affects "how much want to learn certain frequency." They can be used collaboratively and interactively to attain the optimal extent of frequency bias that we need for a problem.
Accuracy
822
S40 Deianit
Accuracy 80
02
Optmat
Optaal
60
Little hightfre. info. cundbe learsed
L Limited deg. of freedomn in the Jow freq, domban
High freqs, get High Eaegs, are litile attention) weighied too much
Figure 5: Two ablation studies the tuning strategics proposed this paper. We train an S4D model with varying parameters of and 8, respectively. On the left, we see that holding B 0 (the default value), the model achieves its best performance when a = 4; on the right, when we fix (the default value), the model performs the best when 3 = -0.5.
(IV) More Experiments. One can find more supplementary experiments in Appendix I on wave prediction (see Figure 1) and video generation.
7 CONCLUSION
We formulated the frequency bias of SSM and showed its existence by analyzing both the initial- ization and training. We proposed two different tuning mechanisms based on scaling the initializa- tion and applying Sobolev-norm- based filter to the transfer function. As future direction, one could develop ways to analyze the spectral information of the inputs a problem and use to guide the selection of the hyperparameters in tuning mechanisms.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like acknowledge the Laboratory Directed Research and Development Program of Lawrence Berkeley National Laboratory under U.S. Department of Energy Contract No. DE-AC02- 05CH11231. In addition, SHL would like to acknowledge support from the Wallenberg Initiative on Networks and Quantum Information (WINQ) and the Swedish Research Council (VR/202) 01X/2021-0360 -03648). We would also like thank the anonymous reviewers for their comments that helped improve this paper.
REFERENCES
Vadim Movsesovich Adamyan, Damir Zyamovich Arov, and Mark Grigor evich Krein. Analytic properties of schmidt pairs for a hankel operator and the generalized schur-takagi problem. Matematicheskii Shormik, 128(1)334-75, 1971.
Naman Agarwal, Daniel Suo, Xinyi Chen, and Elad Hazan. Spectral state space models. arXiv preprint arXiv:23/2.06837 2023.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference on Machine Learning, pp. 1120-1128, PMLR, 2016.
S. Arora, S. S. Du, W. Hu, Z. Li, and R. Wang. Fine-grained analysis of optimization and generaliza- tion for overparameterized two-layer neural networks. In Inter. Conf. Mach. Learn., pp. 322-332. PMLR, 2019.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint 18330118, 2018.
R. Basri, D. Jacobs, Y. Kasten, and S. Kritchman. The convergence rate of neural networks for learned functions of different frequencies. Adv. Neur. Info. Proc. Syst., 32, 2019.
R. Basri, M. Galun, A. Geifmann D. Jacobs, Y. Kasten, and Kritchman. Frequency bias in neural networks for input of non-uniform density. In Inter. Conf. Mach. Learn. pp. 685-694. PMLR, 2020.
A. Bietti and J. Mairal, On the inductive bias of neural tangent kernels. Adv. Neur. Info. Proc. Syst., 32, 2019.
Yuan Cao, Zhiying Fang, Yue Wu, Ding- Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. arXiv preprint 1r1.v1191..01198, 2019.. 2019.
Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi. Antisymmetricrnn: A dynamical system view on recurrent neural networks. In International Conference on Machine Learning, 2019.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference Machine Leurning, 2020.
James J Condon and Scott M Ransom. Essential radio astronomy, volume 2. Princeton University Press, 2016.
W. M. Czarnecki, S. Osindero, M. Jaderberg, G. Swirszez, and R. Pascanu, Sobolev training for neural networks. Adv. Neur. Info. Proc. Syst., 30, 2017.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.2/06/ 2024.
Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE signal processing magazine,
2012.
N Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Ma- honey. Lipschitz recurrent neural networks. In International Conference on Learning Represen- fations, 2021.
11

---

Published as a conference paper at ICLR 2025
Izrail Moiseevitch Gelfand, Richard A Silverman, et al. Calculus of Variations. Courier Corporation, 2000.
Keith Glover. All optimal hankel-norm approximations of linear multivariable systems and their L,co-error bounds. International journal of control, 3966):115--119; 1984.
Walter Greiner and Joachim Reinhardt. Field quantization. Springer Science & Business Media, 2013.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.00072, 2023.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré, s4. httpss://gtttb..comm///////////t http:///sstttttssspessssssss 2021a.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Com- bining recurrent, convolutional, and continuous-time models with linear state space layers. Ad- vances in neural information processing systems, 34:572-585, 2021b.
Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances Neural Information Processing Systems, 35:35971- 35983, 2022a.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022b.
Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. How to train your hippo:
State space models with generalized orthogonal basis projections. International Conference Learning Representations, 2023.
Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 35:22982-2299 2022.
Moritz Hardt, Tengyu Ma, and Benjamin Recht. Gradient descent learns linear dynamical systems. Journal of Machine Learning Research, 19(29):1-44, 2018.
Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. International Conference on Learning Repre- sentations, 2023.
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Adv. Neur. Info. Proc. Syst., 31, 2018
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5165. PMLR, 2020.
Nikita Kitaev, tukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference Machine Learning, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selec- tion. Annals of statistics, pp. 1302-1338. 2000.
Soon Hoe Lim. Understanding recurrent neural networks using nonequilibrium response theory. Journal of Machine Learning Research, 22(47):1-48, 2021.
Fusheng Liu and Qianxiao Li. From generalization analysis to optimization designs for state space models. arXiv preprint arXiv:2405.02670 2024a.
12

---

Published as a conference paper at ICLR 2025
Fusheng Liu and Qianxiao Li. The role of state matrix initialization in ssms: A perspective the approximation estimation cnttimattionnsstttttinnnn tradeoff. ICML 2024 NGSM Workshop, 2024b.
Xinliang Liu, Bo Xu, Shuhao Cao, and Lei Zhang. Mitigating spectral bias for the multiscale
operator learning. Journal of Computational Physics, 506: 12944, 2024. ISSN 0021-9991.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
M. W. Mahoney. Approximate computation and implicit regularization for very large-scale data analysis. In Proceedings of the 31st ACM Symposium on Principles of Database Systems, pp- 143-154, 2012.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023.
Alan Oppenheim. Discrete-ting time signal processing. Pearson Education India, 1999.
Antonio Orvieto, Samuel Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pas- canu, and Soham De. Resurrecting recurrent neural networks for long sequences. arXivv preprint arXiv:2303.06344 2023.
Antonio Orvieto, Soham De, Caglar Gulcehre, Razvan Pascanu, and Samuel L Smith. Universality of linear recurrences followed by non-linear projections: Finite-width guarantees and benefits of complex eigenvalues. In Forty-firsi International Conference Machine Learning, 2024.
Robert G. Parr and Weitao Yang. Density- Density-Functionas Functional Theory of Atoms and Molecules. International
Series of Monographs on Chemistry. Oxford University Press, 1994. ISBN 9780195357738
Biqing Qi, Junqi Gao, Dong Li, Kaiyan Zhang, Jianxing Liu, Ligang Wu, and Bowen Zhou. S4++: Elevating long sequence modeling with state memory reply. 2024.
N. Rahaman, A. Baratin, D. Arpit, Draxler, M. Lin, F. Hamprecht, Bengio, and A. Courville. On the spectral bias of neural networks. In Inter. Conf. Mach. Learn., pp. 5301-5310. PMLR, 2019.
David W Romero, Anna Kuzina, Erik J Bekkers, Jakub M Tomezak, and Mark Hoogendoorn. Ck- conv: Continuous kemel convolution for sequential data. In International Conference on Machine Learning. 2022.
T Konstantin Rusch and Siddhartha Mishra. Unicornn: A recurrent model for learning very long time dependencies. International Conference on Machine Learning, pp. 9168-9178. PMLR, 2021.
Jakub Smékal, Jimmy TH Smith, Michael Kleinman, Dan Biderman, and Scott W Linderman. To- wards a theory of learning dynamics in deep state space models. arXiv preprint arXiv:2407.07279 2024.
Jimmy Smith, Shalini De Mello, Jan Kautz, Scott Linderman, and Wonmin Byeon. Convolutional state space models for long-range spatiotemporal modeling. Advances in Neural Information Processing Systems, 36, 2024.
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for se- quence modeling. In The Eleventh International Conference Learning Representations, 2023.
H. Son. J.W. Jang, WJJ Han, and H.J. Hwang. Sobolev training for the neural network solutions of PDEs. arXiv preprint arXiv:2101 08932, 2021.
Hwijae Son. Sobolev acceleration for neural networks. 2023.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video rep- resentations using Istms. In International conference on machine learning, pp. 843-852, PMLR, 2015.
13

---

Published as a conference paper at ICLR 2025
L. Su and P. Yang. On learning over werrparameeterize neural networks: A functional approximation perspective. In H. Wallach, H. Larochelle, A. Beygekzimer, d'Alché-Buu Buc, E. Fox, and R. Gamett (eds.), Adv. Neur. Info. Proc. Syst., volume 32. Curran Associates, Inc., 2019.
Dennis Sun. Introduction probability.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. International Conference Learning Representations, 2021.
Lloyd N Trefethen. Approximation theory and approximation practice, extended edition. SIAM, 2019.
C. Tsay. Sobolev trained neural network surrogate models for optimization. Comp. Chem. Eng., 153:107419, 2021.
N.N. Vlassis and W. Sun. Sobolev training of thermodynamic-infomeed neural networks for inter- pretable elasto-plasticity models with level set hardening. Compu. Meth. Appl. Mech. Eng., 377: 113695, 2021.
Aaron Voelker, Ivana Kajié, and Chris Eliasmith. Legendre memory units: Continuous-timg repre- sentation in recurret neural networks. Advances in neural information processing systems, 32, 2019.
Shida Wang and Qianxiao Li. StableSSM: Alleviating the curse of memory in state-space models through stable reparameterization. arXiv preprint UrXiv:2311.144933 2023.
Shida Wang and Beichen Xue. State-space models with layer-wise nonlinearity are universal ap- proximators with exponential decaying memory. Advances in Neural Information Processing Systems, 36, 2024.
Z. J. Xu. Frequency principle: Fourier analysis sheds light deep neural networks. Commun. Comput. Phys., 1866617661666 2020.
G. Yang and H. Salman. A fine-grained spectral perspective on neural networks. arXiv preprint 1rXiv:1907.10599 2019.
Cagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. Continuous-time model-based reinforce- ment learning. In International Conference on Machine Learning, pp. 12009-12018, PMLR, 2021.
Annan Yu and Alex Townsend. Leveraging the hankel norm approximation and data-driven algo- rithms in reduced order modeling. Numerical Linear Algebra with Applications, pp. e2555, 2024.
Annan Yu, Yunan Yang, and Alex Townsend. Tuning frequency bias in neural network training with nonuniform data. International Conference on Learning Representations, 2023.
Annan Yu, Michael W Mahoney, and N Benjamin Frichson. There is hope to avoid hippos for long-memory state space models. arXiv preprint 14X1::2405..3955 2024a.
Annan Yu, Arnur Nigmetov, Dmitriy Morozov, Michael W. Mahoney, and N. Benjamin Erichson. Robustifying state-space models for long sequences via approximate diagonalization. In The Twelfth International Conference on Learning Representations, 2024b.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning. pp. 27268-221 27266-27286. PMLR, 2022.
B. Zhu, J. Hu, Y. Lou, and Y. Yang. Implicit regularization effects of the Sobolev norms in image processing. arXiv preprint arXiv:2 2021.
14

---

Published as a conference paper at ICLR 2025
A
MORE ON THE DEFINITION OF FREQUENCY BIAS
In this section, we provide an additional figure that explains the frequency bias of an SSM.
cos(kyt)
cos(k2t)
cos(k3f)
cos(k,t)
Inputs
time domain
Faurier Transform
Transfer Function
kg kg
kg
Inverse Fourier Transform
frequancy Komain
G(ik, ())
G(ikz)cos(k22)
Gliks)cos(ks))
G(iks)oos(k4)
Outputs
Figure 6: If the transfer function has a large total variation in the low-frequency region, then given two different low-frequency input signals, the LTT system sets very different pass rates for them. Conversely, when the transfer function has small total variation in the high- geg-ffreueeccy region, then given two different high-frequency input signals, the LTI system must set similar pass rates. The Fourier transform of cosine wave involves both positive and negative frequency. We drop the negative frequency component for the cleanliness of the figure.
We also provide additional justifications for definition of the frequency bias in section 2. As noted in Yu et al. (2024a), one reason why an LTT system in an SSM degenerates is that its transfer function is too flat. That is, if G can be well-approximmtee by much lower-degree rational function, i.e.,
inf sup G G(ss)
GER SER
small for some d & n, where Rd the space of rationals of degree S d, then the large LTI system can be well-approximated by much smaller one (i.e., given the same input the outputs of the two systems are close to each other). other words, our LTT system is not as expressive as its size may have suggested. If we now restrict attention to only part of the frequency domain, then the same reasoning applics: if the transfer function G flat on [a, 6]. then that means
inf sup |G(is) G(s))
GER* sEja.
is small for some d & n. Therefore, there exists much smaller LTI system I and its actions on the Fourier modes in [a, are very similar those of the original system. Hence, this essentially means that our LTI system is unable to capture "complex patterns" in the frequency interval [a, because all does in [a, b) can also be done by a much smaller system.
B PROOFS
In this section, we provide the proofs of all theoretical statements in the manuscript.
15

---

Published as a conference paper at ICLR 2025
First, we prove the statements about the initialization of the LTI systems. The total variation can be bounded straightforwardly using the decay of the transfer functions.
Proof of Lemma 1. Since G is the real part of G, its total variation is always no larger than the total variation of G. Hence, have
V 2 (G) VV -
Cj
iB ag
B + w
The other bound is similarly obtained.
Proof of Corollary
By Lemma and the Hölder's 's inequality, we have
lc
V= (G)
<
S Wj + B
[ 12//w//
where Wj - 1/(w B). Since Ei -NO 1) i..dd, we have that | C 1 follows the x²- distribution with degree 2n. By Laurent & Massart (2000), we have with probability at least 1 - that
B C
< 2n + 1n + + (1/6) s 2(v (lm(16))
By the definition of B, we have that
||
(B - / 2)2
The result for VGB follows from the last two inequalities. The bound on (G) can be derived similarly.
The initialization Corollary is called the HiPPO-Lin initialization. Using the same idea, we can immediately prove a result for the HiPPO-LegS initialization.
Corollary 2. Assume that the diagonal entries ay are initialized using HiPPO-LegS (Gu et al., 2022a) and Ej: Cy ~ NO, 1) ii.d. where NO0 1) is the standard normal distribution. Then, given B > and 5 > 0, we have
V2n(yn + (n(0)))
V V (G), (G)))
with probability 2 1-6 5.
Proof. The proof is done by noting that every aj is real-valued and mimicing the proof of Corol- lary
We skip the proof of Proposition and defer it to Appendix D when present detailed derivation of the scaling laws. We next prove the statement about the training dynamics the imaginary parts of di.
Proof of Theorem 1. Fix some J S n, we first view the transfer function G wy) as a function of two variables. We compute the derivative of G(s, wy) with respect to wy:
aG(s, Ow
(s - Enug
+D= a
( w,) Eg4g
Ow,
Ow,
0² + - w,
(0) (s w) (0 (0/0w, (S; wa) Eg1z)
(0)) + )2
(sj(s w)) Eju,) (0/0wr (03 3 + (s 10,,)² w2
(0) + (s w 2))
-03 + (s w. ),)
2(Cr(s - w.)) - [, ww)
(0g + - (w)) 22)2
(n) + (8 w2)
G( G(-03 (s w.y 2 2(s w;)2) E, wz)) K,(s).
(0) + - w, 2))2
16

---

Published as a conference paper at ICLR 2025
Since assume that (00 (Is)(") for some p 1, the integral in eq. (4) converges. By Parr& & Yang (1994, Appendix A), eq. (4) holds.
The statement about the training dynamics of wy given Sobolev filter follows immediately from Theorem 1.
Proof of Theorem 2. Since we assume that (O//OG(is) ([s)pp) for some p 1 3, the
integral in eq. (9) converges. The result follows by noting that
O (8,w3) (1+ 1s|) (G(s, wj) (1+ 11) Ks(s) KK.0)
dw,
dw,
and applying the equation in Parr Yang (1994, Appendix A).
C FUNCTIONAL DERIVATIVES
In this section, we briefly introduce the notion of functional derivatives (see Appendix A. in (Lim, 2021) for a more technical overview). To make our discussion concrete, we do it the context of Theorem 1. Consider the transfer function Giis) defined in eq. (3). It depends parameters Vy, Wy, &y. and GJ. In this section, we separate out a single Wj for fixed S j S n, leaving the remaining parameters unchanged. Then, for every w I, we can define (s) to be the transfer function (iss) when Wj = w. Under this setting, the set of all possible transfer functions indexed by w, i.e., F {f(")/w R} is subset of a Banach space, say (R). To avoid potential confusions, we shall remark that f(") is not linear in its index w, i..,, f(an)tw))) * f(()) + (wx2 in general, neither is F a subspace of (R). This does not impact our following discussion.
the model
Now, consider the loss function C. Given a choice of wy and corresponding transfer function f((r), the loss function maps f(() to real number that corresponds the current loss. Hence, L can be viewed as (not necessarily linear) functional of f((), We would like to ask: how does L respond to small change of f(() (s) at some E R? Ideally, this can be measured as
((S(W) + cd.) f(w)
lim e-0
(10)
where 6, is the Dirac delta function at s. However, the loss function C is not defined for distri- butions, making eq. (10) not directly well-defined. To fix this issue, we have to go through the functional derivatives. The idea, as usual in functional analysis, is to pass the difficulty of handling a distribution to smooth functions that approximate it. If there exists a function (0/0((w) ((w) such that the equation
dc (f(w) (s)o(s) ds = lim e00
L(((w) + - -
holds for all smooth O functions & that are infinitely differentiable and vanish at infinity, then (()af()))CC is defined to be the functional derivative of C at f(w), Taking { to be an approx- imate identity centered at s, we recover eq. (10) using (0/af(")) ((s).
One nice thing about the functional derivatives is that they allow to write down continuous analog of the chain rule, which is the meat of Theorem 1. To get some intuition, let us first consider a function [(w) [([i(w)..... ... (w)) that depends on w via intermediate variables f1,..., .., fk- Assuming sufficient smoothness conditions, the derivative of E with respect to wo can be calculated using the standard chain rule:
8/
ac
ac 0f, 0f,
dw
Σ i=1
ac
ac
(II)
dw
05
dw
87
8f.
85x Zw
17

---

Published as a conference paper at ICLR 2025
The only difference in the case of C is that instead of depending & discrete intermediate variables fi,....Ja Jk. our L. depends on continuous family of intermediate variables (s) indexed by s TR. In this case, one would naturally expect that in eq. (I1), the sum becomes an integral, or equivalently, the and the column vectors become the and the column functions. This is indeed the case given our functional derivative:
ac
Oc 0f(w)
a af(w)(s
(s)
ds.
dw
Ow
This formula can be found in Parr & Yang (1994, (A.24)) and Greiner & Reinhardt (2013, sect. 2.3).
D SCALING LAWS OF THE INITIALIZATION
In this section, we expand our discussions in section 5.1 and give the proof of Proposition 1. Throughout this section, we assume that we use the bilinear transform to discretize our continuous- time LTI system. The length of our sequence is L and the sampling interval is At. The bilinear transform is essentially a Mobius transform between the closed left half-plane of the s-domain and the closed unit disk in the z-domain. Hence, it gives us two ways to study this filter - by either transplanting the transfer function G onto the unit circle and analyzing in the discrete domain or by transplanting the FFT nodes from the z-domain to the imaginary axis the s-domain. The two ways are equivalent, but we choose the second method for simplicity.
The output of an LTI system can be computed by
y = iFFT(FFT(()) o G(w)),
where is the transfer function of the discrete system and where
w = [exp (2mi%)
exp (2
is the length-L vector consisting of Lth roots of unity. We do not have direct access to G, but do know G, its continuous analog, in the partial fractions format. They related by the following equation:
2
G(2) = (s),
At +
In that case, the vector ((w) can be equivalently written
G(wy)= = G exp (2T + -11 = G ((112 tan
2
π j
At exp (2 (
This is how we obtained eq. (6). The locations of the new samplers on the imaginary axis are shown in Figure 7, with L = 101 and At 0.01.
Note that the right figure (the s-domain) is on a logarithmic scale and only the upper half-plane is shown due to the scale. We also choose L odd; when L is even, pole is placed "at infinity" in the s-domain, at which any partial fraction vanishes. Why do we go through all the pains to study this bilinear transformation? The reason is that it gives instance, for L 101 and At = 0.01, if a pole has a much larger imaginary part than 10a, then the discrete sequence will hardly see the effect of this partial fraction even though the underlying continuous system will. This corresponds to the intuition behind the aliasing error that we discussed in the main text.
guideline for scaling the poles. For
As in Proposition 1, it suffices to study single partial fraction instead of all. Hence, instead of studying the entire transfer function together, we focus on one component of it:
G(is)
Re(s)
0, Re(a) < 0.
is a'
This is partial fraction in the s-domain. For fixed L and At, this partial fraction corresponds to bounded linear operator G ([L)) + e ([L)) that maps an input sequence to an output sequence, where [7.] (1...., T.). is the set of the first natural numbers. We consider the norm of this
18

---

Published as a conference paper at ICLR 2025
1.5
0.5
03
10
-1.5
100
Figure 7: The poles of the FFT samplers in the z-domain and the samplers in the s-domain for L = 101 and AL 0.01. Due to the essential difficulty of plotting the entire real line logarithmic scale, we only show the semilog plot of the upper half-plane of the s-domain (right), Hence, on the right figure, we omit the samplers corresponding to the lower are of the unit circle on the left.
operator, where will find that [m((s] + oc, the norm of the operator vanishes. The rate of vanishing will guide us in selecting an appropriate range for the pole. So, how can we tell the norm of this operator? By definition, the norm of & is defined by
| = sup
sup ا
|
||u||
where y is the output of the operator G given input i... y Gu, and the second step follows from the Parseval's identity. By Hölder's inequality, we further have
||
sup | G(w)lle S ||g||>>, ||
where G(w) = g is the sample vector of the bilinearly transformed transfer function of in the z-domain, i.c.,
π j 2
g = Gww),
G(w); -(w,) G
tan
At
tan (Tj/L)
Hence, we have
131²=
Re(a)² + (Im(a) 2 tan (rJ/L)/A1}²
When Im(a) > 2 tan (mj/L) / AAt for all j. l is maximized when j |L/2| 1, in which case we have
| Re(a)² +
(12)
(Im(a) (Im()) This gives us the second rule (Law of Zero Information) when scaling the diagonal of A. This is worst-case analysis, where we essentially assume that the Fourier coefficient of is one-hot the highest frequency. In practice, course, this assumption is bit unrealistic; fact, the Fourier coefficients usually decay the frequency gets higher. Therefore, we should derive another rule for the average-case scenario. We consider the operator 9 that maps the Fourier coefficients of the good average-case estimate, because arg max = Gu 1/ule
(2/ tan (L-))((1(L11))))
inputs those of the outputs, then the norm 1191ee=-02
is necessarily at vertex of the simplex defined by | < ! That is, ú; +1 for S S L. Now, using the Hölder's inequality again, we have
| 20
sup ||
| gelee S |ggle
Note that instead of sup because the domain {|bille> 1} is compact.
19

---

Published as a conference paper at ICLR 2025
Hence, instead of studying the -orm of g, consider the e for the average- case estimate. The precise computation the e -norm can hard, but let us write out the full expression:
llglß E
19
Re(a)² + (Im(a) - (t) tan (xj/L))
Given the imaginary part of O, grab all Fourier nodes the ju axis that are below 1m and lower-bound them; we also grab all above Im(a)/2 2 and assume that they collapse to Im(a). This gives an estimate of the let norm:
(Jl(2/ [t) tan (aj/L) < lm(a) (2)
+ ((](//At) tan (mj/L) lm(a)/23]
Re(a)² lm(4)3/4
Re(a)²
L (2arccan(Imm(44/// +1 L(1 2rrclan((m(o)A4/44455
Re(a)²
(13)
(1arcta((((())4444//))) Re(a)² (m /4)
(1- aarcaaaa
Re(a)²
N2
Proof of Proposition 1. Given eq. (12) and (13). Proposition follows immediately.
Let us take a closer look at this expression. Ideally, the e-norm of g should be independent of do not want ||g||// /L to diminish. First, note that
and At; that is, L - do and At - Ni and N2 are independent of L as L - xc. As AL - 0, Ni inevitably vanish, regardless of the location of Im(a). In order to maintain N2 a constant, we would need Im(a) to not blow up. This gives us the first rule (Law of Zero Information) for scaling the poles. We can further work out some constants in O be used in practice. For example, guarantee that Im(a) is smaller than the top 5% Fourier nodes, we would need that
2 arctan
Im (m(a)
50.82
0.95 = Im(a) <
At
In particular, eq. (12) and eq. (13) together give us the proof of the lower bounds in Proposition The upper bounds proved by noting all all derivations in this section asymptotically tight.
E MORE NUMERICAL EXPERIMENTS ON THE ILLUSTRATIVE EXAMPLE
In section 4 and 5.2, we see that using Sobolev-norm based filter with 3 O, one is able to escape from the local minima caused by small local noises. In this section, we present similar set of experiments to show the effect of our filter, even when / < O. We choose our new objective function be
5
0.2
Fiis) = Re (-1 75i)
R.
(-1 251)
Compared to the objective in section we see two differences. First, we remove the sinusoidal noises around the origin. Second, we shift the locations of the two modes in the target: instead of locating at 50 and s = 50, we shift them to have large high-frequeee frehufrreeenyy mode and small low www-freuenccy mode in the ground truth.
and s = 25, respectively. This allows us
We show the results in Figure when we train an LTI system using a Sobolev-norm--aase filter with different values of 3. Note that when 8 = 0, the picture only differs from Figure (middle) in the frequency labels because that case, the gradient (0/Ow) L only cares about the relative difference but not the absolute values of s. From Figure 8, see that as 3 increases, more trajectories converge to the local minimum near (g = 5, w -75). The reason is that a larger 3 favors a higher frequency (see Theorem 2).
20

---

Published as a conference paper at ICLR 2025
-
0
10
6
HTD (i.c., 1.²) Loss
4
2000
Loss
1500
2
1000
600
-75
-50
-25
25
-75
-50
-25 W
0
25
-75
-50
-25
25
W
W
Figure 8: An SSM is trained with a filter based on the Sobolev-norm (see section 5.2). The plots read in the same way those in Figure 1. The transfer function converges to of the two local minima. As 3 ranges from -2 to 2, the transfer function becomes more sensitive to the high- frequency global information rather than the local information. Hence, the edge between the two different convergences shifts rightward.
F IMPLEMENTATION OF THE SOBOLEV-NORN-BASE FILTER
There are two popular ways to evaluate an LTI system: via a convolutional kernll (Gu et al., 2022b) via parallel scans (Smith et al., 2023). The first approach is directly based the frequency domain computation in eq. (8); thus, the implementation is trivial. The parallel sean algorithm does operate in the frequency domain, making the computation of eq. (8) less trivial. However, by noting that
((s) G ( (is)ú(s) G(is) [(1 [s]) (
one imagine that apply the standard parallel scan algorithm the transformed inputs [11 is) ()))) Hence, all we need to do is to preprocess the inputs by applying the filter
(1 + | the Fourier domain and then use the standard parallel sean algorithm.
G CHOICE THE SCALING FACTOR
In this section, briefly provide practical guideline for how a be chosen. We will advocate two ways to select an Omax as an upper bound of a. Then, given Omax one needs to tune hyper- parameter (0, Omax) that gives satisfactory amount of frequency bias. We do not recommend tuning a together with other hyperparameters. Instead, we suggest starting with chosen, one tune using the bisection method,
O'max and
once all the of the hyperparameters which only requires Ollog Omax) many trials. The reason why the bisection method works is that we believe in a unique local minimum of the loss as a changes (see Figure 5).
To select an Amax, we can either be informed by the input data or not. If we choose not to study the input data, then we propose to guarantee that Im(a) is smaller than the top 10% Fourier nodes which we sample the transfer function G (see Appendix D). That is,
2 π
[Im(a)At)
25.26
50.52 anAt
arctan
< 0.9 Im(a) <
At Omaax
The other way is to select Omax based on the input data. That is, one can plot the densities of the FFTs of all input training data and identify edge smax for which [-8maxs Smax] contains most densities. Since the Fourier domain one-dimensional, one can identify 8max by simply eyeballing. Then, we can select amax so that the information in -Smax Smax] can be learned. That is,
Im(a) E
8max
LA
Omax
TnLAt
8max
where I. is the length of the sequence.
21

---

Published as a conference paper at ICLR 2025
H
DETAILS OF THE EXPERIMENTS
H. DENOISING SEQUENTIAL AUTOENCODER
For every image in the CelebA dataset, reshaped have resolution of 1024 256 pixels to allow for higher-frequeng noises. We trained a single-layer S4D model with n = 128 and d_model 3. We dropped the skip connection D from the model. The model was trained using the MSE loss. That is, for every predicted sequence of pixels, we compared the model against the true image and computed the 2-norm of the difference vector.
Horizontal Stripes
Low- eww-Frequency Noises
Flattening
Vertical Stripes
High-Frequency Noises
Image
Sequence of Pixels
Figure 9: When a noisy image is flattened into pixels using the row-major order, the noises induced by horizontal stripes become low-frequency noises while those induced by vertical stripes become high frequencies.
We trained the model on the original images, i.c., those without any noises. When we inferred from the model, we added noises to the inputs, introducing about 10 cycles of horizontal vertical stripes, respectively. Our noises were large, almost shielding the underlying images. When the value of pixel out of range, then we ignored such issue during training; clipped its value to the appropriate range when rendering the image in Figure
To obtain the numbers in Table 2, computed with trained models, where set the inputs to be pure horizontal or vertical noises with no underlying images. Then, we evaluated the size of the output image and took the ratio of the outputs a particular noise. Table 2 shows the ratio between the pass rate of the low-frequency noises over the fiehufire ones. Our model did not have a nonlinear activation function, which made the model linear. Hence, it does not matter what the magnitude of the inputs was.
the inputs. We call this value the "pass rate" of
H.2 LONG-RANGE ARENA
In this section, we present the hyperparameters of our models trained on the Long-Range Arena tasks. Our model architecture and hyperparameters are almost identical those of the S4D models reported in Gu et al. (2022a), with only two exceptions: for the ListOps experiment, we set instead of d_model = 128 to reduce the computational burden. We do not report the dropout rates since they set to be the same those in Gu et al. (2022a). Also, made / trainable parameter.
64, which aligns with Smith al. (2023) instead; for the PathX experiment, we set
I SUPPLEMENTARY EXPERIMENTS
1.1 PREDICT THE MAGNITUDES OF WAVES
In Figure 1, we see an example of the frequency bias of SSMs, where the model better at extract- ing the wave information of low-frequency produce more examples on the same task to show that one is able to tune frequency bias by playing with a and 3 we introduced in section 5.1 and 5.2, respectively.
than a high-free lighffreuencyy one. In this section,
22

---

Published as a conference paper at ICLR 2025
Task
Depth
#Fcatures
Norm
Prenorm
LR
BS
Epochs
WD
A Range
ListOps
8
256
BN
False
0.002
50
80
0.05
(le-3,1e0)
Text
256
BN
True
0.01
32
300
0.05
(Ie-3,1e-1)
Retrieval
6
128
BN
True
0.004
64
40
0.03
(Ic-3,1e-1)
Image
512
LN
False
0.01
1000
0.01
(Ie-3,1e-1)
Pathfinder
6
256
BN
True
3
0.004
64
300
0.03
(Ie-3,le-1)
Path-X -X
128
BN
True
0.001
0.03
(Ic-4.le-1)
Table 4: Contigurations of our S4D model, where LR, BS, and WD stand for learning rate, batch
size, and weight decay, respectively. The hyperparameter a is the scaling factor introduced in sec- tion 5.1. We set the parameter in section 5.2 as trainable parameter to reduce the need for hyper- parameter tuning.
Frequency bias Enhanced
Default
Counterbalanced
Reversed
Mean Error
10
10
10
101
10
109
10
10
5 250
50
100
50
100
50
100 0
50
100
Epochs
Epochs
Epochs
Epochs
Figure 10: Reproduction of the experiment shown in Figure 1, with difference choices of (ax, 3)- From the left to right, our choices of (a, are (0.01, (1, 0) (the default), (10, 0.5), and (100, 1), respectively. The legend applics to all pictures. We can see that these choices allow us to enhance, counterbalance, even reverse the frequency bias.
We see from Figure 10 that by tuning hyperparameters, we can change the frequency bias of an SSM. In particular, when 100 and / = 1, we reversed the frequency bias so that the magnitude of the low-frequee www-freqenccy wave (ts cannot be well-predicted, while high-frequency wave is captured relatively well.
1.2 TUNING FREQUENCY BIAS IN MOVING MNIST VIDEO PREDICTION
In this section, present experiment to tune frequency bias in video prediction task. We show that our frequency bias analysis and the tuning strategics not only work for vanilla SSMs but also their variants. We examine a model architecture called ConvS5 that combines SSMs and spatial convolution (Smith et al., 2024). We apply the model to predict movies from the Moving MNIST dataset (Srivastava et al., 2015). In this dataset, two (or more) digits taken from the MNIST dataset (Deng, 2012) move on a larger canvas and bounce when touching the border. This forms a video over time. In our experiment, we slightly modify the movies by coloring the two digits. In particular, every movie contains two moving digits a fast-moving red one and a slow-moving blue one. The speed of the red digit is ten times that of the blue digit; consequently, the red digit can be considered as "high- frigenfre component, whereas the blue digit is a "low-frequency" component. Our goal in this experiment is to use ConvS5 model to generate up to 100 frames, conditioned on 500 frames. The ConvS5 model applies LTI systems to the time domain (i.e., the axis of the frames), but in the meantime incorporates spatial convolutions in the LTI systems, where the LTI systems are still initialized by the HiPPO initialization.
In this experiment, we train two models using two different initializations. The first initialization we use is the default HiPPO initialization. Then, we try another initialization, where for every wy that is the imaginary part of an eigenvalue of A, we transform wy by
Wy + sign (w)) (1wr 200).
(14)
That is, we shift every wy away from the origin by 200. This does not correspond to any >> 0 that we introduced in section 5.1, but our intuition is still based on our discussions in section and 4. That is, when we move away every wy that is contained in [-200, 200), our model is incapable
23

---

Published as a conference paper at ICLR 2025
of handling the low frequencies. This is indeed observed in Figure 11: when the original HiPPO initialization, the high-frequency red digit cannot be predicted, whereas when we modify the initialization based eq. (14), we well-predicter the red digit but the low- wwwfrequency blue digit is completely distorted.
0
1
2
3
498
499
500
501
502
598
599
True
7
5
75
5
7
5
7
7
7
5
5
5
5
5
5
7
7
Model2 Modell
7
75
5
7
5
7
7
7
7
5
5
7
75
5
7
7
7
5
5
5
5
5
5
Conditioning
Prediction
Figure 11: A ConvS5 model is trained to predict the Moving MNIST videos. In "Modell", the original HiPPO initialization; in "Model2", we modify the initialization based on eq. (14), We that when use the HiPPO initialization, only the slow-moving blue digit can be generated; the other hand, pushing all eigenvalues of A to the high-frequency regions (see eq. (14)) allows us predict the fast-moving red digit.
24


Reviews:
[{"review_id": "MnmiPkfHLK", "reviewer": "Reviewer_joFK", "summary": "This paper studies the frequency bias for State-space-models (SSMs). With strategies to select better initialization and Sobolev filtering (during training), this problem could be mitigated. In the experimental study, this manuscript also provides extensive results to show the effectiveness of their proposed strategies.", "questions": "1. The authors may need to define HiPPO in line 92 Page 2 before mentioning it. \n\n2. How will these discoveries help for general neural network settings? \n\n3. Would the Sobolev filter difficult to compute in practice (when doing the SSM training)? \n\n4. How to interpret Figure 4 when having low-freq. noise and high-freq. noise. What is the difference? Additionally, why low-freq. noise can be denoised (with higher-frequency-pass filter) but the images in the bottom rows will be reconstructed?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper presents a remarkable analysis of the frequency bias in state-space models (SSMs). It has been commonly observed that neural networks tend to fit low-frequency information first, often filtering out high-frequency data in many cases. This paper critically emphasizes the importance of initialization and introduces a strategy to mitigate this problem using a Sobolev filter. The figures presented in the experiments are highly inspiring. As a theoretical work, this paper provides sufficient experimental results, demonstrating strong performance.", "weaknesses": "The weaknesses of this paper mostly come from two parts. \n\n1. The analysis is based on SSM which is potentially hard to be generalized on general neural networks. \n\n2. Even though the reason is getting clearer, these strategies of this paper could still be suboptimal for practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L3BaQzAssW", "reviewer": "Reviewer_XLdv", "summary": "In this paper, the authors find that when training state space models with canonical initialization methods, there is an implicit bias such that the models are implicitly toward to capture low-frequency components instead of high-frequency ones.\nTo encourage the models to capture a more broader frequency pattern, the authors propose two mechanisms: one is scaling the initialization and another method is to apply a Sobolev-norm-based filter on the state space models.\nBy tuning the frequency bias, the state space models are shown to be able to have better performance on the long range arena benchmark.", "questions": "For the numerical experiments (Table 3) on the long range arena benchmark, how is the $A$ matrix initialized? Is $A$ initialized by the method in Corollary 1? There are some ablation studies on the scaling of the imaginary part of $A$ in the S4D paper [1]. It is shown in [1] that scaling all imaginary parts by a constant factor substantially reduce the performance of S4D, so I am curious that if we only scaling the imaginary part of $a$, does it help to improve the performance on the long range arena benchmark? It would be more convincing if the authors can provide more experiment results of only scale $a$, only train $\\beta$, the results in Table 3 is the case for scale $a$ + train $\\beta$.\n\n\n\n\n\n\n[1] Gu, Albert, et al. \"On the parameterization and initialization of diagonal state space models.\" Advances in Neural Information Processing Systems 35 (2022): 35971-35983.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The organization and presentation of this paper is smooth and clear, and it provides a better understanding on the training mechanisms of state space models on sequence modeling.\n\n2. I find the paper to be well-written and easy to follow. The overall topic of state space models is important.", "weaknesses": "1. In the end of Section 2, the authors state that the larger the total variation of the transfer function is, the better an LTI system is at distinguishing the Fourier modes. This claim is not intuitive for me, and the total variation cannot distinguish the following two different cases: the first one is large amplitude with low frequency and the second one is small amplitude with high frequency. For example, $G$ is an impulse response vs $G$ is a sinusoidal wave with a small amplitude. From my understanding, these two LTI systems have different ability on distinguishing the Fourier modes.\n\n2. The statement after Lemma 1 that small $|y_i|$ induces small total variation seems to be wrong. From the upper bounds in Lemma 1, if $|y_i|$ decreases, the the upper bounds increase, which means that the total variation will be large.\n\n3. The initialization method in Corollary1 is not a commonly used method. For S4D, the initialization methods for $a$ are mainly S4D-Legs and S4D-Lin. Why not choose these two initialization methods instead?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pbGBHVci1v", "reviewer": "Reviewer_uuaF", "summary": "This paper studies the frequency bias of state space models, where the authors identify a tendency, under generic initialisations of the state space matrix, to favor the response to low frequency signals. The authors give a definition of the frequency bias through the total variation of the transfer function, in that a low TV over an interval in frequency space is understood as a bias to not capture this region effectively. The authors then show that generic initialisations indeed gives lower TV for high frequency regions, and moreover that this cannot be fixed by optimisation (in the sense that the gradient on the frequency component is small if there is little initial overlap). \n\nTwo solutions are proposed to mitigate this issue: better initialisation where a hyper-parameter is introduced to weigh higher frequencies differently, or changing the loss function by adding a weighting function in frequency space to promote training larger frequencies (or vice versa). Improvements are observed on both a synthetic denoising task and benchmarks from long range arena.", "questions": "1. Can the authors explain more clearly why the rate of change of the transfer function is a proper measure of frequency bias? Traditionally (e.g. low pass filters) would consider the support of the transfer function, or its magnitude over frequency intervals, as a measure of its response to frequency components. Some proper definition of \"frequency sensitivity\" would make the definition of frequency bias in terms of TV more transparent.\n2. After proposition 1, Rule II: what happens in the limit $L\\to\\infty$?\n3. Eq 7: I find it strange that the weighting factor is $(1+|s|)^{2\\beta}$ instead of $(1+|s|^2)^{\\beta}$, as in the usual Bessel potential spaces. What's the motivation for this departure? In any case, the name \"the Sobolev norm\" is unclear since there are many Sobolev spaces. \n4. For the mitigation algorithms, what are some general tuning strategies for $\\alpha$ and $\\beta$? Fig 5 seems to say $\\alpha$ is more sensitive than $\\beta$. Some comments on this would be helpful.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very clearly written, and the motivation and implications of the theoretical results are clearly explained. The identification of the frequency bias is useful for problems where high frequency components need to be extracted. The proposed methods to mitigate the frequency bias appear simple to implement, and are principled.", "weaknesses": "1. The paper can be improved by clarifying some notational and definition issues, and these are detailed in the questions below.\n2. Notational suggestion: the notation of using $y_j$ to denote imaginary part of the diagonal of $A$ can be changed to avoid confusion with the output of the network, also called $y$.\n3. Since the SSM transfer function is differentiable, it may be simpler to define the total variation as the integral of its derivatives to avoid unncessary complications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3jITNamRRY", "reviewer": "Reviewer_tRCa", "summary": "This paper addresses the frequency bias of State Space Models (SSMs) and provides methods to adjust this bias. The authors introduce a formal framework to characterize the frequency behavior of SSMs and reveal that standard SSMs, as commonly described in recent literature, tend to learn low frequencies. While this low-frequency bias is often advantageous, the authors argue that it is possible to improve the model's frequency response.\n\nTo achieve this, they propose two different approaches: the first modifies initialization to control which frequencies the model can learn and the second alters the transfer function to adjust how much the model emphasizes particular frequencies.\n\nAlthough I am not a expert in SSMs, I found the paper well-structured and effective in explaining complex issues with clear, illustrative examples.", "questions": "Could you provide some guidelines for tuning the α and β parameters? How much tuning was necessary to achieve the results in Table 3?\n\nHow could the analysis be extended to non-diagonal systems?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Well-written with effective illustrative explanations.\n\nProvides a theoretically sound analysis of a complex problem.\n\nPractical relevance is high.", "weaknesses": "The analysis is limited to diagonal systems.\n\nTuning the additional hyperparameters, α and β, may pose challenges in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)