UnHiPPO: Uncertainty-aware Initialization for State Space Models
Authors: Marten Lienen
Abdullah Saydemir
Stephan Günnemann
Venue: ICML 2025 poster (2025)
Keywords: state space
uncertainty
hippo
mamba
kalman
noise
filter
Primary area: deep_learning->sequential_models_time_series
TL;DR: HiPPO extension based on linear stochastic control theory and the Kalman filter making SSMs more robust against noise

Abstract: State space models are emerging as a dominant model class for sequence problems with many relying on the HiPPO framework to initialize their dynamics. However, HiPPO fundamentally assumes data to be noise-free; an assumption often violated in practice. We extend the HiPPO theory with measurement noise and derive an uncertainty-aware initialization for state space model dynamics. In our analysis, we interpret HiPPO as a linear stochastic control problem where the data enters as a noise-free control signal. We then reformulate the problem so that the data become noisy outputs of a latent system and arrive at an alternative dynamics initialization that infers the posterior of this latent system from the data without increasing runtime. Our experiments show that our initialization improves the resistance of state-space models to noise both at training and inference time.

UnHiPPO: Uncertainty-aware Initialization for State Space Models
Marten Lienen Abdullah Saydemir'
Stephan Günnemann' 12
Abstract
high-order polynomial projection operator (HiPPO) initial- ization for their dynamics introduced by Gu et al. (2020). The HiPPO dynamics compress sequence into finite- dimensional state that represents the projection of the se- quence onto a basis of Legendre polynomials. Importantly, maintaining this state requires only the current state and the next value in the sequence and thus independent of the sequence length. This enables the construction of SSMs that efficiently capture long-range dependencies with linear computational complexity. Such SSMs are well-suited for deep learning applications where sequence length can be substantial, such as language (Gu & Dao, 2023) or video modeling (Park et al., 2025).
State space models are emerging dominant model class for sequence problems with many re- lying on the HiPPO framework to initialize their dynamics. However, HiPPO fundamentally as- sumes data to be noise-free; an assumption often violated in practice. We extend the HiPPO theory with measurement noise and derive an uncertainty- aware initialization for state space model dynam- ics. In our analysis, we interpret HiPPO as a linear stochastic control problem where the data enters as a noise-free control signal. We then re- formulate the problem so that the data become noisy outputs of a latent system and arrive an alternative dynamies initialization that infers the posterior of this latent system from the data with- out increasing runtime. Our experiments show that our initialization improves the resistance of state-space models to noise both at training and inference time.
However, the original HiPPO framework assumes that the observed data is noise-free an assumption that is eas- ily violated. In many practical scenarios, measurements are contaminated with noise due to sensor imperfections, environmental factors, or inherent variability in the data- generating process. Thus, this noise-free assumption limits the applicability of HHPPPO-iitializze SSMs, as they may perform suboptimally when exposed to noisy observations commonly encountered in real-world applications.
1. Introduction
We address this limitation by extending the HiPPO frame- work to explicitly account for measurement noise. By rein- terpreting HiPPO as a linear stochastic control problem where data emerges as noisy observations of a latent sys- tem, we derive the uncertainty- HiPPO (UnHiPPO) initialization for SSMs. With UnHiPPO, an SSM implicitly performs posterior inference in linear dynamical system, making it robust against noise in the data without increasing computational complexity or changes to the model structure.
In 2019, Voelker et al. proposed novel memory cell, the Legendre Memory Unit (LMU), for recurrent neural net- works. Their cell keeps continuously updated represen- tation of input in a sliding window in terms of orthogonal polynomials. After Gu al. (2020) generalized LMUs to variable length input windows, they were further extended as state space models (SSMs) by Gu et al. (2021) and applied across a multitude of domains. Their ability to capture tem- poral dependencies over long sequences made them a great fit for natural language processing (Mehta et al., 2022), time series analysis (Patro Agneeswaran, 2024b), speech gen- eration (Goel et al., 2022) and more (Patro Agneeswaran, 2024a).
Our contributions can be summarized as follows:
We provide a thorough description and derivation of HiPPO and its application to SSM initialization in Sec- tions A to 3.
A significant contribution to the success of SSMs is the
In Section 4, we analyze HiPPO with respect to its noise robustness using the framework of linear stochas- tic control theory. We modify HiPPO based on this analysis so that it performs implicit posterior inference under the assumption of observation noise and propose a regularization technique to make this numerically stable.
'Department of Computer Science, Technical University of Munich MMunich Data Science Institute, Technical University of Munich. Correspondence to: Marten Lienen <m.lienen de>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

UnHipPO: Uncertainty-aware Initialization for State Space Models
In Section 5, we present our uncertainty-aware exten- sion of HiPPO and show how apply it as initial- ization in the Linear State Space Layer (LSSL) model (Gu et al., 2021). Furthermore, we analyze the effect of applying the time-varying UnHiPPO dynamics a time-invariant way and why this can be interpreted as operating model on different time scales.
functions on [-1, and let represent any such function
D ( P(()). f(x)= = ==0 1B1
(1)
f is the best possible approximation the distance induced by (.-) ) in terms of polynomials up to degree D and can be seen the projection of f onto the space spanned by the basis functions ,o...., Pp. Fig. I illustrates these concepts.
f with respect to
Our experiments in Section 6 demonstrate how the Un- HiPPO initialization improves the robustness of SSMs against noise using the example of LSSL.
In this sense, the vector c; (f, P.). o..... D is the best possible compression of in the I,² space into D + numbers it lets us reconstruct f f. For more back- ground polynomial approximation of signals in general and Legendre polynomials in particular, we refer the reader to (Vetterli et al., 2014; Arfken & Weber, 2008).
Find our implementation
Notation We use O-based indexing for the coefficient vec- tors and transition matrices to stay consistent with the natu- ral numbering of polynomial basis functions where di of degree i. Furthermore, & IR denotes the current point in time and [0, t) another point in the signal up to time t. is point in the subset of R where the polynomials under consideration form an orthogonal basis, e.g. x E [-1,1] for Legendre polynomials. [n]] refers to the set integers I.
3. HiPPO
Let's say, we observe scalar signal f(r): IR+ - IR and want to process it in online fashion with downstream model h RN X. for example time series classifica- tion or a rumning forecast. Since processes fixed-length vectors, we need a vector representation length N the signal history fs: == S J/jo.0] up to the current time that expressive and informative.
2. Legendre Polynomials
Degree
16
Gu al. (2020) propose the HiPPO that represents fsx. as the coefficient vector e₂ of the projection fce of fst onto a basis of slightly modified Legendre polynomials gi up degree D N 1. While this representation is expressive, its computation via Eq. (1) would require the complete signal fst at each point t. This would make both runtime and memory requirements scale with 4, rendering this representation inefficient for long time series.
1.0
0.5
0.0
00.5
Legendre Polynomials P
-1.0
Crucially, Gu al. (2020) show how the coefficients Ctas1 can be computed approximately from coefficients Ct2 an earlier time and the value of the signal at time 44+1 with a linear update step. This means that only need to remember the current coefficients C₁2 instead of the com- plete signal history fs: to update the compressed representa- tion of the signal when a new observation Ut.+ = f(+++1) comes in. As a result, HiPPO gives us expressive features describing fst without having store the complete history. In fact, the memory requirements of HiPPO are ((N) and thus independent of the length of the sequence.
1.00
0.75
0.50
0.25
0.00
0.25 0.50
0.75
1.00
1.5
Approximating cos( (mrm(w (w= 1/4)²)
1.0
0.5
0.0
-0.5
-1.0
-1.00
00.75
0.50
-0.25
0.00
0.25
0.50
0.75
1.00
In the following, we will present condensed description of HiPPO to introduce the formal background for our extension in Sections 4 and 5. See Section for a complete and expository derivation.
Figure 1. Function approximation with Legendre polynomials.
The Legendre polynomials P; R + R, E No are set of polynomials of increasing degree i orthogonal with
respect to the 2 inner product (f.g) = S [x) g(x) dr. Together, they form basis of the space of T i.2-interralle
Shifting and scaling As a first step. the Legendre poly- nomials have to be shifted and scaled form an or-
2

---

UnHipPO: Uncertainty-aware Initialization for State Space Models
thonormal basis on the interval [0, J. For that, Gu et al. (2020) choose the time-dependent inner product (f a/s So (xx g(x) dæ, under which the basis polynomials
Such system has a latent state a, that we can influence with scalar control signal ue via its linear stochastic dynamics in Eq. (7). At discrete times fg. we take noisy measure- ments Vt, of the system in Eq. (8). d,3 is white noise, Et, is independent Gaussian noise and Φ, I and V are param- eter matrices. Note that Eqs. (7) and (8) include exactly two sources of noise: in the state dynamics and during the measurements.
(T) 2i + + (P)()((T))
are orthonormal. (e(T) 27/2 1 maps [0, t] linearly onto [-,,1]. This gives us the vector representation
Cr (t.)
(3)
If we compare the HiPPO dynamics in Eq. (4) to the above system, we see that e₁ takes the role of the system state Zt. In contrast to what one might expect, the observed sig- nal f(t) does not correspond to the observations yts, in HiPPO and, instead, it corresponds to the control signal ut, which is assumed to be noise-free in linear stochastic control theory. This means that HiPPO fundamentally assumes that there is no measurement noise the signal f(t) nor any noise in the system dynamics themselves and that e₁ is a deterministic function of fgt.
of the complete history of the signal f until time i.
Online update To maintain the coefficient vector o as time passes and we observe more of the signal f. Gu et al. (2020) take the time derivative of Eq. (3) to find the vector ordinary differential equation (ODE)
de - Auci BnJ((),
- di -
(4)
However, in many applications, these assumptions do not hold. This includes basically all time series modeling where the data is derived from a physical process such as a tem- perature, wind speed or power output, but also extends to exactly measurable but inherently noisy data such as user interactions and financial transaction logs.
which describes the evolution of the coefficient vector in continuous time under the influence of the signal [(( The HiPPO matrix Ah and vector BH are given by
12i+1
if < i, if j = i, j >i
AH.ü 1+1
(5)
Guarding the system against noise requires us to take it ex- plicitly into account in the model. Consequently, we design a continuous-discete LDS based on the HiPPO dynamics in Eq. (4) but with ft) modeled via noisy observations in- stead of noise-free control signal. In the end, f()) will not appear directly in the model at all. Instead, we will infer the posterior distribution p(c= Mraa), filtering out the effect of noise in the measurements and transitions.
and BH /2i 1.
Discretization Applying the update to discrete data ob- served at times tı, ...... requires a discrete analog of Eq. (4). To this end, Gu al. (2020) discretize the equation into the recurrence
To begin, we need to specify a model for yt, = ((t) + E linear Cty, Since Ct,, represents the approximate signal fsa, until t., we predict (fxx with
Ctys1 Anci, + B
(6)
as described in Section A.3. Eq. (6) can then serve as the
basis for recurrent neural network layer, similar to GRU (Cho et al., 2014) or LSTM (Hochreiter & Schmidhuber, 1997).
N
fen. (1x) =
Σ
Ctai Sta = BICt..
(9)
See Section A for an explanatory derivation of the results in this section.
and model
yt. BTCc2 tE
where Bμ.1 = /22 +1. (10)
4. From Control to Inference
Secondly, we need to make the HiPPO dynamics indepen- dent of ft) while keeping them linear. For that, we substi- tute Eq. (9) for [(() in Eq. (4) and get
To understand the behavior of HiPPO under measurement noise, we will analyze it as linear stochastic control prob- lem (Aström, 1970). In this theory, a continuous-discred linear dynamical system (LDS) with scalar control and ob- servations is described as
dc, 1 dt t B An)
(11)
(7) which we can simplify with the identity B A =
dx. (EP. + Tut)dt + dB
AT which follows directly from the definitions in (8) Eq. (5). Since the HiPPO dynamies are exact, the dynamics
Vt2, Ve + Exu-
3

---

UnHliPP: Uncertainty-aware Initialization for State Space Models
in Eq. (11) give us the exact evolution of e if the true signal Is is just the polynomial fgt-
retaining the shape of f, we preserve the information em- hedded in Ct.
By combining the model for Vt, and the data-fres free dynamics,
Formally, we can express our desired behavior of fge as
we get the first version of our noise-resistant LDS.
dfg((t) afge(t) and d dfgg(t) 0. (13)
dc; = - (AA f)e,dl + dB
dt dr
dt dr
(12)
Ut. BHc,, + tb
In words, the first condition demands that the value at of the reconstruction fgt, i.c. the current value of our signal, follow the gradient of f&e as we extend the domain and predict into the future. If you compare to Fig. 2, we want that the rightmost value of fce follows the dashed line as increase t. The second condition says that this gradient shall not change. Remember that the parameter of fsi is so the two derivatives in the conditions refer to the subscript and the parameter, respectively.
4.1. Extrapolation
Inferring e, from data with Eq. (12) will come down to two steps. First, we solve Eq. (11) forward in time from 4 to 1++1 to get coefficients C that represent the extrapolation of fsg from to to f++1- Second, we update the forecast Ctu+1 with the observed data Mtu+1. In contrast, HiPPO uses the data directly in the forward solving of Eq. (4).
With Eq. (9), we see immediately that
A complication with our first step is that Legendre poly- nomials and by extension fsu. are terribly unfit for extrapolation. While each It, is well behaved within [0, t]. they diverge rapidly towards too outside of it like an i-th degree polynomial be exact.
dfs((t)
B de, dt
(14)
dt
The r-derivative of fs((T) at T = becomes
1.5
dfs((t)
N
dg
1.0
fs
dr
Σ
Ct
dr
t
QTa
(15)
i=0
0.5
afsy, (tt) de
where Q; = 2i II
2 + ( which can
de
0.0
be derived from Eq. (2) and (Arfken & Weber, 2008, Eq. (12.24))
f fan
-0.5
dP.(x) dx = iP: + dP:-1(x)
-1.0
(16)
dx
-1.5
Time
Now, can combine Eqs. (11) and (13) into the condition
Figure 2. The Legendre polynomial fee diverges rapidly beyond its domain [0, fx]. If we extrapolate fe4, with Eq. (11), fsim will equal fsa, but on the extended domain [0, t.+a]. By regular- izing the dynamics with the conditions in Eq. (13), we can get the fste shown here, that extrapolates fst, lincarly until fa+1-
I
[A] I 2QT
de
B Q
Ct.
(17)
dt
Q
Since this is overdetermined, we cannot solve for de ex- actly, but can find approximate solution with the pseudo-inverse denoted by f.
This is problem from two perspectives. First, when we model fzr with. for example, N = 256 coefficients, fse. diverges like a degree 255 polynomial outside of [0, tx] and any predicted Yt.., will almost surely be a horrible predic- tion of the true 11) Second, numerical computations with forecast of such magnitude will produce enough float- ing point cancellation invalidate any results.
AT I
dc: di
1 7
B Q
2Q
(18)
A
To reduce the order of divergence, we propose to regular- ize the system dynamics. In particular, we are looking for dynamics that retain the shape of f as much as possible up to tú but evolve c such that the slope of remains roughly constant from la onwards. Fig. visualizes this goal. By
We call Ar the regularized HiPPO matrix and define the regularized LDS
1 de; = ARc,dt + d3,
Vi = BIcn, + Esc- (19)

---

UnHliPP: Uncertainty-aware Initialization for State Space Models
4.2. Discretization
and update the posterior parameters accordingly.
At this point, we move from the semi-discretized the fully discretized case. As described by Särkkä & Solin (2019, Section 6), the discrete-time model equivalent to Eq. (19) is
= yk -- mm
Sk B P. B + +
K₃ 1/0, Pk B
(23)
C++1 = AR,+1CH 9%
mR = m + KgUk
(20)
= Blcc +
Pk- = PE 82 K K
Since there is no O-th data point and thus no fo. we set to t to compute AR,1, giving AR,1 = I.
where AR,tI is the transition matrix from Ey to t4+1, i.e.
1+11 - AR
Numerical Stability Direct application of the Kalman filter can be numerically challenging, because the covari- ance matrix Ps often has some eigenvalues close zero. The smallest eigenvalues to floating point cancellation in the matrix difference in Eq. (23), losing the positive semi memi-definiteness of PA. Fur- thermore, because floating point addition and multiplication are non-assoce -anasssccative (Goldberg, 1991). Ps can also lose its symmetry, making the filter diverge. In our experiments, we solved the former issue by computing the Kalman fil- in double precision and the latter with symmetrization Ps == (Px + P, 1/2 after each update.
AR, I+
(21)
then flip into the negative due
Ik and Ea are zero-mean Gaussian noise variables with covariance and o², respectively, as hyperparameters.
In contrast to HiPPO, the dynamics in Eq. (18) do not de- pend on the observations. This means that we can evaluate them, and in particular 1/r AR. at any i, not just the tu where we have observed data. Therefore, we can use any ODE solver to approximate the transition matrix via Eq. (21).
We can, for example, approximate AR,K+1 with any integra- tion rule considered by Gu et al. (2021), e.g. forward Euler AR, a backward Euler step (I
5. Uncertainty-aware Initialization
step &
An) or 27c AR) where
At
le+1
the trapezoidal rule (I instead of upproximations. We show in Section B that it produces the most stable linear recur- rence. Note that it is matrix exponential, not a component- wise application.
26++1 AR) (I+
Upon closer inspection of the posterior distribution of ce we notice two things. First, the posterior covariance con- tains no information about the data consequence of the assumptions of the Kalman filter. Second, the whole procedure in Eqs, (22) and (23) into single linear equation for the posterior mean my.
At f++1 fg- However, contrast to HiPPO, there ex- ists an exact closed-form solution exp(log( (sp(lo((+++//))) Ar) to
Eq. (21), which
combine
m, = (I K3 AR. m + K2 Vkk
(24)
4.3. Posterior Distribution
Au.
Bu.
Next, we connect the model specified in Eq. (20) to the data. Since both dynamics and observations are linear, we can compute the posterior p(cx 11**) = N(m,, Pk) in closed form via the Kalman filter (Kalman, 1960; Särkkä & Solin, 2019).
We call Au, k.kk and Bu. the uncertainty-aware HiPPO (Un-
HiPPO) matrix and vector, respectively.
Eq. (24) is an uncertainty-aware equivalent the discretized HiPPO dynamics in Eq. (6). Fig. 3 compares the two in the face of noise. It shows a random function sampled from a Gaussian process as the ground-truth signal and noisy ob- servations derived from the signal by adding independent Gaussian noise. On top of the data, we see the reconstruc- tions fst. created from the coefficients gathered from the HiPPO and UnHiPPO dynamics, respectively. This lets us visually compare the effect of the observation noise on the features that these dynamics extract from the data. We see that the noise introduces a lot of spurious high-frequency signal in the HiPPO representation of the data, while the UnHiPPO dynamics filter out the majority of the noise and extract a close approximation of the ground-truth signal.
For that, put standard Gaussian prior N(m,, Po) i.e. mg 0 and Po = I. Then, we can compute p(ck Vi.+) = M(m, Pk) through the following recur- rence. First, we roll out the dynamics for one step to extend feu to feun
mi = AR,
(22)
PE = AR. AT R,k +
Then, we compare the prediction the observed data yk
5

---

UnHiiPO: Uncertainty-aware Initialization for State Space Models
Observations 5%
Signal
HiPPO
f5
UnHiPPO
sequence y via latent state C₁ RN and the recurrence
C₁ Ack-1 + Bug
(25)
yk Cc2 Dug
where A E RNN, B BERN.CER' and D E RR are parameters.
Gu et al. (2021) create a building block for neural sequence models from this by stacking H independent copies of Eq. (25) for H-dimensional input, i.e. the output of the previous layer. To improve the expressiveness of the system, they introduce M latent channels by increasing the dimen- sion of the output yı to M. Since simple stacking means that each feature of the input ug is processed independently, they also introduce a position wise map to mix the channels. This map consists of a GELU nonlinearity followed by a linear layer that maps yk R the final output of dimensionality H, i.e. the weight matrix of the linear layer has dimensions (H M) x H. The stacked parameters be- come A E N B E R HXN, C E " and D E R HHM XM
-2-
0
10
Time т
Figure 3. The uncertainty-aware HiPPO dynamics filter out the
majority of the noise.
The strength of the denoising is controlled by the o² hyper- parameter through its influence on the so-called innovation covariance 9y in the Kalman update step in Eq. (23). Fig. 4 visualizes the effect spectrum of o², Too small values mean that the signal modeled closely including noise, while too large values make the dynamics ignore the data. For optimal filtering, o² needs to be adapted to the data. Note the surprisingly large scale of o² in Fig. 4. This is necessary because o² needs to overcome the B PÆ Bu term in 8 which is large because of the magnitude of entries of the HiPPO matrix in Eq. (5). An unfortunate side effect is that o² cannot be interpreted as the noise variance of the data directly.
The initialization of A and B is essential as Gu al. (2021) have shown and the HiPPO initialization produced much stronger results than random initialization. They discretize the HiPPO dynamics to
Ck AH.&CE-T + BII
(26)
where
At
1014
AH,
I+
Ah
I
Ah
(27)
2t
2t
1012
and
Noise Variance o²
At
10:00
BI =
I
An
B
(28)
21
fx
10
See Section A.3 for a derivation this discretization. Then, they initialize A, = An,; and B, = Bu, where they set
At 1 and vary t4 log-uniformly between Emin for and fmax for i H.
10
-2
-3
10a
We adapt this into the uncertainty-aware LSSLL (UnLSSL) by initializing
Time
Figure 4. The o² hyperparameter controls the level of filtering.
A, AC.(U and B, BU.
(29)
where varies in the same way as for LSSL. One difference to LSSL is in how these initializations are computed. In con- trast LSSL where
For the transition uncertainty, we always choose Σ = I. In our experiments, had no appreciable effect except for instability if it was very small.
can get the discretized dynamics any /. directly, for UnL we compute them for all integer steps E [tmax] and then select subset. In theory, we could jump to any directly the Kalman update Eq. (23), but that would increase the uncertainty P. as if there was no data before i, changing the dynamics. Instead, we also
5.1. Uncertainty-aware Linear State Space Layer
The LSSL (Gu et al., 2021) is a simple sequence model that maps a scalar input sequence u4, [n] to a scalar output
6

---

UnHiPPO: Uncertainty-aware Initialization for State Space Models
compute all intermediate steps, which mirrors the more re- alistic setting where we also observe data 2,..... , 1. Note that this only happens once for initialization and has negligible runtime cost.
Training Noise p
0
10
10-2
10-
10"
AAccuracy [%)]
10
FSD
Gu et al. (2021) improve the runtime of LSSL by rewriting Eq. (26) from a recurrence to a convolution with Krylov kernel that can be constructed in (log n) time. Since UnLSSL is just a different initialization for the same dy- namies, the same optimization applies. Note that the Krylov kernel for UnLSSL needs to be constructed in double pre- cision, though it can be cast to single precision before the convolution.
5
10
10
10
10
Accuracy [%)
20
SC10
10
A; and B, can be fine-tuned with gradient descent during model training. However, Gu et al. (2021) have shown that training A; and B; has only minor effect on LSSL per- formance in sequence classification, but prevents caching the Krylov kernel and increases training time significantly. Therefore, we keep the SSM parameters fixed in our experi- ments.
10
10
10
Evaluation Noise p
Figure Accuracy difference between UnLSSL and LSSL speech classification trained on a noise level p' and evaluated on range of noise levels
Time-invariant Dynamics Both the UnHiPPO dynamics in Eq. (24) and the HiPPO dynamics in Eq. (25) are time- varying. However, the LSSL. and UnL.SL. layers fix the dynamics point in time and then apply them repeatedly. We understand the effect of applying the dynamics in time- inverinvariant way visually from Fig. 5.
[-1, 1] into [-1, 1-,1-2//++1 and B fills the freed up inter- val [1 1 2/k+1, 1] with the new data. So, repeated application of As+1 squeezes the information encoded in fsu. by k/k+1 each time. Fig. 5 shows this as the 250-fold application of A500 has compressed the initial value of f exactly by factor of (499/500)29) 250 The fact that the fca, for large k contain the whole signal while for small they contain mostly the recent past, illuminates why initializing and B with Aş and B, for various k in LSSL and UnLSSLL lets the model take multiple timescales into account as Gu et al. (2021) report.
Signal f
= 500
k 100
k
20
1.0
0.5
0.0
-0.5
6. Experiments
-1.0
We experiment with a multi-layer LSSL and UnLSSL ar- chitecture with linear encoder and decoder as described by Gu al. (2021). Section C gives further details on the hyperparameters.
10 Time T
100
250
8
10
Figure 5. Reconstruction fs,, of signal obtained through re- peated application of Aş and B. for fixed k. Data observed at 250 cquispaced locations between 0 and 10. Each application of A compresses the reconstruction fsu, by k-1/e.
We evaluate UnLSSL on two sequence classification datasets, the Free Spoken Digits dataset (FSD) (Jackson et al., 2018) and a 10-class subset the Speech Commands dataset (SC10) (Warden, 2018). FSD contains 3000 record- ings of speakers pronouncing each digit from to 9 at a sample rate of 8000. We loop recordings shorter than one second and then cut them so that each sample is univariate sequence of length 8000. SC10 is multi-speaker dataset consisting of one second clips each being recording of one spoken word. Each sample is a univariate sequence of 16 000 steps representing the raw waveform of the audio.
Semantically, the dynamics A of either HiPPO or Un- HiPPO take the reconstruction fre of the signal from 0 to t represented by e₁ and extend by one step. Then, B++1 updates the extended reconstruction, that it fits the data at 1 To understand the effect of repeated application of the same A it is instructive to imagine what happens if we map the signal back to the domain [-1, 1] of the Legendre polynomials with or from Section 3. In that view, Ag+1 squeezes the data that is encoded in fse. as function on
For our experiments, where we measure the effect of noise in

---

UnHliPP: Uncertainty-aware Initialization for State Space Models
Effect of o² a² in the Kalman filter in Eq. (23) influences how much noise the UnHiPPO dynamics filter out. This lets the user adapt UnHiPPO to the expected level of noise in the data. See Fig. 8 for an evaluation where we have trained UnL.SSL. on SC10 with Gaussian noise of = 10-1 and varied the o² parameter. The results show that UnLSSL achieves a better on noisy data than LSSL.
FSD
A Accuracy [%6
-2
10
10 2
10
10
Discretization Methods While LSSL relies on a trapezoidal-like discretization for the HiPPO matrix, we use the closed- form solution in terms of the matrix exponential for the UnHiPPO matrix in Eq. (21). Ta- ble 1 shows that the closed-form solution is only 50% slower than the trapezoidal approximation, which is negligible since this only needs to be computed once to initialize the SSM. An UnLSSL model initialized with the resulting UnHiPPO matrices reaches comparable performance for the closed-form solution and stable backwards Euler approximation. Both the trapezoidal and forward Euler approximations produce an SSM initialization that diverges on the long sequences of SC10. Section B has a visualization of the effect of the discretization methods on the HiPPO and UnHiPPO matrices.
SC10
AAccuray [%]
Table 1. Time compute discretized AR. for N = 256 and resulting UnLSSL accuracy on SC10 with p 10 noise.
Method
Time
Accuracy
-1
2
10a
closed-form trapezoidal
2.8 ms
79.3%
Noise
1.8 ms
Figure 7. Accuracy difference between UnLSSL and LSSL in speech classification for models trained and evaluated on a range of noise levels p.
forward
0.1 ms
backward
1.6 ms
79.8%
Accuracy [56]
77.5
75.0
72.5
LSSL
UnLSSL
100
16a
104
10c
10a
1010
10
10
2²
Figure 8. The o" parameter adapts the noise filtering the level of noise in the data on SC10 with p 0.1. Error bars denote standard deviation.
7. Related Work
Yu et al. (2023) introduce diagonalization of the HiPPO matrix through an approach called "perturb-then- diagonalize". Their parametrization improves runtime per- formance over a non-diagonal one and is more robust against noise than the diagonalizations proposed by Smith et al. (2023); Gu al. (2022b). Yu et al. (2025) propose an al- ternative parametrization for SSMs detached from HiPPO and instead based on Hankel operator theory. Their HOPE scheme improves the training stability of SSMs and makes them robust against noise appended to the end of a sequence. However, they do not evaluate noise on the data itself. Agar- wal et al. (2023) derive an SSM from the spectral filtering algorithm (Hazan et al., 2017) which is based on an LDS with noisy outputs similar to our setup in Section 4. Zhou et al. (2024) describe an SSM that parametrizes Kalman filter with a learnable dynamics matrix. They accelerate their model by rewriting the recurrence as a convolution with a Krylov kernel similar to (Gu et al., 2021). However, they do not connect it to HiPPO theory or SSM initialization in general.
the data, we add independent Gaussian noise ò ~ NO(0 to the training and/or test data.
Noise Robustness For Fig. 6, we trained LSSL and UnLSSL on data with varying noise levels p' and evaluated the classification accuracy on a test set across spectrum of noise levels p. The results show that the UnHiPPO initial- ization improves the model's robustness against other noise levels than it was trained on, both lower and higher.
Training Noise In the real world, training and evaluation data often have similar levels of noise. To investigate this case, we trained LSSL and UnLSSL on range of noise levels / and then evaluated their accuracy on the test at the same noise level. Fig. 7 shows that the UnHiPPO ini- tialization hinders the model's 's performance slightly at low noise levels while improving it as the noise level in the data increases.
8

---

UnHipPO: Uncertainty-aware Initialization for State Space Models
Gu et al. (2022c) present additional background on the mech- anism behind HiPPO. Gupta al. (2022) explore simpli- fied alternative to HiPPO with a diagonal transition matrix. Liu & Li (2025) analyze the role of initializations in state space models discretized with the zero-order hold approxi- mation.
entific guidance. All authors reviewed and discussed results, both theoretical and empirical, throughout the project.
Software
For our results, we rely on excellent software packages, notably numpy (Harris et al., 2020), scipy (Virtanen et al., 2020), pytorch (Paszke al., 2019), einops (Ro- gozhnikov, 2022), matplot (Hunter, 2007), hydra (Yadan, 2019) and jupyter (Granger & Pérez, 2021).
8. Limitations
Gu et al. (2022a) have proposed an alternative parametriza- tion of the HiPPO matrix Aи in terms of diagonal and low-rank matrix, which they call a structured SSM. With this parametrization, the matrix multiplication in the SSM recurrence can computed O(N) instead of O(?)). A similarly structured parametrization of either the regular- ized HiPPO matrix AR in Eq. (18) or the uncertainty-aware HiPPO matrix Au, in Eq. (24) eluded us, unfortunately, because of the pseudo-inverse and Kalman filter equations, respectively.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
References
Agarwal, N., Suo, D., Chen, X., and Hazan, E. Spectral
9. Conclusion
State Space Models, 2023.
We analyzed HiPPO as a linear stochastic control problem and understood why it is not robust against noise. By rewrit- ing the control problem
Arfken, G. and Weber, H. J. Mathematical Methods for Physicists. Elsevier Academic Press, Amsterdam, Hei- delberg, 6. ed., [print., international ed.] edition, 2008. ISBN 978-0-12-0598762..
system controlled by a latent state with noisy observations, we were able to derive the UnHiPPO dynamies that perform posterior inference in that system and thereby filter out noise. The resulting UnHiPPO initialization for SSMs offers a configurable amount noise filtering without any modifications to the model structure runtime costs. We have furthermore shown in periments how such built-in noise filtering can improve the robustness of SSMs against noise in the real world.
Aström, K. J. Introduction to Stochastic Control Theory. Number v. 70 in Mathematics in Science and Engineer- ing, Academic Press, New York, 1970. ISBN 978-0-12- 065650-9.
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk. H., and Bengio, Y. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation, September 2014.
Acknowledgments
MI. was funded by the Bavarian State Ministry for Sci- ence and the Arts within the framework of the Geothermal Alliance Bavaria project. We want to thank the Munich Cen- ter for Machine Learning for providing compute resources. Furthermore, we are thankful to Marcel Kollovieh for his insightful feedback and his support with Fig. 2.
Goel, K., Gu, A., Donahue, C., and Re, C. It's Raw! Au- dio Generation with Space Models. In Interna- tional Conference on Machine Learning, pp. 7616-7633, PMLR, June 2022.
Goldberg, D. What every computer scientist should know about floating foating-point arithmetic. ACM Computing Surveys, 23(1):5-48, March 1991. ISSN 0360-0300. doi: 10.1 103162.
Authors' Contributions
MI. conceived the project idea, developed the theory, implemented the UnHiPPO initialization and wrote the manuscript. AS implemented the majority of the pipeline, designed and executed the empirical evaluation (i.e. selected the datasets and implemented model training and data load- ing to generate the quantitative results for visualization and analysis), identified a technical correction in the mathemati- cal formulation, and corrected (Un)LSSL implementations. SG contributed to the design of the method and provided sci-
Granger, B. E, and Pérez, Jupyter: Thinking and Sto- rytelling With Code and Data. Computing in Science & Engineering, 23(2):7-14, March 2021. ISSN 1558-366X. doi: 10.1 10.1109/MSS..222..00
Gu, A. and Dao. T. Mamba: Linear- Time Sequence Model- ing with Selective State Spaces, December 2023.

---

UnHliPP: Uncertainty-aware Initialization for State Space Models
Gu, A., Dao, T., Ermon, S., Rudra, A., and Re, C. HiPPO: Recurrent Memory with Optimal Polynomial Projections. In Neural Information Processing Systems, 2020.
(1):35-45, March 1960. ISSN 0021-9223. doi: 10.1115/ 1.3662552.
Liu, F. and Li, Q. Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models. In International Conference on Learning Representations, 2025.
Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and Ré, C. Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers. In Neural Information Processing Systems. arXiv, October 2021. doi: 10,485550/rXxvv211100 13985.
Mchta, H., Gupta, A., Cutkosky, A., and Neyshabur, B. Long Range Language Modeling via Gated State Spaces. In The Eleventh International Conference on Learning Representations, September 2022.
Gu, A., Goel, K., and Ré, C. Efficiently Modeling Long Sequences with Structured State Spaces. In International Conference Learning Representations. arXiv, 2022a.
Park, J., Kim, H.-S., Ko, K., Kim, M., and Kim, C. Video- Mamba: Spatio- Tempooll Selective State Space Model. In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), European Conference Computer Vision, pp. 1-18, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-72698-9 doi:
Gu, A., Gupta, A., Goel, K., and Ré, C. On the Parameteri- zation and Initialization of Diagonal State Space Models, June 2022b.
Gu, A., Johnson, I. Timalsina, A., Rudra, A., and Re, C. How to Train your HIPPO: State Space Models with Generalized Orthogonal Basis Projections. Interna- tional Conference on Learning Representations, Septem- ber 2022c.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An Imperative Style, High- Sigh-Performance Deep Learning Library. In Neu- ral Information Processing Systems, 2019.
Gupta, A., Gu, A., and Berant, J. Diagonal State Spaces are as Effective as Structured State Spaces. In Advances in Neural Information Processing Systems, October 2022.
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E.,, Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M.. Haldane, A., del Río. J. F., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, T. E. Array programming with NumPy. Nature, 585(78255 September 2020. doi: 10. 00988444158-000-24449229
Patro, B. N. and Agneeswaran, S. Mamba-360- Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges, April 2024a.
Patro, B. N. and Agneeswaran, V. S. SiMBA: Simplified Mamba-Bat Based Architecture for Vision and Multivariate Time series, April 2024b.
Hazan, E., Singh, K., and Zhang, C. Learning Linear Dy- namical Systems via Spectral Filtering, In Guyon, I,, Luxburg, U. V.. Bengio, S., Wallach, H., Fergus, R., Vish- wanathan, S., and Gamett, R. (eds.), Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Rogozhnikov, A. Einops: Clear and Reliable Tensor Ma- nipulations with Einstein-like Notation. In International Conference on Learning Representations, 2022.
Särkkä, S. and Solin, A. Applied Stochastic Differential Equations. Cambridge University Press, April 2019. ISBN 978-1-108- 978-1-108-18673-5.
Hochreiter, S. and Schmidhuber, J. Long Short-Term Mem- ory. Neural Computation, 9:1 13770017700, 1997.
Smith, J. T. H., Warrington, A., and Linderman, S. W. Sim- plified State Space Layers for Sequence Modeling. In International Conference on Learning Representations, 2023.
Hunter, J. D. Matplotlib: A 2D graphics environment, Com- puting in Science & Engineering, 9(3):90-95, 2007. doi: 10.1 109100/E.22000.5555
Vetterli, M., Kovačević, J., and Goyal, K. Foundations of Signal Processing. Cambridge University Press, Cam- bridge, 2014. ISBN 978- 978-1-107-03866-8.
Jackson, Z., Souza, C., Flaks, J., Pan, Y., Nicolas, H., and
Thite, A. Free-spoken-digii--aaatstt V1.0.8. Zenodo, August 2018
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D.,, Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M.,
Kalman, R. E. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering,
10

---

UnHipPO: Uncertainty-aware Initialization for State Space Models
Wilson, J., Millman, K. Mayorov, N., Nelson, A. J.,
Jones, E., Kern, R., Larson, E.,, Carey, C. J., Polat, I., Feng, Y., Moore, E. W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F,, van Mulbregt, P., and SciPy 1.0 Contributors. SciPy 1.0: Fundamental algorithms for scientific computing in python. Nature Methods, 17:261-272, 2020. doi: 10.1038/64159--019-0688
Voelker, A., Kajić, I., and Eliasmith, C. Legendre Mem- ory Units: Continuous- Time Representation Recurrent Neural Networks. In Neural Information Processing Sys- tems, 2019.
Warden, P. Speech Commands: A Dataset for Limited- Vocabulary Speech Recognition, April 2018.
Yadan, O. Hydra A framework for elegantly configuring complex applications. Github, 2019.
Yu, A., Nigmetov, A., Morozov, D., Mahoney, M. W., and Erichson, N. B. Robustifying State-space Models for Long Sequences via Approximate Diagonalization, In International Conference on Learning Representationss arXiv, October 2023. doi: 10.448550ar2xx..330000011
Yu, A., Mahoney, M. W., and Erichson, N. B. HOPE for Robust Parameterization of Long-memory State Space Models. In International Conference on Learning Repre- sentations. arXiv, 2025. doi: 10:485550arrxix.2455.333
Zhou, Z., Guo, X., Jie, Y., and Xia, C.-M. Kalman- Modeling long-term time series with kalman filter struc- tured state spaces. IEEE Signal Processing Letters, PP: 1-5, January 2024. doi: 10. 11911099/SPP2 1422444449.222.445777....

---

UnHiiPP: Uncertainty-aware Initialization for State Space Models
A. HiPPO Derivation
complete derivation can, of course, also be found in the original work of (Gu et al., 2020). However, the original derivation includes several additional degrees of freedom support multiple HiPPO variants. Since the variant based on Legendre polynomials is the only one relevant our work and also the one most relevant to follow-up works (Gu al., 2021; 2022a), give specialized derivation here that eschews unnecessary symbols.
A.1. Choice of Basis
Let's consider the case where we observe continuous scalar signal f(r): IR R and at any point in time we want to encode the history f&e fl [o. of the signal that we have observed so far into N dimensional vector Ct- We consider ca good representation of fSt, if we can turn it back into approximate version fse of fst. approximation framework for signal processing (Vetterli et al., 2014), In this framework, we begin by choosing an inner product (-, between functions on [0, to induce distance via d(f. 9) = vllf - gli and Il gll = (f g. g) and thus define exactly what f&r = f&e should mean. Equipped with this inner product, orthonormal polynomials 9.. i ...... can approximate the continuous signal fg.. Finally, we can find e₂ by projecting f&e onto 9i, i.e. Ct (fet. gi).
rely the polynomial
then derive set of N 1 on [0, to form the basis of finite- dimensional function space in which
Our derivation of the Legendre-based HiPPO begins with an inner product between two functions f and on [0, t). We will choose
(g.)) - - 7 L [(T) ((F)
(30)
which is just the L inner product between functions except for constant factor 1/1. While the constant factor has no effect on orthogonality, it will be useful later when we update the representation 6₂ with new data. In terms of function approximation, this inner product induces the distance
(dU,s))² - 1. - -(())
(31)
This means that the approximation fse we are looking for minimizes the squared difference to fsr across the segment from O to & with equal weight given to each r.
Next, we need the polynomial basis. Since the Legendre polynomials P. are orthogonal basis on [-1, 1] for the L inner product, they are also orthogonal on [0, t) with respect to our shifted and scaled inner product if we shift and scale them linearly from [-1, 1] to [0, t). For this shifting, we define (T( = 27/1 - that takes 7 E [0, f] to x E [-1, 1]. This gives us (T) as an orthogonal basis on [0, t). However, the projection of f&r onto the basis is simplified if the basis orthonormal, so we need to compute the normalization constants of this shifted basis. With d((r) = 2/c and substitution
integration, we get
= t - 1 ((((((T)) dr = 1 / P.(x)² dx 2
1
(P((((())
(32)
2i +
The last step uses the normalization constant of the unscaled Legendre polynomials of 2/26+1. This gives us (T. - i= 1.....,
(33)
as basis of orthonormal polynomials on [0, |] up any degree D.
Computing the representation C₁ E IRN thus means evaluating Cr, = (fsi, s(,1); with Eq. (30) for i = 0, O...., -11- 1.
A.2. Linear Dynamics
How does the representation e, of fer change observe more of the signal? In the continuous case, the answer to this question is the ODE
dea. d di di
( ((T) = = ( ((T) f(T) + 1 di
d
di L' f(T) (())).
-
(fsn
(34)
12

---

UnHipPO: Uncertainty-aware Initialization for State Space Models
The first part is just Cr, but the derivative of the integral in the second part needs some more work. We can exchange differentiation and integration with Leibniz's rule and get
di L [(T) (T) (())(())) + S. [(T)
dgen(T) dr. dt
(35)
To progress, need to some properties of Legendre polynomials. For the first part, use that P.(1) 1 for all and therefore (t,;(t) = 2 + For the second part, we use the identity (see (Gu et al., 2020, Eq. (8)) and (Arfken & Weber, 2008, Eqs. (12.23) and (12.24)))
d P((x)= Σ f=1
+ 1)
+) 11
(36)
dx
to derive 4 (T). Note that
d 2T
2T
- { -1+1) (7(66) +1).
1 1
2T
(c(T)
(37)
dt
dt
12
With Eqs. (36) and (37) and recognizing (T) as I, we get
dt (t,(T) = .22++ +1 1 d
dt (T(
+2 (T) P(()((T))
dt Ot
1 - + (o)(T) + 1) 8 0t
P((e((T))
(38)
1 12i +1 (P.( Σ (2 - 5)+ + 1) P. Pi-s(oa(T))
1 (T) + v 2i +1
Σ
/2 -) 1) +1 19.1-1(7)
/=1
When we plug this into the second part of Eq. (35), we see that
L S()) (T)
dr = - 1 1. ((T) (T) + 222 +
22 - 5) +
gt. 1).--((T) dr
dt
/=1
(39)
= - -[ +V 2i +1 Σ
2) +1 ( (.
j=1
5
If we plug Eq. (39) into Eq. (35) and that in turn into Eq. (34), we see that d
de,, df 82 I f(() f(1))
1
dt / f(r) g()())
(40)
= - + 1 122 lic.s + 2i Σ 2(i ()
Now can separate terms involving and and write the N instances of Eq. (40) for the N components of single vector differential equation. This gives the full continuous coefficient dynamics from (Gu et al., 2020)
de dt
Anci Baf(()
(41)
with the HiPPO matrix and vector
2i +
V
2j
+
i <
An.5j
i+1
if i
and
B1g /2i +1,
(42)
0
if > i
respectively.
13

---

UnHliPP:: Uncertainty-aware Initialization for State Space Models
A.3. Discretization
Usually, we observe the signal at discrete time points & instead of truly continuously, Therefore, we cannol evaluate the exact, continuous dynamics in Eq. (41) and have discretize them instead, that only ever need to evaluate the discrete observation points tj.
With a continuous signal, we would compute Ctu+. from Ct2 by integrating Eq. (41) from l to 1 i.e.
f4+1 fram Amr + BB)(()
Ctal1 = Cta. +
(43)
For discrete observations (f1), f(2))., we need to approximate the integral on the right-hand side in way that only requires f(4)) and [(1x+1).
(Gu et al., 2020) describe three approximations for the integral, forward Euler, backward Euler and the trapezoidal rule of which they use the latter for their experiments. Forward Euler approximates the value of the integrand with its value the lower integration bound, i.e. at lx, and gives
1 ( AHc, + +- =BB((()) = =(1
(
-
At Cta
At
Cies1 FJ Cta. + (1+-1- - (x)
Ah
+
BHJ( Lx
(44)
Li
la
1k
=i
Backward Euler is the other extreme that approximates the integrand with its value at 1u+ i.e.
Cra-s s Cca + At (-
(-1
AHC +
Buf Daat(ttt))
(45)
1x+1
16+1
Solving for Ca. gives
At
At
B (tamr))
Ctu+1
I+
Ah
Cix
+
Bu
(46)
ba+1
l4+1
For the trapezoidal rule, we approximate the integrand the average of its values ta and fx+1, i.e.
C Cra. + A1
(
1
Anci + tx
But((t)) + (- f
( Aucru + txt1
Bu BuJ (n-(1-
(47)
t
After simplifying and solving for Cle+1 this becomes
Cr++1
20
I+
Al
An
"[(
Al
An Ct.
+
Al
Bui
( 1
- f((tt)
+ s(ta+))
(48)
2t
2t
2
th
tk+1
Note that (Gu et al., 2020) approximate Eq. (48) as
At
At
At Buf Btf(t
Clz+1
I+
Ah
An C²2
+
(49)
2tx+1
2tx+1
tx+1
ie. they approximate f(xx) + 1/tot1ffffff of (tx+1) as of This, in effect, pretends that the dynamics in Eq. (41) would constant in time and is required by LSSL for the numerical stability of the accelerated evaluation via convolution with Krylov kernel. this work, we discretize the LSSL baseline with Eq. (49), too. Note that for our model closed-form solution in terms of the matrix exponential exists, which we instead of numerical approximation.
Each of Eqs. (44), (46), (48) and (49) can be put into the form
Cta AH.t, Ct. + BH.as + BH
(50)
by collecting terms appropriately.
14

---

UnHliPPO: Uncertainty-aware Initialization for State Space Models
B. Discretizations Visualized
Recurrent roll-out an SSM or the construction of the Krylov kernel equate to repeated matrix multiplication with HiPPO or UnHiPPO matrix taking powers thereof. Therefore, successful training and evaluation on long sequences requires the discretization to chosen so that matrix powers do not diverge. Fig. shows the effect of repeated applications of HiPPO and UnHiPPO matrices discretized with the closed-form solution and the approximations considered by (Gu et al., 2021). The HiPPO matrix is stable under the backward discretizations and has negligible degree of divergence with the trapezoidal rule. The UnHiPPO matrix cannot be stably discretized with the trapezoidal rule, but its closed-form discretization is stable and retains the information encoded in the shape of the reconstruction f almost perfectly.
Closed-form
Trapezoidal
Forward Euler
Backward Euler
UnHiPPO
Reconstruction fig. Eec
HiPPO
A crN
Figure 9, Visualization of repeated application of HiPPO and UnHiPPO matrices various discretization methods through the polynomial representation of Magnitudc of the plotted curves correlates with |
C. Experiment Details
We use the parameters listed Table for all models. For the range tx the initialization of LSSL, we set tmin = 10 and fmax 1000 to cover range of time scales.
Table 2. Hyperparameters of the LSSL architecture used the SC10 experiments.
Parameter
Value
Layers
4
N
128
Linear Embedding Size
128
Latent Channels
4
Dropout
0.1
UnHiPPO o²
1000
Training Steps
100000
Batch Size
16
15


Reviews:
[{"review_id": "rTpxktLplf", "reviewer": "Reviewer_JbxW", "summary": "This paper investigates state space models (SSMs) through the lens of linear stochastic control theory and proposes a novel initialization method to enhance robustness against input noise. The authors first reformulate the linear recurrence in SSMs as a homogeneous linear dynamical system with noise, replacing the input signals with their closed-form reconstructions. Next, they derive a regularized HiPPO formulation by enforcing the online approximator to extrapolate linearly and maintain a consistent time derivative with the closed-form approximation at the boundary. Building on this new dynamic system, the authors compute the posterior mean estimate under Gaussian noise, which ultimately yields an improved initialization for the state transition matrix in SSMs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review.\n\n**Time-dependence of matrices**\n\nIt is correct that SSMs learn $A$ and $B$ and discretize them on the fly. However, at least for LSSL, the time step at which the learned matrices are discretized is fixed per feature to make the model adapt to multiple timescales and does not actually vary with time. Therefore, it is factually equivalent that we learn $A_k$ and $B_k$ directly. \n\n**Experiments**\n\nWe did not evaluate on LRA, because its synthetic tasks have discrete input data, which does not fit the assumptions of our derivation. In there, we assume that the noise is Gaussian and therefore continuous. An adaptation for discrete data and discrete noise would be a nontrivial adaptation.\n\n**Typo and related work**\n\nThank you for your careful reading. We have added the missing \"by\" to clarify that the GELU nonlinearity comes first and included the two works you mentioned in our related work section.\n\n**Compatibility with efficient parametrizations**\n\nAs we described under Limitations (Section 8), we were unfortunately not able to derive a similarly structured representation of our initialization, because the pseudo-inverse and the Kalman filter equations pose a significant challenge, which we elaborate in the following.\n\n$\\\\mathbf{A}_{\\\\mathrm{R}}$ is not a normal matrix, therefore, there is no trivial diagonalization of it. Previously, Gu et. al. (S4) used correction matrices to obtain skew-symmetric matrices which are in turn a special case of normal matrices. Here, though, finding a correction matrix is not trivial due to pseudo-inverse. Considering the definition of pseudo-inverse for matrices constituting of linearly independent columns,\n\n\\\\begin{equation} \\\\mathbf{M}^{\\\\dagger} = (\\\\mathbf{M}^{\\\\ast}\\\\mathbf{M})^{-1}\\\\mathbf{M}^{\\\\ast}\\\\end{equation}\n\nwhere $(\\\\cdot)^{\\\\ast}$ denotes the conjugate transpose that is equivalent to transpose for a real matrix. Since $\\\\mathbf{A}_{\\\\mathrm{R}}$ is real, we can safely use,\n\n\\\\begin{equation}\\\\mathbf{M}^{\\\\dagger} = (\\\\mathbf{M}^{\\\\mathsf{T}} \\\\mathbf{M})^{\\-1}\\\\mathbf{M}^{\\\\mathsf{T}}\\\\end{equation}\n\nThen, pseudo-inverse of (19) becomes\n\n\\\\begin{equation}\n\\\\begin{pmatrix}\\\\mathbf{I}\\\\\\ \\\\mathbf{B}^{\\\\mathsf{T}}\\_{\\\\text{H}}\\\\\\ \\\\mathbf{Q}^{\\\\mathsf{T}}\\\\end{pmatrix}^{\\\\dagger} =                       \\\\begin{pmatrix}\\\\begin{pmatrix}\\\\mathbf{I}\\\\\\ \\\\mathbf{B}^{\\\\mathsf{T}}\\_{\\\\text{H}}\\\\\\ \\\\mathbf{Q}^{\\\\mathsf{T}}\\\\end{pmatrix} \\\\begin{pmatrix}\\\\mathbf{I} \\\\\\ \\\\mathbf{B}^{\\\\mathsf{T}}\\_{\\\\text{H}}\\\\\\ \\\\mathbf{Q}^{\\\\mathsf{T}}\\\\end{pmatrix}\\\\end{pmatrix}^{\\-1} \\\\begin{pmatrix}\\\\mathbf{I}\\\\\\ \\\\mathbf{B}^{\\\\mathsf{T}}\\_{\\\\text{H}}\\\\\\ \\\\mathbf{Q}^{\\\\mathsf{T}}\\\\end{pmatrix}^{\\\\mathsf{T}}\\\\nonumber\n\\\\end{equation}\n\nand $\\\\mathbf{A}_{\\\\mathrm{R}}$ can be explicitly written as\n\n\\\\begin{equation}\n\\\\mathbf{A}\\_{\\\\mathrm{R}} = \\\\begin{pmatrix}\\\\underbrace{{\\\\begin{pmatrix}\\\\mathbf{I} \\\\\\ \\\\mathbf{B}\\_{\\\\text{H}}^{\\\\mathsf{T}} \\\\\\ \\\\mathbf{Q}^{\\\\mathsf{T}}\\\\end{pmatrix}}^{\\\\mathsf{T}}{\\\\begin{pmatrix}\\\\mathbf{I} \\\\\\ \\\\mathbf{B}\\_{\\\\text{H}}^{\\\\mathsf{T}} \\\\\\ \\\\mathbf{Q}^{\\\\mathsf{T}}\\\\end{pmatrix}}}\\_{\\\\text{First part}} \\\\end{pmatrix}^{-1} \\\\underbrace{{\\\\begin{pmatrix}\\\\mathbf{I} \\\\\\ \\\\mathbf{B}\\_{\\\\text{H}}^{\\\\mathsf{T}} \\\\\\ \\\\mathbf{Q}^{\\\\mathsf{T}}\\\\end{pmatrix}}^{\\\\mathsf{T}} \\\\begin{pmatrix}\\\\mathbf{A}\\_{\\\\text{H}}^{\\\\mathsf{T}} - \\\\mathbf{I} \\\\\\ 2\\\\mathbf{Q}^{\\\\mathsf{T}} \\\\\\ \\\\mathbf{0}\\\\end{pmatrix}}\\_{\\\\text{Second part}}\n\\\\end{equation}\n\nFirst part gives us a symmetric and square matrix. Therefore, we know that inverse of this will be also symmetric, and this part would not be a problem for diagonalization if it were to stand by itself, or alongside another symmetric matrix.\n\nSecond part does not result in a symmetric, skew-symmetric or in general a normal matrix. Therefore, the overall result needs a correction matrix to become diagonalizable matrix.\n\nHowever, hand calculation requires taking the inverse of the first part. Even after that, multiplication with a non-normal matrix needs to be computed, together which makes it too complicated to reveal the structure of the matrix, so that it can be manipulated to have a normal form.", "claims_and_evidence": "Most claims in the paper are clear and well-supported. However, I have some reservations regarding certain steps in the derivation of the UnHiPPO initialization:\n\nTo obtain data-free dynamics, the authors substitute $f(t)$ with $\\hat{f}_{\\le t}(t)$. While this choice eases the derivation, it introduces a discrepancy between the dynamics used for initialization and those used during actual signal processing. Further justification would strengthen this step\n\nAdditionally, as far as I understand, models such as LSSL and S4 do not learn time-dependent matrices $A$ and $B$ directly. Instead, they learn static matrices $A$ and $B$ which are then converted into time-varying forms during training or inference. In contrast, the proposed initialization appears to generate a separate $A_k$ and $B_k$ for each timestep. It remains unclear whether the proposed method requires time-wise parameterization in practice, and if not, how the initialization is reconciled with the time-independent parameterization commonly used in these models.", "methods_and_evaluation_criteria": "The primary goal of this paper is to enhance the robustness of state space models (SSMs) against input noise. The proposed method is an initialization scheme that provides a posterior estimate of the signal under Gaussian noise at the initialization stage. The derivation appears rigorous and well-grounded. However, when integrated into an SSM, the scheme seems to instantiate time-dependent parameters $A_k$ and $B_k$, which may diverge from the common practice of using time-invariant parameters $A$ and $B$ that are converted to $A_k$ and $B_k$ on the fly during training or inference. The experiments focus primarily on evaluating the robustness of the proposed approach under noise perturbations, which is well-aligned with the stated objective.", "theoretical_claims": "All derivations look correct to me.", "experimental_designs_or_analyses": "The experimental setup feels somewhat limited. Unlike many recent works on SSMs that evaluate on the Long Range Arena benchmark [1], this paper validates the proposed approach on a relatively small-scale dataset. Additionally, the comparisons are restricted to LSSL with HiPPO initialization, without including other competitive baselines. It would strengthen the empirical evaluation to include comparisons with more recent and widely adopted models such as S4 [2], DSS [3], or other state-of-the-art SSM variants.\n\n[1] Tay et al., Long Range Arena: A Benchmark for Efficient Transformers\n\n[2] Gu et al., Efficiently Modeling Long Sequences with Structured State Spaces\n\n[3] Gupta et al., Diagonal State Spaces are as Effective as Structured State Spaces", "relation_to_broader_scientific_literature": "The proposed method is highly relevant to recent advancements in sub-quadratic sequence modeling, which aim to improve the efficiency and effectiveness of long-sequence processing. The underlying dynamical system studied in this work forms the core foundation for this class of models.", "essential_references_not_discussed": "Two core works on the initialization and parameterization of SSMs appear to be missing:\n\n[1] Gupta et al., Diagonal State Spaces are as Effective as Structured State Spaces\n\n[2] Gu et al., How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections", "other_strengths_and_weaknesses": "Strengths:\n\n+ Analyzing SSMs through the lens of stochastic linear control theory provides a novel perspective and a powerful theoretical tool. I found Section 4.1 particularly insightful, in particular, the idea of regularizing the extrapolation behavior of SSMs using boundary conditions. The authors may also want to emphasize that this regularized formulation contributes directly to improving robustness against noise.\n\nWeaknesses:\n\n- The writing could benefit from greater clarity. For example, it took me some time to fully understand how $\\hat{f}_{\\le t}$ depends on $\\tau$. If I understand correctly,  $\\hat{f}$ should be expressed as two functions of $t$ and $\\tau$ separately rather than ambiguously as a function of one variable.\n\n- While the derived initialization is theoretically interesting, the empirical analysis could be strengthened. I recommend evaluating the method on more diverse tasks, such as language modeling, to demonstrate broader applicability. Additionally, it would be valuable to discuss how this initialization connects with recent advances in SSMs, such as Mamba [1], and whether it can be integrated into these newer architectures.\n\n[1] Gu et. al. Mamba: Linear-Time Sequence Modeling with Selective State Spaces", "other_comments_or_suggestions": "I caught one typo:\n\nLn 279-280, a GELU nonlinearity followed a linear layer -> a GELU nonlinearity following a linear layer", "questions_for_authors": "A question is whether the proposed initialization remains compatible with computationally efficient parameterizations, which serve as a core advantage of SSMs. The original HiPPO matrix has been shown to admit a normal-plus-low-rank decomposition, enabling fast computation. Subsequent work further simplifies this structure by approximating HiPPO with diagonal matrices. It remains unclear whether the UnHiPPO formulation retains or admits similar structural properties.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jIa6emAswP", "reviewer": "Reviewer_a4H9", "summary": "The paper proposes to extend high-order polynomial projection operators (HiPPO) that are used to initialise the dynamics of recent state space models. HiPPO theory is agnostic to measurement noise. The paper extends HiPPO operators to capture uncertainty arising from measurement noise.   Specifically, the paper proposes to infer the posterior pdf of the HiPPO coefficients conditional on the noisy observations. The parameters of the pdf are estimated using a Kalman filter.  The updated Kalman state (i.e., the mean of the posterior) itself is a first-order difference equation, where the transition of states from time step t-1 to t is captured by a transition matrix, and where the excitation is projected into state-space by an input vector. The transition matrix and input vector are therefore synonymous to the HiPPO matrices, but – owing to their Bayesian formulation – capture uncertainty in the measurement noise. The paper embeds the proposed matrices within LSSL (Gu & Dao 2021). The results of the resulting uncertainty-aware LSSL are compared against LSSL for a 10-class subset of the Speech Commands dataset. Results indicate improvements of up to approximately 1.5 percentage points  of the proposed model over the baseline.\n\n## Update after rebuttal:\nI thank the authors for their rebuttal. I particularly appreciate their effort in providing new results for different datasets to address my concerns. In my opinion, the idea of capturing uncertainty to improve robustness to noise is interesting and very promising. Unfortunately, similar to the results in the paper, the new results provided during the rebuttal also demonstrate only small improvements (< ~3%) compared to the LSSL baseline. In my opinion, more convincing results are required to evidence the paper's claim, i.e., that the proposed initialisation improves robustness against noise. I therefore maintain my score. Again, I think this is a promising approach and I would encourage the authors to undertake a more thorough experimental investigation with the aim to identify scenarios in which the proposed approach clearly outperforms LSSL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n**How was the noise added? How were the signals normalized?**\n\nWe follow LSSL implementation and the code from repository of the authors. After the audio files are loaded, signals are divided by 32k to be normalized (see [here)](https://github.com/state-spaces/s4/blob/e757cef57d89e448c413de7325ed5601aceaac13/src/dataloaders/datasets/sc.py#L407). After the noise is added, we apply z-score normalization.\n\n**What type of noise was added?**\n\nThe two noise sources in equations (7) and (8), namely $\\mathrm{d}\\mathbf{\\beta}$ and $\\varepsilon_{t_k}$, are Gaussian. In line with the theory, we sampled Gaussian noise with 0 mean and fixed variance. We have not experimented with other noise sources.\n\n**Computation of parameters**:\n> P. 6, paragraph following (29): “In contrast to LSSL where we can get the discretized dynamics at any t directly, for UnLSSL we compute them for all integer steps $t ∈ [t_{max}]$ and then select a subset.” – I assume that this is necessary since the uncertainty-aware transition matrix and input vector are obtained from the discrete-time update equation in (24). Please can you clarify this point?\n\nThat is correct. In the end, the parameters are tied to Kalman filtering in Equation (23), which is a linear-dynamical system (LDS) evolves with the transition matrix computed for each time step. We briefly mentioned this in the next sentence: \"In theory, we could jump to any directly in the Kalman update in Equation (23), but that would increase the uncertainty as if there was no data before, changing the dynamics.\"\n\n> Same paragraph: “Instead, we also compute all intermediate steps, which mirrors the more realistic setting where we also observe data at 1, 2, . . . , t − 1.” – Is the initialization of UnLSSL therefore data dependent?\n\nNo, it just means that we compute the parameters for the case where we observe data at each integer time step. Otherwise, the computed parameters would be computed as if we had not observed any data for longer stretches of time and therefore increase the uncertainty estimate and smoothing.\n\n**How were the noise experiments conducted? What type of noise was added? At what SNR?**\n\nFor each audio signal, we compute the standard deviation of the signal and add a random noise with times the computed standard deviation. We used 10 different values: 0.0, 1e-7, 1e-6, 3.16e-6, 1e-5, 1.77e-5, 3.16e-5, 5.62e-5, 1e-4. We observed values larger than 1e-4 repress most of the audio signals in SC10 and leaves nothing but noise for this dataset.\n\nSince we do not add the noise based on amplitudes, and instead based on the standard deviation, we do not have a constant SNR value. For example, an audio with constant signal would never be effected by noise even when is non-zero as the standard deviation of the signal itself is 0.\n\nWe run the LSSL model with HiPPO and UnHiPPO initialization on the same data and report averages of three different seeds. The complete configuration for the noise experiments is available in the supplementary material in `config/experiment/sc-raw-noise.yaml`.", "claims_and_evidence": "The main claim of the paper is that the performance and noise robustness of models initialised with HiPPO can be improved by capturing explicitly uncertainty in the measurement noise. The mathematical formulation is presented in a clear and concise manner. Convincing examples are provided in Section 5 to illustrate the benefits of the proposed method.", "methods_and_evaluation_criteria": "In my opinion, the proposed method provides an elegant and simple approach to embed measurement uncertainty in HiPPO. The dataset that was selected is appropriate to validate the proposed methodology, given the inherent challenges arising from speech in noise. While the results in the presented in the paper are sufficient for *validation*, the results showcase only small improvements in accuracy. To help identify the benefits (and potential limitations) of the proposed model, I would have expected a more thorough study involving different tasks and datasets for *evaluation*. For example, most SSMs since LSSL are evaluated on the LRA benchmark.", "theoretical_claims": "See “Claims and Evidence”", "experimental_designs_or_analyses": "In general, the experimental design provides the necessary information about the models to reproduce the studies. However, there is some information missing about the distortion of the speech signals:\n\n1)\tBased on the information provided in Section 6, I gather that the authors added noise to Speech Commands to investigate performance under varying conditions. Is this correct? If so, my main concern is that Speech Commands was recorded in varying acoustic conditions, i.e., the clips already include background noise at varying noise levels. The results in Figure 6 seem to indicate LSSL actually performs equally well as UnLSSL at low noise levels (i.e., when no additional noise was added to the noisy Speech Commands utterances). This seems contradictory to the main claim of the paper.\n2)\tIt is unclear how noise was added. Were the signals normalised to a particular level prior to adding noise? What is the target SNR? How is this handled considering that clips often contain long periods of silence / background noise with very short speech utterances?\n3)\tIt is unclear what type of noise was added. The type of noise will have a significant impact on the results. For example, white noise is relatively easy to remove. However, realistic noise sources are rarely white. Have the signals been distorted with realistic noise sources, such as speech-like noise or music? Which noise dataset was used?\n\nProvided that my understanding of the noise experiments is correct, I recommend to repeat the experiments with a dataset that is recorded in anechoic or studio conditions (e.g., TIMIT, VCTK). It would also be possible to estimate the noise levels in Speech Commands from periods of speech inactivity.", "relation_to_broader_scientific_literature": "A timely paper that fits will within recent work on the initialisation of SSMs, including, e.g., [1] below. In my opinion, the novel contribution of the paper is the extension of HiPPO to a framework that embeds uncertainty. \n\n[1] Liu & Li, “Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models”, ICLR 2025", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "*Strengths:* This is a very well written paper, that provides clear motivation and justification for assumptions, coherent and linear explanations, and helpful illustrative examples. I particularly appreciated the concise summary of HiPPO in Section 3. \n\n*Weaknesses:* The Figures in Section 6 provide nice visualisations of the general trend of accuracy and error curves. However, considering the small differences in performance between UnLSSL and LSSL, Tables are required to provide a precise comparison in performance. For example, in Figure 7 at \\sigma^2 = 10^{10}, I struggle to determine if the difference in accuracy is 1 percentage point, more or less.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. P. 6, paragraph following (29): “In contrast to LSSL where we can get the discretized dynamics at any t directly, for UnLSSL we compute them for all integer steps t ∈ [tmax] and then select a subset.” – I assume that this is necessary since the uncertainty-aware transition matrix and input vector are obtained from the discrete-time update equation in (24). Please can you clarify this point?\n2. Same paragraph: “Instead, we also compute all intermediate steps, which mirrors the more realistic setting where we also observe data at 1, 2, . . . , t − 1.” – Is the initialization of UnLSSL therefore data dependent?\n3. How were the noise experiments conducted? What type of noise was added? At what SNR?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VR8XH6Fvjt", "reviewer": "Reviewer_ZXjz", "summary": "The paper extends HiPPO by incorporating uncertainty-awareness, enhancing the robustness of state space models (SSMs) to noise.\n\n- The study is limited to SC10; testing on additional datasets and tasks would improve generalizability, a limitation acknowledged in the paper.  \n- The choice of regularization method is not thoroughly analyzed, though the proposed approach is discussed in detail. Exploring alternative strategies would strengthen the contribution.  \n- The $\\sigma^2$ hyperparameter is crucial but lacks a systematic selection method across different datasets, despite some analysis of its impact.  \n- Claims of negligible runtime cost are supported by experiments, but further evaluation on larger datasets would provide a clearer comparison with standard HiPPO.  \n- No structured formulation for the UnHiPPO matrix is provided, which may impact computational efficiency—an issue noted by the authors.  \n- The closed-form solution is preferred over the trapezoidal rule, and while its instability in UnHiPPO is noted, further explanation could clarify this choice.  \n- More analysis on the impact of different $\\Sigma$ structures on performance would be beneficial.  \n- Empirical comparisons with existing methods would help contextualize UnHiPPO’s advantages and limitations beyond theoretical discussion.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Please see Figure 9 in Appendix B for a visualization of the effect of different discretizations. It demonstrates the instability of some methods in both HiPPO and UnHiPPO and in particular the remarkable stability of the closed-form solution.", "claims_and_evidence": "See summary", "methods_and_evaluation_criteria": "See summary", "theoretical_claims": "See summary", "experimental_designs_or_analyses": "See summary", "relation_to_broader_scientific_literature": "See summary", "essential_references_not_discussed": "See summary", "other_strengths_and_weaknesses": "See summary", "other_comments_or_suggestions": "See summary", "questions_for_authors": "See summary", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LFLpBnPxgC", "reviewer": "Reviewer_NaeU", "summary": "This paper studies the HiPPO framework with noisy data. While the original HiPPO(-LegS) framework is based on projecting a function onto Legendre polynomials, it assumes that the input function is noise-free. The paper proposes an alternative way of formulating the HiPPO framework, called UnHiPPO, which is based on modeling the posterior of the states given the noisy measurements. The UnHiPPO framework introduces a new hyperparameter $\\sigma$ that is related to the noise level and ablation studies have been carried out to show its necessity.\n\n## update after rebuttal\nI thank the author(s) for their response and would maintain my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and careful reading.\n\n**Phrasing**\n\nWe will clarify Section 4 based on your feedback. The phrase \"observations do not exist in HiPPO\" refers to the fact that the data does not take the role of an observation in HiPPO, but rather of a control signal. We will update the section to make this clear. We have also adopted your clarification \"best possible compression of $f$ in the $L^2$ space\". \n\n**Effect of noise level**\n\nhttps://figshare.com/s/132436b00e91612513fd\n\nWe ran two experiments similar to Figure 7, but this time the noise level $\\rho^2$ is set to a lower and a higher value. The best results were obtained for $\\sigma^2 = 10^{10}$. However, as we increase the noise level $\\rho^2$, we notice that higher $\\sigma^2$ values perform comparatively well, as opposed to what happens in Figure 7, when $\\sigma^2 = 10^{14}$.\n\n**What is the effect of ignoring the time-variance of the true dynamics?**\n\nWe examined the effect of this in Figure 5 and the surrounding text in the paragraph \"Time-invariant Dynamics\" empirically, but we did not analyze it theoretically.\n\n**How can we select $\\sigma^2$?**\n\nAt the moment, our best strategy for choosing $\\sigma^2$ is empirical. The reason that $\\sigma^2$ cannot be chosen as the noise in the data comes down to $\\Sigma = I$ in the Kalman filter. Because this adds uncertainty to all degrees of the polynomial representation at each step, it also increases the uncertainty in the high-degree components. Even thought the dynamics are regularized, the highest degrees still grow quickly, so $\\sigma^2$ needs to be of a similar order of magnitude as $B_H^T P_k^{-}B_H$ in $s_k$ to have an effect. Choosing to the diagonal of $\\Sigma$ to fall quickly should be able to counteract this effect, though it is unclear how quickly it would need to fall exactly.\n\n**Can UnHiPPO be thought of as a regularization scheme, where $\\sigma^2$ controls the magnitude of regularization?**\n\nYes, as we have shown in Figure 4, $\\sigma^2$ controls directly how sensitive the system is to high-frequency components / noise in the data.\n\n**Can UnHiPPO be extended to HiPPO variants other than LegS?**\n\nWe decided to focus on the LegS variant of HiPPO, because it is the most relevant one in the literature. Extending the approach to other polynomial bases should be possible, because we only use properties of Legendre polynomials explicitly in Section 4.1 to derive the regularized HiPPO matrix. The same derivation should be theoretically possible in other bases, thought it might turn out to be either unnecessary if the basis behaves well under extrapolation or not produce a closed-form of $Q_i$.", "claims_and_evidence": "I find the claims made in the submission clear and convincing, modulo the first point I raised in the weaknesses section.", "methods_and_evaluation_criteria": "The proposed method makes sense for the problem of interest.", "theoretical_claims": "I checked the derivations of the formulas in the main text. I did not check the mathematics in the supplementary material.", "experimental_designs_or_analyses": "I agree with the soundness of the experimental designs. I do have some questions about the analyses, which I raise in the questions section.", "relation_to_broader_scientific_literature": "This paper studies the noise handling of a specific initialization scheme of LSSL/SSM. Both the objective (making the model noise-aware) and the class of models are of interest to many sequential and time series problems.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "I am ambivalent about my recommendation of this manuscript. On the one hand, the derivation of the UnHiPPO framework in this paper is beautiful and principled; on the other hand, I am not totally persuaded by the significance of the contribution to the application of LSSLs/SSMs, as there is a gap between the theory of the (Un)HiPPO framework and its practical success in such sequence models (as outlined in the questions section). In the end, I vote for weak acceptance, keeping in mind that such a gap is not an intrinsic flaw of the manuscript itself and that the UnHiPPO framework is potentially useful in fields other than LSSLs/SSMs or even deep learning.\n\nStrengths:\n   * The noisy-aware modeling is an important topic and has not been considered in the HiPPO literature.\n   * The derivation of the UnHiPPO framework in this paper is beautiful and principled.\n\nWeaknesses:\n   * While the UnHiPPO framework is derived from mathematical principles, there lacks a theoretical comparison of how UnHiPPO and HiPPO handle noises. That is, I would like to see a clear and convincing theorem that shows UnHiPPO is more robust to noises than HiPPO.\n   * The UnHiPPO state transitioning matrix $\\mathbf{A}$ is void of a simplified representation (e.g., diagonal-plus-low-rank or diagonal). This makes it harder to apply this initialization in the more efficient S4/S4D/S5 models.\n   * The presentation of the UnHiPPO framework can potentially be improved. See the comments section.", "other_comments_or_suggestions": "1. I suggest clearly stating the issue with a noisy function $f$ and the objective of designing a noise-aware system before starting to derive it.\n2. The manuscript is very well-written and easy to follow up to the end of section 3. In section 4, the discussion is a bit confusing and required several reads for me to figure out the core idea. That is, (7)-(8) requires more explanation of how it connects to (4). The UnHiPPO systems are not derived in a way that an ML researcher standardly imagine, where one seeks parameters of an autoregressive system with given inputs. The idea is more of \"input matching given the guessed states.\" This can be made more straightforward at the beginning of section 4 to avoid confusion.\n3. I cannot understand the following sentence in section 4: \"In contrast to what one might expect, the observations $y\\_{k\\_t}$ do not exist in HiPPO and, instead, the signal $f(t)$ corresponds to the control signal $u\\_t$.\" Please consider rephrase or further explain it.\n4. At the end of section 2, I suggest expanding \"best possible compression of $f$\" to \"best possible compression of $f$ in the $L^2$ space\" to make the statement more precise.", "questions_for_authors": "1. My main question is about the application of the HiPPO in LSSL. As mentioned earlier, I think there is a gap between theory and practice. The following subquestions may be useful to consider:\n   1. In Figure 7, you showed an ablation study that changing $\\sigma^2$ can vary the performance of the model. If you look at the model more carefully, then only a small proportion of $\\sigma^2$ leads to a better performance than the model initialized by HiPPO. This made me wonder: is it the noise-aware mechanism or something else that determines the performance of the model as the hyperparameter varies. By \"something else\", I mean things like the magnitudes of the eigenvalues of the discrete-time matrix $\\mathbf{A}$ that controls the system's memory (e.g., [1] and [2]), the imaginary parts of the eigenvalues of the continuous-time matrix $\\mathbf{A}$ that controls the frequency bias and the approximation-estimation (e.g., [3] and [4]), and the stability of the parameterization of the system (e.g., [5]). Can the authors show more empirical studies of whether the noise-awareness is really *the* thing? For example, it is useful to show Figure 7 multiple times given different $\\rho^2$ and see how the optimal $\\sigma^2$ changes.\n   2. As mentioned in the manuscript, what is used in theory (a time-varying system with a $1/t$ scaling) is different from what is used in practice (a time-invariant system that drops $1/t$). How does the theory account for this change and what is the noise-awareness of the time-invariant system?\n   3. HiPPO and UnHiPPO are only initializations. You have to train the systems eventually. How stable is UnHiPPO when trained?\n2. On page 6, you mentioned that \"An unfortunate side effect is that $\\sigma^2$ cannot be interpreted as the noise variance of the data\ndirectly.\" While I understand that one does not have $\\sigma^2 = \\rho^2$, I wonder if there is a connection between them. That is, if I know the noise level in the input, is there a principled way for me to select or scale the hyperparameter $\\sigma^2$?\n3. Can UnHiPPO be thought of as a regularization scheme, where $\\sigma^2$ controls the magnitude of regularization? This seems pretty clear in Figure 3-4. I wonder if some theory can be derived in this direction.\n4. The UnHiPPO discussed in this paper is only for HiPPO-LegS, while there are also many variants of HiPPO (e.g. HiPPO-LegT). Can the work in this paper be extended to those? To be clear, I do not look for detailed derivations of each of them. I only wonder if the analysis in this paper is generic or ad hoc to HiPPO-LegS.\n\n[1] Antonio Orvieto et al., Resurrecting recurrent neural networks for long sequences, International Conference on Machine Learning, 2023.\n\n[2] Naman Agarwal et al., Spectral state space models, arXiv preprint arXiv:2312.06837, 2023.\n\n[3] Annan Yu et al., Tuning frequency bias of state space models, International Conference on Learning Representations, 2025.\n\n[4] Fusheng Liu and Qianxiao Li, Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models, International Conference on Learning Representations, 2025.\n\n[5] Shida Wang and Qiaoxiao Li, Stablessm: Alleviating the curse of memory in state-space models through stable reparameterization, International Conference on Machine Learning, 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper has been well received by all the reviewers, who also point out that it is theoretically sound and easy to follow.

The main point of criticism has to do with the moderate improvement over the HiPPO variant, LSSLs/SSMs.
I agree with that concern, but at the same time I also see why the reviewers are focusing on the elegance of the proposed approach and the "principled formulation".

I would suggest the authors to take the time to incorporate the reviewers feedback, together with the additional experiment in the final version of the manuscript.

Congratulation on getting your paper accepted.
Decision: Accept (poster)