WaLRUS: Wavelets for Long range Representation Using State Space Methods
Authors: Hossein Babaei
Mel White
Sina Alemohammad
Richard Baraniuk
Venue: NeurIPS 2025 poster ()
Keywords: State-space models
SSM
HiPPO
Mamba
wavelet
safari
time series
sequence modeling
online function approximation
online memory
+
Primary area: general_machine_learning
TL;DR: We introduce WaLRUS, a wavelet-based SSM leveraging SaFARi for improved accuracy and stability in modeling non-smooth, transient signals, outperforming traditional HiPPO-based models.

Abstract: State-Space Models (SSMs) have proven to be powerful tools for online function approximation and for modeling long-range dependencies in sequential data. While recent methods such as HiPPO have demonstrated strong performance using a few polynomial bases, they remain limited by their reliance on closed-form solutions for specific, well-behaved bases. 
The SaFARi framework generalizes this approach, enabling the construction of SSMs from arbitrary frames, including non-orthogonal and redundant ones, thus allowing an infinite diversity of possible "species'' within the SSM family. In this paper, we introduce WaLRUS (Wavelets for Long-range Representation Using SSMs), a new species of SaFARi built from Daubechies wavelet frames. We instantiate two variants, scaled-Walrus and translated-Walrus, and show that their multiresolution and localized nature offers significant advantages in representing non-smooth and transient signals. We compare Walrus to HiPPO-based models and demonstrate improved accuracy, better numerical properties, and more efficient implementations for online function approximation tasks.

WaLRUS: Wavelets for Long-range Representation Using SSMs
Hossein Babaei Mel White Sina Alemohammad Richard G. Baraniuk Department of Electrical and Computer Engineering, Rice University (hb26,men mel. white, sa86, richb) edu
Abstract
State-Space Models (SSMs) have proven be powerful tools for modeling long- range dependencies in sequential data. While the recent method known as HiPPO has demonstrated strong performance, and formed the basis for machine learning models S4 and Mamba, it remains limited by its reliance on closed-form solutions for few specific, well-behaved bases. The SaFARi framework generalized this approach, enabling the construction of SSMs from arbitrary frames, including oothorthogonal and redundant ones, thus allowing an infinite diversity of possible "species" within the SSM family. In this paper. we introduce WaLRUS (Wavelets for Long-range Representation Using SSMs). We compare WaLRUS to HiPPO-bar based models, and demonstrate improved accuracy and more efficient implementations for online function approximation tasks.
1 Introduction
Sequential data foundational to many machine learning tasks, including natural language processing, speech recognition, and video understanding [133 These applications require models that can effectively process and retain information over long time horizons. A central challenge this setting the efficient representation of long-range dependencies in way that preserves essential features of the input signal for downstream tasks, while remaining computationally tractable during both training and inference 4
Recurrent neural networks (RNNs) are traditional choices for modeling sequential data, but struggle with long-term dependencies due to vanishing or exploding gradients during backpropagation through time 44 While gated variants like LSTMs 7 and GRUs [8] mitigate some issues, they require significant tuning and lack compatibility with parallel processing. hindering scalability.
State-space models (SSMs) offer linear and principled framework for encoding temporal information, and have re-emerged as a powerful alternative for online representation of sequential data [2H66]. By design, they enable the online computation of compressive representations that summarize the entire input history using fixed-size state vector, ensuring a constant memory footprint regardless of sequence length. major breakthrough came with HiPPO (High-order Polynomial Projection Operators), which reformulates online representation as function approximation problem using orthogonal polynomial bases (9). This approach underpins state-of the-ahtt art models like S4 and Mamba, enabling compact representations for long- range dependencies [I]
However, existing SSMs primarily rely Legendre and Fourier bases, which, although effective for smooth or periodic signals, struggle with non-stationary and localized features [9] 10). These challenges especially evident in domains such audio, geophysics, and biomedical signal processing, where rapid transitions and sparse structure are common.
To address this limitation, the SaFARi framework (State-Space Models for Frame-Agnostic Represen- tation) extends HiPPO to arbitrary frames, including orthoonho and redundant bases B 14 17
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

HiPPO (Legendre)
HiPPO (Fourier)
WaLRUS
Figure 1: An input signal comprising three random spikes is sequentially processed by SSMs and reconstructed after observing the entire input. Only the wavelet-based SSM constructed using WaLRUS can clearly distinguish adjacent spikes.
This generalization enables SSM construction from any frame via numerical solutions first-order linear differential equations, preserving HiPPO's memory efficiency and update capabilities without closed-form restrictions.
In this paper, we leverage the SaFARi method with wavelet frames to introduce a new model, WaLRUS (Wavelets for Long-range Representation Using SSMs). We derive our model using Daubechies wavelets with two variants: scaled-WaLRUS and translated-Wo MallaatedWWaRRUSSS designed for capturing non-smooth and localized features through compactly supported, multiiresslluion wavelet decompositions [18]. These properties allow WaLRUS to retain fine-grained signal details typically lost by polynomial-bee based models.
We also provide a comparative analysis of WaL and existing HiPPO variants (see Fig. D Empirical results demonstrate that the wavelet-based WaLRUS model consistently outperforms Legendre and Fourier-based HiPPO models in reconstruction accuracy, especially on signals with sharp transients. Furthermore, WaLRUS has been experimentally observed to be stably diagonalizable, which is the key enabler of efficient convolution-basel implementations and parallel computation 14
These results highlight the practical advantages of WaLRUS models, particularly in scenarios where signal structure varies across time and scale. By bridging multiscale signal analysis and online function approximation, WaLRUS opens new directions for modeling complex temporal phenomena across disciplines.
2
Background
Recent advances in machine learning, computer vision, and large language models have pushed the frontier of learning from long sequences of data. These applications demand models that (1) generate compact representations input streams, (2) preserve long-range dependencies, and (3) support efficient online updates.
Classical linear methods, such as the Fourier transform, offer compact representations in the fre- quency domain (12223 However, they are ill-suited for online processing: each new input requires recomputing the entire representation, making them inefficient for streaming data and limited in their memory horizon. Nonlinear models like recurrent neural networks (RNNs) and their gated variants (LSTMs, GRUs) have been more successful in sequence modeling, but they face well-known issues such vanishing/cxploding gradients and limited parallelization [4 g Moreover, their representations are task-spe,ific, and not casily repurposed across different settings.
To resolve these issues, the HiPPO framework [9] casts online function approximation a continuous projection of the input u(t) onto linear combination of the given basis functions G. At every time T. it produces a compressed state vector (() that satisfies the update rule:
dT [T) (T) + BiTy
(I)

---

Here, A(T) and B(T) are derived based on the choice of polynomial basis and measure j(tt, which defines how recent history is weighted. Two commonly used measures are:
1 7 1 (l)(1) 1
Mer (1r(1)
The translated measure Mer emphasizes recent history within sliding window of length 6, while the scaled measure Mse compresses the entire input history into a fixed-length representation.
Despite its strengths, HiPPO is restricted to only few bases (e.g., Legendre, Fourier), and deriving A()) and (1) in closed form only tractable specific basis-measure combinations.
SaFARi addressed this limitation by generalizing online function approximation to any arbitrary frame [7]. A frame (() is of elements {()) such that can reconstruct any input g(() by knowing the inner products (t(,), (t())). For given frame its complex conjugate Φ, and its dual T the scaled-SaFARi produces an SSM with and given by:
① oT (T) -
1 - Ad(T) + + Bu(T), A1,= + T. (rr B== (3)
(r')tr,,
or
1=
while the translated-SaFARi produces an SSM with the A and B given by:
OT (T) - - AC() + Bu(T), A. = -,0,,,0) + Ot t=
(t') B, -1 (4)
In the appendix, we provide some theoretical background on Eq. and Eq. from 17
Incremental update of SSMs: The differential equation in Eq. can be solved incrementally.
Following [9] we adopt the Generalized Bilinear Transform (GBT) 24 given by Eq. 5 for its
superior numerical accuracy in first order SSMs.
dt + At) = (1 + StaAA4+t) " - +
(5)
Diagonalization of A: Each GBT step involves matrix inversion and multiplication. If A(t) has time-independent eigenvectors (e.g., A(t) = (t) it can diagonalized as A(t) = VA(VVV- allowing a change of variables 7 = V-le and B B-1)((), yielding:
i c BB(t) Ōt
(6)
This reduces each update to elementwise operations, significantly lowering computational cost.
2.1 Wavelet Frames
Wavelet frames offer multiresolution analysis that captures both temporal and frequency character- istics signals, making them particularly effective for representing non-stationary or long-range dependent data (25. Initiated by [26] and formalized by 27), wavelet theory gained prominence with Ingrid Daubechies' seminal work (28), which introduced compactly supported orthogonal wavelets.
Since then, wavelets have played central role in modern signal processing (29).
Wavelet analysis decomposes a signal f(t) into dilations and translations of a mother wavelet (t)), enabling simultaneous localization in time and frequency. The discrete wavelet transform is
W k)) - f (()))a dt, (5,((t)
1 V
k
2-
Unlike global bases such as Fourier or polynomials, which struggle with localized discontinuities, wavelets provide sparse representations signals with singularities, such as jumps or spikes [18] 30
Their local support yields small coefficients in smooth regions and large coefficients near singularities, enabling efficient compression and accurate reconstruction. These properties make wavelet frames natural and powerful choice for time- time-frequency analysis in wide range of practical applications.

---

SSM-based based model
SSMs as Function Approximators (Can be constracted via SaFARi)
HiPPO Legendre
HiPPO Fourier
WaLRUSS
SSM Layer
Focus of this paper
Figure 2: A diagram of the relationships between HiPPO, SaFARi, WaLRUS (this work), and SSM- based models such as S4 and Mamba. The focus of this work is on the development a wavelet-baa based SSM in function approximation task, which could later be used as drop-in replacement for the SSM layer in a learned model.
Dushechies-22 22 Wavelet Frome
WaLRUS
Transiated
40
20
20
09
40
660
Figure 3: Left: Elements of Daubechies-22 22 wavelet frame, with father wavelet 6. mother wavelet v/. and two scales. Right: The scaled and translated A matrices for WaLRUS with N 21.
3 WaLRUS: Wavelet-based SSMs
Daubechies wavelets 18, 28] provide a particularly useful implementation of a SaFARi SSM. While there are different types commonly used wavelets, Daubechies wavelets are of particular interest in signal representation due to their maximal vanishing moments over compact support.
To construct the frame, the mother wavelets by a factor of two at each level. For each scale, different shifts along the x-axis introduced. Compressive wavelet frames are truncated versions of wavelet frames that contain only few of the coarser scales, and introduce overlapping shifts to keep the expressivity and satisfy the frame condition (See Mallat, (29). The interplay between the retained scales and the minimum required overlap to maintain the expressivity is extensively studied in the wavelet If there is excess overlap in shifts, the wavelet frame becomes redundant, and redundancy has ad advantages in expressivity and robustness to noise.
the usual dyadic scaling for multiresolution analysis; that is, scaling
literature [8 28 29
Figure left, gives visual representation of how construct such frame. The frame consists of shifted copies the father wavelet & at one scale, and shifted copies of a mother wavelet vb at different scales, with overlaps that introduce redundancy. Figure. right, shows the resulting A matrices for the scaled and translated WaLRUS
Some recent works [31 32) has conceptually connected the use of wavelets and SSM-based models (namely Mamba). These efforts are fundamentally distinet from ours in that they perform multi- resolution analysis on the input to the model only. No change is made to the standard Mamba SSM layer.
This work, on the other hand, is the first to challenge the ubiquity of the Legendre- based SSM, and present alternative wavelet- based machinery for the core of powerful models like Mamba. WaLRUS could be used as drop-in replacement for any existing SSM- -asedaaed framework. However, before simply substituting part in a larger system, we must first justify how and why different SSM can improve performance. This paper presents tool that stands alone an online function approximator, and also provides foundational building block for future integration SSM-based models.
Code to generate the matrices is available at the following repository:
walrus

---

3.1 Redundancy of the wavelet frame and size of the SSM
In contrast to orthonormal bases, redundant frames allow more than one way to represent the same signal. This redundancy arises from the non-trivial null space of the associated frame operator, meaning that multiple coefficient vectors can yield the same reconstructed function. Although the representation is not unique, it is still perfectly valid, and this flexibility offers several key advantages in signal processing. particular, redundancy can improve robustness to noise, enable better sparsity for certain signal classes, and enhance numerical stability in inverse problems 35
We distinguish between the total number of frame elements Nfull and the effective dimensionality Nef of the subspace where the meaningful representations reside. In other words, while the frame may consist of Nfull vectors, the actual information content lies in a lower-dimensional subspace of size Nen- This effective dimensionality can be quantified by analyzing the singular-vall value spectrum of the frame operator 29 33
For the WaLRUS SSMs described in this work, we first derive ANua using all elements of the redundant frame. We then diagonalize A and reduce it to a size of Neft. This ensures that different frame choices, whether orthonormal or redundant, can be fairly and meaningfully compared in terms of computational cost, memory usage, and approximation accuracy. The exact relationship between the wavelet frame and the resulting Noff of the A matrix depends not only on the overlap of the shifts in the frame, but also on the type (and order) of chosen wavelet, and number of scales. Determining the "optimal" overlap or New is application-specifi and an area for future research.
3.2 Computational complexity of WaLRUS
For sequence of length I., scaled-SaFARi has (N complexity due solving an N-dimensional linear system at each step, while Manslated-SaFARR can reuse matrix inverses, and thus has O(N*L) complexity, assuming
diagonalization [17]. When the state matrix A is diagonalizable, the complexity reduces to O(N and can further accelerate to O(L) with parallel processing on independent scalar SSMs.
We observe that each of the scaled and translated WaLRUS SSMs we implemented, regardless of dimension, were stably diagonalizable. Further research is required to determine whether Daubechies wavelets will always yield diagonalizable SSMs. Legendre- based SSMs, on the other hand, are not stably diagonalizable (9). Although [9] proposed fast sequential HiPPO-LegS update to achieve O(NT.) complexity, 117 showed that it cannot be parallelized to (L)) Moreover, no efficient sequential update exists for HiPPO-LegT, leaving Legendre-basse based SSMs at disadvantage during inference when sequential updates are needed.
As sequence length increases, step-wise updates become bottleneck, especially during training when the entire sequence is available upfront. This can be mitigated by using convolution kernels instead of sequential updates. Precomputing the convolution kernel and applying it via convolution accelerates computation, leveraging GPU-based parallelism to achieve O L) run-time complex- ity for diagonalizable SSMs. This optimization is feasible for both WaLRUS and Fourier-based SSMs. Although Legendre-ber based SSMs can attain similar asymptotic complexity through structured their nondiagonal nature prevents decoupling into N independent SSMs.
algorithms [0 12]
3.3 Representation errors in the translated WaL.US
Truncated representations SSMs inevitably introduce errors, as discarding higher-order compo- nents limits reconstruction fidelity [17. SaFARi only investigated these errors for scaled SSMs, leaving their approximation accuracy unquantified. Visualizing the convolution kernels generated by different SSMs offers some insight into the varying performance of different SSMs on the function approximation task. An "ideal" kernel would include a faithful representation for each element of the basis or frame from T 0 to T = W, where W is the window width, and it would contain no non-zero elements between W and L. However, certain bases generate kernels with warping issues, as illustrated in Fig.
The HiPPO-LegT kernel loses coefficients due to warping within the desired translating window (see areas B and C of Fig... For higher degrees of Legendre polynomials, the kernel exhibits an all-zero region at the beginning and end of the sliding window. This implies that high-frequency information in the input is not captured at the start or end of the sliding window, and the extent of this dead zone
5

---

Window
N coefs.
B
0
t = 2000
Figure 4: The kernel generated by HiPPO-LegT with window size W 2000 and representation size N 500. Three key non-ideal aspects of the kernel noticeable. A) poor localization due to substantial non-zero values outside W. B) coefficient loss from at bottom left of the kernel, and C) coefficient loss the bottom right of the kernel for E (1500, 2000).
Ideal Kernel
Actual Kernel
WaveT
Leg'T
FouT
Figure 5: Left: The ideal kernels, which yield zero representation error are shown for Translated- WaL (using the D22 wavelet), HiPPO-LegT, and HiPPO-FouT. Right: The corresponding kernels generated by the translated models are presented for comparison. WaveTT has superior localization within the window of interest compared HiPPO-LegTT and HiPPO-FouT.
increases with higher frequencies. The translated Fourier kernel primarily suffers from the opposite problem: substantial nonzero elements outside the kernel window indicate that LegT struggles to effectively "forget" historical input values. Thus contributions from input signals outside the sliding window appear as representation errors. LegT also has this problem, to lesser extent-see area of Fig. for closer view of the kernel.
visual inspection of Fig. revelss that the translated-aaLRUS kernel closely matches the idcalized version, whereas both FouT and LegT exhibit significant errors in their computed kernels. We emphasize that the issues observed with LegT and FouT arise from inherent limitations of the underlying SSMs themselves and are not due to the choice of input signal classes.
4 Experiments
The following section deploys the WaLRUS SSM on synthetic and real signals for the task of function approximation, comparing its performance with extant models in the literature. We will evaluate performance in MSE as well as their ability to track important signal features like singularities, and show that using WaLRUS can have an edge over the state-of-the-ar polynomial-basee SSMs.
To benchmark WaLRUS against state-of-the offthh--ae art SSMs, we implement two variants: Scaled- WaLRUS and Translated- WaLRUS, which will call WaveS and WaveT respectively, following HiPPO's convention. These models are compared against the top-peut por-forfonnng HiPPO-bace based SSMs. Further details the wavelet frames used in each experiment are provided in Appendix A.2.4 and code can be found at wt/https://githie com/ ttthbabb/walrree
We conduct experiments the following datasets:

---

1 LegS
i FouS WaveS
0.6
Dataset
LegS
FouS
WaveS
MSE
M4
0%
0.47%
99.53 %
0.4
Speech
4.25%
0%
95.75 %
Blocks
0%
0%
100 %
Spikes
0%
0%
100% %
Bumps 0%
0%
100 %
Piecepoly
1.00%
0%
99.00 %
M4
Speech
Blocks
Spikes
Bumps
PiecePoly
Figure Comparing reconstruction MSE be- tween WaveS, LegS, and FouS. Error bars repre- sent the first and third quantile of MSE. WaveS produces the lowest MSE in each dataset.
Table I: Percent tests where each basis had the lowest overall MSE.
M4 Forecasting Competition [36]: diverse collection of univariate time series with varying sampling frequencies taken from domains such as demographic, finance, industry, macro, micro, etc.
Speech Commands (37 : A dataset of one-second audio clips featuring spoken English words from a small vocabulary, designed for benchmarking lightweight audio recognition models.
Wavelet Benchmark Collection [38]: A synthetic benchmark featuring signals with distinct singular- ity structures, such as Bumps, Blocks, Spikes, and Piecewise Polynomials. We generate randomized examples from each class, with further details and visualizations provided in Appendix A.2.2
4.1 Comparisons among frames
We note that frame is universally optimal for all input classes, different classes of input signals exhibit varying decay rates in representation error. However, due to the superior localization and near-optimal eтror decay rate of wavelet frames, wavelet-based SSMs consistently show advantage over Legendre and Fourier-based SSMs across a range real-world and synthetic signals. These experiments position WaLRUS powerful and adaptable approach for scalable, high-fidelity signal representation.
4.1.1 Experimental setup
The performance of SSMs in online function approximation can be evaluated several ways. One metric the mean squared error (MSE) of the reconstructed signal compared the original. In the following sections, we compare the overall MSE for SSMs with a scaled measure, and the running MSE for SSMs with a translated measure.
Additionally, in some applications, the ability to capture specific features of a signal may be of greater interest than the overall MSE. As extreme case, consider signal that is nearly always zero, but contains a few isolated spikes. our estimated signal is all zero, then the MSE will small, but all of the information of interest has been lost.
In the experiments, we use equal SSM sizes Nett as described in Sec. 3.1
4.1.2 Function approximation with the scaled measure
In this experiment, we construct Ballud, HiPPO-LegS, and HiPPO-FouS with equal effective sizes (see Appendix A.2.4). Frame sizes are empirically selected to balance computational cost and approximation error across datasets.
Fig. shows the average MSE across random instances of multiple datasets. Not only is the average MSE lowest for WaLRUS for datasets, but even where there is high variance in the MSE, all
methods tend to keep the same relative performance. That is, the overlap in the error bars in Fig. b does not imply that the methods are indistinguishable; rather, for given instance of a dataset, the MSE across all three SSM types tends to shift together, maintaining the MSE ordering WaveS
7

---

Dataset:
Spikes
Bumps
Basis/Prame:
Legendre
Fourier
Wavelets
Legendre
Fourier
Wavelets
Peaks missed
2.5%
0.62%
0%
0.29%
0.30%
0%
False peaks
1.6%
1.6%
0.01 %
0.3%
1.9%
0%
Scaled
Instance-wist wins
76%
92.9%
100 %
97.1%
96.9%
100%
Relative amplitude
16.2%
11.8%
5.5%
12.4%
16.2%
6,5%
Average displacement
18.8
32.0
10.0
12.7
33.7
7.1
Peaks missed
6.4%
13.0%
0.27%
1.12%
29.76%
0.08 %
False peaks
1.1%
0.05% %
0.222%
0.43%%
0.28%
0.20 %
Translated
Instance-wise wins
36.9%
13.65%
99.95
85.1%
0.2%
100 %
Relative amplitude error
19.6%
28.4%
3.5%
6.9%
28.4%
2.5 %
Average displacement
6.0
5.4
4.3
5.5
5.8
4.8
Table Performance comparison of WaLRUS- -aaellesss, HiPPO-Legendre, and HiPPO-Fourier for peak detection with the translated measure. WaLRUS shows significant advantage in successfully remembering singularities over HiPPO SSMs.
LegS < FouS. To highlight this result, the percentage of instances where each SSM had the best performance also provided in Table
The representative power WaLRUS is attributed to its ability to minimize truncation and mixing errors by selecting frames that capture signal characteristics with higher fidelity. See [17 for further details.
4.1.3 Peak detection with the scaled measure
In this experiment, we aim to detect the locations of random spikes in input sequences using Scaled- WaL.USS FouS, and LegS, all constructed with equal sizes. We generate random spike sequences, add Gaussian noise (SNR = 0.001), and compute their representations with Daubechies wavelets, Legendre polynomials, and Fourier series. The reconstructed signals are transformed into wavelet coefficients, and spike locations are identified following the method in [3).
To evaluate performance, compare the relative amplitude and displacement of detected spikes with their ground truth (see 7). This process repeated for 1000 random sequences, each containing 10 spikes. Table summarizes the average number of undetected spikes for each SSM and the instance-wise win percentage, representing the number of instances where each SSM had fewer or equal misses peaks than the other SSMs. Note that these percentages do sum to 100, some instances result in identical spike detection across all models.
As shown in Table 2 WaveS misses significantly fewer spikes than FouS and LegS, with lower displacement errors and reduced amplitude loss. Figure [ illluttraess an example where WaLRUS successfully captures closely spaced spikes that are missed by LegS and FouS, demonstrating its superior time resolution.
True
Est.
amp.
False
Peuk
Th
err.
Not Counted
Missed
Peak
displaccment
Figure 7: Illustration of the metrics to evaluate performance of SSMs on different datasets in Table

---

M4
Speech
Blocks
10
MSE
-2
2.5
10-8
10
10
Bumps
Piecepoly
Spikes
100
MSE
10
1..4
1
Basis:
WaveT
LegT
FouT
Figure 8: For each dataset, the median and (0.4, 0.6) quantile of running reconstruction MSE across different instances is demonstrated in different colors for WaveT, LegT, and FouT. WaveT captures information in the input signals with a higher fidelity than LegT and FouT.
4.1.4 Function approximation with the translated measure
In this experiment, construct WaveT, LegT, and FouT SSMs, all with equal effective sizes (see Appendix A.2.4). The chosen effective sizes are smaller than those we used for the scaled measure since the translated window contains lower frequency content within each window, making it possible to reconstruct the signal with smaller frames. Then, for each instance of input signal, the reconstruction MSE each time step calculated and plotted in Fig.
For each input signal instance, we compute the running MSE at each time step, as shown in Fig. g This plot represents how the MSE evolves over time across multiple instances, providing comparison of running MSEs for each SSM. The results demonstrate that Translated- WaLRUS consistently achieves slightly better fidelity than LegT and significantly outperforms FouT across all datasets.
As discussed in Section 3.3 the reconstruction error stems from two main factors: (1) non-idealities in the translated SSM kernel, affecting its ability to retain relevant information within the window while effectively forgetting data outside (see Fig. 4 and (2) the extent which these fundamental non-idealities are activated by the input signal. For example, signals with large regions of zero values are less impacted by kernel inaccuracies, as the weights outside the kernel contribute minimally to reconstruction.
WaveT achieves modest, and in some cases negligible MSE improvement over LegT (e.g.. M4 and Blocks). However, the kernel- based limitations highlighted in Section 3.3 may have a more pronounced effect on longer sequences or different datasets.
4.1.5 Peak detection with the translated measure
In this experiment, we evaluate the ability of Wave'T, FouT, and LegT to retain information about singularities in signals, following the setup in Section but with translated SSM. We generate 2,000 random sequences, each containing 20 spikes. The average number of undetected spikes for each SSM, along with instance-wise win percentages, is reported in Table As the scaled measure experiment, the percentages do not sum to 100 due to ties across SSMs. Table shows that WaveT consistently outperforms FouT and LegT, with fewer missed peaks, reduced displacement, and less amplitude loss.

---

Limitations
In this work we have implemented only one type of wavelet (Daubechies-22) as our purpose is to introduce practical and theoretical reasons replace polynomial SSMs with wavelet SSMs. Other wavelets (biorthogonal, coiflets, Morlets, etc.) could also be used, with some caveats. First, we require differentiable frame 177. nondifferentiable wavelets like Haar wavelets or other lower-order Daubechies and Coiflets cannot be used with this method. Second, the redundancy of the frame (and the resulting Nef of the A matrix) depends the shape of the wavelet's function and the chosen shifts and scales this function. Other wavelet types, and other choices of shift and scale, may exhibit better worse performance and dimensionality reduction, and this is an important question for future work.
Additionally, we emphasize that the choice of frame is pmplication-dependeenn If the signal is known be smooth and periodic, ravelet-based SSM is not likely to outperform a Fourier-based SSM, for example. The introduction of WaLRUS is not intended to be a one-size-fits-al model, but rather appllala tool that combines compressive online aunction-aprooximation SSMs with the expressive power of wavelets.
6 Conclusions
We have demonstrated in this paper how function approximation with SSMs, initially proposed by [9] and subsequently extended to general frames, can be improved using wavelet-based SSMs. SSMs constructed with wavelet frames can provide higher fidelity in signal reconstruction than the state-of-the-ar Legendre and Fourier-based SSMs over both scaled and translated measures. Future work will explore alternate wavelet families, and the trade-offss in effective size, frequency space coverage, and representation capabilities of different frames.
Moreover, since the Legendre-based HiPPO SSM forms the core of S4 and Mamba, and WaLRUS provides a drop-in replacement for HiPPO, WaLRUS could be used initialize SSM-based machine learning models- potenntilly providing more efficient training. As AI becomes ubiquitous, and the demand for computation explodes, smarter and more task-tailored MI. architectures can help mitigate the strain on energy and environmental resources.
Acknowledgments
Special thanks to T. Mitchell Roddenberry for fruitful conversations and insights. This work was supported by NSF grants CCF- 191 1094, 11S-1838177, and IIS- 1730574; ONR grants N00014- 18-12571, 100001420-111 1-2334, N00014- 1-047, and MURI N00014-20-1-27877 AFOSR grant FA9550-22- 109550-22-100666 10060; DOE grant DE-SC0020345; and ONR grant N00014-18- 1-2047. Additional support was provided by Vannevar Bush Faculty Fellowship, the Rice Academy of Fellows, and Rice University and Houston Methodist 2024 Seed Grant Program.
References
[1] Nikola Zubić, Mathias Gehrig, and Davide Scaramuzza. State space models for event cameras. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
[2] Sina Alemohammad, Hossein Babaei, Randall Balestriero, Matt Y. Cheung, Ahmed Imtiaz Hu- mayun, Daniel LeJeune, Naiming Liu, Lorenzo Luzi, Jasper Tan, Zichao Wang, and Richard G. Baraniuk. Wearing a mask: Compressed representations of variable-length sequences using recurrent neural tangent kernels. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2950-2954, 2021.
[3] Eric Nguyen, Karan Goel, Albert Gu. Gordon Downs, Preey Shah. Tri Dao. Stephen Baccus, and Christopher Ré. S4ND: Modeling images and videos as multidimensional signals with state spaces. In Advances in Neural Information Processing Systems, 2022.
[4] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9735:1735-1880 1997.
10

---

[5] Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):179-2) 1990.
[6] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE Transac- tions on Signal Processing, 45((1):2673-2681, 1997.
[7] Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional LSTM and other neural network architectures, Neural Networks, 18(5):022-610, 2005.
[8] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dæmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. arXiv preprint arXiv: 1406. 1078. 2014.
[9] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. HiPPO: Recurrent memory with optimal polynomial projections. In Advances in Neural Information Processing Systems, 2020.
[10] Albert Gu, Karan Goel, and Christopher Re Efficiently modeling long sequences with structured state spaces. In International Conference on Leurning Representations, 2022.
[11] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00772, 2023.
[12] Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Re. How to train your HiPPO: State space models with generalized orthogonal basis projections. In International Conference on Learning Representations, 2023.
[13] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. In Advances in Neural Information Processing Systems, 2024.
[14] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In Advances in Neural Information Processing Systems, 2022.
[15] Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In International Conference Learning Representations, 2023.
[16] Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. In International Conference on Learning Representations, 2023.
[17] Hossein Babaei, Mel White, Sina Alemohammad, and Richard G Baraniuk. SaFARi: State-space models for frame-agnostic representation. arXiv preprint arxiv:2505.00977, 2025.
[18] Ingrid Daubechies. Ten lectures on wavelets. SIAM Press, 1992.
[19] Alan V Oppenheim. Discrete- Signal Processing. Pearson, 1999.
[20] Agostino Abbate, Casimer DeCusatis, and Pankaj K Das. Wavelets and Subbands: Fundamen- tals and Applications. Springer, 2012.
[21] George EP Box, Gwilym M Jenkins, Gregory Reinsel, and Greta M Ljung. Time Series Analysis: Forecasting and Control. John Wiley & Sons, 2015.
[22] John G Proakis. Digital Signal Processing: Principles, Algorithms, and Applications. Pearson, 2001.
[23] Paolo Prandoni and Martin Vetterli. Signal Processing for Communications EPFL Press, 2008.
[24] Guofeng Zhang, Tongwen Chen, and Xiang Chen. Performance recovery in digital implemen- tation of analogue systems. SIAM Journal on Control and Optimization, 4526):200-2223. 2007.
[25] Patrice Abry, Patrick Flandrin, and Murad S. Taqqu. Self-similarity and long-range dependence through the wavelet lens. In Paul Doukhan, George Oppenheim, and Murad Taqqu, editors, Theory and Applications of Long-Rang Dependence, pages 527-556. Birkhäuser, 2003.

---

[26] Alfred Haar. Zur Theorie der Orthogonalen Funktionensystemm. PhD thesis, University of Göttingen, 1909.
[27] A. Grossmann and Morlet. Decomposition of Hardy functions into square integrable wavelets of constant shape. SIAM Journal Mathematical Analysis, 15(4):723-736, 1984.
[28] Ingrid Daubechies. Orthonormal bases of compactly supported wavelets. Communications Pure and Applied Mathematics, 41(7):909-996, 1988.
[29] Stéphane Mallat. A Wavelet Tour of Signal Processing: The Sparse Way. Academic Press, 3rd edition, 2008.
[30] Stephane Mallat and Wen Liang Hwang. Singularity detection and processing with wavelets. IEEE Transactions on Information Theory, 38(2):617-6433 1992.
[31] Tianpei Zhang, Yiming Zhu, Jufeng Zhao, Guangmang Cui, and Yuchen Zheng. Exploring state space model wavelet domain: An infrared and visible image fusion network via wavelet transform and state space model, 2025.
[32] Wenbin Zou, Hongxia Gao, Weipeng Yang, and Tongtong Liu. Wave-mamba: Wavelet state space model for ultra-high-definttiee low-light image enhancement. In Proceedings of the 32nd ACM International Conference Multimedia, page 1534-1543, Association for Computing Machinery, 2024.
[33] O Christensen. An Introduction to Frames and Riesz Bases. Birkhauser, 2003.
[34] Karlheinz Gröchenig. Foundations of Time-Frequency Analysis. Springer, 2001.
[35] Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations
over learned dictionaries. IEEE Transactions on Image processing, 15(12)33736-3745, 2006.
[36] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54 74, 2020.
[37] Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint 1804v 03200, 2018.
[38] David I. Donoho and Iain M Johnstone. Ideal spatial adaptation by wavelet shrinkage. Biometrika, (1))):425-44) 455, 1994.
12

---

A Appendix
A.1 SaFARi derivation for arbitrary frame
Where HiPPO 9 provided closed-form solutions to construct and B for few polynomial bases, SaFARi [17] introduced method build A and B from any arbitrary frame. The derivations
provided below follow [17], and are given here as convenient reference for the reader.
Take signal f and frame vb. To get vector of weights representing signal on basis, the inner product:
C. f f())(()ftt
(A.1)
So at some time T, we scale the magnitude of f(t) and stretch the basis to match the length of f:
Gn (T) ( ( f(1) Lo
- t0 T to
Φ
dt
(A.2)
T T
We are actually interested in the change in c. We will take the partial derivative with respect to T, since the coefficients update at each new time T. Call the start time fy: this is 0 for the scaling case, and to varies with the windowed case. If we call the size of the window 8 then to = B. The derivation below will be generic version, then we will separate the two cases.
- 1 f(t)
d
1 T T-
lo to
rT (T)
(T)
dt
(A.3)
JI J
tg
We note that this is the partial derivative of an integral bounded by two variables, Thus we call on Leibniz' integration rule and find:
1
(n()) = S(T) ( 40
Φ
(T)
f(40)
(00) 8T
(40)
8T
T to
8
to
di (A.4)
1(1) 5T
to
T 10
to
h(t)
Some manipulation of the htt) term yields:
8(to)
t - to
-(1
5(to)
t t to
t- to
)
h(t)) = T to
Φ
8T
T
to
- to
8T
T to
T
1
)
5(fo)
L - t0
8T
Φ
T - to
T
T
(A.5)
Our h(t) term now has the derivative of our basis (d') it, but we'd like to be able to combine combine terms with Φ. Therefore we can make a mapping from Øi - & using the dual, Ø:
Φ' t t-to to T T- 10
0 t
to 40
to ) Φ T to
(A.6)
10
p
Likewise:
(t to
t tu T to
to
to
(t- 1o)o'
(T to
=
(t to) 0¹
T 60
Φ
T - to
(A.7)
13

---

This lets us do another simplification of h(t), and group all the functions of ø. Let's also call to save some space.
T to )
1 to
8(10) 8T
5(1a) 5T
(1
8(4)) 8T
]
((1) = A%
0 P:
(A.8)
P is not function of t, so it can be moved outside the integral. For the measures we are looking at, at is always constant with respect to t - is either O or 1. We can substitute then group follows:
Now we can return to Eq. A.4
d dT
T -
1
[(MTAT)) ((1o)((0)
57 (1o)
8
Ca (T)
1
5(to)
(1-
5(to)
Pt - (1- 5(tu)
}
T
to
P
f(t)
dt
T
to
8T
8T
8T
T to
T to
c(T)
(A.9)
Noting that the final term in this equation contains Eq.
A.2
we can simplify further:
((T))(1) - ((1))((0)) 8(to)
dT (T)
= T to
8T
+
(A.10)
1 T tu a(T)
[ [-((1)) P- (1 5(l0)
8(1)) Pe - (1 5T }
8T
8T
Unfortunately, we still have a term fto) that we don't have access to; this is the value of the function the start of our window. But we have not stored this value; that would defeat the point of an online update in the first place. Instead, we will approximate it based on our current coefficient vector and our known basis.
(p.
f = (5.)) c)
f(t) (
We now have update rule for that depends only the frame o, the current value of c(T', and the new information from the signal, f(T):
1 T to
((T) - T(0)) 5(to)
dT
8T
(A.11)
T - to )
1
((T))
8(tu) 8T
p
P+ (1-
5(to)
Pi+ + (1- 5(to)
8T
8T
A.1.1 The scaled case
In the case scaling, to = 0 and
8 (to) =
57
dT (T) =(t) S(T) - (T)
5(10)
(A.12)
AT
- -( ( c(T)
6(ta))
6(ta))
8(tor)
P
1
P.+
1
(A.13)
/T
AT
/
dT (T) = (1)
-1
((T)
c(T)(P:+1) +1)
(A.14)
T
T
14

---

The A matrix acts on the coefficient vector and B acts on the current input, f(T). Expressed in matrix notation:
d
1
CT(
T ( S(1)
(A.15)
dT
T
Equivalently,
d
1 T
Cn (T)
c ( +
+1
S(T)
(A.16)
dT
T
B
A
A.1.2 The translated case
Now T' - to f where 0 is the window size, and 6 8T (to) = 1. Following the same procedure as the previous section:
J7 (T = -(1) ((T)(1)) - - ())) + +P]
d
(A.17)
(T) = 1
dT cx())
(P + (()) c(T) + f(I)
(A.18)
J (T)
1
(0)0(0) c(T) 6(1) f(T)
(A.19)
A
A.2
Experiments
A.2.1
Datasets
In this paper, we conducted our experiments on these datasets:
M4 forecasting competition: The M4 forecasting competition dataset [36] consists of 100,000 univariate time series from six domains: demographic, finance, industry, macro, micro, and other. The data covers various frequencies (hourly, daily, weekly, monthly, quarterly, yearly) and originates from sources like censuses, financial markets, industrial reports, and economic surveys. It is designed to benchmark forecasting models across diverse real-world applications, accommodating different horizons and data lengths. We test on 3,000 random instances.
Speech commands:
The speech commands dataset [37] is set of 400 audio files, each containing a single spoken English word or background noise with about one second duration. These words are from small set of commands, and are spoken by a variety of different speakers. This data set designed to help train simple machine learning models.
Wavelet benchmark collection: Donoho 38) introduced a collection of popular wavelet benchmark signals, each designed to capture different types of singularities. This benchmark includes well- known signals such as Bumps, Blocks, Spikes, and Piecewise Polynomial. Following this model, we synthesize random signals belonging to the classes of bumps, blocks, spikes, and piecewise polynomials. Details and examples of these signals can be found in Appendix A.2.2
A.2.2 Wavelet Benchmark Collection
Donoho [38] introduced a collection of popular wavelet benchmark signals, each designed to capture different types of singularities. This benchmark includes well-known signals such Bumps, Blocks, Spikes, and Piecewise Polynomial.
Following this model, synthesize random signals belonging to the classes of bumps, blocks, spikes, and piecewise polynomials in our experiments to compare the fidelity of DaubS to legS and fouS, and also to compare the fidelity of DaubT to LegT and FouT.
Figure demonstrates a random instance from each of of the classes of the signals that we have in wavelet benchmark collection.
15

---

2.0
10-
0.5
250
Paccoply
Spilkes
12
0.20
1.0
0.15
0.8
0.10
0.05
0.
00
0 50
100 150
200
250 300
50
300
150
200
250
300
Figure Instances of different signal types in the wavelet benchmark collection. Top Left: Blocks is piecewise constant signal with random-hight hight sharp jumps placed randomly. Top Right: Bumps is collection of random pulses where each pulse contains cusp. Bottom Left: Piecepoly is piecewise polynomial signal with discontinuity in the transition between different polynomial parts. Bottom Right: Spikes collection of rectangular pulses placed randomly with random positive hieght.
A.2.3 Description of metrics for 'Spikes' and 'Bumps' experiments
Peaks Missed The number of true peaks signal is Ntp- and the number of detected peaks (that is, where the estimated signal surpasses amplitude threshold T Thamp), is Ndp- Napltp the number of detected peaks where a true peak is also within displacement threshold (Thase) of the detected peak.
-(1
Napltp
Peaks Missed
100%
Nep
False Peaks The metric False Peaks is calculated as the percentage of detected peaks that occurred when there was true peak within the displacement threshold. The number of detected peaks when there was no true peak is represented by N Ndp
Ndpli
False Peaks
x 100%
Ndp
Instance-wise Wins In each K time-series instances Each SSM m gets the instance win over other SSM models if captures more true peaks than the other models.
Instance-wise Wins
wk 100%
K
1, if Peaks Missed,, & Peaks Missed
Wy
0, Ow
cases where multiple models achieve same maximum, each tied model receives credit for that time series instance. As result, the sum of instance-wise wins for different SSMs may exceed 1.00.
Relative Amplitude Error The relative amplitude error is calculated as the average percent error in the estimated amplitude of detected peaks, including false peaks.
16

---

Maplep
Atp.
Adplepaa
Relative Amplitude Error
x 100%
Nap
Aep.n
Average Displacement The location of detected peak where true peak was within displacement threshold is given by Xdpttp- The location of the true peak is denoted as Xtp-
Nay
Average Displacement
Xip.n
Xdpltn.nl
Ndv
A.2.4 Wavelet frames used for each experiment
Unlike HiPPO-based SSMs, which fully characterized by their state size N, WaLRUS employs redundant wavelet frames that require additional parameters for identification. Once the wavelet frame is defined, the SaFARi framework constructs the unique A, B matrices corresponding that frame. The key parameters for specifying redundant wavelet frame in WaLRUS are as follows:
Wavelet Function:
Wavelet frames are built from a mother wavelet and a father wavelet, which capture high-frequency details and low-frequency approximations, respectively. Dif- ferent families such as Daubechies, Morlet, Symlet, and Coifflet provide varied wavelet functions. For this work,
the D>2 wavelet from the Daubechies family.
L (Frame Length): This represents the length of the wavelet frame. Increasing L increases numerical accuracy in the calculation of the A and matrices at the cost of additional computation time. However, this initial computation need only be performed once, is best to choose a large L. For the experiments in this work, we set L 19
Scale min and Ner: The minimum scale sets the smallest feature of the signal that can be represented by the frame. This parameter should be chosen based on knowledge about the signal of interest and its component frequencies. Note that the size of the smallest feature relative to the length of the signal under consideration, so this value may differ under scaling and translating measures.
For wavelets, scale min also controls the effective rank, Nan. Each new lower scale intro- duces factor of two in the effective rank of the frame, owing to the additional shifted elements each scale. Fig. showss two scales, where there are father wavelets (oo) and 3 coarse-scale mother wavelets (1h1). The next scale introduces 6 scaled and shifted mother wavelets (4/2), the next would include 12, and so on. Table also illustrates this pattern, Netr of 26, scale min of corresponding Net of 27, and so on, with some margin of error for numerical accuracy and truncation.
with scale min of 0 corresponding
Our code includes another variable, scale max. Since smaller scales can also combine to represent larger scales, scale max in fact has no impact on Nefr (see [29] for further information). Fig. [0 demonstrates on example implementation that varying scale max does not impact the size of Net- It also easily shown that varying scale max results same diagonalized A; numerical accuracy in the calculation of A, however. We do not include scale max in Table 3, but we do provide it in our code with each experiment for reproducibility.
code supplement. Adding coarser scales can help improve
520
480
460
-3
2
Scule Max
Figure 10: Effective Rank of WaLRUS A matrix with Scale Min=-3, shift=0.01
Shift: scale i, 2 overlapping shifts are applied to the wavelets, where < m S is shift constant. Setting 1 corresponds to dyadic shifts. As our wavelet frames typically
17

---

Experiment
Basis/Measure
scale min
shift
Neff
WaveS
-3
0.01
501
Scaled M4
LegS
500
FouS
500
WaveS
-5
0.01
1995
Scaled Speech
LegS
1995
FouS
1995
WaveS
-3
0.01
501
Scaled synthetic
LegS
500
FouS
500
WaveS
0.01
65
Scaled peak detection
LegS
65
FouS
65
WaveT
0.01
128
Translated M4
LegT
128
FouT
128
WaveT
33
0.0025
500
Translated Speech
LegT
500
FouT
500
WaveT
-1
0.01
128
Translated synthetic
LegT
128
FouT
128
WaveT
0
0.01
65
Translated peak detection
LegT
65
FouT
Table 3: Parameters for the redundant wavelet frame used by WaLRUS in different experiments. All of the above experiment share the parameters -= 2 and rcond 0.01.
only contain few dilation levels, using m can mean that the constructed of vectors no longer satisfies the frame condition, and is lossy. We choose small value (0.01 for most experiments), and tune this as needed.
rcond: This parameter controls the numerical stability of the pseudo-inverse calculation for the dual frame. Singular values smaller than rcond x m are discarded during the inversion process maintain numerical stability.
Note that all the above parameters solely identify the redundant wavelet frame, and that WaLRUS does not introduce any new parameters. Table Yummmiiss the settings for all experiments, alongside the SSM sizes for HiPPO-Legendre and HiPPO-Fourier.
A.2.5 Computational resources
Within the scope of this paper, no networks were trained and no parameters were learned. Only CPU resources utilized, but speed could be improved with parallel resources GPU. Using WaL to find representation has two different stages:
Pre-computing:
Computing SSM matrices and diagonalizing them. This step can be computationally intensive, but need only be calculated once.
Computation: Using SSM A matrices to find representations of signals.
For all our experiments except Scaled-Speech, the pre-computing stage takes less than 10 minutes. For scaled-speech, the pre-compute time is on the order of hours. Once the matrices are computed and stored, run time is the same for all experiments.
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our abstract and introduction states that we introduce the use of wavelets in state-space models for online function representation, and show how these can outper- form Hate-of-thhearr aht polynomial models for certain data types. Section 3 describes the construction of wavelet-based SSMs, and section 4 experimentally supports our performance claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 5 describes limitations, both in terms of what have implemented in this work, as well as limitations in the use of our method.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All necessary theoretical background is given in Sec. 2 and full support for our results are sections 3-4.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The Experiments section thoroughly describes what metrics were tested and how they were evaluated, as well as the publicly available datasets used. Scripts replicate the experimental results are available at htthtps:// batttps://gthutt com/ wwweebaahaaaaaaeeeeeeeeeeeeeaaae walrus
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
20

---

reproducible, they some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code and data are available at http:://oo...oo//h/////eeeee (Bc33b777622665414448888800
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https https///ooSSuubiss...o
/ //nipscco wwtppcpcccggpcicecccccooocccccce
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the required information both the datasets, and the exact experimen- tal setting required to recreate the wavelet frame, are provided in the Appendix. This information can also be found in our code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars and quantiles are provided in figures and 7, and explanations of their source in the text and captions of the figures. Since MSE is not normally distributed, we chose to use quantiles and percentiles to reflect the distribution more accurately. We also provide tables and describe additional nuances of the comparison data.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Since work did not involve any training, GPU computation was
necessary. More discussion is available in the Appendix (Sec.A.2.. A.2.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have conducted this research with integrity and reported our findings with honesty. The link to the Code of Ethics provided is broken, and have instead consulted this provisional copy of the document: https: /ttts://oppnreviie net net/forum?id- ZVoy8kAFKPT
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
22

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This work is a basic mathematical result that does not have a targeted end use. We do note in our conclusion that improved function approximators, like the one we present here, can reduce the computational resources required for training certain types of neural networks - resources that have recently become a major environmental concern.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This is foundational and theoretical work that is primarily mathematical in nature: a compressive online approximation of time-series signals over a wavelet frame. The potential use cases for such a tool are similar in scope to that of a Fourier Transform; that is, is too broad to responsibly hypothesize specific use cases or create guidelines.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
23

---

Answer: [Yes]
Justification: the M4 dataset does not have required license: https::/paperswittie com/dataset/mgq The SpeechCommands dataset has CC BY license, allowing for unrestricted use, with attribution to the author: hettps:/huugingeee cttifaacecco/oceao co/ co/datasets/ commaccaccccccccdssssssssaaa google/ sppeeeh, commands The four other data types we test on are generated by code
that is made available with this paper, and based on [38
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwpssspppeeseeeeeeeeeseepp thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
An implementation WaLRUS is provided with code.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
There were human subjects in this theoretical work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
24

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: There were no study participants this theoretical work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: have used LLMs only to assist writing and polishing the grammar.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy for what should or should be described.
25


Reviews:
[{"review_id": "pucsEOaXa3", "reviewer": "Reviewer_UUkH", "summary": "The paper proposes WaLRUS, a wavelet-based formulation of SSMs. For this the authors start from the SaFARi framework and leverage Daubechies wavelet frames. Experimental evaluations of WaLRUS on online function approx tasks show improved performance compared to two baselines: HiPPO-LegT and HiPPO-FouT.\nThe paper argues that the wavelet-based kernel has better localization, and is better at resolving sparse signal (e.g two close by signal peaks as shown in Fig. 1).\n\nDisclaimer: I'm not an expert either in SSMs nor in wavelets. Thus, I may not be familiar with some shared assumptions in these subfields. However, my review may still add value as a sample of the potential attendant distribution (so for exposure, understandability etc).", "questions": "1) Why does the paper omit a direct comparison with SaFARi, even though WaLRUS is built on it? Could the reported gains simply stem from inheriting SaFARi’s advantages rather than from the wavelet basis itself?\n\n2) Do the current benchmarks really demonstrate WaLRUS’s benefits over existing SSMs (and maybe even Transformers)? I believe an experiment on more community-relevant settings a small-scale long-context language-model or long biological sequences could provide a clearer picture.\n\n3) Given earlier papers that combine wavelets with SSMs (see [1, 2] above), what specific gap does WaLRUS fill, and how does its contribution differ from or improve upon these prior approaches? Would these need to be shown experimentally as well?\n\n4) How reusable is the proposed Walrus code? I.e. from reading the paper I'm not sure how easy it will be to try this approach out as drop-in replacement in existing SSM frameworks.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Walrus seems to be quite efficient: under certain conditions the runtime over a sequence of length L is claimed to be O(L) using parallel processing.\n- Overall, I think it's a great idea to propagate insights from signal processing / numerical analysis like wavelets through to recent ML techniques like SSM.\n- To my understanding, the wavelet representations add the nice property of compact support - whereas many other bases would lead to non-zero values also outside of the window of interest.\n\nWeaknesses:\nOn a high level, I'm confused about three main parts:\n1) The narrative of the abstract/intro is: There is HiPPO --> SaFARi --> WaLRUS (based on SaFARI). But the experiments compare WaLRUS only against two HIPPO variants. Why not compare against the direct baseline SaFARi? (as any benefit could be due to the advantage of building on SaFARi otherwise.\n2) The experimental evaluations are on tasks that I personally do not find super relevant to showcase the benefit of a new SSM variant:\n- i.e. evaluating on some basic time series dataset and peak detection may be fair - but I expect the community would be more curious to see if walrus can offer benefits in the typical SSM vs transformer settings (e.g. long-range / long-context language modelling, long biological sequences etc.). Why not do a tiny-scale language model experiment by training small models (e.g. GPT-2 scale) regular transformer vs SSM (best prior) vs SSM (Walrus).\n3) There seem to be few works [1], [2], that do Wavelets with SSM, I haven't checked them in careful detail, but it begs the quesiton whether Wavelet + SSM is novel - or what are the gaps that this works fills exactly? \n\nSmaller points:\n- Fig 1: only shows HiPPO (two version) against WalRus - but not SaFARi baselines.\n- It would be nice to provide a bit more exposure/explanation to Eq 3,4 which use notation that not everbody in this ML venue may be familiar with.\n- Purely aesthetic: but I would remove the vertical lines from Tab 2 \n\nReferences:\n[1] https://arxiv.org/pdf/2408.01276\n[2] https://arxiv.org/pdf/2503.18378", "was_revised": true, "final_justification": "Rebuttal adressed my concerns. Willing to raise my score by 1; but still not a super strong submission in my view due to limited clarity.", "rebuttal": "Dear Reviewer,\n\nThank you for your thorough and insightful feedback.  We would like to address some of your concerns about this work in order to strengthen our paper and better tailor it to a NeurIPS audience.  Brief summaries of your feedback are in bold, with our responses below.  \n\n**“It would be nice to provide a bit more exposure / explanation to Eq 3,4 which use notation that not everybody in this ML venue may be familiar with.”**\n\nWe agree that since this is very recent work that readers may not be familiar with, it will be helpful to provide deeper insight into these equations.  We have written a new appendix section which provides a quick and convenient reference for readers on the derivations of Eqns 3 and 4 following reference [17].  This section starts from a simple projection of a signal onto a basis via inner product, and develops the full SSM models step by step with text narrative to build intuition along with the mathematical framework.  \n\n**“The narrative of the abstract/intro is: There is HiPPO --> SaFARi --> WaLRUS (based on SaFARI). But the experiments compare WaLRUS only against two HIPPO variants. Why not compare against the direct baseline SaFARi?”**\n\nWe have added clarifying text to Section 1 to emphasize the point that SaFARi is a method to generate models, whereas WaLRUS is a model instance:\n\n“In this paper, we leverage the SaFARi method with wavelet frames to introduce a new SSM instance, WaLRUS (Wavelets for Long-range Representation Using SSMs).”  \n\nWe have also created a block diagram to illustrate the relationships between WaLRUS, HiPPO, SaFARi, and Mamba, which also helps to highlight the unique contributions of this paper and the distinction from other works.\n\n**“How reusable is the proposed Walrus code? I.e. from reading the paper I'm not sure how easy it will be to try this approach out as drop-in replacement in existing SSM frameworks.”**\n\nWaLRUS is indeed a drop-in replacement for existing SSM frameworks using HiPPO; the new figure in our revision explains this.  Our code generates A and B matrices of size NxN and Nx1 respectively, which can be directly substituted into any code that uses SSM layers made from A and B matrices (code supplement provided in footnote on p.1).  \n  \n**“There seem to be few works [1], [2], that do Wavelets with SSM… it begs the question whether Wavelet + SSM is novel - or what are the gaps that this works fills exactly?”**\n\nWe did not cite extensively from the vast and growing literature on Mamba, since it is outside the scope of our paper, except as motivation and contextualization.  Yet, given the overlap in the titles of these papers and the focus of our work, we expect that other readers will have the same question.  Therefore, we have include these citations in Section 2 with the following text: \n\n\"Some recent work [32, 33] has conceptually connected the use of wavelets and SSM-based models (namely Mamba). These efforts are fundamentally distinct from ours in that they perform a multi-resolution analysis on the input to the model only. No change is made to the standard Mamba SSM layer; it is still using Legendre-polynomial based SSMs.\n\nThis work, on the other hand, is the first to challenge the ubiquity of the Legendre-based SSM, and present alternative wavelet-based machinery for the core of powerful models like Mamba.  WaLRUS could be used as a drop-in replacement for any existing SSM-based framework.  However, before simply substituting a part in a larger system, we must first justify how and why a different SSM can improve performance. This paper presents a tool that stands alone as an online function approximator, and also provides a foundational building block for future integration in SSM-based models.”\n\n**“Do the current benchmarks really demonstrate WaLRUS’s benefits over existing SSMs (and maybe even Transformers)? I believe an experiment on more community-relevant settings a small-scale long-context language-model or long biological sequences could provide a clearer picture.” “The experimental evaluations are on tasks that I personally do not find super relevant to showcase the benefit of a new SSM variant.”**\n\nWaLRUS, like HiPPO, does not learn any parameters; it is a purely signal-processing approach that does not compare directly to learned models like transformers.  It holds the promise of building better machinery for ML models like Mamba, but it also stands on its own as a function approximation technique.  Our goal in choosing experimental cases was to highlight the power of WaLRUS in function approximation as a first step in developing a new tool, and as such, the only appropriate comparison is to HiPPO.  We again use our new diagram to emphasize this focus.  \n\nThe use cases you suggested – language models and biological sequences – are excellent candidates for future work, e.g. by comparing a WaLRUS-based Mamba to a HiPPO-based Mamba.  In fact, we hope that this work will inspire the community to apply WaLRUS in such scenarios by building on this foundational work.  \n\n**Formatting of Table 2:**\nWe agree that without vertical lines it is much nicer; we have removed these in our most recent revision.\n\nWe believe our updated manuscript is significantly improved and better contextualizes the work.  We thank the reviewer again for their time and for the valuable feedback on our paper, and look forward to any additional comments you would provide."}, {"review_id": "f19DmJvYwH", "reviewer": "Reviewer_obbg", "summary": "The paper builds upon the established SaFARi framework, which does frame-agnostic representations for function approximation using state-space models. The authors introduce WaLRUS, a novel extension that replaces the Legendre or Fourier frames in SaFARi with wavelet frames. They provide an analysis of the theoretical advantages and potential limitations of this approach, including \na discussion on computational complexity and approximation error. The experimental evaluation, done on both real and synthetic datasets as well as a new \"spike detection\" task, demonstrates the overall superiority of wavelet frames.", "questions": "1. The critique in section 3 on other frames seems to focus on the Legendre translation (LegT) frame, other frames occupy less room. Can you balance this more?\n2. The metrics for the \"isolated spikes\" task are only described with an image. Could you provide some formulas as well?\n3. What is the effect of varying the effective dimension N_eff? Could you plot MSE over N_eff?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- well written\n- clear objective of the paper\n- theoretical motivated\n- empirically proven\n\nWeaknesses:\n- The authors switch a part in an existing framework. While this is shown to be better, the overall novelity is not of the highest quality.\n- While outperforming existing methods, the shown improvements in MSE are not that high. The relevance of those improvements to down stream tasks ramain uncertain.\n- Wavelet frames come with more parameters, e.g. scale, shift N_eff. It is totally unclear how to set them. There are some justifications for the set parameters, but it is still unclear how to set them properly. This is however mentioned in the limitations.\n- A.1.3 lists all the chosen parameters, which vary from experiment to experiment. That seems like hyperparameter optimization and making the significance of the result somewhat questionable.", "was_revised": true, "final_justification": "The rebuttal has improved clarity of the proposed method and my questions were overly answered.\nI stay with the finding that the improvements in MSE exist but are not that high. The relevance of those improvements to down stream tasks remain not fully convincing. Nevertheless, replacing the Legendre or Fourier frames in SaFARi with wavelet frames is interesting and the results are thorougly validated. So I stay with my positive assessment.", "rebuttal": "Dear Reviewer,\n\nThank you very much for your thorough and insightful feedback.  We would like to address some of your concerns about this work in order to strengthen our paper.  Brief summaries of your feedback are in bold, with our responses below.  \n\n**“The authors switch a part in an existing framework. While this is shown to be better, the overall novelty is not of the highest quality…  the shown improvements in MSE are not that high. The relevance of those improvements to downstream tasks remain uncertain.”**\n\nWe have framed our work as a drop-in replacement for SSM-based ML models as motivation and context, but this work also stands alone as a lightweight online function approximation tool.  More importantly, it is *the very first work* offering a replacement for the Legendre-based SSM that is taken for granted when utilizing off-the-shelf Mamba/S4 implementations.  To date, no other work has ever gone back to first principles to understand the functionality of the core machinery running these models, let alone fundamentally alter it.  \n\nBased on results from prior and current work, we believe that using better-performing function-approximating SSMs in neural networks should improve performance in downstream tasks; however, we first must show how and why these SSMs perform better on their own.  The focus of this paper is thus on developing a core building block that will enable those downstream applications, and hopefully be of use and interest to members of the community.  \n\nOur revised manuscript includes a new figure--a simplified block diagram illustrating the relationship between our work, HiPPO, SaFARi, and Mamba.  We have also included some additional text in Section 2 that contextualizes our work in the space of Mamba and other such models:\n\n\"Some recent work [R1,R2] has conceptually connected the use of wavelets and SSM-based models (namely Mamba). These efforts are fundamentally distinct from ours in that they perform a multi-resolution analysis on the *input to the model only*. No change is made to the standard Mamba SSM layer; it is still using Legendre-polynomial based SSMs.\n\nThis work, on the other hand, is the first to challenge the ubiquity of the Legendre-based SSM, and present alternative wavelet-based machinery for the core of powerful models like Mamba.  WaLRUS could be used as a drop-in replacement for any existing SSM-based framework.  However, before simply substituting a part in a larger system, we must first justify how and why a different SSM can improve performance. This paper presents a tool that stands alone as an online function approximator, and also provides a foundational building block for future integration in SSM-based models.”\n\n**“Wavelet frames come with more parameters, e.g. scale, shift N_eff. It is totally unclear how to set them. There are some justifications for the set parameters, but it is still unclear how to set them properly. This is however mentioned in the limitations.”**\n\nScales for Daubechies wavelets are always dyadic, scaling by a factor of 2; we have updated the equation in Section 2.1 to make this clearer:  \n$$\nW(j,k) = \\int_{-\\infty}^\\infty f(t) \\psi^*_{j,k}(t) \\, dt,\n\\quad\n\\psi_{j,k}(t) = \\frac{1}{\\sqrt{2^{-j}}} \\psi\\left(\\frac{t-k}{2^{-j}}\\right)\n$$\nThe choice of shift amount introduces some interesting complexity that we agree could be given deeper treatment, and we have added the following text to Section 3: \n\n“To construct the frame, we use the usual dyadic scaling for multiresolution analysis; that is, scaling the mother wavelets by a factor of two at each level.  For each scale, different shifts along the x-axis are introduced. Compressive wavelet frames are truncated versions of wavelet frames that contain only a few of the coarser scales, and introduce overlapping shifts to keep the expressivity and satisfy the frame condition (See Mallat, [18]).  The interplay between the retained scales and the minimum required overlap to maintain the expressivity is extensively studied in the wavelet literature [18,20,30].  If there is excess overlap in shifts, the wavelet frame becomes redundant, and redundancy has advantages in expressivity and robustness to noise.\n\nThe exact relationship between the amount of overlap and the resulting dimensionality of the A matrix depends not only on the overlap of the shifts, but also on the type (and order) of chosen wavelet, and number of scales.  Determining the “optimal” overlap or N_eff is application-specific, and an area for future research.”   \n\n**“A.1.3 lists all the chosen parameters, which vary from experiment to experiment. That seems like hyperparameter optimization and making the significance of the result somewhat questionable.”**\n\nSome parameter choices are heuristic and some are codependent, but none were chosen to be optimized, or to otherwise unfairly advantage one method over another.  We acknowledge that the varying parameters as presented in Table 3 could reasonably raise this concern for readers, however, and so we have added additional text to clearly explain how these parameters were chosen.  We have also simplified our table to remove the wavelet type (D22, which is the same for every experiment in this work and is already described in the same appendix), and the “scale max” variable, which does not impact the rank of the frame, as explained in the added text below.\n\n“*Scale Min* and $N_{\\rm{eff}}$: The minimum scale sets the smallest feature of the signal that can be represented by the frame. This parameter should be chosen based on knowledge about the signal of interest and its component frequencies, as one would for a filter bank.  Note that the size of the smallest feature is relative to the length of the signal under consideration, so this value may differ under scaling and translating measures.\n\n*Scale min* also controls the effective rank, $N_{\\rm{eff}}$. Each additional lower scale introduces a factor of two in the effective rank of the frame, owing to the additional shifted elements in each scale.  To illustrate, Fig. 2 shows two scales, where there are three father wavelets ($\\phi_0$) and three coarse-scale mother wavelets ($\\psi_1$). The next scale introduces six scaled and shifted mother wavelets ($\\psi_2$), the next would include 12, and so on. Table 3 also shows this pattern, with a minimum scale of 0 corresponding to $N_{\\rm{eff}}$ of $2^6$, minimum scale of $−1$ corresponding to $N_{\\rm{eff}}$ of $2^7$, and so on, with variance resulting from the shift combined with truncation of frame elements during construction.\n\nOur code includes another variable, *scale max*. Since smaller scales can also combine to represent larger scales, *scale max* in fact has no impact on $N_{\\rm{eff}}$.  (See Mallat [18] for further details on this topic.)  Adding coarser scales can help improve numerical accuracy in the calculation of $A$, however.  We do not include *scale max* in Table 3, but we provide it in our code with each experiment for reproducibility.\"\n\nTo support this additional text, we also have generated some quantitative results.  Fig. 10 (in the appendix of our revised paper) takes an example implementation of WaLRUS and varies *scale max*, and we observe that the size of $N_{\\rm{eff}}$ remains constant.  It is also easily shown that varying scale max results in the same diagonalized $A$ matrix.  \n\n**“What is the effect of varying the effective dimension N_eff? Could you plot MSE over N_eff?”**\n\nWe agree that analyzing the impact of varying $N_{\\rm{eff}}$​ on MSE would be insightful.  However, due to limited time, we cannot include this experiment in the current revision, and plan to pursue it in future work.\n\n**“The critique in section 3 on other frames seems to focus on the Legendre translation (LegT) frame, other frames occupy less room. Can you balance this more?”**\n\nWe have added the following text in Section 3 of our revised manuscript after the discussion of the translated Legendre kernel:\n\n\"The translated Fourier (FouT) kernel primarily suffers from the opposite problem: substantial nonzero elements outside the kernel window indicate that LegT struggles to effectively \"forget'' historical input values. Thus contributions from input signals outside the sliding window appear as representation errors.  LegT also has this problem, to a lesser extent--see area A of Fig. 3.\"\n\nWe keep the primary focus on Legendre-based kernels, as this is the most ubiquitous (even if hidden!) SSM used today in ML architectures.  \n\n**“The metrics for the \"isolated spikes\" task are only described with an image. Could you provide some formulas as well?”**  \n\nYes; we have added a section in the appendix of our revised manuscript with these formulas.    \n\nWe thank the reviewer again for their thoughtful questions, which spurred interesting discussions among the authors and provided concrete ways to better communicate the content and context of this work.  We believe our updated manuscript is significantly improved, and look forward to any further feedback you would provide.  \n\n---\n\nReferences:\n\n[R1] Zou, W., Gao, H., Yang, W., & Liu, T. (2024, October). Wave-mamba: Wavelet state space model for ultra-high-definition low-light image enhancement. In Proceedings of the 32nd ACM International Conference on Multimedia (pp. 1534-1543).\n\n[R2] Zhang, T., Zhu, Y., Zhao, J., Cui, G., & Zheng, Y. (2025). Exploring state space model in wavelet domain: An infrared and visible image fusion network via wavelet transform and state space model. arXiv preprint arXiv:2503.18378."}, {"review_id": "qkveb2obPY", "reviewer": "Reviewer_tu5D", "summary": "The paper proposes and demonstrates replacing Legendre and Fourier basis sets for State Space Modeling of dynamic trajectories with Wavelet basis functions. The theory is described in general terms and experiments are performed on benchmark problems involving many dynamic systems and spike recognition problems. Comparisons with other basis set formulations (Legendre polynomials and Fourier series) for State Space Modeling show that the wavelet basis often outperforms.", "questions": "Can you provide a clearer overview of the theory, to include precise justifications of all claims?\n\nCan you provide comparisons with other, non state-space models?\n\nCan you provide a code supplement for reproducibility?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The wavelets clearly outperform the comparison bases on a broad number of test problems.\n\nHowever, there is not much theoretical development. In particular it is not shown how to calculate the transition matrices $A$ and $B$ using the wavelet basis. It is claimed that wavelet bases always yield diagonalizable state-space representations, but this is not justified. Furthermore the introduction of the state space method leaves something to be desired, as it is unclear how the minimization on line 70 yields a state vector.\n\nMore serious is the fact that only state-space methods are compared with each other, while competing methods (for instance, RNNs, but also many other methods for dynamic systems, in particular Markov methods) are left out. Finally there seems to be no code supplement for reproducibility purposes.", "was_revised": true, "final_justification": "I missed the repo in the footnote. I am also not a part of the SSM culture -- maybe that community is large enough that an advance like this completely within that context is enough for acceptance at NeurIPS. I would assume that some comparisons with methods outside of SSM would be requried but I'll leave that to the chairs.", "rebuttal": "Dear Reviewer,\n\nThank you for your feedback.  We would like to address some of your concerns about this work in order to strengthen and improve our paper.  Brief summaries of your notes are in bold, with our responses below.  \n\n**“There seems to be no code supplement for reproducibility purposes.”**\n\nA link to the code supplement was provided in our paper twice.  Please see the footnote on page 1, and the NeurIPS checklist, item 5.  \n\n**“...it is not shown how to calculate the transition matrices A and B using the wavelet basis.”**\n\nEquations 3 and 4 give the construction of A and B explicitly.  These equations and their derivations can be found in reference [17], as noted in the text.  There is also an algorithmic implementation in our code.  \n\nThe construction of wavelet frames is well-covered in the literature (see references [18],[20],[29],[30]).  Figure 2 contains an illustration of a partial Daubechies wavelet frame, providing readers with key information (scales, shifts, and function shape) required to understand the contribution of this paper, without needing full knowledge of the vast literature on wavelet frame construction.\n\nTo facilitate easy access to key information for our readers, we have also added a section in the Appendix of our revised manuscript with details from [17] on the derivations of A and B.  \n\n**“It is claimed that wavelet bases always yield diagonalizable state-space representations, but this is not justified”**\n\nWe did not claim that wavelet bases “always yield” diagonalizable SSMs; we were careful to note that wavelet-based SSMs were found to be *empirically* stably diagonalizable (line 47-51) and *observed* to be diagonalizable in our experiments (line 137). Prior work has not proved any theory regarding which bases/frames or measures produce stably diagonalizable $A$ matrices, and have thus far only reported empirical observations, as we have done.  \n\nCritically, the only significant previous work on this subject (HiPPO) found that even at fairly low rank, Legendre-based matrices are *not* stably diagonalizable.  This problem was a major roadblock in the development of S4 and ultimately Mamba, and provided the motivation for multiple subsequent follow-up papers on how to make diagonal *approximations* to the Legendre-based $A$ matrix (see refs [12-14]).  In fact, if one examines the code for the state-of-the-art Mamba model (https://github.com/state-spaces/mamba), the SSM layers are initialized with one of these approximations, and not the true $A$ matrix.  Thus, we believe it is valuable to note our empirical findings of diagonalizability, even in the absence of a full theoretical proof.\n\nTo make very clear the limits of our claims, we have made the following wording adjustments in our revised manuscript:\n\nLine 47-51, Section 1: “Empirical results demonstrate … WaLRUS enjoys diagonalizability, which is the key enabler of efficient convolution-based implementations and parallel computation [13, 14].” \n\nChange to → “Empirical results demonstrate … WaLRUS has been consistently experimentally observed to be stably diagonalizable, which is the key enabler of efficient convolution-based implementations and parallel computation [13, 14].”\n\nLine 137, Section 3.2: “We observe that both scaled and translated WaLRUS are stably diagonalizable.” \n\nChange to → “We observed that each of the scaled and translated WaLRUS SSMs we implemented, regardless of dimension, were stably diagonalizable.  Further research is required to determine whether Daubechies wavelets will always yield diagonalizable SSMs.”  \n\nWhile a complete theoretical proof of diagonalizability is outside the scope of this particular work, we agree that it would be a significant contribution to the literature on this topic.  We hope that our paper inspires members of our community to join us in exploring this problem.\n\n**“Only state-space methods are compared with each other, while competing methods (e.g., RNNs, Markov methods) are left out.”**\n\nRNNs and Markov methods are not competing methods with online basis projection.  Our paper proposes an alternative basis for state-of-the-art *function approximation* SSMs that play a key role in some of the fastest-growing models in ML today; it does not propose a *time-series predictor*.  \n\nHiPPO / SaFARi / WaLRUS only store *previous* information by iteratively (and deterministically) projecting a signal onto a basis; probabilistic methods such Markov methods are designed for predictive tasks.  Most importantly, neither RNNs nor Markov methods produce a model compatible with use in S4, Mamba, and Mamba’s variants, a driving motivation for this work.  Thus, the only baseline for comparison is existing SSMs for function approximation (namely HiPPO).  \n\n**“it is unclear how the minimization on line 70 yields a state vector”**\n\nThe results in line 70 were a key finding of the original HiPPO paper [9]; we do not reproduce it in our paper, only noting the results pertinent to our work.  To ensure that this paragraph does not cause confusion for readers unfamiliar with HiPPO, we have made the following wording change in our revised manuscript, removing the reference to the minimization: \n\nLine 70: “To resolve these issues, the HiPPO framework [9] casts online function approximation as a continuous projection of the input u(t) onto a linear combination of the given basis functions G. At every time T, it ~~solves ming (T )∈G ∥uT − g (T) 70 (t)∥µ , producing~~ produces a compressed state vector c(T) that satisfies the update rule:” \n\nWe thank the reviewer again for their time and the opportunity to clarify and improve our paper."}]

Author rebuttal:

Meta-review: The paper proposes an extension of the SaFARi framework, using state-space models for long-range representation learning. The method offers improvements in representing non-smooth and transient signals compared to HiPPO models.

The reviewers found the approach to be interesting, finding that replacing the Legendre or Fourier frames in SaFARi with wavelet frames does result in a performance improvement. They also appreciated the efficiency of the method. During the discussion phase, questions were raised about the relevance of the experimental setup to validate the claims and the chosen contenders. However, the authors clarified that "WaLRUS, like HiPPO, does not learn any parameters; it is a purely signal-processing approach that does not compare directly to learned models like transformers" and is to be used for function approximation. Thus, they convincingly argued that the comparison against HIPPO is sufficient to validate the claims. The authors also answered questions about the parameters of the model and about wavelet-based SSMs being "empirically stably diagonalizable" diagonalizable state-space representations. Finally, all reviewers determined that the contributions of the paper are over the threshold for acceptance.

As long-term modeling of time series is an important problem and this paper has some degree of methodological novelty and interesting insights, I recommend its acceptance at NeurIPS.
Decision: Accept (poster)