MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling
Authors: Mahdi Karami
Ali Behrouz
Peilin Zhong
Razvan Pascanu
Vahab Mirrokni
Venue: COLM 2025 (2025)
Keywords: MS-SSM: Sequence Models
Language models
State Space Model
Multi-Scale
Multi-Resolution
TL;DR: We introduce MS-SSM which enhances traditional SSMs by modeling sequence dynamics at multiple resolutions using independent SSMs, scale-dependent initialization, and an input-dependent scale-mixer.

Abstract: State-space models (SSMs) have recently attention as an efficient alternative to computationally expensive attention-based models for sequence modeling. They rely on linear recurrences to integrate information over time, enabling fast inference, parallelizable training, and control over recurrence stability.  However, traditional SSMs often suffer from limited effective memory, requiring larger state sizes for improved recall. Moreover, existing SSMs struggle to capture multi-scale dependencies, which are essential for modeling complex structures in time series, images, and natural language. This paper introduces a multi-scale SSM framework that addresses these limitations by representing sequence dynamics across multiple resolution and processing each resolution with specialized state-space dynamics.  By capturing both fine-grained, high-frequency patterns and coarse, global trends, MS-SSM enhances memory efficiency and long-range modeling. We further introduce an input-dependent scale-mixer, enabling dynamic information fusion across resolutions.
The proposed approach significantly improves sequence modeling, particularly in long-range and hierarchical tasks, while maintaining computational efficiency. Extensive experiments on benchmarks, including Long Range Arena, hierarchical reasoning, time series classification, and image recognition, demonstrate that MS-SSM consistently outperforms prior SSM-based models, highlighting the benefits of multi-resolution processing in state-space architectures.

Published as a conference paper at COLM 2025
MS-SSM: A Multi-Scale State Space Model for Efficient Se- quence Modeling
Mahdi Karami 1 Ali Behrouz ! Peilin Zhong' 1 Razvan Pascanu 2, Vahab Mirrokni
'Google Research, "Google DeepMind
(mahdika, al ibehrouz, pei weilinz, razp, mirrokni @google.coom com
Abstract
State-space models (SSMs) have recently attention as an efficient alternative computationally expensive attention-based models for sequence mod- eling. They rely on linear recurrences to integrate information over time, enabling fast inference, parallelizable training, and control over recurrence stability. However, traditional SSMs often suffer from limited effective memory, requiring larger state sizes for improved recall. Moreover, existing SSMs struggle to capture multi-scale dependencies, which are essential for modeling complex structures time series, images, and natural language. This paper introduces multi-scale SSM framework that addresses these limitations by representing sequence dynamics across multiple resolution and processing each resolution with specialized state-space dynamics. By capturing both fine-grained, high-frequency patterns and coarse, global trends, MS-SSM enhances memory efficiency and long-range modeling. We further introduce an input-dependent scale-mixer, enabling dynamic information fusion across resolutions. The proposed approach significantly improves sequence modeling, particularly in long-range and hierarchical tasks, while maintaining computational efficiency. Extensive experiments on benchmarks, including Long Range Arena, hierarchical reasoning, time series classification, and image recognition, demonstrate that MS-SSM con- sistently outperforms prior SSM-based models, highlighting the benefits of multi-resolution processing in state-space architectures.
1 Introduction
Over the past few decades, numerous deep neural network architectures have been devel- oped for sequence modeling- Early approaches like recurrent neural networks (RNNs) (El- man, 1990) and their variants, such as Long Short- Memory (LSTM) networks (Hochre- iter et al., 1997) and Gated Recurrent Units (GRUs) (Cho et al., 2014), were proposed to handle sequential dependencies by maintaining hidden states over time. However, these models struggled with long-range dependencies and computational inefficiencies. With the advent of attention mechanisms (Bahdanau et al., 2015; Vaswani et al., 2017), the Trans- former architecture emerged the de facto standard for many sequence modeling tasks. The Transformer's 's self-attention mechanism enabled the modeling of complex relationships across sequences without relying on recurrence, allowing for parallel computation and better handling of long-range dependencies which enabled breakthrough advances across wide range of applications. However, inference in transformer can be expensive due to the quadratic complexity of the attention mechanism, hindering its ability to handle even longer context tasks efficiently or run in low resource settings. These limitations has motivated the exploration of alternative scalable sequence modeling approaches with comparable expressiveness.
Recently, state-space models have generated renewed interest efficient attention-free sequence models. Deep state-space models (SSMs), a class of RNNs that use linear recur- rences, provide scalable training and inference capabilities, proving particularly effective for long-range dependency modeling (Gu et al., 2020a). These methods typically rely on block structure similar to transformers, where the linear recurrences do sequence mixing,

---

Published as a conference paper at COLM 2025
while MLPs are used for feature mixing (Orvieto et al., 2023). To gain expressivity, similar to transformer, many such blocks are typically stacked on top of each other (Orvieto et al., 2024). The linearity allows to reformulate the recurrence
convolution (Gu et al., 2020a; 2022a; 2021b; Mehta et al., 2022) or the use of associative scan (Smith et al., 2023; De et al., 2024), making SSM on par to transformer in terms of training cost. Recent architectures also typically use gating mechanisms, similar to LSTMs and GRUs, which can also be viewed as relying on input-dependent model parameters, increasing their expressivity (Gu & Dao, 2023; Orvieto et al., 2023; Dao & Gu, 2024; De et al., 2024; Beck et al., 2024), along with long convolution models (Karami Ghodsi, 2024). They demonstrate considerable potential in various applications, including natural language processing (Gu & Dao, 2023; Karami & Ghodsi, 2024), computer vision (Liu et al., 2024; Karami & Ghodsi, 2024; Behrouz et al., 2024a), DNA modeling (Nguyen et al., 2024; Gu & Dao, 2023), and graph data (Behrouz & Hashemi, 2024).
However, traditional SSMs lack the inherent ability to capture multi-scale patterns prevalent in many real-world signals, such as image, audio, and time series data. Moreover, the effective memory of these linear RNNs, which is inversely proportional to the distance of the eigenvalues from the unit circle (Agarwal et al., 2023), is limited, requiring larger state sizes for improved recall. To address these limitations, we propose incorporating Multi- Resolution Analysis (MRA) into SSMs. By decomposing the input sequence into multiple scales, our approach allows the SSM to capture both aine-graindd details and broader trends simultaneously. This multi-scale representation enables SSM to effectively capturing historical patterns at multiple levels of granularity.
Multi-resolution analysis plays a crucial role in understanding and modeling complex patterns across diverse datasets, including audio (Van Den Oord et al., 2016), images (Long et al., 2015), time series (Deznabi & Fiterau, 2023), graph generation (Karami, 2024), and text (Tamkin al., 2020; Tai et al., 2015; Bowman et al., 2016). The importance of this approach stems from the multi-scale properties inherent in these data types, where patterns and structures manifest at various levels and timescales. For instance, natural language data exhibit multi-scale patterns ranging from subword to word, phrase, sentence, para- graph, and document levels. Similarly, the multi-scale structure of images and videos can reveal details from pixel-level to higher-level scene interpretation. Recently evidence from neuroscience further underscores the significance of multi-resolution analysis, particularly in language processing. Specifically, Caucheteux et al. (2023) provide evidence supporting hierarchical predictive coding in language, showing that the human brain predicts speech in hierarchical manner, with different brain regions responsible for different levels of predic- tion. This aligns with earlier observation that the brain continuously predicts a hierarchy of representations tionns across multiple timescales in the cortical hierarchy (Wacongne al., 2011). Consequently, modern language models augmented with hierarchical predictions across multiple timescales can improve their alignment with human brain responses. Further- more, even in data without explicit multi-scale characteristics, this modeling approach can efficiently capture long-range dependencies (Shi et al., 2023).
Several approaches have been proposed to incorporate multi-resolution analysis into se- quence modeling. For instance, Nawrot et al. (2021) introduce a hierarchical Transformer architecture that processes information across multiple levels of abstraction in language mod- eling tasks. This approach explores various strategies for downsampling and upsampling activations in Transformers, achieving efficient computation and improved performance on various benchmarks. The Clockwork RNN (Koutnik et al., 2014) enhances traditional RNNs by partitioning the hidden layer into modules that operate at different temporal frequencies. This structure allows for a more efficient processing of sequences with varying temporal dynamics, thereby improving performance on complex tasks. In the context of Fourier-based multiresolution models, techniques such as FNet (Lee-Thorp et al., 2021), Prism (Tamkin et al., 2020), and Orchid (Karami & Ghodsi, 2024) operate in both the spatial and frequency domains. However, these methods are inherently non-causal, as the Fourier transform is applied across the entire sequence, also Fourier transform is poor in time localization of the representation in the frequency domain. Shi et al. (2023) proposed multi-resolutionn convolution as an efficient pattern memorization, utilizing learned con- volution kernels with dilations shared across multiple timescales. However, similar to
2

---

Published as a conference paper at COLM 2025
other short convolution-based architectures, this model's effective receptive field is limited. Additionally, Fan et al. (2024) has utilized the intrinsic granularity present in data to design more stable and accurate forecasting methods using diffusion.
In this work, we introduce MS-SSM, which integrates an efficient multi-resolution analysis into the state space architecture, decomposing the dynamical system into multiple time scales. This enables the overall SSM to operate at different resolutions. We show the effectiveness of our methods on Long Range Arena (Tay et al., 2020b) well as other sequential tasks. In section 2 we describe in detail the proposed method, providing our empirical evaluation in section 3.
Method
The proposed sequence model is composed of two core components: 1) a multi-scale decomposition and 2) an array of state space models (SSMs). These components work together to capture patterns and temporal dynamics at different resolutions. Each will be explained in detail in the following sections.
2.1 State Space Models
SSM. State Space Models (SSMs) are linear time-invariant systems that map input se- quence x(t) E RR_ to response sequence y(t) E R- (Aoki, 2013) using a latent state k(t) E NXL, parameter A E R (a.k.. state transition matrix), and projection pa- rameters B RNx1, C R1*N, That is: ('()) A hf) + B xx(),, y(t) Ch(f). Discrete space state models (Gu et al., 2020a; Zhang et al., 2023) is obtained by discretizing at step size A through high accuracy Zero-Order-Hold (ZOH) method:
h; À h xt
(1)
y/1 Cit,
where B = (AA) (exp (AA- 1)) AB and Ā = exp (AA).
These models can be interpreted as both CNNs and RNNs and are equivalent to the con- volution K = (CB, CAB, CAL-IB), and so y = x K (Gu et al., 2020a). Leveraging the convolution theorem and Fast Fourier Transform (FFT) algorithm for this long convolution formulation, its training complexity scales quasi-linearly with sequence length and can be parallelized, while it enjoys linear complexity at inference time using its recurrence form.
Structured SSM (Gu et al., 2022a) relies on a diagonal parametrization A, enabling efficient computation of the discretization in (1) and its convolution formulation. Combined with the use of associative scan techniques Smith et al. (2023), this allows for efficient parallelization of computation even when using the recurrent form. Newer architectures such as Mamba (Gu & Dao, 2023) or Griffin (De et al., 2024), typically have moved away from the convolutional formulation.
Input-Dependent SSM. Recently, Gu & Dao (2023) introduced the 56 block, a structured State Space Model (SSM) with a selective scan mechanism. This input-dependent gating mechanism enables S6 to selectively propagate or forget information along the sequence dimension by allowing specifying the parameters as:
B, 5y= Linears (x:), C. = (x = (inearc(x)),, A = sa (xx) = Softplus (Lineara (x:))
where Linear(-) a linear projection and Softplus(-) log(1 (1 + exp(.)). This approach adds context-awareness to SSMs and a similar form is used in other works, e.g. (De et al., 2024). Despite its more expressive power, in contrast to S4, this time- and input-variant model prevents the of the convolutional formulation. But as mentioned above, computation can still be parallelized by using the associative scan (Martin & Cundy, 2018; Smith et al., 2023; Orvieto et al., 2023). Also, it allows for more hardware-aware implementations.
Limitations: While the linear formulations of SSM allows to greatly improve scalability of the system and to control its stability (Orvieto et al., 2023), it also limits the architecture. From
3

---

Published as a conference paper at COLM 2025
Figure 1: (a) Block diagram of the MS- SSM model. (b) Multi-scale convolu- tion layer. The mul ti-scale conv layer, which decomposes the signal into mul- tiple scales, is composed of nested con- volution layers Convld defined in (3). The scale-mixer combines the scales through an input-dependent weighted summation defined (4).
Mazes
Lices
Mad sokie Com
(a)
(b)
an expressivity point view, a single linear recurrent layer is limited in what it can represent. Deep SSM architectures recapture expressivity by stacking multiple blocks Orvieto et al. (2024). Additionally, the system can only exhibit fading memory, where the time to live for information is inversely proportional to the distance of the sigenvalues from the unit circle (Agarwal et al., 2023), requiring an increase in the state size in order to improve the ability of the system recall.
2.2 Multi-Scale Decomposition
Multi-resolution analysis (MRA) is a mathematical framework that enables the analysis of signals at multiple scales resolutions. A powerful tool for performing MRA is the Discrete Wavelet Transform (DWT), which decomposes a signal into different levels of approximation and detail by recursively applying pair of filters-a low-pass filter and high-pass filter, denoted by q and 4,,
followed by downsampling.
A major limitation of the standard Discrete Wavelet Transform (DWT) is its lack of translation-invariann
meaning that even small shifts in the input signal can result in significant changes to the resulting wavelet coefficients. To address this issue, several DWT variants have been developed that use redundant signal representations. One such approach is the Dual-Tree Complex Wavelet Transform (DTCWT) (Selesnick et al., 2005), which provides approximate translation-invariinne by using two parallel DWT trees with slightly different filters. In contrast, the Stationary Wavelet Transform (SWT) (Nason & Silverman, 1995) achieves true canslation-invariinne by skipping the downsampling step at each decomposition level. Given an input signal a & the SWT decomposes it recursively into approximation and detail coefficients at each scale E {1,2, ..., $}, as follows:
a" [' A (a- * 1 20 =
Σ
a d'11 1 [
d' A (a-11 (1p r 22)
Σ
d - 2--1--1-1]]]]
(2)
In essence, the coefficients at level s are obtained by convolving the upsampled filters, (Φ t 1) and (yp f 2 1), with the approximation coefficients from the previous level, a² of the signal after levels consists of the set of detail coefficients at all scales, (f)..., [f)), along with the final approximation coefficients, a a$(t), which together can perfectly reconstruct the original signal. This transformation of the signal provides information about both the frequency content and the time localization of the signal and also captures both the smooth, global trends and the fine-grained details, enabling wide range of applications in signal processing. One key advantage of the SWT is that it maintains the same sequence length at each decomposition level, producing redundant representation of the signal. This redundancy is key to achieving translation- invariance, which leads to significant performance improvements in applications such as signal denoising (Kumar et al., 2021), image resolution enhancement (Demirel & Anbarjafari, 2010), and feature extraction (Zhang et al., 2010). However, the trade-off for this improved performance is the increased computational cost and memory usage compared to the standard DWT.
The complete multi-scale
|Continuous form of multi-scale analysis such as continuous wavelet transforms are normally discretized with finite dyadic set {2>]5=]
4

---

Published as a conference paper at COLM 2025
The specific form of the filters @ and y depends on the choice of wavelet basis. Different wavelet families, such as Haar, Daubechies, and Symlets, have distinct filter coefficients, resulting in different properties for the wavelet transform (Daubechies, 1992). While choos- ing an orthogonal wavelet basis ensures perfect reconstruction of the signal, this property is not always desirable in deep sequence modeling. As observed in recent research (Shi et al., 2023), employing trainable filter weights instead of fixed wavelet bases offers greater flexibility and model expressiveness. This approach enables the model to learn optimal filter coefficients for specific tasks, potentially leading to enhanced performance in range of applications. The filtering operation at level s, as defined in equation 2, can be efficiently implemented using& causal depthwise 1D convolution (Conv1d) with two output channels, kernel length of K, and a dilation factor of 2 As a result, the input-output relationship in (2) can be specified by" 2
la"; d'] = (o, 2, L, 2 1aa" 11.
(3)
In this model, the multi-scale block utilizes convolution kernels with dedicated weights for each scale.
This recursive process leads a nested multi-scale decomposition block that transforms 1-dimensional sequence into set of sequences across different scales, which can be collected into multi-diimensionll representation vector, i.e. X; E R + &; E S Each dimension in this representation corresponds to a different resolution, capturing signal features from fine-grained details to coarse global trends, enabling analysis of the signal across varying levels of granularity. The higher the scale value, s s-which corresponds to deeper levels in the recursion tree of (3) more noarse-grained the information represented at that scale. This follows the recursive principle (Pauwels et al., 1995), whereby larger values of result in increasingly blurred (less sharp) representations of an image (Worrall & Welling, 2019).
At each time scale s, the dilated convolution filter captures patterns of length up to 2 x K, meaning that xē represents local patterns within a limited window preceding the time index t. In other words, akin to the localized spectro-temporal representation in the Discrete Wavelet Transform, the scale components of X,, with limited number of scales, capture only recent local structures. However, for non-local patterns that span larger intervals, such as those found in auditory signals (Romero et al., 2020), it is essential to model long- range temporal correlations within each scale representation. To address this, we apply independent -WMMshhichh maintain a global receptive field-to each scale representation, as well to the original signal, in order to capture the temporal dynamics within the scales. The proposed models, named MS-SSM, specializes distinct SSMs for different time scales. This setup results in an array of (S + 2) SSMs operating in parallel, with each SSM having a latent state size of N. Consequently, the effective latent state size per input channel becomes (S + N) To obtain comparable state dimension in the proposed model, we set this effective state size to match the recurrent state size of other models, thereby maintaining consistent latent dimensions across different architectures. Additionally, this SSM array can be implemented in parallel, making their overall computational complexity comparable to architectures operating at a single resolution. The MS-SSM block is illustrated in Figure
Initialization. The eigenvalues of the state transition matrix (AA((A))) play a critical role in determining the stability and memory capacity of State Space Models. To ensure stability in discrete SSMs, these eigenvalues must lie within the unit circle, while for continuous-time SSMs, the eigenvalues of A must be in the left half-plane. Figenvalues of A that are closer to 1 enhance the model's ability to capture long-range dependencies (Gupta et al., 2022; Orvieto et al., 2023). In essence, the effective memory of an SSM, which quantifies how long past information influences the present state, is inversely proportional to the distance of the eigenvalues from the unit circle. Formally, when eigenvalues satisfy [A,(A) < 1 the effective memory is on the order of 1 (Agarwal et al., 2023).
To balance between capturing long-range dependencies and maintaining different effective memory at each resolution, we employ a scale-dependent initialization scheme. Previous works
2In PoTorch, this operation can be simply realized with the following code: torch. nn. Co, 2, kernel size=L, (sl-)).
5

---

Published as a conference paper at COLM 2025
observed that real-valued SSMs perform on par with or even outperform complex- valued counterparts (Ma et al., 2022; Gu & Dao, 2023), hence, we adopt a diagonal-structured recurrence matrix with real values.
For lower resolutions (higher value of s in hi- erarchy), which contain coarse-grained infor- mation, we initialize the diagonal elements of A with values closer to to enhance the model's ability to capture long-rang depen- dencies within these scales. In contrast, for higher resolutions containing fine-grained details, we initialize (A) with smaller val- ues to prioritize shorter effective memory and focus on local dynamics at initialization. Specifically, the diagonal elements of the state transition matrix at scale E {O...., 5 1}, diag(A) (A),, are initialized uniformly within the interval (- N(S + s), N(S s)] (or equivalently diag(As (A=) E (cl-AoN(5+1-5)) (N-NAp(S-s)) ), where N is the state size per scale. By concatenating all latent states into large state [4:; ... h h5*11, the overall state transition matrix becomes A (alag(diag(
Figure Initialization scheme for 3 different scales with N = 3 and 4 0.2.
..., diag ( Then, this real-valued initialization aligns with that in the S4D-real (Gu et al., 2022a) which is grounded in the HiPPO theory (Gu et al., 2020a), where the n-th element of diag (A) is initialized An example of this initialization scheme illustrated in Figure 2.
(I + 1).
Scale Mixer. After independently modeling the temporal dynamics at each specific scale, the array of (S + 2) SSMs produces outputs that are collected into the vector y: E I To effectively merge these multi-scale representations, the model requires a mechanism that encodes cross-scale interactions, enabling information to flow between scales and ultimately combines them into a single-dimensional output. To achieve this, we combines the scales through weighted summation applying input-dependent projection matrix E, E R (2)
2 = scale-mixer(yj:xt = E, yı, where E, = sx(x, = Linearg
(4)
This approach allows the model to dynamically adjust the contribution of each scale based on its input.
Input-dependent Parameterization. In S6 (Gu & Dao, 2023), an input-dependent parame- terization is employed for the SSM, allowing the model to selectively propagate forget information along the sequence based on the input token of the SSM, functioning similarly to gating mechanism in RNNs. In this work, for the s-th SSM operating on scale s, make the parameters functions of the original input x,. Specifically, the parameters of the s-th SSM, are modeled as B; = (x C. = (xt), and A; = s (xx). Through empirical studies, presented in Appendix C, observe that gating based on the raw input, X;, more effective than gating based the scale-specific representations (B, sgg(),,, C (Xf), and A = s (2.)). Using the raw input for controlling the parameters results in a more effective mixing each scale's representation with the raw input information.
Complexity. The multi-scale convolution operation introduces linear time computation overhead of (LKS) and require an additional O (KS) parameters per layer. However, this overhead is minimal compared to the overall model size, given the small convolution kernel size, K, and the limited number of scales, 5.
3 Experiments
We evaluate our proposed architecture across image classification tasks, where images are converted into a sequence of patches (lmageNet-1k) or pixels (sCIFAR), as well as hierarhical reasoning and time series classifications. In all experiments, we report the results of two variants of our approach, i.e., MS-SSM (S4) and MS-SSM (S6), in which we use S4 (Gu et al., 2021a) and S6 (Gu & Dao, 2023) blocks as the recurrent module, respectively. Comparison of these two, as two instances of fata-dependent and data-ina lata-indpendent recurrent models, shows that MS-SSM perfoommee does not rely on the S6 block and supports the significance of our design.

---

Published as a conference paper at COLM 2025
Table I: Results on SCIFAR (Shi et al., 2023) and Im- ageNet (Deng et al., 2009). Missing results mean that the performance of the model is not reported on ImageNet-IK in the original reference.
Table 2: Performance of predicting outcomes
of list operations in ListOps dataset of Tay et al. (2020b). Mamba 2X Param and Mamba 2X State denote Mamba model with double model size and double state size, respectively.
Method
CCAARR
ImageNet-1K
Model
Accuracy (%)
Transformers
Transformers
Transformer (Vaswani et al, 2017)
622
78,9
Transformner (Vaswani et al., 2017)
36.37
Recurrent Neural Networks (RNNs)
Local Attention (Tay et 20206) 20220)
15.82
Linear Trans. (Katharopoulos et al,, 2020)
16.13
HIPPO-RNN
2020a)
61.1
Linformner (Wang et al., 2020)
16.13
L.TMM (Hochneiter et al., 1997)
63.0
Sparse Transformer (Child
2019)
17.07
r-LSTM (Trinh et 2018)
72.2
Performer (Choromanski et al,, 2020)
18.01
UR-GRU (Gu al., 2020b)
74.4
Sinkhorn Transformer (Tay et al., 202(Ua
33.67
LipschizzNNN (Erichson al., 2020)
64.2
Longformer (Bellagy al, 2020)
35,63
BigBird (Zaheer
20200
State Space Models (SSMs)
Lama-256 (Ma et 2021)
37.25
S4 (Gu et 2022b) 2022))
91.1
79.1
Relormer (Kitaev el al, 2020)
37.27
$4D (Gu et 2022a)
89.9
80.4
HH-Tranffomee-- (Zhu Soricut, 2021)
85 (Smith et 2023)
89.7
77.9
Convolutions
Liquid-St (Hasani et al., 2022)
92.0
Mamba (Gu & Dao, 2023)
90.1
80.5
CDIL (Cheng et 2023)
SCConv (1i ct al., al,
61.45
Convolutions
MounResNer (Shi et 2023)
62.75
CKConv (Romers et al., 2021)
63.7
SSMs
MULTIRESNET (Shi et al., 2023)
93.1
Orchid (Karami & Ghodsi, 2024)
93.0
80.2
S4 (Gu al., 2022b)
DS8 (Gupta 2022)
89.60
Convolution SSMs
SID (Gu et al., 2022a)
60.52
MS-SSM (S4) MS SSM (56)
90.3
79.7
Liquid-S4 (Hasani
Si (Smith et 2023)
2022)
62.75
62.15
93.3
81.3
Griftin (De et al., 2024)
32.34
Mambo (Gu & Dao, 2023)
38.02
Mamba Param
Mambe 2x State
42.14
Convolutions + SSMs
MS-SSM (S4)
62.83
M5 M5-SSM (56)
63.04
Image Classification. We evaluate the performance of MS-SSM in two image classification tasks: ImageNet-1K (Krizhevsky et al., 2012) and SCIFAR (Shi et al., 2023). We use ImateNet to compare the performance of MS-S5M with baselines in modeling the sequence of image patches. In sCIFAR task, however, each image is treated as 1D sequence of pixel and so the models are not using any 2D inductive bias from the images. Therefore, the model must be able to capture long-range dependencies and patterns at different resolutions. Results are reported in Table MS-SSM shows outstanding performance compared to all other sequence models in both tasks and more specifically in capturing long range and multi-resolution modeling of pixels in SCIFAR. The superior performance compared to Mamba (Gu & Dao, 2023) and similar SSM-based models (Smith et al., 2023; Gu et al., 2022b;a) comes from the multi-resolution convolutions that helps MS-SSM to capture the dependencies at different levels of granularity. Compared to multi-resolution methods, e.g,, MULTIRESNET (Shi et al., 2023), the superior performance of MS-SSM highlights the significance of SSMs and our scale-mixer module.
Time Series Classification. Time series classification is one of the important tasks in se- quence modeling that requires capturing dependencies different resolutions. We use PTB-XL (Wagner et al., 2020), commonly used dataset of electrocardiogram (ECG) in the time series literature. This dataset has 21,837 ECG recordings, each which with 12 channels, from 18,885 patients. Each recording has at least one label from 71 total ECG labels obtained from SCP-ECG standard. In this experiment, the dataset is partitioned into six subsets of "all", "diagnostic", "diagnostic subclass", "diagnnsttc superclass", "form", and "rhythm"" Following previous studies (Behrouz et al., 2024b; Shi et al., 2023), we use the 100Hz version of the dataset, in which each time series has 1000 timesteps. Table reports the results on ECG classification tasks. MS-SSM outperforms all the baselines, even specialized models for time series (e.ge, SpaceTime (Zhang al., 2023)).
Hierarchical Reasoning. To evaluate the MS-SSM's ability in reasoning about hierarchical structures, we perform experiments on the long ListOps dataset from the Long Range Arena benchmark (Tay et al., 2020b). This dataset consists of sequences with hierarchical structures and operators such as MAX, MIN, MEDIAN, and SUM MOD, which are enclosed by brackets to indicate nested operations. A short example of sequence from this dataset is as follows:
7

---

Published as a conference paper at COLM 2025
Table 3: AUROC for ECG multi-label/mulli-hkee classification the PTB-XL dataset.
Model (AUROC)
All
Diag
Sub-diag
Super-diag
Form
Rhythm
Transformer (Vaswani et al, 2017)
0.857
0.876
0,882
0,887
0.771
0.831
MULT (Shi et al. 2023)
0,938
0.939
0.934
0.934
0.897
0.975
Spacetime (Zhang et al., 2023) $4 [Gu et al, 2022b)
0,936
0.933
0.929
0.883
0.967
0.938
0.939
0.929
0.931
0.895
0.977
Inception (Ismail Fawaz al,, 2020)
0.925
0.931
0,930
0,921
0.899
0.953
ISTM (Hochreiter et al,, 1997)
0.907
0.927
0.928
0.927
0.851
0.953
Wavelet features (Strodthoff et al., 2020)
0.849
0.835
0.859
0.874
0.757
0.890
Mamba Dao, 2023)
0.915
0.929
0,905
0.912
0,876
0.952
MS-SSM (54)
0.939
0.939
0.935
0.930
0.899
0,980
MS-SSM (56)
0.939
0.941
0.936
0.935
0.901
0.979
Table 4: Performances Comparison on the Long Range Arena benchmark (Tay et al., 2020b). The
baselines results are reported by Qin et al. (2024).
Model
Text
Retrieval
Image
Pathfinder
Path-X
AVG.
Transformer (Vaswani et al,, 2017)
61.95
80.69
40.57
65.26
62.12
cosFormer (Qin 2022)
67.70
83.15
51 51
71.96
68.51
FLASH (Hua et al. 2022)
64.10
86.10
47.40
70.25
66.96
S4 (Gu al,, 20226)
86.82
90.90
94.20
96.35
91.38
DSS (Gupta 2022)
84.80
87 87.80
85,70
84.60
87,80 $7.80
86.13
DSSEXPP (Gupta et al,, 2022)
84.60)
87
84.90
84.70
85.60
85.47
DSSEXP-NO-SCALE (Gupta et 2022)
42.40
86.00
81.20
81.30
66.46
TNN (Qin et 20233)
87.90
90.97
88.24
93.00
96.10
91.24
S5 (Smith et 2023)
89.31
91.4
88.00
95.33
98.56
92.52
Mega (Ma et al, 2022)
90.43
91. 01.25
90.44
96.01
97.98
93.22
SGCony (Li et al., 2022)
89.2
91.11
87.97
95.46
97.83
92.31
LRU (Orvieto al., 2023)
89.40
89.90
89.00
95.10
94.20
91.52
Mamba (Gu & Dao, 2023)
82.98
72.14
69.82
69.26
67.32
72.30
Griffin (De 2024)
71.75
66.58
73.38
69.53
66.47
MS-SSM (S4)
87.22
91
89.15
94.90
97.12
91.89
MS-SEM (S6)
85.70
83.21
89.83
87.24
87.70
86.73
INPUT: [MAX
[MIN 6
[MEDIAN 1 711
OUTPUT: 7
Table 2 reports the performance of MS-SSM and baselines on ListOps dataset. MS-SSM achieves the best results compared all baselines. Notably, MS-SSM achieves x2 accuracy compared to Mamba (Gu & Dao, 2023), which shows the significance of multi-resolution modeling of the sequence.
Additionally, the performance improvement is achieved without increasing computational complexity or parameter count. When compared to Mamba models with double parameter count and double state size, MS-SSM consistently exhibits superior performance, highlight- ing its effectiveness and efficient utilization of its multi-timescale memory capturing long hierarchical structures.
Long Range Arena. We further evaluate the performance of MS-SSM on additional tasks from the Long Range Arena benchmark (Tay et al., 2020b). The results, summarized in Table 4, highlight the advantages of MS-SSM over similar data-dependent SSM-based architectures such as Mamba and Griffin. While these models exhibit poor performance long-range tasks, MS-SSM achieves a significant 14.42% performance improvement over its closest counterpart, Mamba, which shares a similar SSM architecture." This performance boost is attributed to the integration of multi-scale convolutions, which enhances MS-SSM's capacity to capture dependencies across various scales and over long sequences.
Ablation Studies. In this section, we evaluate the significance of model design and the made choices by performing an ablation study on ListOps and PTB-XL datasets. To this end, we change the main components of the MS-SSM, one at a time, to evalu- ate its contribution in the performance of MR-SSM. We use the following variants: (1) is the main variant of MS-SSM, when using S6 block as the recurrent module, (2) re-
The primary of this work goal is not to achieve SOTA results on LRA and other benchmarks. As it share similar SSM architecture with Mamba, fair comparison is conducted against it.
8

---

Published as a conference paper at COLM 2025
places the S6 block with S4, (3) removes the recurrent module, (4) removes the mul- tiresolution convolution and instead uses Conv1D, (5) is the original gating for scales, (6) for each scale, we use its own imput for the
Table 5: Ablation the architecture of MS-SSM.
gating, (7) is the gating where each scale is
Method
PTB-XI.
ListOps
gated with the original input, (8) is the orig- inal scale mixing module used in MS-SSM, (9) uses simple linear layer for mixing dif- ferent scales, and (10) uses non-linearity in the gating (dtta-dppendency) of scale mix- ing. The results are reported in Table 5, indi- cating that all components contributes to the performance gain, where main contribution comes from the multiresolution convolution. Additional experimental results and ablation studies (on the types of initialization) are dis- cussed in Appendix C.
Base
1
MS-SSM (S6)
0.939
63.04
MS-SSM (S4)
3
Remove S6/S4
0,936
62.39
Remove Multi. Conv.
0.916
37.98
Gating (Input, Based on)
5
(selí scale, original input)
0,939
63.04
(self scale, self scale)
0,938
62.91
(original input, self scale)
0.939
62.95
Scale Mixing
input-dependent
0.939
63.04
9 None-linear tMax gate
input-independent
0.932
61.28
0.921
61.42
3.1 Effective Receptive Field
We introduce the concept of the mean mixing distance metric to quantify the effective receptive field (ERF) in our model, drawing inspiration from the receptive field in convo- lutional networks. This definition is inspired by the average attention distance defined in self-attention models (Dosovitskiy et al., 2020).
The normalized attention scores between each pair of tokens defines the mapping between each output token and all tokens in the input sequence." Using this, the average attention distance (Dosovitskiy et al., 2020) is defined as: d(m, n) = Σn A(x) m,n X (m n1) where each row of the attention matrix forms a probability distribution over distances (Ben-Kish et al., 2024), as they lie in the (I. - 1) sisimpeex (i.e. the rows sum to 1). In contrast, expressing a closed-form mapping between input and output tokens for y = MS-SSM(x)) = f(x) is not straightforward. Therefore, we rely on the Jacobian of the output with respect to the input to describe how the sequence transformed by a MS-SSM layer. We define the mean mixing distance for MS-SSM as:
d(m,n) n) = 111
mxx)m D mx))
(m 11)
(5)
As the results in Table 6 highlights, MS-SSM achieves significantly higher mean mixing distance than Mamba, indicating its superior ability to attend to distant contexts, thereby capturing long-range dependencies in the sequence more effectively.
4 Conclusion and Discussion
In this paper, we introduced MS-SSM, multi-resolution state-space model for sequence modeling that integrates multi-scale analysis into state space models (SSMs). By decom- posing the system into multiple time scales and incorporating independent SSMs at each resolution, MS-SSM is able to capture dependencies at varying levels of granularity, address- ing key challenge in long-range sequence modeling. The of specialized convolutions and scale-specific parameter initialization enhances the model's ability to efficiently handle both local and global temporal dynamics.
Our extensive experiments across multiple benchmarks, including image classification, hierarchical reasoning, long-context tasks, and time series tasks, demonstrate the effective- ness of the proposed approach. MS-SSM consistently outperforms state-of-the-art SSM architectures, such as Mamba and Griffin. The results in the Long Range Arena benchmark further validate that MS-SSM can handle effectively long-range dependencies, showing sig- nificant improvements over similar data-dependent SSM models. One of the key strengths of MS-SSM lies in its parallelized implementation and minimal computation and model
4Frr simplicity, we assume the value projection is
9

---

Published as a conference paper at COLM 2025
parameters increase, which ensures computational efficiency despite the increased capacity in capturing multi-scale structures.
While MS-SSM is highly effective in capturing multi-resolution and long range dependencies, there remain several avenues for future research. First, extending the MS-SSM framework to other sequence domains, such natural language processing, where hierarchical structures are prevalent, could further validate its generality. Another potential direction is the exploration of multi-resolution in the most recent form of RNN such as LRU (Orvieto al., 2023) and xLSTM (Beck et al., 2024) and analyze how it improves the system's 's memory in such RNN/SSM models.
10

---

Published as a conference paper at COLM 2025
References
Naman Agarwal, Daniel Suo, Xinyi Chen, and Elad Hazan. Spectral state space models. arXiv preprint arXio:2312.06837, 2023.
Masanao Aoki. Stale space modeling of time series. Springer Science Business Media, 2013.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Internationll Conference on Learning Representations, 2015.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prud- nikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended long short-term memory. arxiv preprint ar Xii::2405.04517, 2024.
Ali Behrouz and Farnoosh Hashemi. Graph Mamba:
Towards learning graphs with state space models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 119-130, 2024.
Ali Behrouz, Michele Santacatterina, and Ramin Zabih. MambaMixer: Efficient selective state space models with dual token and channel selection. ar Xiv preprint arXiv:2403.19888 2024a.
Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Chimera:
Effectively modeling multivariate time series with 2-dimensional state space models. In Thirty-eighth Conference on Advances in Neural Information Processing Systems, 2024b. URL https://arxxii.. orgg/bs/r////////////////oo/. 2406. 04320.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document trans- former. Xiv preprint ar Xii::2044005150, 2020.
Assaf Bin-Kish, Itamar Zimerman, Shady Abu-Hussein, Nadav Cohen, Amir Globerson, Lior Wolf, and Raja Giryes. Decimamba: Exploring the length extrapolation potential of mamba. arXiv preprint ar arXio:2406.14528, 2024.
Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. fast unified model for parsing and sentence understanding. ar Xiv preprint ar (xXx:16603.0000 2016.
Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. Evidence of predictive coding hierarchy in the human brain listening to speech. Nature human behaviour, 7(3): 430-441, 2023.
Lei Cheng, Ruslan Khalitov, Tong Yu, Jing Zhang, and Zhirong Yang. Classification of long sequential data using circular dilated convolutional neural networks. Neurocomputing, 518:50-59, 2023.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. Xiv preprint ar ttt:1904.10009 2019.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arxiv preprint MxXiu:149..1259 2014.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.1477 2020.
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024.
Ingrid Daubechies. Ten lectures on wavelets. Society for industrial and applied mathematics, 1992.
11

---

Published as a conference paper at COLM 2025
Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. ar Xrviv preprint ar XtXt:22422.14427 2024.
Hasan Demirel and Gholamreza Anbarjafari. Image resolution enhancement by using discrete and stationary wavelet decomposition. TEEF. transactions on image processing, 20 (5):1458-1460, 2010.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp.
Ieee, 2009.
Iman Deznabi and Madalina Fiterau. Multiwave: Multiresolution deep architectures through wavelet decomposition for multivariate time series prediction. In Conference on Health, Inference, and Learning, pp. 509-525. PMLR, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. Xiv preprint ar XrXi:22000.1192, 2020.
Jeffrey Elman. Finding structure in time. Cognitive science, 14(2):179 -1111199111111 1990.
N Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Mahoney. Lipschitz recurrent neural networks. arXiv preprint arXiv:2006.12070 2020.
Xinyao Fan, Yueying Wu, Chang Xu, Yuhao Huang, Weiqing Liu, and Jiang Bian. Mg- tsd: Multi-granulariiy time series diffusion models with guided learning process. arXiv preprint ar Xxx::2403.0555. 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arviv preprint arXivv2312.00752, 2023.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. HiPPO: Recurrent memory with optimal polynomial projections. Advances in neural information processing systemss, 33:1474-1487, 2020a.
Albert Gu, Caglar Gulcehre, Thomas Paine, Matt Hoffman, and Razvan Pascanu. Improving the gating mechanism of recurrent neural networks. In International conference machine learning, pp. 3800-3809. PMLR, 2020b.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arv preprint arXiv:2111.00396, 2021a.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-tin time models with linear state space layers. Advances in neural information processing systems, 34:572-585, 2021b.
Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Information Processing Systems, 2022a. URL http:::://pperevvv// wwwperrwewww..neeeeeoeeeeeeeeeee nett/oormmiddyyJEiiS...
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022b. URL https://openrew/ wwwpppeereevvieeeee www...www..
Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are effective as structured state spaces. Advances in Neural Information Processing Systems, 35:22982-2295 222994, 2022.
12

---

Published as a conference paper at COLM 2025
Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. arviv preprint arXiv:2209.129511 2022.
Sepp Hochreiter, Jürgen Schmidhuber, et al. Long short-term memory. Neural computation, (8):1735-1780, 1997.
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In International conference on machine learning, pp. 9099-9117. PMLR, 2022.
Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel Schmidt, Jonathan Weber, Geoffrey 1 Webb, Lhassane Idoumghar, P'ierre-llinn Muller, and François Petitjean. Inceptionlime: Finding alexnet for time series classification. Data Mining and Knowledge Discovery, 34(6):1936-1962, 2020.
Mahdi Karami. HiGen: Hierarchical graph generative networks. In The Twelfth International Conference on Learning Representations, 2024.
Mahdi Karami and Ali Ghodsi. Orchid: Flexible and data-dependent convolution for sequence modeling. In Thirty-eighth Conference on Advances in Neural Information Processing Systems, 2024. URL httpp:://aree http::////aaii.oogaa////////// wwtp/os///////////// 18508.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pp. 5156-5165. PMLR, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ar Xiv preprint ar arXiv:2001.044: 044451,,, 2020.
Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. A clockwork rnn. In International conference machine learning, pp. 1863-1871. PMLR, 2014.
Alex Krizhevsky, Tlya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Ashish Kumar, Harshit Tomar, Virender Kumar Mehla, Rama Komaragiri, and Manjeet Kumar. Stationary wavelet transform based ecg signal denoising method. ISA transactions, 114:251-262, 2021.
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. Xrv preprint ar xrXie:2105.03824, 2021.
Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. What makes convolu- tional models great on long sequence modeling? In The Eleventh International Conference on Learning Representations, 2022.
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. aiv preprint ar Xiie2200.110166 2024.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3431-3440, 2015.
Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. Luna: Linear unified nested attention. Advances in Neural Information Processing Systems, 34:2441-2453, 2021.
Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: moving average equipped gated attention. Xiv preprint ar XrXx:2220.11065 2022.
Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In International Conference on Learning Representations, 2018. URL https: https: //opnreviee.e..www. net/foruniid-lee hi=hhmwwuiH-H--c.....
13

---

Published as a conference paper at COLM 2025
Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces, arXiv preprint (rXiv:2206.13947, 2022.
Guy P Nason and Bernard W Silverman. The stationary wavelet transform and some statistical applications. In Wavelets and statistics, pp. 281-299. Springer, 1995.
Piotr Nawrot, Szymon Tworkowski, Michat Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. arviv preprint arXiv:2110013711, 2021.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch- Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems,, 36, 2024.
Antonio Orvieto, Samuel Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pp. 26670- 26698. PMLR, 2023.
Antonio Orvieto, Soham De, Caglar Gulcehre, Razvan Pascanu, and Samuel L. Smith. Uni- aagl
versality of linear recurrences tollowed by non-linear projections: Finite-width guarantees and benefits of complex eigenvalues. In International conference machine learning, 2024.
Eric Pauwels, Luc] Van Gool, Peter Fiddelaers, and Theo Moons. An extended class of scale-invariant and recursive scale space filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(7):691-77) 1995.
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. arXiv 2022.
preprint ar
Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Toeplitz neural network for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023. URL https: //openreview. httpe/iwwww..eeeeeee weett/oormm/ii-xximsndeeeee
Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXio:2404.0790% 2024.
David W Romero, Erik Bekkers, Jakub M Tomczak, and Mark Hoogendoorn. Wavelet networks: Scale-translation equivariant learning from raw time-series. ar preprint drXio:2006.0525,, 2020.
David W Romero, Anna Kuzina, Erik J Bekkers, Jakub doom. Ckconv: Continuous kernel convolution for sequential data. In International Conference on Learning Representations, 2021.
Jakub Mikolaj Tomezak, and Mark Hoogen-
Ivan W Selesnick, Richard G Baraniuk, and Nick C Kingsbury. The dual-tree complex wavelet transform. IEEE signal processing magazine, 2236)1133 2005.
Jiaxin Shi, Ke Alexander Wang, and Emily Fox. Sequence modeling with multiresolution convolutional memory. International Conference on Machine Learning, pp. 31312-31327. PMLR, 2023.
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023. URL https::t///tt httt::::///ppeeviiii www..nee........ net net/forun i d 8Hw3AXqks..
Nils Strodthoff, Patrick Wagner, Tobias Schaeffter, and Wojciech Samek. Deep learning for ecg analysis: Benchmarks and insights from ptb-xl. IEEE journal of biomedical and health informatics, 2519)1151, 2020.
14

---

Published as a conference paper at COLM 2025
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic rep- resentations from tree-structured long short-term memory networks. arXiv preprint #rXiv:1503.00077, 2015.
Alex Tamkin, Dan Jurafsky, and Noah Goodman. Language through prism: A spectral ap- proach for multiscale language representations. Advances in Neural Information Processing Systems, 33:5492-5504, 2020.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. Proceedings of ICML, 2020a.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint ar Xiv:200 Xi0:2001.044066, 2020b.
Trieu Trinh, Andrew Dai, Thang Luong, and Quoc Le. Learning longer-term dependencies in rnns with auxiliary losses. In International Conference on Machine Learning, pp. 4965-4974. PMLR, 2018.
Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu, et al. Wavenet: A generative model for raw audio. arXiv preprint wxXi:1600.004999, 12, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Catherine Wacongne, Etienne Labyt, Virginie Van Wassenhove, Tristan Bekinschtein, Lionel Naccache, and Stanislas Dehaene. Evidence for hierarchy of predictions and prediction errors in human cortex. Proceedings of the National Academy of Sciences, 10865)):20554-20733 2011.
Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima Lunze, Wo- jeiech Samek, and Tobias Schaeffter. Ptb-xl, large publicly available electrocardiography dataset. Scientific data, 7(1):1-15, 2020.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self- attention with linear complexity. arXiv preprint arXiv:2006.04766 2020.
Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. Advances in Neural Information Processing Systems, 32, 2019.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, el. Big bird: Transformers for longer sequences. Proceedings of NeurlPS, 2020.
Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao, Karan Goel, and Christopher Re. Effectively modeling time series with simple discrete state spaces. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevieeeee forum?1d-2EpjkjzdcM.
Yudong Zhang, Shuihua Wang, Yuankai Huo, Lenan Wu, and Aijun Liu. Feature extraction of brain mri by stationary wavelet transform and its applications. Journal of Biological Systems, 18(5-ec01):115-1322, 2010.
Zhenhai Zhu and Radu Soricut. H-transformer-ld: Fast one-dimensional hierarchical attention for sequences. arXiv preprint arX:::2107.11906, 2021.
15

---

Published as a conference paper at COLM 2025
A Details
A.1 Notation definition
Notations
Brief definition and interpretation
Ni Mr, W
the sequence R and R are input and output of layer, while matrices denoted by bold uppercase letters, such as layer weight matrix W.
A, A, B
discretization step size and parameters of the discrete SSM: Á exp (AA) (state transition matrix), B (AA) (exp (AA- []) .BB.
A A'
the superscripts denotes the index of a scale: s E (O...., representation of x,, and A is the SSM parameter applied to that scale.
S 9+ 1}. 8j is the s-th scale
Concatenation vectors {K²,
h 1
C Linearc(x:)
input-depente dependent parameter modeled by Linearc(x)) - We
Convid(1, 2, L, 21-1)
causal depthwise 1D convolution (Convid) with two output channels, kernel length of and dilation factor of 2 applied each feature dimension.
convolution: y(t| (h x] Σ hlt []x({]
I x
element-wise multiplication (Hadamard product): y(d] (I o x)[(]]
k[[] x(1]
diag(A), diag(a)
diag(A) vector containing the diagonal elements of square matrix A, and diag(a):
square matrix formed by entries a on its diagonal.
Softplus(.)
the nonlinearity defined as: log(1 exp(-))
sof smf(ma
Softmax activation function defined as: sof tefttta((u u);
expla;)
Li exp(u;)
A.2 Model Architecture
A.2.1 Scale Mixing
We explored differnt approaches for scale mixing within the proposed architecture: (i) data-dependent scale mixing module, as defined in equation 4, (ii) a simple trainable linear layer for scale mixing that is data-independent, and (iii) a data-dependent scale mixing module, similar to the one in equation 4, but uses non-linearity in its gating, expressed as E, = (F(x) = Sof (Linearp(x:)).
The ablation study results, reported in Table 5, indicate that the data-dependent scale mixing with the linear parameterization from equation achieves the best performance among these methods.
A.3 Effective Receptive Field
We introduce the concept of the mean mixing distance metric to quantify the effective receptive field (ERF) in our model, drawing inspiration from the receptive field in convo- lutional networks. This definition is inspired by the average attention distance defined in self-attention models (Dosovitskiy et al., 2020).
For a length-L sequence of tokens (x1, x1), the self-attention layer transforms the sequence by computing a weighted sum of token embeddings, as follows:
QKT
y SA SoftMax
A(x)x, X,
Vdz
where
Wo or K
WK.
In this equation, the matrix A(x) contains the normalized attention scores between each pair of tokens. Which defines the mapping between each output token and all tokens in
16

---

Published as a conference paper at COLM 2025
Table 6: Comparison of Mean Mixing Distance between Mamba and MS-SSM the ListOps dataset. The metric (m, L), as defined (6), averaged across all channels and layers the model.
Meur Miking Stitanee Lepereeenn tellee
Hoats
Method
Mean Mixing Distance
Mamba
38.84
MS-SSM (56)
94.90+64..2
the input sequence." Using this, the average attention distance (Dosovitskiy et al., 2020) is defined as:
d(m, n) = Σ Axx)ma X (m - 1)
where each row of the attention matrix forms probability distribution over distances (Ben- Kish et al., 2024), as they lie the (L - simmple (i.e. the rows sum to 1).
In contrast, expressing closed-form mapping between input and output tokens for y = MS-SSM(x) = f(x) is not straightforward. Therefore, we rely on the Jacobian of the output with respect to the input to describe how the sequence is transformed by MS-SSM layer. The Jacobian matrix defined as the collection of the gradient of each output token with
f1
respect to the input sequence: 15
We define the mean mixing distance for MS-SSM
fi.
as:
J m
d(m, n)
Σ
(m n)
(6)
m Kk= mxx)
where the Jacobian is normalized row-wise to form a probability distribution over the distance analogous to attention-based models. In classification tasks, compute d(m, L), the mean mixing distance for the last token, as a measure of the ERF, capturing how far dependencies extend across the sequence in MS-SSM.
As the results in Table 6 highlights, MS-SSM achieves a significantly higher mean mixing distance than Mamba, indicating its superior ability to attend to distant contexts, thereby capturing long-range dependencies in the sequence more effectively.
A.4 Efficient Implementation of Multi-scale Decomposition Layer
While computation of multi-scale decompasition (2) requires sequential application of convolution layer, this filtering scheme is actually linear time-invariant (LTI) and can be implemented using linear convolution layers. Composing two linear convolution layers @i and @2 with kernel sizes K1 and K2, respectively, yields a single linear convolution layer P1:2 91 22 with an effective kernel size of K1 K2 - 1. This property enables us to transform this sequential linear convolutions into a parallel application of array of filter banks during inference. When the filter length and number of levels are limited, this ap- proach can potentially accelerate multi-resolution decomposition by leveraging specialized implementations of convolution units available on modern hardware accelerators, resulting in a more hardware-efficien solution.
For simplicity, assume the value projection
17

---

Long Range Arena: We use the settings from Long Range Arena benchmark (Tay et al., x2 of the number of layers for Published as a conference paper at COLM 2025
B Experimental Details
For all the experiments, we use the same experimental setup as Smith et al. (2023) and Shi et al. (2023). The results of baselines are either from the original papers, or are reported by Shi et al. (2023) / Qin et al. (2024).
B.1 Image Classification
We employ the Vision Transformer (ViT) architecture (Dosovitskiy et al., 2020), integrating MS-SSM the core block. The models are evaluated two image classification tasks: SCIFAR (Shi et al., 2023) and ImageNet-1K (Krizhevsky et al., 2012).
SCIFAR-10: For the SCIFAR-10 dataset, each image is transformed into a sequence of pixels with size 1024 and 3 channels, and the model is built using ViT architecture (Dosovitskiy et al., 2020) consisting of 10 layers with a hidden size of 256 and filter size of 2. The Adam optimizer with standard settings (ß1 0.9, B2 0.999) and a learning rate of 0.0045 was used, along with a linear warmup over the first epoch. A weight decay of 0.01 was applied as regularization. We use = 3 and N 128. The model was trained A6000 GPUs for 250 epochs with a batch size of 50.
the case of ImageNet- images were divided into patches of x 16 pixels, and we trained ViT-base architecture (Dosovitskiy et al., 2020) with 24 layers and hidden size of 256. Training was conducted using the Adam optimizer with a base learning rate of 1e-3 and its standard settings (B1 0.9, B2 0.999). The learning rate scheduler included linear warmup for the first 10 epochs, followed by a cosine decay. MS-SSM was trained for 300 epochs using 4xA6000 GPUs with a batch size of 1024, Each MS-SSM layer consists of multi-scale convolution with = 3 scales, each convolution having a length of K = 4, and SSMs with latent state size of N 128.
ImageNet-1K:
We use the setting of Long-range Arena (Tay et al., 2020b) benchmark and pad all sequences to the length of 2048 and then use an embedding layer to encode them into 128 channels. We 20 layers of MS-SSM to mach the number of parameters of other models in the benchmark study. In MS-SSM we choose filter size as 4 and dimension of 128. The model is trained for 100 epochs with batch size of 50. Following Shi et al. (2023), A optimizer with a weight decay rate 0.03, learning rate of 0.003 after 1 epoch of linear warmup, and a dropout rate 0.1. The batch normalization is used instead of layer normalization. We use S = and N = 128.
ListOps:
use
2020b) but to match the number of parameters, Transformers.
this dataset, we have 12 channels, each of which has 1000 timestamps. All the architectural setting for this experiment is the same as the CIFAR10 but instead of batch normalization, we use layer normalization. We use dropout rate of 0.2 and the AdamW optimizer with weight decay rate 0.06. The network is train for 5 warmup epochs and then 95 epuchs of cosine learning rates.
PTB-XL
C Additional Experiments and Ablations
C.1 Ablations
In this section, we compare our initialization with the Mamba's initialization. The results are reported in Table 7. As expected, the scale-dependent initialization scheme proposed in this work is more effective and MS-SSM achieve better performance when using such initialization.
18

---

Published as a conference paper at COLM 2025
Table 7: Ablation studies on the initialization of MS-SSM.
Method
PTB-XI.
ListOps
Base
MS-SSM
0.939
63.04
Mamba's Initialization
0..288
57.49
19


Reviews:
[{"review_id": "qw91NXa2cl", "reviewer": "Reviewer_xBk6", "summary": "This work introduces an extension to the State Space Model (SSM) by adding multi-scale convolution layer. The motivation for this additional layer is to capture multi-scale dependencies in order to better model complex structures. \nFirst, the input signal is decomposed into multiple scales and later combined with a scaler-mixer. The additional components introduce a small overhead in terms of parameters and computation.\nExperiments were conducted on well-known, SSM-related tasks such as sCIFAR, ImageNet-1K and Long Arena benchmark. The introduced model yields superior results over Mamba and other models.\n\nThis work is well motivated and easy to follow. The experimental setup is quite vanilla and this paper could be even more convincing by providing results on a range of popular (NLP-related) downstream tasks (similar to (Gu & Dao, 2023)).\nFurthermore, it would be great to quantity “minimal computation and model parameters increase”.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well written and motivated\n- Simple and effective method to extend SSMs", "reasons_to_reject": "- Vanilla SSM-related experimental setup only", "questions_to_authors": "- Can you quantity “minimal computation and model parameters increase”?\n- Please highlight best (and second best) results in Table 4", "extra_scores": {}}, {"review_id": "KxZXxhy8bV", "reviewer": "Reviewer_FfTf", "summary": "This paper discuss the multi-scale state-space models in the sequence modeling task.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Extensive numerical experiments are conducted, including classical sequential CIFAR, LRA, and ECG benchmarks. An ablation study is also provided to illustrate the contribution of each component.\n2. The metric proposed in Section 3.1 is an interesting concept. Although its relationship with attention mechanisms and practical optimization applicability is underexplored, introducing new metrics is valuable for enhancing the understanding of sequence models.", "reasons_to_reject": "1. The theoretical connection between the multi-scale argument and the model's final performance is limited, as the discussion mainly focuses on input-dependent parameterization for improved multi-scale approximation. It is recommended to extend the theoretical analysis to parameterization theorems, similar to the theorem presented in StableSSM (https://arxiv.org/abs/2311.14495). If it's not achievable, maybe it's better to demonstrate a theorem to show that the improvement from multi-scale resolution is significant even when the models get larger. \n2. A minor issue is the absence of substantial NLP tasks for language modeling and downstream evaluation. Although these tasks are computationally demanding, it is still advisable to train a reasonably scaled model (e.g., 120M parameters) for comparison with GPT-2, Mamba, and GLA.", "questions_to_authors": "NA.", "extra_scores": {}}, {"review_id": "SX6dirJm9v", "reviewer": "Reviewer_x51h", "summary": "This paper proposes MS-SSM, a multi-scale state-space model (SSM) for efficient sequence modeling. It enhances traditional SSMs by processing sequences at multiple resolutions, combining specialized dynamics with a learnable scale mixer to capture both fine-grained and coarse patterns. The authors claim improved memory efficiency and long-range modeling, supported by experiments on Long Range Arena, time series, and image tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The multi-scale SSM framework is a novel extension, with innovations like the scale mixer and specialized dynamics. However, the significance is limited by the lack of consistent performance gains across all benchmarks. More rigorous comparisons with non-SSM baselines (e.g., attention, RNNs) would better demonstrate its advantages. The concept is promising, but the empirical validation needs strengthening.", "reasons_to_reject": "1. The paper focuses primarily on accuracy, but efficiency in sequence modeling involves more than just accuracy-speed, memory usage, and scalability are equally important. The experiments lack comparisons on these aspects, weakening the claim of \"efficient\" modeling. Additionally, the performance gains are inconsistent: Table 1 shows marginal improvements, while Table 2 reports large gains. Without robustness and generalization analysis, the overall impact remains unclear.\n2. The motivation is too broad—while multi-scale modeling is important, the specific gaps in SSMs are not sharply defined. A clearer justification for why existing SSMs fail at multi-scale tasks would strengthen the contribution. The writing is clear, but the experimental results need deeper discussion to explain the inconsistent improvements across tasks.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Z7FOpZpJP", "reviewer": "Reviewer_Rtcz", "summary": "This paper introduces MS-SSM, a multi-scale state space model for efficient sequence modeling. The method decomposes the input into multiple temporal resolutions using a wavelet-inspired convolutional layer, applies a separate SSM at each scale, and combines their outputs through an input-dependent mixing mechanism. This design aims to address the limitations of standard SSMs in modeling long-range and hierarchical dependencies. The model is evaluated on a diverse set of tasks including image classification, time series classification, and hierarchical reasoning, achieving consistent gains over prior SSM-based models such as S4 and S6, and outperforming Mamba in several benchmarks. Ablation studies highlight the importance of both the multi-scale decomposition and the scale mixer.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The multi-scale decomposition addresses known limitations of standard SSMs in modeling long-range and hierarchical structure, and the overall architecture is clean, modular, and intuitive.\n\n2. The model consistently outperforms prior SSM-based baselines on a diverse set of tasks including vision, time series, and structured reasoning.\n\n3. The paper carefully isolates the contributions of the multi-scale convolution and the scale mixer, providing insight into which components are most important for performance.\n\n4. The paper is well-organized and easy to follow, with sensible baselines and reproducible experimental details.", "reasons_to_reject": "**1. Limited applicability to selective scan models**\n\nThe proposed method is evaluated only with S4 and S6, which use standard recurrence or input gating. However, recent state-space models such as Mamba rely on selective scan, where recurrence parameters are computed from the input at each position. Applying the multi-scale framework to such models may require running a separate selective scan per scale, which increases memory usage, compute cost, and may lead to training instability. The paper does not address how MS-SSM could be extended to this class of models, leaving open questions about its compatibility with the latest SSM architectures.\n\n---\n**2. No evaluation on long-context NLP tasks**\n\nThe paper focuses on improving long-range and hierarchical sequence modeling, but it does not include any experiments on natural language (NLP) tasks where these challenges are most critical. Long-context benchmarks such as lost-in-the-middle are commonly used to evaluate this ability. Without such evaluations, it is difficult to assess the model’s effectiveness in NLP settings where long-context reasoning plays a central role.", "questions_to_authors": "- Extension to selective scan models like Mamba\n- Evaluation on long-context NLP benchmarks", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces MS-SSM, a multi-scale state-space model framework that addresses limitations in traditional SSMs by processing sequences at multiple resolutions with specialized dynamics and an input-dependent scale mixer. The work presents a well-motivated and technically sound approach to enhance SSMs' ability to capture both fine-grained patterns and coarse global trends, demonstrating consistent improvements across diverse benchmarks including Long Range Arena, time series classification, and image recognition tasks. The multi-scale decomposition is intuitive and addresses known limitations of standard SSMs in modeling hierarchical dependencies. The experimental evaluation is comprehensive with proper ablation studies that isolate the contributions of key components, and the writing is clear and well-organized throughout.

Pros: Well-motivated multi-scale framework addressing clear limitations in existing SSMs; consistent performance improvements across diverse tasks (vision, time series, structured reasoning); comprehensive ablation studies demonstrating the value of each component; clean, modular architecture that integrates naturally with existing SSM designs.

Cons: Limited evaluation on natural language processing tasks, particularly long-context NLP benchmarks where multi-scale modeling would be most critical. The work could be improved with experiments in standard language-model tasks.
Decision: Accept