Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation
Authors: Yehjin Shin
Seojin Kim
Noseong Park
Venue: ICLR 2026 Poster (2026)
Keywords: State Space Model
Mamba
Graph Signal Processing
Adaptive Filter Bank
Primary area: foundation or frontier models, including LLMs
TL;DR: HADES reinterprets Mamba2 as a graph-based adaptive filter bank, achieving efficient and interpretable sequence modeling with fewer parameters.

Abstract: State-space models (SSMs) offer efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, uses selective input gating and a multi-head structure, enabling parallel computation and strong benchmark performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we propose a novel method called **H**ierarchical **AD**aptive filter bank for **E**fficient **S**SMs (*HADES*), a Graph Signal Processing (GSP)-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. Our hierarchical architecture introduces two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved through structured bias on the parameter $\Delta$. *HADES* achieves comparable performance to baseline models including Mamba2 across various benchmarks in language modeling, commonsense reasoning, and long-context retrieval, while using only **58.9%** of the original parameters. In this regard, *HADES* bridges GSP and neural sequence modeling, enabling efficient, hierarchical, and interpretable filtering within state-space models.

Published as a conference paper at ICLR 2026
GRAPH SIGNAL PROCESSING MEETS MAMBA2: ADAPTIVE FILTER BANK VIA DELTA MODULATION
Yehjin Shin', Seojin Kim,, Noseong Park
KAIST
(yehjin..tt shin, seojinkim, (oseong)@kaisse
ABSTRACT
State-space models (SSMs) offer efficient alternatives attention with linear-time recurrence. Mamba2, a recent SSMM-based language model, uses selective input gat- ing and multi-head structure, enabling parallel computation and strong benchmark performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we propose a novel method called Hierarchical Daptive filter bank for Efficient SSMs (HADES), a Graph Signal Processing (Gsp)-inspired framework that reinterprets Mamba2 as an adaptive filter bank on line graph. Our hierarchical architecture introduces two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved through structured bias on the parameter A. HADES achieves comparable performance to baseline models including Mamba2 across various benchmarks in language modeling, commonsense reasoning, and long-context retrieval, while using only 58.9% of the original parameters. In this regard, HADES bridges GSP and neural sequence modeling, enabling efficient, hierarchical. and interpretable filtering within state-space models.
INTRODUCTION
Transformer architectures have emerged as the dominant approach for sequence modeling across a range of tasks, in- cluding text generation and machine translation. However, their inherent limitations, most notably the quadratic com- putational complexity, have motivated the development of more efficient, sub-quadratic alternatives (Gu et al.
2.5
Effective Rank
2022. Yang et al.
2024b| Smith et al.) 2023: Poli et al.
2023 Peng et al.
2023 Sun et al., 2024). In particular, Mamba (Gu & Dao
(223) and Mamba2 (Dao & Gu, 2024
1.5
have demonstrated that continuous-time SSMs can match surpass transformer baselines in diverse sequence mod-
cling tasks.
HADES
Mamba2
Despite their empirical success, the internal structure of Mamba2-especcally the role of multi-head recur- rence- recerremiin explored Prior works have focused on improving long-con context performance by delta modula- tion (Ben-Kish et al., 2025: Azizi al. 2025 Ye et al., 2025). Another line of work (Wang et al.. 2025) has critically examined the architectural limitations of SSMs and Mamba, highlighting issues such as recency bias and information bottlenecks. The authors proposed polarization as potential solution to these challenges. However, their work mainly relies simplified experimental settings, leaving its effectiveness on real-world tasks unexplored. Moreover, it is not well understood how different heads contribute to the overall representation, whether they exhibit complimentary dy- namics. In Fig. I effective rank analysis reveals that the heads learned by Mamba2 collapse into
Distribution of layer-wise Ef- fective Rank from the spectral responses of Mamba2 and HADES
Figure 1:
"Equal contribution. Corresponding Author.

---

Published as a conference paper at ICLR 2026
low-rank spectral subspaces, suggesting that most heads operate in highly overlapping frequency regimes rather than functioning as diverse, complementary filters (see Section 4.2 for details).
To address this issue. reinterpret and enhance Mamba2 within the framework of Graph Signal Processing (GSP). Specifically, we model the input sequence as a signal on a line graph, where tokens serve nodes and their temporal connections form edges. In this view, each recurrent head in Mamba2 functions as a graph filter applied to this signal. This perspective naturally leads to a filter bank interpretation, where individual heads can be understood as specialized filters, each capturing distinct spectral characteristics of the input.
Building this formulation, further propose hierarchical filter bank model architecture, HADES, which allows adaptive and efficient information flow. HADES organizes filters into two functional categories: (1) shared filters, which perform globally consistent filtering across the sequence, and (2) expert filters, which adapt their filtering behavior on a per-token basis.
HADES demonstrates competitive performance across a diverse set of benchmarks, including two language modeling tasks, eight zero-shot commonsense reasoning tasks, and long-context retrieval, while utilizing only 58.9% of the parameters compared to Mamba2. For interpretability, we further examine HADES through case study and spectrum analysis, demonstrating how our filter bank approach affects the model's internal dynamies. By seamlessly integrating principles from GSP into neural sequence modeling, our method offers scalable, hierarchical, and transparent filtering mechanism within state-space models.
Our main contributions are follows:
GSP-Inspired Adaptive Filtering: We establish novel theoretical framework by reinter- preting Mamba2 as a graph filter bank operating over line graph, creating principled bridge between state-space models and GSP that enables more effective sequence modeling.
Hierarchical Filter Architecture: We design an adaptive filtering system that optimally combines shared and expert filters through GSP-inspired delta modulation and bias mecha- nisms, enhancing model expressivity while maintaining computational efficiency.
Efficient and Scalable Performance: Our approach achieves superior results across various benchmarks while using 58.9% of the parameters required by Mamba2. Through compre- hensive spectral analysis, we demonstrate how our adaptive filtering strategy effectively captures both local and global dependencies in sequence data.
2 BACKGROUND
Our method, HADES, is based on reinterpretation of structured sequence models from the perspective of Graph Signal Processing (GSP). We view the multi-head state-space model (SSM) as a learnable graph filter bank, where each head captures distinct frequency-selective dynamics. This section outlines the necessary background on SSMs and GSP to support this perspective.
2.1 STATE SPACE MODELS (SSMS) AND MAMBA
Structured state space models represent new category of sequence models in deep learning, drawing connections to RNNs, CNNs, and traditional state space models. These models are motivated by specific continuous system that processes one-dimensional input sequence E R into an output sequence y E gº via an implicit latent state R'NN, Eq. is a fundamental representation of organized SSMs.
(((() An(() + Bx(() ut) Ch(t) + Da(()
h A 1 Br
(1)
yt = Ch2 + Dr
where A, NNN B, NXXI C, E R IxN This continuous SSMs in Eq. | are discretized to Eq. through fixed formulas: A fA(A, = fn(A, For the remainder of this paper, we will omit the parameter D for exposition (or equivalently, assume D 0) because the term Da, can
2

---

Published as a conference paper at ICLR 2026
be viewed skip connection and is easy compute.
K [CB, CAB CA*B] y = x # K
In S4 (Gu et al. system parameters A, B, C do not change over time. The resulting sequence model can be computed either as a linear recurrence or global convolution using the kernel K in Eq. 3
2.22). the authors refer this formulation linear time-invariant (LTTT), meaning the
Using definitions from Dao & Gu (2024), we describe Mamba's internal dynamics. Each vector is designated as a row vector. Assuming that U [ui, U2,.., ur] T E R2'xd, that is, un E Rd, is discrete time sequence of ' tokens, the inner equation for the t-th token of each head of the Mamba layer can be understood as follows:
h4 A,h1- + B E ERNXP, y1 = C.ly E ER
0g = W, WNoorm( 9 W2)))) R.
where t is current time step, 2t, Ye E R" are projected input representation and output hidden representations of t-th token respectively, Norm denotes RMS normalization (Zhang & Sennrich 2019), W. E R Pxd W. E RR&PP are trainable parameters. Especially, in Mamba2, A, is scalar- identity matrix, i.e. A, a,I. We denote d for hidden representation dimension, N for state size, P for dimension of each head, T for sequence length.
(6) By A, Mamba implements imput- meput-dependent selection mechanism. A decides the discretization step size in Mamba, which used formulate SSM parameters A,; B,. Detailed parameterization of A, B₁ Ct, are deferred to Appendix
b,ssc W ba RR A, Softplus(Ar,es!! R
2.2 GRAPH SIGNAL PROCESSING (GSP)
Graph Signals and Filtering. Graph Signal Processing (GSP) provides tools for analyzing and processing data defined over graph structures. In GSP, signal is defined as a vector x E TRA, where each element is associated with a node a graph of N nodes. One of the core operations in GSP is graph filtering, which can be viewed as form of graph convolution. This operation emphasizes or suppresses specific frequency components of the signal based on the graph topology. Given a typically chosen the adjacency matrix the (normalized) graph Laplacian-a linear graph filter G is often defined as a polynomial in S:
shift operator S E I Nx Nypicllyl
K
y= = En.s
k=0
where x is the input graph signal, hg are the filter coefficients (also called filter taps). and K the filter order. This convolution operation aggregates information from neighboring nodes up to K hops away, as determined by powers of the shift operator. This filtering can also be interpreted as linear time-invariant (LTT) system graphs, where the filter coefficients ha determine the system's 's impulse response under the graph structure. This system-theoretic view enables a conceptual connection to structured sequence models such as SSMs, which we explore in the following sections.
Graph Filter Banks. A graph filter bank applies multiple filters to a graph signal and combines their outputs to form unified representation. Given graph signal x E R and graph shift operator NNN, the filter bank output can be expressed as:
M + (1E K=
y = + ((x00)),))
(6 h
Sk
where 1,(0) are the coefficients of the i-th filter, K is the filter order, M is the number of filters in the bank, and (c-) denotes the aggregation function over the filter outputs (e.g., concatenation, summation, or projection). This general form enables the system to capture diverse frequency characteristics of the graph signal through multiple learned filters. Our method adopts this perspective to reinterpret the heal SSM as a learnable graph filter bank, where each head corresponds to a distinct frequency response. While our model does not explicitly compute the graph spectrum, this filter bank perspective serves as conceptual tool for understanding the role of the learned dynamic filters.
3

---

Published as a conference paper at ICLR 2026
2.3 SSMs IN THE PERSPECTIVE OF GSP
SSMs as Graph Filters. A one-dimensional token sequence can be naturally represented as signal defined on line graph (i.e., linearly connected graph), where each token corresponds node and edges connect adjacent tokens in the sequence. This perspective enables the application of GSP tools sequential data. particular, the line graph admits natural notion of convolution, where filtering operations over token sequences can be interpreted as graph convolutions. This provides a principled foundation for analyzing state-space models from a GSP perspective.
Specifically, the S4 model can be viewed as a LTI system operating on line graph, where its kernel acts as the convolutional filter. This interpretation allows the SSM to be expressed graph convolution over the input sequence, offering a unified framework that bridges sequence modeling and GSP framework Eq.
K
y == = Σ (CA'B) k=0
S*x
hv
In contrast, Mamba can be interpreted as a linear time-varying (LTV) system operating on a line graph. Unlike an LTI system, which applies the same filter across all nodes, Mamba applies distinct, input-dependent filters at each node, enabling more flexible and adaptive sequence modeling. This formulation can be written as:
K yr Σ K=0
(10)
where A I A; means cumulative product of A, from shift start index for l hops. For more explanation on GSP and Mamba, refer to Appendix B.T
Multi-Head SSMs as Filter Banks. rences, one per head, formulated as:
Mamba2 employs multiple parameterized space recur-
h A (
B(D Ity yi
(11)
where i E [M] indexes the heads. This structure can be interpreted as a filter bank, with each head i acting distinct filter applied the input signal It.
M (0) + ((x(0)
K (f
M
BQ
B s
(12)
1-11
where ()) E RNN B(d)) E RXI ((U) E RIIN are parameters of SSM equations and M denotes the number of filters. Likewise, we can interpret multi-head architectures into graph filter bank. In Fig. 2(a) we illustrate multi-head SSMs interpreted as graph filter banks.
Due to the use of head-specific recurrence parameters and potentially time-varying coefficients, the model is capable of exhibiting diverse temporal and spectral responses across heads. Nonetheless, Mamba2 imposes no explicit structural constraints or functional differentiation among heads. The learned filters are not directed toward specific frequency bands or contextual roles, resulting in an unstructured and static filter bank. This lack of coordination may hinder the model's ability to jointly capture both global and local dynamics in the input sequence. To address these limitations, we introduce structured and adaptive filter bank design that encourages functional diversity across heads while enhancing the model's capacity to capture both global and local sequence.
3 PROPOSED METHOD
3.1 HADES:: HIERARCHICAL ADAPTIVE FILTER BANK FOR EFFICIENT SSMS
From the perspective of adaptive filtering, a key challenge lies in how to effectively select and combine diverse filters. To enhance the structural expressivity of Mamba2 without compromising its efficiency, we propose an adaptive filter bank architecture based on GSP principles. Our approach decomposes the multi-head structure into complementary components: shared filters and expert
4

---

Published as a conference paper at ICLR 2026
Linear
Lincaz
Expert
Expect
Repertty
Eapert
Sehocted
Top-
Not Selected
HAD
Shandl
Si3M
PilLep
X
Linenn
Router
Cony
A
Coav
Libear
Linear
Mraa
(a) Mamba2
(b) HADES
Figure Architectural Comparison between Mamba2 and HADES. Mamba2 applies all filters uniformly to every input token, whereas HAAESS employs routing mechanism that selects and activates filters conditioned on the spectral residual ry and A.
filters. router is employed select the Top-Q expert filters, where the expert scores computed based on the spectral residual and the characteristics of the input sequence.
Fig. iillustrates how method funetions architecture, where all filters are always utilized regardless of context. In contrast, the proposed method in Fig. 2(b) selects a subset of filters to be used each timestep f. Among them, the shared filters are always applied, independent of the router's selection. This yields efficiency with fewer parameters than Mamba2:
filter bank. Fig. 2(a) shows the general Mamba2
defer analyses in Appendix F In our method, from M filters of the general Mamba2 architecture, H filters are selected at each timestep. These H filters are composed of shared filters and expert filters, which are dynamically chosen based on the routing mechanism. The final output is computed weighted linear combination of the selected filter outputs.
Expert Filters. To enable token-level adaptivity, we introduce a router that assigns a subset of expert filters each token based its frequency characteristics. Specifically, for each token at time step 1, we compute the spectral residual as r; = 2a The base delta parameter A Ar,u concatenated with the
where 14 a running mean across the
sequence, i.e., 14 (n..n(1,...... (.,,,...,33 residual rt, and the resulting vector is passed through a linear projection to compute selection scores s, for the expert filters:
= fe([Ar, [t]), Tt x2
142
(13)
where fe is a function that computes expert selection scores based on both the base Arb and the token's is spectral residual and -I denotes vector concatenation. The resulting score vector 8, E F contains a scalar score for each the E expert filters. The Top-Q filters with the highest scores then selected and applied to the token. While expert filters are not explicitly assigned to specific frequency bands, their distinct configurations induce varied update dynamics, implicitly shaping their responses based on the token's frequency characteristics.
The residual rt is used not only for expert selection, but also for modulating the delta value itself. Specifically, it introduces frequency-sensitive bias that adjusts A in accordance with token-level frequency characteristics, spectral bias:
ALHADDS Softplus A base + - re]))
(14)
where fo is function that generates a content-aware adjustment to Ar.me based on the token's ' residual Tt. and a scaling hyperparameter that controls the strength residual-basse modulation. We use single-layer linear projection for fo and fo in our implementation.
Shared Filters. Shared filters are always applicd regardless of the router's 's selection, without addi- tional bias and relying solety the hase A base- Designed process globally smooth components

---

Published as a conference paper at ICLR 2026
present throughout the sequence, they do not incorporate per-token modulation like expert filters, which explicitly respond to high-frequency deviations such as I μ.. While not explicitly constructed as low-pass filters in the spectral domain, their uniform and content-agnostic operation tends to preserve low-frequency patterns and attenuate high-frequency variations. This behavior mirrors the role of fixed low-pass filters in GCN-style models (Dong et al., (02T) and structure-pres pressevii averaging in (Wu et al.) 2002), both of which apply smoothing without input-dependent bias. Such design establishes a stable spectral foundation and reduces the risk of over-adaptation.
3.2 TRAINING LOss TERMS
To ensure effective learning of the adaptive filter bank, it is crucial that the model utilizes diverse set of filters rather than overfitting subset. Without appropriate regularization, the model may converge to using only a few filters, leaving others underutilized. Such underutilized filters may fail to generalize effectively, leading underfitting, where some filters are insufficiently trained due to limited exposure diverse sequence patterns. To address this challenge, we introduce dual loss mechanism that encourages balanced filter utilization during training. Specifically, we apply two complementary objectives:
Load Balance Loss. To prevent the model from collapsing to a small subset expert filters, we add regularization term that encourages more balanced usage of all available experts. Specifically, we compute the squared coefficient variation over the selection scores to penalize high variance in expert preference:
Var(s:)
balance
(15)
(E|s,)) +
where 8, Se([ (r((Artta babe r:))) )) is the vector of selection scores for the E experts time step I, and is a small constant for numerical stability. Minimizing this loss encourages the model to distribute its attention more uniformly across different experts.
Diversity Loss. This term ensures that each filter not only gets selected but also effectively con- tributes to the model's output. We achieve this by introducing variance-based regularization on the filter responses, encouraging their outputs to be decorrelated. Concretely, we compute the pairwise similarity of normalized filter outputs and penalize deviations from orthogonality:
[1 if Laiversiyy = Es, ((s...3)) -5,5)² 5.)))] 5. = 0 otherwise
(16)
where y; denotes the fz-nomallieed output of the i-th expert filter. This loss encourages filter outputs to be mutually dissimilar, promoting specialization and functional diversity across experts.
Final Loss Term. The final training objective combines these two components:
L = Lusk + A Liaatane + Xa Ldiversity
(17)
where Lusk is the primary task loss (cross-entropy loss for language modeling), and Ay. Ag are hyperparameters controlling the strength of the selection and diversity losses respectively. This dual loss mechanism ensures that the model effectively learns a diverse set of filters, each specialized for different aspects of the input sequence.
4 EMPIRICAL STUDIES
4.1 EVALUATION
Setup. Our experiments encompass comprehensive comparison of recent state-of-the-arr architec- tures. We evaluate against the following baselines: Linear Transformer (Katharopoulos et al. 2020
RetNet (Sun et al. 2024), Mamba (Gu& & Dao 2023), Mamba2 (Dao Gu| 2024). and DeltaNet YYngg
et al.
2024b). We trained all models using approximately 200B tokens from the Pile dataset (Gao
et al. 2020). We adopt all baseline implementations from
Yang
& Zhang 2024). For model,
used hyperparameter set of H 16, S 8, t 1c
A Ie 3. 7 = 25e 2. For ablation studies, we maintain all hyperparameter configurations
constant, modifying only the target under evaluation. Detailed settings are in Appendix C

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison on language modeling and zero-shot common-sense reasoning. The best results are highlighted in bold, while the second-bes results are underlined. Avg. denotes the average of accuracies and normalized accuracies over tasks. With only 58, 9% of parameters compared to baseline models, HADES achieves comparable even better performance.
Model
Train.
Wiki.
LMB.
BoolQ
Hella.
Wino. acc
ARC-c acc
PIQA acc
OBQA.
Avg. tasks
ppl
ppl
ppl
acc T
acc
acc_n]
acc
acc_n
Lincar Transformer
2.49
45.43
73.93
24.06
61.30
28.20
51.30
42.05
21.76
60.55
27.60
39.63
RetNet
62.40
DeltaNet
2.29
33.25
26.82
35.75
54.07
31:40
49.96
44.11
22.18
63.60
29.60
41.33
Mambal
22.43
Mambu2
2.33
31.34
24.38
36.46
53.88
32.62
50.83
45.29
24.15
63.44
26.40
41.63
HADES (Ours)
2.31
31.48
21.74
39.24
58.84
32.82
$2.64
45.03
22.01
63.93
28.80
42.91
Language Modeling and Commonsense Reasoning. In Table [ we present the performance of each model across multiple benchmarks, including language modeling perplexity and zero-shot accuracy on commonsense reasoning benchmarks for models with 370M parameters. Even with 58.92% % of parameters (218M), HADES consistently outperforms other baselines, including Linear Transformer, RetNet, Mamba, Mamba2, and DeltaNet. More experiment results are in Appendix D.2
Long-context Retrieval. To evalu- ate the long-range memory capacity, we adopt the passkey retrieval task, where a key-value pair is planted at various depth in a long sequence and queried the end. Experimental de- tails are in Appendix C.2 In Fig.
10
10
Depa Percent
Deph Percent
Score
Scor0
1100 50
>00
16k
16k
Taken Limit
Token Limit
(a) Mamba2
(b) HADES
the results show that our model sigmil- icantly outperforms Mamba2, demon- strating the effectiveness of our GSP-inspinee adaptive filtering in retaining distant dependencies.
Figure Passkey retrieval result of Mamba2 and HADES
4.2 IN-DEPTH ANALYSIS
Why Graph Signal Processing is Effective? Our method fundamentally grounded GSP, which offers principled framework for understanding and designing filtering behavior over graph-structured sig- nals. This spectral view brings clarity the model's s internal dynamics: shared filters serve as filters that capture smooth, global trends across the sequence, i.e. low frequency components, while expert filters, modulated by local signal variations, act filters that adaptively detect sharp, localized changes, i.e., high frequency components, or occasionally choose to observe low frequency components. The GSP per- spective bridges the gap between theoretical signal processing and practical neural architecture design. Our model is not just a collection of independent recurrent heads, but adaptive filter bank, where each head (filter) can specialize in processing spe- cific frequency bands and complementary each other. This structured filtering hierarchy aligns with the spectral nature of graph signals, while maintain- ing the scalability and efficiency of our model.
Task Description
100
Dummy Text
Token Index
Passkey
200
Dummy Text
300
400
Query
12 14 16
22
Filter Index
2
3
Not Selected
Top-K Rank
Figure 4: Expert filter selection Passkey Retrieval task
Filter Selection Analysis. To investigate filter selection patterns of HADES, we analyze the selection tendencies of each filter under the passkey retrieval task, which provides relatively clean separation specifc and irse-inrel tokens. The prompt used for passkey retrieval is divided into four semantic regions: Task Description, Passkey, Query, and Dummy Text. The exact textual
7

---

Published as a conference paper at ICLR 2026
Sogancee
Segunsce
Norml hnd Magmitude
Nomo Roud
Fresuency
000
-200
Prequency
Mamba2
(b) Shared Filter (HADES)
(c) Expert Filter (HADES)
Spectrum of filter inputs and outputs from Mamba2 and HADES. The x-axis represents the Fourier frequency bins, and the y-axis shows the normalized magnitude of the Fourier coefficients, with larger values indicating stronger frequency components (see Appendix E.4 for details).
Figure 5:
contents of each part are provided in Appendix C.2 Among these, Passkey and Query are directly
related to the task and thus can be considered task-specific content. As shown in Fig. we observe distinct selection patterns across filters depending on the regions of the input prompt. Filter 0, 11, and 14 are predominantly activated in task-specific regions, such as the Passkey and Query regions. In contrast, Filter 4, 7, and 20 are tend to be selected primarily within Dummy Text region, suggesting a possible specialization for irrelevant noisy input. Notably, the Task Description region, located at the beginning of the prompt, shows broader diversity in filter selection compared to other parts. This behavior suggests that in the initial stage of the prompt, the model may leverage multiple filters to encode general context and task intent before narrowing down to more specialized filters in later segments. This transition from diverse focused selection suggests an adaptive routing mechanism, where filters self- organizge to capture both high-level instructions and low-level execution signals.
Output Spectrum Analysis. For output spectrum analysis, we apply Fourier transform to filter outputs obtained from a randomly sampled sentence from the Pile dataset. In Fig 5 the kernel characteristics directly influence the information each model learns; while Mamba2 outputs in Fig. 5(a) mainly preserve low-freque fewffrrquency information, HADES captures a more diverse range of signals through the shared and expert filters. The shared filters (Fig. 5(b). shaped by smooth kernels, consistently emphasizing low-frequency components, aligning with their role in capturing stable, global information across the sequence. In contrast, expert filters in Fig. [(c) learned by rippled kernels, demonstrate more dynamic filter response, highlighting their adaptive specialization in capturing localized details. This spectral distinction reflects our model's 's design: shared filters ensure stable contextual foundation, while expert filters dynamically adapt to fine fine-grained variations.
Filter Frequeney Response Analysis. To characterize the in- trinsic spectral behavior of our model, amalyze the frequency response of its learned filters, examining how the kernels them- selves react to different frequency components. For the filter frequency response, we analyzed the frequency response of its filters (cf. Eq. [2p, following the procedure proposed in Wang iet al.] 20022) Our analysis in Fig. reeeelss that the majority of Mamba2's learned filters behave as smooth kernels. This bias toward smooth filtering implies that Mamba2 tends to prioritize low-frequency or long-range information while insufficiently capturing high-frequency variations. As result, many heads converge to similar, general-purpose behaviors, leading to sub- stantial redundancy across filter outputs. A detailed analysis of this output redundancy is provided in Appendix E.3
HRDES (Exper)
Normakoed Megritude
(MDES (Shanes)
3.2
Frequancy
Munba2
Nomakaad Mageitude
0.0
0.02
In contrast, HADES exhibits more diverse set of filtering behaviors: alongside smooth kernels, we also observe clear rippled kernels that respond to higher-frequency components. The presence of ripple kernels provides enhanced sensitivity to high-frequency bands, and through our modulation and expert-
Frequancy
Figure 6: Filter Spectral Responses of Mamba2 and HADES (see Ap- pendix E.5 for details).
8

---

Published as a conference paper at ICLR 2026
Expen Ouent Spactral Inpit Sequence
Sspert fiber Outour inpur Sequence
18h
0.8
Normal bodd Magnuude
Courn Iteg Sccale)
Noma
100
05
Precquee suneenyy
Prequency
A S.Shift by Spectral Bias
(a) With Spectral Bias
(b) Without Spectral Bias
(c) Histogram of A,HADES A.
Figure 7: (a-b) Comparison on frequency spectrum of filter outputs from expert filter with bias and expert filter without spectral bias. The x-axis represents the Fourier frequency bins, and the y-axis shows the normalized magnitude of the Fourier coefficients, with larger values indicating stronger frequency components (see Appendix E.4 for details). (c) Histogram of spectral bias.
selection mechanisms, the model learns information at multiple spectral resolutions. As a result, HADES leverages not only low-frequency dominant global context but also high-frequency driven fine-grained structure in more balanced manner. This indicates that our high-frequency-aware modulation and routing design enables the model to capture a wider range of spectral patterns. Consistent with this observation, HADES shows noticeably higher effective rank than Mamba2, suggesting that it learns more expressive and less redundant filter bank.
Effect of Spectral Bias. We further explore the impact of spectral bias on expert filters. Specifically, Fig. ) shows the frequency spectrum of the output generated by using the original delta without spectral bias, A. In contrast, Fig. 7(a) illustrates the effect of applying spectral bias to expert filters, ALMADES, where clear upward shift in frequency distribution is observed. This shift indicates that the delta values are tuned to capture higher- tigher-freueecy details, enabling the model learn finer-grained information. Fig. 7(c) presents log-scale histogram of the difference between A, HADES and A,- calculated over 25 randomly sampled sentences from the Pile dataset, totaling approximately 38,000 tokens. Throughout our analysis, we observe that the spectral residual bias is generally positive, which encourages larger delta values and enables the model to effectively capture high-frequency information, which aligns with Fig. [7a) Occasionally, the bias becomes negative, reducing the step size for certain tokens and allowing the model to better capture global context. This adaptive mechanism allows HADES to flexibly balance the extraction of local and global information, adjusting to the needs of each token in context.
4.3 ABLATION STUDIES
In this section, we conduct ablation studies on our model, HADES, to systematically evaluate the im- pact of each component and design choice on model performance. For more ablation result, refer to Ap- pendix I We first ablate on two auxiliary losses: Ldiversity and Lhalance As shown in Table 2 the best performance is achieved when both losses are applied together. Interestingly, without Loatance. performance drops as filter selection becomes overly concentrated on few filters, leaving the rarely selected filters under-trained and preventing them to learn dynamics effectively when they are eventually chosen. We also evaluate the impact of different filter configurations: Using Only Shared Filters and Using Only Expert Filters. Using only shared filters outperforms using only expert filters, as shared filters consistently capture global lowffrequency information, while expert filters adaptively capturing low and high frequency information.
Table 2: Ablation Studies
Methods
Wiki.
LMB.
Avg.
ppl
ppl 4
8 tasks f
HADES (Ours)
31.51
21.74
42.91
w/o Laalance w/o Laveesty
34.73
26.84
41.57
33.83
27.40
42.15
Only Shared Filters
34.55
27.64
42,21
Only Expert Filters
36.34
30.12
41.68
Fixed Random
34.55
27.64
42.21
35.78
32.77
41.03
Pos. Bias No Bias
30.23
21.93
42.15
34.57
28.79
41.11
We conducted additional ablations on filter selection and delta modulation. Fixed uses a constant set of filters; and Random randomly samples filters without considering the input. We further

---

Published as a conference paper at ICLR 2026
evaluated three variants in Appendix E.1 For delta modulation, Position Bias (Pos. Bias) adds token-wise positional information; and No Bias removes modulation entirely. In filter selection, Fixed outperformed random selection, and Input-only achieved reasonable performance. However, spectral-basd -aase selection (HADES) produced the best results by adapting to input characteristics. Pos. Bias helped, regarding delta modulation, but original bias was more effective, it captures relative importance beyond absolute positions.
5 RELATED WORKS
Graph Signal Processing in Language Modeling. Recent work has explored interpreting Trans- former architectures through GSP. In this view, the self-attention mechanism functions as a graph filter, where the attention matrix acts a learned adjacency matrix, with tokens as nodes and attention weights defining edges. GFSA (Choi et al. (024) explicitly models self-attention as a graph filter on fully connected graph, while ContraNorm (Guo al.) (023) treats it as normalized adjacency matrix, connecting it to Graph Neural Networks. The Anti- Anti-verrmmothhin framework (Wang et al. low-pass filter, highlighting its smoothing effect in
2022) further characterizes self-attention the spectral domain. Unlike Transformers, represented with fully connected graphs, SSMs operate sequentially, best represented by a line graph with unidirectional information flow. This insight motivates our GSP-based filter bank approach for Mamba2, recent SSM-based model.
Adaptive Filtering and Mixture of Experts. Recent methods have improved multi-head atten- tion efficiency by dynamically selecting or weighting attention heads. Mixture of Attention Heads (MoA) (Zang et al.) (022) treats each head as an independent expert, with a router dynamically selecting a subset of K heads per token, enhancing efficiency by focusing on the most relevant heads. Interpreted through GSP lens, MoA functions as adaptive filtering, where tokens selectively activate the most suitable filters (heads). Building on this, Mixture-of Heads Attention (MoH) (Jin al., 2024) further advances this approach by using router to assign weights to all heads, rather than selecting a subset. This allows each token to receive a weighted combination of all head outputs, offering greater flexibility. Unlike MoA, which treats heads independently, MoH uses shared set of heads with adaptive weights, providing a more direct form of adaptive filtering where filter weights (MoE) frameworks (Shazeer al.) 2017] Fedus et al.] [022 Dai et al. 2024) route tokens across multiple expert networks, expanding capacity at the cost of greater computation and memory. Our approach embodies the MoE principle of conditional computation in a lightweight form: routing remains within a single architecture, with each head functioning as a filter in filter bank. In this sense, is structurally closer MoA/MoH. yet conceptually aligned with the adaptive utilization underlying MoF.
continuously adjusted. Conventional Mixture-of-Exper
Modulation of SSMs. Mamba's recursive state update leads to information loss as context length increases, problem noted in various studies. DeciMamba (Ben-Kish al. (025) addresses this by measuring information loss using Effective Receptive Field (ERF) and removing less important tokens with low A values. MambaFxtend (Azizi et al., 2025) improves on this by offering training- scaling method, adjusting A values directly to enhance long-come long-context performance. LongMamba (Ye) jet al., 2025) further refines this by separating global and local channels, using token filtering in global channels to improve memory efficiency and extend the receptive field. Another work tries to emphasize polarization of A, thereby allowing SSMs to capture vanishing influence of earlier tokens in long sequences (Wang al. 2025
CONCLUSION
In this work, we proposed a hierarchical adaptive filtering architecture for SSMs, bridging the gap between SSMs and GSP. Our method, HADES, reinterprets Mamba2 as a GSP-inspired filter bank, introducing a novel separation of shared and expert filters via delta modulation and spectral residual bias. This design enables efficient, frequency-adaptive filtering, significantly improving performance across language modeling, commonsense reasoning, and long-context tasks, while maintaining parameter usage at 58.9% of baseline models including Mamba2. Our approach not only advances the understanding of SSMs from a GSP perspective but also demonstrates the effectiveness of structured. adaptive filtering in neural sequence modeling. We leave limitations and future work in Appendix G
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was supported by the MSIT (Ministry of Science, ICT), Korea, under the Global Re- search Support Program in the Digital Field program RKS-2024-0043668) Developing Sustainable and Al Research Hub
Collaborative Multi-modal Lifelong Learning Framework Project ---0244-00458882 supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation). This project supported by Microsoft Research Asia.
ETHICS STATEMENT
This work does not involve human participants, sensitive information, or proprietary datasets. All experiments conducted on publicly available benchmarks such the Pile. Although advances in long-context language modeling may broaden downstream applications, we recognize potential risks of misuse, including harmful misleading content generation. Our contributions are purely methodological, and we encourage responsible use of this technology in accordance with the ICLR Code Ethics.
LLM STATEMENT
In preparing this manuscript, we used a large language model (LLM) solely to refine the language and presentation. The tool was applied to enhance readability and polish the text, without influencing the conceptual development or technical contributions the work. The responsibility for the research content and findings remains entirely with the authors.
REPRODUCIBILITY STATEMENT
Our implementation is publicly available at attt.tps://githhi www.:ttwwb.oooeeejnnnn-iittteeee It contains source code, evaluation scripts, and configuration files for all reported experiments. All datasets used in this study are publicly available. Experimental setups and detailed hyperparameters are documented in Appendix
REFERENCES
Seyedarmin Azizi, Souvik Kundu, Mohammad Erfan Sadeghi, and Massoud Pedram. Mambaextend: A training-free approach to improve long context extension of mamba. In The Thirteenth Inter- national Conference on Learning Representations, 2025. URL htRp:htts:///peee
ew. n
forum?id-LgzRon
Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein, Nadav Cohen, Amir Globerson, Lior
Wolf. and Raja Giryes. Decimamba: Exploring the length extrapolation potential of mamba. In The Thirteenth International Conference on Learning Representations, 2025. URL https
//openreview.oo net. forum? d=s.:rmmmmmi:is15222.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019, URL https: / http://://axavvor/////// 1911
11641
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-conted large language models. In The International Conference on Learning Representations (ICLR), 2024.
Jeongwhan Choi, Hyowon Wi, Jayoung Kim, Yehjin Shin, Kookjin Lee, Nathaniel Trask, and Noseong Park. Graph convolutions enrich the self-attention in transformers! Advances in Neural Information Processing Systems, 17752891-52936 2024.
Christopher Clark, Kenton Lee, Ming-Wes Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-292 2936, 2019.
11

---

Published as a conference paper at ICLR 2026
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URL
Damai Dai, Chengqi Deng, Chenggang Zhao, Rx Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, al. Deepseckmoe: Towards ultimate expert specialization in mixture- of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1280-1297, 2024.
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024.
Yushun Dong, Kaize Ding, Brian Jalaian, Shuiwang Ji, and Jundong Li. Adagnn: Graph neural networks with adaptive frequency response filter. In Proceedings of the 30th ACM international conference on information & knowledge management, pp. 392-401, 2021.
Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, and Yu Cheng. Mom: Linear sequence modeling with mixture-of-memories. arXiv preprint arXiv:2502. 13665,, 2025.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39 2022.
Fernando Gama, Brendon G Anderson, and Somayeh Sojoudi. Node-variant graph filters in graph neural networks. In 2022 IEEE Data Science and Learning Workshop (DSLW), pp. 1-6. IEEE, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101 00027, 2020.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.00772, 2023.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL https 7/openreview n
Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning per-
spective on oversmoothing and beyond. In The Eleventh International Conference on Learning Representations, 2023.
Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. Moh: Multi-head attention as mixture-of-head attention, 2024. URL Rtt.hhtts:///arxttt orgbass//o//////o//o/oo/o/../a /241 0. 11842
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5156-5165. PMLR, 13-18 Jul 2020.
Simon Korbliith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 2519-
PMIR, 2019.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning. Christopher Re Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji. Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Ieard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. ISSN 0835-8856. URL htthstps:: //tpr///eepeeeeeeeeeemm net/ forum? farum?id=i0 d0.00:104..70 b Featured Certification, Expert Certification.
12

---

Published as a conference paper at ICLR 2026
Shahar Lutati, Itamar Zimerman, and Lior Wolf. Focus your attention (with adaptive iir filters). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 12538-12549, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https://
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2381-2391, 2018.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Pro- ceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1525- 1534, Berlin, Germany, August 2016. Association for Computational Lin- guistics. doi: 10. 10..1863/vvll. 1/P16 144. URL htt.t:tp///// http://///caaaaaooooooooooo /tt.://oooo.//////o///o/ooaaoao. ott.tt/...///c....t..o...o.o/ee.
Guilherme Penedo, Hynek Kydlfček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:308 1-30849, 2024.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanistaw Wožniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14048-14077, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/w1/2023, 10086655/1122333.ddnnng---mmmm findngg--emnlp.93.. URL /stps::/acllantholl www.g..orgaa.
Finaings-emnip 936/
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: towards larger convolutional language models. In Proceedings of the 40th International Conference on Machine Learning. ICML'23, '3, JMLR.org, 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande; An adver- sarial winograd schema challenge at scale, 2019. URL httpp::/arxxv..o.mmmm// org. http::::///aail...o////////////
10641
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gatee mixture-of- layer. In International Conference on Learning Representations, 2017.
Jerome Sieber, Carmen Amo Alonso, Alexandre Didier, Melanie Zeilinger, and Antonio Orvieto. Understanding the differences in foundation models: Attention, state space models, and recurrent neural networks. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL /ttttps://ppeeree...e. wwwp://wnne.//oooidiiii.. watprruumniidd1111mmmxetttt
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023. URL U.R.httpp:://peeeeiiii
net. forum? BHWA1BHWWAAAq
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and
Furu Wei. Retentive network: A successor to transformer for large language models, 2024. URL
https: htttps://openrevieee net.
http://wwwweeeeeeeueneeeeee www.moormmmooooo d==U99 cwbhin
Peihao Wang. Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Anti-oversmoothing in deep
vision transformers via the fourier domain analysis: From theory to practice. In International Conference Learning Representations, 2022.
13

---

Published as a conference paper at ICLR 2026
Peihao Wang, Ruisi Cai, Yuehao Wang, Jiajun Zhu, Pragya Srivastava, Zhangyang Wang, and Pan Li. Understanding and mitigating bottlenecks of state space models through the lens of recency and over-smoothing, In The Thirteenth International Conference Learning Representations, 2025.
wtt.httts:///ppee
net / forum? 1d=pmmdpppppppppppm
Yuhan Wu, Xiyu Meng, Huajin Hu, Junru Zhang, Yabo Dong, and Dongming Lu. Affirm: Interactive mamba with adaptive fourier filters for long-term time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chenggi Zhang. Beyond low-pass filtering: Graph convolutional networks with automatic filtering. IEEE Transactions on Knowledge and Data Engineering. 35(7):6687-666 2022.
Songlin Yang and Yu Zhang. Fla: A bistnn library for hardware-efficient imt.:fccctttiiiimmmmmmmmtttttttt of linear attention mechanism, January 2024. URL httpp:://gttuu..com/fffff
Lash-lirear-ateeee
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In Forty-first International Conference on Machine Learning, 2024a. URL (tKLitttps///peerer///e
Songlin Yang. Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length, 2024b. URL httpsss//aa.....rgg////e///////e 2406 06484
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks; Improving mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations, 2025. URL
:ttt:s://openreviiee
net /ttp//ooommminnH/mhme
Zhifan Ye, Kejing Xia, Yonggan Fu, Xin Dong, Jihoon Hong, Xiangehi Yuan, Shizhe Diao, Jan Kautz, Pavlo Molchanov, and Yingyan Celine Lin. Longmamba: Enhancing mamba's 's long- context capabilities via training-fie ffree receptive field enlargement. In The Thirteenth International Conference on Learning Representations, 2025. URL
net/ forrrm?
id=[MbLszV018
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P11 1472.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, and Zhang Xiong. Mixture of attention heads: Selecting attention heads per token. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
14

---

Published as a conference paper at ICLR 2026
A FULL MAMBA2 ARCHITECTURE
Given an input sequence U = [U1, U2, uT] E RTxx a Mamba2 block with channels built
on top of the S6 layer via the following formula, generating output sequence O = [0t, 02, .... ØT E I
h A + BAT E R yt Cche E R
(18)
02 = W. (Norm(( O W2u))) E
(19)
where Wz. W E RdxP, W. E RPdd are trainable parameters. Each Mamba2 block consists of M heads, so that M x == d, which are computed in parallel, the result of which is summed together. We can specify how each matrices are created for each head:
A. a,I E RNN
C, (Coonv(Woou)
RRNN
0g = exp(- (xp(-A)) ER
A base = Wau + ba E R
(20)
(21)
B, A,B, E N
A, Softplus( (,sss) E R
B, (Conv(WBBu)) RR
Zt = oCConv (W,on)(,,)))) E R Px1
where WB, Wc E Nxd WA E gixd, denotes SiLU activation function and Conv(-) denotes channel-wise one-dimensional convolution. By A, Mamba2 implements input-dependent selection mechanism. As performs as decay-ratio
is cumulatively multiplied. DeciMamba (Ben-Kish et al.) 2025) elaborates the condition of dr For computational stability. A > 0 and A < 0 is guaranteed in original implementation. Therefore, we can conclude 0, (0, 1).
Using this Mamba2 block, we can derive layer-wise Mamba2 architecture with 7. layers as below. For initial input, input sequence is I = [Ão, f1..., kr] RR" where in E [V] and we have (((-1)- (u!'=1), u2 ... '1 as input sequence for the /-th layer. O |o ob ,...o"1 serves as (1-1) [L- 1/
output sequence of 1-th Mamba2 layer Mamba (1) V denotes vocab size and P E TR denotes final logits.
U(0) Embedding, (7) E Rxx
(22)
(() Mamba (2) (Norm(U D)) E Txd
(23)
P Embedding (Nrrii (NNorm(O0))
TXV
(24)
Here, the output of the I-th layer is used as the input for the / + 1-th layer, i.e. 0t = UOO)
B MORE DISCUSSION
B. DETAILED EXPLANATION ON HADES
Detailed explanation on connection of GSP and HADES. While it is true that most GSP-based analyses traditionally assume inverinvriiant dynamics (e.g.. S4). we emphasize that GSP can indeed be extended to handle time-variant systems. In Eq. 10 we explicitly formulate Mamba Time-Variant (LTV) system operating over line graph. From the GSP perspective, this corresponds to Mode-Variana Graph Filter (NVGF) (Gama al. 2022), where each node (i.e., each time step in the sequence) is associated with its own filter coefficients modulated by input-dependent dynamics via the state-space parameters A, B, C.
Linear
In the node-invariant case, corresponding to an LTI system such as S4, the output can be expressed as
K
Σ
(25)
R=
with coefficients given in $4 by hig izes LTV systems such as Mamba, the filtering operation becomes
CA By contrast, in the node-variant case, which character-
K
diag ( s*x,
(26)
A=
15

---

Published as a conference paper at ICLR 2026
where each h2 (hg h/ h )) Mamba in particular, these coefficients satisfy
(0)
R- assigns distinct filter values each node. For
h C.A.t-BBI-.-
(27)
This formulation is grounded in established work on NVGF in GNNs [1], and our approach uniquely applies these principles to 1D sequences via the line graph interpretation. Thus, our work extends GSP tools to LTV systems like Mamba, offering principled and novel perspective that complements traditional LTI analyses such as those for S4.
Mamba can also be understood as form of linear attention. Conversely, linear attention itself be reformulated using the state-space model equations employed in Mamba (Dao & Gu 2024] Siieeer al. 2024). For Mamba viewed as Linear Time-Varian (LTV) system, the state-space equations are given by
h4 A,h;-1 Ba
V1 C,h, + D,us-
(28)
In the of linear attention, these parameters can be expressed in the same form, with
A; (clu(g:-1) + + 1) (elu(q:) +1) ビj (elu((k;))
+ 1)
(29)
+ ((oelu()))
B; (elu(q:-1) + 1) (e 1)
Ia x (elu(k)) + 1),
(30)
C = la x (elu(g:) + +).
(31)
Under this formulation, linear attention can also be interpreted in Mamba's ' equation form. Ultimately, both models can be framed linear time-varian systems within the context of GSP, offering unified analytical view of these scemingly distinct architectures.
Pseudo Code. In this paragraph, we display pseudo-code for method on prefill and decode stage respectively.
forward seglen,Kone, seq_ Nox=None, cu_seq eng-None, inference params-Nonel Check for Inference eache inference_parent exists:
update sache params go to decode function
1. Lineer projection of input in_proj(u)
zxbe, dt - split (2xbedt into festures and dt terma)
11
2. Compute spectrol residual spectral rrattral_reedddd doal udt concat (epectrai_residuat, dt)
12
sumulative_mee mean (t)
13
14
Project routing
16
his h_proj (udt) b, spectrei_ban
17
split
scores and bies termst
18
19
4. MoB routing
20
select_ids top
21
shared_ids shared expert ids (broadcasted)
concat (select_ids, shared_ids)
25
dt gather dt using ids
apectrai_biss apply gamma pad at dt dt_blas spectral moe_loss sv_squared(()
25
26
27
5. Combine features again 28
concat (zxbe,
30
31
out mamba_asm k
32
Inout = sxbedt, weights cenvld weights, dt, norm and activation cenfig
33
34
35
37
out = reshapa to (3, b, diveraity_ lowrrtty_las_vwaa vols_val diveresty_losaccor
38
16

---

Published as a conference paper at ICLR 2026
out reshape to (8,
out = out_praj (cut) 40
42
diversity _orreitt__ss__aa_ diversslty y (B, D, P)) out put_proj
43
44
45
return out, Imoe_ diversity_lose_vall
Listing 1: Forward function in Prefill stage
def stetes, conv_state, ssm_etate, sumsum_state, t_pos):
only taken time (saquence length squeeze hidden_states
1. Input projection
in_proj
split exbedt into z0, x0, XBC, dt
Spectral residual routing penetrra__resddall - teumsun_atate / (t_poe - 11)
10
concat lapectral_residueee dt)
12
his h_proj (udt)
h, spestral_hie shb]
14
15
3. NoE tep-k selection
select_ids topk
17
sdsced_ids fixed shared expert ids
concat (select_ids, sharedLids)
19
dt gather dt using ids
spectral _aaatral_hia___ _hiaa gamma coneat (spectral_blag seros)
dt dt apectral_bias
22
Update cumaum cunsun_state 1-
spectral residual
24
26
5. Convin Step
27
KBC causal_convil wec.ooond_pppeeeee _uppdteeppeeeeeeeeee (x3C, conv_atate, convid weights and bias)
split xBC into
C
29
SLate Space
Step
31
Ne use Expanded version for group/state-aware update
32
repest/reshape A, dt, dt_bias, D, E, reshape to (batch, heada, head_din)
33
34
serettive_ttte_upeeee (ssn_state) 8, de, &, B, C, D, 2, dt_bias)
reshape (batch, total_dim)
36
Output projectien
38
out = out_proo return out untguageuu (11, conv_state, asn_state, aumoum_stats
Listing 2: Forward function in Decode stage
B.2 COMPARISON TO OTHER FIELDS
Comparison with MoE. While our proposed method draws partial inspiration from the MoE framework, its core contribution lies in the filter bank interpretation from GSP perspective. Specifi- cally, our model interprets each head within single architecture as a distinct filter and adaptively selects among them, making it most analogous to MoA (Zhang et al.) 2022) among existing related works. In contrast, conventional MoE approaches route tokens across multiple separate architectures, leading to significantly larger model capacity and computational overhead. In this sense, our approach focuses on efficient utilization within single architecture, whereas MoE methods entail learning and managing multiple parallel networks.
Regarding the loss function, were inspired by the load balancing losses commonly used in MoE settings (Jin et al. 2024] Zhang et al. 2022 Dai et al. 2024). To ensure balanced selection, we apply loss terms both before and after routing, encouraging equitable utilization of filters.
From an interpretability standpoint, our GSP-based filter view allows the model's s internal mechanisms to be understood more clearly adaptive filtering. While implicit, this behavior manifests in observable differences in filtering effects across tokens and tasks (see Fig. 5), demonstrating the effectiveness of formulation.
17

---

Published as a conference paper at ICLR 2026
Comparison with Adaptive Filtering. To better illustrate the effectiveness of our method, present discussion comparing adaptive filtering approaches to HADES. Affirm (Wu et al., (2025) and our method differ mainly in their filtering strategies: Affirm uses explicit filtering by applying FFT-based domain transformation, enabling frequency-domain operations. In contrast, our approach leverages implicit filtering mechanism, interpreting heads filters from GSP perspective. Without requiring domain transformation, our model adaptively selects token-specific filters, yielding performance improvements through flexible and interpretable routing, Focus (Lutati et al. 2023) takes DSP-inspired approach by modeling SSMs as Infinite Impulse Response filters, transitioning from a Linear modified STFT (chunked-FFT) and hypernetwork that generates filters dynamically. While both Focus and our method introduce adaptivity via routing, the key difference that Focus generates filters, whereas we select from pre-defined heads interpreted as filters. Moreover, Focus applies explicit frequency-domain filtering, while our method remains implicit, operating entirely in the original domain.
to Time- Variant perspective. It enables adaptive filtering through a
C EXPERIMENTAL SETUP
C.I TRAINING DETAILS
We adopt all baseline implementations from Elash-linear-atten 2024). For fair comparison, all models are trained under identical conditions with 370M parameters exculding readout head on 200B tokens from the Pile dataset (Gao et al. 2020). Starting from 370M configuration, HADES yields smaller parameter count 218M We use the Adam W optimizer with a peak learning rate of 48e-4, weight decay of 0.1, 3 E [0.9, 0.95] following Mamba2, and gradient clipping of 1.0. The learning rate follows cosine annealing schedule with a warm-up phase of 375M tokens and a batch size of 2 tokens (# sequences sequence length) and the number of training steps as 47,042 (# tokens/ # tokens in one batch) steps. All models employ the GPT-NeoX tokenizer with a vocabulary size of 50,277. For sequence modeling, we set the training length to 2K tokens. Our experiments were conducted on computing server equipped with AMD EPYC 9654 CPU (2 sockets, 192 cores, 384 threads, 1.5-3.7 GHz, L3 cache 768 MiB) and four NVIDIA A100 80GB PCIe GPUs with CUDA version 12.4. For our model, used hyperparameter set of H = 16, S 8, A1 = le 3, A2 = le 3, 7 25e- - 2.
(Yang & Zhang
C.2 EVALUATION
Language Modeling and zero-shot Commonsense Reasoning. Following prior works (Gu & Dao 2023 Yang et al.) 2024a)), we evaluate our method against five baseline models across two evaluation categories: WikiText (Wiki.) perplexity and zero-shot commonsense reasoning tasks. The
commonsense tasks include LAMBADA (LMB.: et al. (2066). PIQA Bisk et al. (2019)
HellaSwag (Hella.; Zellers et al. (2019)). WinoGrande (Wino.; Sakaguchi et al. | (2019)) ARC-
easy (ARC-e) and ARC-challenge (ARC-c) Clark al. (2018), BoolQ Clark al. (2019). and
OpenbookQA (OBQA.: (Mihaylly et al. (2018).
We measure perplexity (ppl) on Wiki'Text and LAMBADA, normalized accuracy(acc_
HellaSwag and ARC-challenge, and standard accuracy (acc) on the remaining tasks (as normalized accuracy provides higher scores for most models on these tasks). Avg. denotes the averaged result of the accuracies and normalized accuracies of eight tasks together. All evaluations are conducted using In-evaluationn-aal
(Liang et al. 2023). We provide details of the evaluation tasks below.
WikiText (Merity et al., 2017): A dataset consisting of high-quality, clean text extracted from Wikipedia articles, commonly used to evaluate language modeling tasks by measuring a model's s ability to predict and generate coherent and fluent text.
LAMBADA (Paperno et al. 2016): A text completion task that measures a model's 's ability to predict the final word of a passage, requiring comprehension of the context, commonsense reasoning, well the ability generate text coherently.
PIQA (Bikk et al. 2019): A physical commonsense reasoning task focused on selecting the most plausible solution to everyday scenarios.
18

---

Published as a conference paper at ICLR 2026
HellaSwag (Zellrs al. 2019): A multiple-choice task that evaluates model's ability to select the most coherent continuation of a given situation based on commonsense and narrative reasoning.
WinoGrande (Sakaguchi et al. 2019): An expanded version of the Winograd Schema Challenge: a pronoun resolution task designed to test commonsense reasoning by identifying which noun pronoun refers to in given sentence.
OpenbookQA Mihhaylvv et al. 201 8): A multiple-choice question answering task designed to test model's s understanding of elementary-level science facts and ability to apply this knowledge to novel scenarios requiring reasoning and inference.
ARC-easy (Clark et al.
2018): A subset of the A12 Reasoning Challenge focusing on questions that require basic scientific and commonsense knowledge.
2018): A more difficult subset the A12 Reasoning Chal- lenge that tests advanced reasoning and deep understanding of scientific and commonsense knowledge.
ARC-challenge (Clark et al.
BoolQ (Clark al. 2019): A yes/no question answering dataset with 15,942 examples, derived from Google search queries, paired with Wikipedia passages.
Passkey Retrieval. For the passkey retrieval task, we adopt the task formulation from Chen et al. (2024). The evaluation is conducted across context lengths from 1K to 16K, with the target digit hidden at depths of 0% to 100% with the gap 10% each of these sequences. Assuming that each correct retrieval receives score of and each incorrect retrieval receives score of O, compute the retrieval score as count out of 10, across all the depths overall context lengths. We did not apply any fine-tuning with longer sequences. We structure the prompt for the passkey retrieval task into four distinct components: task description, passkey, query, and dummy text.
Task Description: This section defines the task for the model, instructing it to identify and memorize specific important information within large amount of irrelevant text.
There is important piece of information hidden inside lot of irrelevant text. Find it and memorize it. I will quiz you about this important information.
Passkey: This section provides the critical information (the passkey) that the model is required to memorize and retrieve.
The pass key 15921. Remember it. 15921 the pass key.
Query: This part contains direct question prompting the model to recall the passkey it memorized.
what is the pass key? The pass
key is
Dummy Text: This section consists of irrelevant text that serves as a placeholder, repeated until the full prompt length reaches the designated sequence length.
The grass is green. The sky is blue. The sun is yellow. Here go. There and back again.
D DETAILED BENCHMARKS AND MORE EVALUATIONS
D.1 FULL RESULT LANGUAGE MODELING AND ZERO-SHOT COMMONSENSE REASONING
In this subsection, we report the full result of language modeling and zero-shot commonsense reasoning with standard error. For metrics aggregated using the mean (accuracy and normalized accuracy), the standard error was calculated using the conventional formula Standard Error Vn
19

---

Published as a conference paper at ICLR 2026
where denotes the sample standard deviation, and is the sample size. In contrast, for Perplexity, due to its potentially non-normal distribution, we employed a bootstrap method with 100 resampling iterations to estimate standard error, calculating the standard deviation of the resampled values.
Table 3: Performance comparison language modeling and zero-shot common cemmon--ense reasoning with standard error (values in (-)). The standard error values are are rounded to three decimal places. The best results are highlighted in bold, while the second-bess results underlined. Avg. denotes the result of accuracies and normalized accuracies over 8 tasks.
Model
Wiki.
LMB.
LMB.
BooiQ
Hella.
Wino.
ARC-e
ARC-e
PIQA
OBQA.
Avg.
ppl 4
ppl
acc
acc f
acc_ n 1
acc 1
acc 1
acc 1
acc 1
acc f
tmikks f
Linear Transformer
45.43
73.93
24.06
61,50
28.20
$1.30
42.05
21.76
60.55
27.60
39.63
(NIA)
(4.168)
(0.006)
(0.009)
(0.005)
(0.014)
(0.010)
(0.012)
(0.012)
(0.020)
(0.011)
34.12
29.46
35.36
55.57
31.31
$1.70
44.49
23.46
62.40
28.00
41.54
RetNet
(N/A)
(1.046)
(0.007)
(0.009)
(0.005)
(0.014)
(0.010)
(0.012)
(0.011)
(0.020)
(0.011)
DeltaNet
33.25
26.82
35.75
54.07
31.40
49.96
44.11
22.18
63.60
29.60
41.33
(NIA)
(0.908)
(0.007)
(0.009)
(0.005)
(0.014)
(0:010)
(0.012)
(0.011)
(0.020)
(0.011)
47.51
85.53
22.43
62.17
28.71
50.67
42.09
22.35
60.72
39.47
Mambal
(N/A)
(3.321)
(0.006)
(0.009)
(0.005)
(0.014)
(0.010)
(0.012)
(0.011)
(0.020)
(0.011)
Mamba2
31.34
24.38
36,46
53.88
32.62
50.83
45.29
24.15
63.44
26.40
41.63
(N/A)
(0.820)
(0.007)
(0.009)
(0.005)
(0.014)
(0.010)
(0.013)
(0.011)
(0.020)
(0.011)
HADES (Ours)
31.48
21.74
39,24
58.84
32,82
52.64
45.03
22.01
63.93
28 80
42.91
(NA)
(0.727)
(0.007)
(0.009)
(0.005)
(0.014)
(0.010)
(0.012)
(0.011)
(0.020)
(0.011)
D.2 MORE EXPERIMENTS
Larger scale experiment. For generality, we conduct bigger scale experiment of our model in T To ensure a fair comparison, all models are trained under the same setup: 1.3B parameters and 30B tokens drawn from the Fine FideEue-Edu dataset Penedo et al. (2024). We adopt the AdamW optimizer with a peak learning rate of 4e-4, weight decay of 1. and apply gradient clipping at 1.0. The learning rate schedule uses cosine annealing with warm-up phase of 1B tokens, and the batch size is fixed at 0.5M tokens. All models are trained with the Llama2 tokenizer, which has a vocabulary size of 32,000. For sequence modeling, the training sequence length is set 4K tokens. HADES shows strong performance against baseline models with only 71.4% of parameters.
Performance comparison language modeling and zero-shot common sense reasoning. The best results are highlighted in bold, while the second-best results are underlined. Avg. denotes the average of accuracies and normalized accuracies over 8 tasks. With only 71.4% of parameters compared to baseline models, HADES achieves comparable or even better performance.
Table 4:
Model
Wiki.
LMB.
LMB.
PIQA
Hella.
Wino.
ARC- acc 1
ARC-E
BoolQ
OBQA.
Avg. 8 tasks
ppl
ppl J
acc
acc
acc_
acc f
acc 1
acc_n"
RetNet (1.3B)
22.45
21.84
38.70
69.04
47.73
52.72
63.68
33.36
60.61
36.60
50.31
Mamba2 (1.3B)
19.47
17.40
40.68
70.29
53.24
56.04
69.87
36.35
55.81
37.40
52.46
DeltaNet (1.3B)
19.77
16.64
41.78
70.95
51.09
54.70
67.63
34.47
61.19
38.40
52.53
HADES (IB)
20.41
17.22
41.18
71.33
51.85
56.35
68.48
34.81
60.73
38.60
52.92
Comparison mixture variant. We additionally compare our method against a mixture variant, MoM (Du et al. 2025). the mixture-of-experts extension of Gated DeltaNet (Yang et al.) (025).. itself an adaptation of Mamba2. For fairness, we use the same 370M configuration (dmodel 1024, Plaayer 24), and we follow the official configuration of MoM, which employs 4 experts. All other training and evaluation settings follow our original setup in Appendix Across all benchmarks, in Table HADES consistently achieves comparable higher average performance compared to MoM, demonstrating the effectiveness of our approach even relative to mixture-hased variants.
Robustness over seed sweep. To further assess robustness of HADES, we conducted an additional seed sweep. Specifically, we evaluated HADES and the primary baseline, Mamba2, across three random seeds. All other training and evaluation settings follow original setup in Appendix C The results are summarized in T where our model consistently maintains strong performance with low variance across seeds, demonstrating robustness to initialization.
20

---

Published as a conference paper at ICLR 2026
Table 5: Performance comparison on language modeling and zero-shot common-sens' reasoning. The best results are highlighted in bold, while the second-bess best results are underlined. Avg. denotes the average of accuracies and normalized accuracies over tasks.
Model
Wiki.
LMB.
LMB.
PIQA
Hella.
Wino.
ARC-e acc
ARC-c
BoolQ
OBQA.
Avg. 8 tasks
ppl 1
ppl
acc
DCC
acc__t T
acc
acc 1
acc 1
acc
MoM
31.58
23.28
40,40
62.57
32.99
52.64
44.91
23,89
52.78
27.20
42.17
HADES
31.48
21.74
39.24
63.93
32.82
52.64
45.03
22.01
58,84
28.80
42.91
Table 6: Performance comparison on language modeling and zero-shot common- sense reasoning with mean and standard error three seeds (values in )). The standard error values rounded to three decimal places. The best results are highlighted in bold, while the second-best results are underlined. Avg. denotes the result of accuracies and normalized accuracies 8 tasks.
Model
Wiki.
LMB.
LMB.
BoolQ
Hella.
Wino.
ARC-e
ARC-c
PIQA
OBQA.
Avg. 8 tasks
ppl 4
ppl
acc
acc 1
sec_n
acc
acc 1
acc ?
acc
acc_pt
30.64
22.57
37.85
54.63
33,27
51.15
45.34
27.93
Mamba2
(0.752)
(1,662)
(0.014)
(0.007)
(0.007)
(0.006)
(0.004)
(0.009)
(0.004)
(0.014)
(0.004)
33.41
25.57
37.10
59.67
31.70
51.85
22.67
27.87
HADES
(L.721)
(3.482)
(0.022)
(0.009)
(0.010)
(0.011)
(0.008)
(0.006)
(0.007)
(0.013)
(0.005)
E EXTENDED MODEL ANALYSES
ABLATION STUDIES
In this subsection, report the full result of ablation studies. We variations of our model with same training and evaluation setting in Appendix C In Table 7 our ablation studies demonstrate both the robustness and tunability our model.
Table 7: Full result for ablation studies. Avg. denotes the averaged result of the accuracies and normalized accuracies of eight tasks together.
Methods
Wiki.
LMB.
LMB.
PIQA
Hella.
Wino,
ARC-e
ARC-e acc
BoolQ OBQA.
Avg.
ppl 4
ppl
acc_.n
acc f
acc_n
tasks
HADES (Ours)
31.48
21.74
39.24
63.93
32.82
52.64
45.03
22.01
58.84
28.80
42.91
w/o Cestuces
34.73
26.84
36.77
62.68
30.96
50.75
43.22
22.70
59.27
26.20
41.57
wfo Laiveniy
33.83
27.40
36.04
62.46
31.48
51.38
44.87
22.61
59.94
28.40
42.15
Only Shared Filters
34.55
27.64
35.75
62.40
31.39
52.88
44.23
60.83
26.00
42.21
Only Expert Filters
36.34
30.12
34.89
61.53
30.30
52.41
44.49
22.70
58.29
28.80
41.68
25 % Shared
35.24
29.53
34.64
62.79
30.89
50.43
43.35
23.81
59.48
28.20
41.70
75 % Shared
35.92
31.67
34.08
62.35
30.12
50.83
42.89
23.21
61.80
28.40
41.71
Fixed
34.55
27.64
35.75
62.40
31.39
$2.88
44.23
24,23
60.83
26.00
42.21
Random Routing
35.78
32.77
33.17
61.97
30.47
52.49
43.31
23.12
55.72
28.00
41.03
Input-only
34.17
23.95
38.40
63.71
31.60
51.22
43.56
23.04
58.35
27.80
42.21
Gumbel Soltmax Top-K
34.83
27.21
36.95
62.19
31.39
50.51
43.43
22.35
58.07
28.20
41.64
Weighted aggregation (MoH)
36.73
32.03
34.95
61.75
30.19
50.75
44.36
22.53
60.92
28.00
41.68
Position Bins
30.23
21.93
38.50
63.93
33,08
51.70
43.73
22.27
53.39
30,60
42.15
No Bias
34.57
28.79 79
34.91
63.38
31.24
50.67
42.68
22.35
56.85
26.80
41.11
On Auxiliary Losses and Filter Configurations. Interestingly, without Loalanee performance drops as filter selection becomes overly concentrated on few filters, leaving the rarely selected filters under-trained and preventing them to learn dynamics effectively when they are eventually chosen. We also evaluate the impact of different filter configurations: Using Only Shared Filters and Using Only Expert Filters. Using only shared filters outperforms using only expert filters, shared filters consistently capture global low-frequency information, while expert filters adaptively capturing low and high frequency information.
21

---

Published as a conference paper at ICLR 2026
On Filter selection and Delta modulation. We tried more ablation filter selection and delta modulation. For filter selection, "Top-Q" refers to the routing mechanism used in our proposed method, HADES, where the top-ranked filters are adaptively selected. "Fixed" denotes the setup with no routing-i.e. a fixed set of filters is always used regardless of input. "Random" indicates that filters are selected at random without regard to token-specific information. We additionally performed "Input-only", "Gumbel Softmax Top-K" and "Weighted Aggregation". "Input-only" refers to the routing mechanism which only use input sequence to get top-ranked filters. "Gumbel Softmax Top-K" is where Top-K K filters are selected with Top-K K selection itself is being trained. "Weighted Aggregation" means output aggregated via linear projection instead of simple aggregation, which can be interpreted as variant of MoH (Jin et al. 2024).
For delta modulation, "Spectral Bias" refers to the biasing scheme originally used in HADES, which modulates A based on learned spectral residual. "Position Bias" incorporates positional information of each token into the bias term, enabling location- aawrree-wwwreeeeeee modulation. "No Bias" denotes the variant where additional modulation is applied to A.
On filter selection, we observed that average performance of "Fixed" was better than that of purely random selection. Also, "Input-only" showed reasonable performance with simple selection. However, leveraging the token-level select score to guide the selection (HADES) yielded the strongest results, as it allowed the model to adapt to the specific characteristics of the input. Regarding delta modulation, introducing a positional bias was more beneficial than using no bias at all. Yet, instead of relying solely absolute positions, incorporating delta-based information- thereby reflecting the relative importance of the input-proved to be more effective in achieving superior performance.
E.2 SENSITIVITY STUDIES
Table 8: Sensitivity Studies. Avg. denotes the averaged result of the accuracies and normalized accuracies of eight tasks together.
Hy per
Wiki.
LMB.
LMB.
PIQA
Hella.
Wino.
ARC-e
ARC-c ucc f
BoolQ acc
OBQA.
Avg. 8 tasks
param.
ppl 4
ppl L
acc f
acc
acc_n 1
acc 1
acc
acc_n
8
39.52
37.58
32.93
60.88
29.63
50.28
42.00
22.27
61.10
24.60
40.46
H
16
31.48
21.74
39.24
63.93
32.82
52.64
45.03
22.01
58.84
28.80
42.91
33.11
26.73
35.40
62.57
31.99
50.75
44.36
23.29
48.93
28.60
40.74
0.15
34,81
29.07
34.78
61.86
31.33
51.22
43.64
23.04
60.34
26.60
41.60
0.25
31.48
21.74
39.24
63.93
32.82
52.64
45.03
22.01
58.84
28.80
42.91
33.96
28.42
35.84
31.69
52.33
43.52
22.87
50.09
28.20
40.95
We conduct sensitivity analysis to assess our model's robustness to hyperparameters. First, We train our model varing Y E [0.15, 0.35] in increments of 0.1 while other settings are fixed (See Appendix (CT)). We use same evaluation settings in Appendix C.2 Table show that performance on language modeling and zero-shot commonsense reasoning benchmarks remains stable, even as higher y increases bias influence, demonstrating model robustness. We then examine the sensitivity of hyper- parameter H by varying the number of active filters among the total of 32. We test H E {8, 16, 24) while keeping other settings fixed. The best performance is achieved with H 16, followed by 24 and 8, supporting our hypothesis that an optimal number of filters enhances information flow. As shown in Table 8 even with a drastically reduced model size of approximately 38.64% (143M) in the H = 8 setting, our model maintains performance comparable to the optimal hyperparameter configuration and even outperforms it on two tasks. is worth noting that more filters does not mean better performance: H = 24 failed to outperform both H/ = 8, Hl 16 setting. This result highlights that selective filter activation can effectively reduce redundancy without sacrificing performance, demonstrating the efficiency of our filter bank approach.
E.3 CKA ANALYSIS ON MAMBA2 AND HADES
We additionally analyze the organization of dynamic filtering behaviors using linear centered kernel aligmment (CKA) (Kombliih et all, 2019) on both Mambu2 and HADES. Given two filter outputs X, Y Raxd where each row corresponds to sequence position and each column feature
22

---

Published as a conference paper at ICLR 2026
dimension, first center the features: X X 1X y= = 1Y meam- The linear CKA between the two filter outputs computed as:
IX CK | T|I
(32)
For each layer, we compute CKA across all filter output pairs, and use the mean off-diagonal CKA A as a single redundancy score.
As shown in Fig. [(d) the CKA heatmap of Mamba2 reveals that Mamba2 contains substantial
number of redundant filters, with many filter pairs exhibiting high similarity. This redundancy indicates that, although Mamba2 possesses dynamic filtering mechanism, a large portion of its filters operate in overlapping regions and fail to specialize effectively. In contrast, HADES shows disappearance of these repetitive structures, suggesting that our model avoids redundant filters and instead selects the filters that are genuinely needed. When comparing the overall similarity distributions, HADES is noticeably skewed toward lower similarity values, further demonstrating that it achieves more diverse and well-differentiated filter selection.
E.4 MORE VISUALIZATION ON SPECTRUM INPUT AND OUTPUT SEQUENCES
Details. To validate the filter behaviors, we analyze the spectrum of the input and output sequences processed by HADES and Mamba2. We compute the sequence spectrum for Fig. 7(a) 7(b) 9 and 12 in the following way. Given sequence I, we obtain its frequency representation I where denotes the ID discrete Fourier transform applied along the temporal dimension. We then measure the amplitude of each frequency component quantify how the input and output sequences differ in their spectral distribution. For ease of comparison, each spectrum is normalized by its maximum amplitude (max scaling). This enables us assess how the shared and expert filters modify the frequency content the processed sequences.
More Visualization. In Fig. 9 we provide additional spectrum visualizations of the input and output sequences processed by HADES across various layers. The input sequence is taken from randomly sampled sentence from the Pile dataset.
E.5 MORE VISUALIZATIONS THE FREQUENCY RESPONSE OF FILTER
Details. To analyze the frequency behavior of the filter itself, we compute the frequency response of the HADES filter and compare it with that of Mamba2. The frequency responses shown Fig. T Fig. and Fig. I0 computed as follows. Following the formulation in Mamba2 (Dao & Gu 2024). both Mamba2 and HADES can be interpreted as linear sequence-to-sqqueeee entire computation is equivalently captured by transformation matrix M. This matrix plays the same role as the attention matrix in Transformers, serving as the full linear operator acting on the sequence, and therefore its frequency response an attention matrix (Wang et al., 2022). Given a filter represented as a transformation matrix M, we characterize its frequency behavior through its Fourier-domain representation A FM-- where F denotes the discrete Fourier transform operator. The magnitude of each frequency response is evaluated using the norm |AA/l2 allowing us assess how strongly the filter amplifies suppresses each frequency component. For fair comparison across filters, the spectra of the processed sequences are additionally || normmaaall ensuring that differences arise from the filters themselves rather than scale variations in the underlying sequences.
operators whose
be analyzed in exactly the manner as that of
More Visualization. We provide additional filter frequency response visualizations in Fig. 10 including responses of Mamba2 and HADES from multiple layers. Mamba2's learned filters remain low-pass and highly redundant across layers, collapsing into nearly identical spectral kernels. HADES introduces rippled high-free seghfrequency responses and broader spectral variability, enabling the model to capture finer structure and richer local detail.
23

---

Published as a conference paper at ICLR 2026
10
10
Mambaz umean-d 0681
25
HADPS (meerrad
08
0.6
4.6
RIN
0.4
0.4
10
02
0.2
28
Fiber
Filer
CKA Similarity
9th Layer
24
HADES [mean=0.050
FEE
50
24
12
10 Fitar
12
01
0.4
Fibtr
CKA Similarity
(b) 24 Layer
14
Nambaz mmsan-0.120
MADES (moa)=02.00))
0.4
76
Similarity
(c) 31 layer
1.0
1.0
88
3,6
Filter
13
2.4
24
12
92
Filter
Filter
CKA Simierrtt
(d) 41 layer
Figure 8: CKA analysis on filter outputs of Mamba2 and HADES. Left: Mamba2. Center: HADES.
Right: comparison distribution.
EFFICIENCY AND COMPUTATIONAL COMPLEXITY
F.1 ANALYSIS ON COMPUTATIONAL TIME AND MEMORY USAGE
We evaluate the efficiency of our method in terms of computation inference time and memory usage. Specifically, measure the inference time and memory consumption at every 10 steps for 100 steps,
24

---

Published as a conference paper at ICLR 2026
1.0
Shared Fit Output Sequance
Exper: Fitter Output Spectral
Expeet Fiter Output
Normalped Mognitude
Frput Sequence
Normaliad Moeettute
win Spectral Dlac Sang Jence
0.0
Normatzed
400
Frequency
Frequency
(a) 12 Layer
Shared Fot Output
Excert Fiter Oliputt
Expert Fiter Ouiput
Normalred Magninuse
Input Sequence
Magiitude
Normalzed Mapnitude
Cormalized
Frequency
Frequency
Prequency
(b) 29th Layer
1.0
Par Output Sequence
Fittr Output Spectral B
Expert Output
Normalired Hagnitude
0.0
Cormalbeed Magnituude
0.8
Inguct Sequercce
Jormaltad Moontbude
0.2
Spectral
Sequuncce
202
200
400
420
200
420
400
250
202
400
Frequency
Frequency
Frequency
(c) th Layer
Figure 9: Frequency spectrum analysis of filter outputs. Left: Shared filter outputs. Center: Expert filters with spectral bias outputs. Right: Expert filters without spectral bias outputs.
Table 9: Inference Time and Memory Usage Comparison.
Model
Linear Transformer 370M
RetNet 370M
DeltaNet 370M
Mamba 370M
Mamba2 370M
HADES
Params.
218M
Inference Time (sec)
2. 2.
2.86 6.17
2.57
4.73
3.43
2.49
Memory (GB)
5.41
5.59
5.56
6.20
4.50
using a sequence length 1024 and a batch size of 32, and report the average values in Table 9 In this scenario. HADES(21 18M) demonstrates a 1.37x speed improvement and 1.37x lower memory usage compared to Mamba2. Furthermore, when compared to other baselines, our approach not only achieves faster processing speeds but also significantly reduces memory consumption, highlighting its efficiency. Additionally, we also examined our model matching the 370M parameter configuration for direct comparison (Time: 3.45s, Memory: 5.9. However, this setting requires using 90 layers, about 1.8 times larger number relative to Mamba2. We argue that such a configuration is less relevant to the practical setting where HADES is used parameter- effecceeeeffiettttt drop-in replacement for a given 370M model. Therefore, our primary evaluation focuses on configurations with matched hidden dimensions and equal number of layers.
25

---

Published as a conference paper at ICLR 2026
3.4
homabber Moythar
Wommateed Magattue
Nor Mormateee Mogifhute
040
0.05
700
0.2
000
0.00
3.6
Frequency
Requueccy
Fresuency
(a) 5thh Layer of Mamba2
(b) 15th Layer of Mamha2
(c) 46 layer of Mamba2
Sqpat flarr
Squat Fii Shand
Zqpat Fla
Shanel fhe
Normatoon Reggitude
Nommstred Magettuue
Kormatted Mlggiitutee
0.20
0.20
12
0.33
3.0
3.60
-020
(d) 5th Layer of HADES
(e) 15 Layer of HADES
(f) 46 Layer of HADES
Figure 10: Frequency responses of Mamba2 and HADES.
F.2 LATENCY ANALYSIS
For more analysis on different model size and sequence length, we recorded the processing time for both prefill and decode stage and reported the average over 21 runs. We report results in Table 10 Also, we evaluated routing overhead in the prefill stage of sequence length 2048, averaged over 100 steps of forward operation in Table I1
Table 10: Average prefill and decode time under various configuration settings. Latency is reported in seconds. We utilize half of the selected filters as shared filters and the remaining half as expert filters. The selection ratio denotes the percentage of filters chosen out of the total available filters.
Selection Ratio
Model
Setting
50%
25%
75%
Prefill (1K)
0.3997
0.3374
0.4969
Prefill (2K)
0.5936
0.4572
0,7869
370M
Prefill (3K)
0.6043
0.4765
0.8033
Decode
0.0004
0.0003
0.0004
Prefill (1K)
0.9981
0.5165
0.7742
Prefill (2K)
1.4274
1.9154
1,9185
1.3B
Prefill (3K)
1.4337
1,9697
1.9614
Decode
0.0067
0.0021
0.0045
Table 11: Latency overhead of running routers (seqlen 2048, averaged over 100 step)
Metric
Value
Avg Routing Time (ms)
0.0071
Avg Total Time (ms)
0.2900
Avg Routing Time Ratio (%)
2,4371 %
26

---

Published as a conference paper at ICLR 2026
F.3 THEORETICAL COMPUTATIONAL COMPLEXITY ANALYSIS
For comprehensive analysis of overall efficiency, we calculated the computational complexity of our method in Table 12 Our architecture introduces additional operations for filter selection and Delta modulation, but otherwise performs the same computations as the original Mamba2. Crucially, unlike the original model that utilizes the entire set of filters, our approach employs reduced number of filters, resulting in a corresponding reduction in computational cost (i.c. И << M). Since the total complexity mainly depends on the hidden dimension d, the introduced filter selection and Delta modulation computations incur only minimal overhead relative to the savings, preserving the overall efficiency of the model. Here, T denotes the input sequence length, d the hidden dimension, M the total number of filters, P the filter dimension, H the number of selected filters, S the number of shared filters, the number of expert filters, deony the convolution kernel dimension, Cin the number of input channels, and Cout the number of output channels. We also analyze step-by-step breakdown of Filter selection and Delta modulation operation.
Table 12: Complexity comparison for each operation in prefill and decode stages. Arrows (-+) indicate improved complexity.
Operation
Prefill
In Projection
O(Td)) O(TMPd) - ((THP))
ID Convolution
O d) + O(T(H + N)
SSM Kernel
O(TlogT - O[Tlo
Out Projection
O(Td)) = O(TMP) > O(THPd)
RMS Norm
O(Td) = O(TMP) - OTHHP
HADES Ops.
O(T(d M) + - -2S)
Operation
Decode
Projection
O = O(MPd) + (HPdd)
ID Convolution
O Coundoon ) O(MP N) -> O(HHP deo
SSM Kernel
((MN) O(HN)
Out Projection
O(d)) O(MPd) + O(HPd)
RMS Norm
O(d) O(MP) O(HP)
HADES Ops.
O(dd+ MM( + Il - 25))
HADES Operation
Complexity
Residual calculation
O(Td)
Projecting selection score
O M) H --2))) O log E) O(TH) O(TH)
Top-Q selection
Spectral bias calculation
Delta Modulation
F.4 PARAMETER REDUCTION ANALYSIS
As you' 've previously mentioned, the majority of parameters in Mamba2 originate from the linear and convolution layers. In our approach, since parameters are instantiated only for the candidate filters selected the time of filter count determination, we are able to construct a significantly lighter-weight model compared to the original. A detailed parameter breakdown is provided in Table [3] Here, we use the following notation: T' denotes the sequence length, the hidden dimension, M the total number of filters, P the filter dimension, H the number of selected filters, S the number of shared filters, and deenv the convolution dimension.
In of 370M parameters, d = 1024, M 32, H = 16, P 64, N = 128, deoav 4, Nlyayr 48. Therefore, resulting parameter size would be 368, 346, 624 - 150, 407, 424 217, 939, 200 218M.
F.5 FLOPS ANALYSIS
We analyze the computational complexity of HADES by decomposing it into two primary components: the mixer complexity and the routing overhead. The mixer complexity follows the Mamha2 structure
27

---

Published as a conference paper at ICLR 2026
Table 13: Parameter complexity of each component. Arrows (-+) indicate reduction from M to H.
Component
Parameters
in_proj linear
(2 2d + 2N + M) (2MP 2N + M) - d- d2H2 2N + M)
convld
(2d N) deay = (MP P N) dea + (II + N) ddem
out_proj
2d--d MP.d
HP.d
rms norm
2d= = MP + HP
ssm params
3M 3H
Added params in HADES
(d M) (M + 1I 25) +2
Name
Mixer Parameters
Mamha2
M[3 deam + N(2d + deoo))
HADES
H P + dom + +1) 3 N(2d deas)) + M) + 4-22 -2S) +
Reduction
(M (P + deam
1) d + + MM(M) + H 28)
Table 14: Per-token FLOPs of Mamba2 and HADES (excluding IIADES selection module).
Operation
Mamba2
HADES
A (Reduction)
In-projection
22M + 2N + M)
2d(2HP + 2N M)
4dP(M - H)
ID Convolution
2(MP Nydeam
2(HP + N)dcom
2(M P
Out-projection
2MPd
2HPd
2 H)Pd
RMS Norm
MPP
Cona CmnH HP
Cons (M H)
SSD (SSM Core)
G log N
OsaH log N
Casa M H)N log
operates on reduced set of H active filters (H M), serving as the dominant cost factor. The routing overhead encompasses the lightweight operations introduced by the selection mechanism, such as score computation and delta modulation. The following sections detail each component and quantify the overall efficiency gain compared to Mamba2.
Mixer Complexity. The core Mamba2 mixer consists of several filter-wise components including in-projection, ID convolution, out-projection, RMSNNrm, and the SSM kernel. Each filter maintains its own parameters and state, so these computations are applied independently for every filter, making their cost linearly proportional to the filter count M. The FLOPs of the Mamba2 mixer
FLOPS2 T (M FLOP + FLOPS
FLOP 2P3(3d + deonv + 1) + Cssa.N log N
FLOPS 2N + d
HADES uses only H filters from the original M. Since all filter-dependent computations scale linearly with the number of filters, replacing M with yields:
FLOPSHADES-misrr = T (H FLOPS + FLOPS)
Routing Overhead. Beyond the mixer, HADES performs several additional but lightweight per- token operations: residual computation, selection score projection, top-Q selection, spectral bias, and delta modulation. Their individual FLOPs per token are summarized in Table [5] Since these terms small and heterogeneous, denote their total cost over the sequence length 7 as FL.OPSHADES ops rather than collapsing them into a single closed-form expression.
Complexity Comparison. Combining the mixer and routing costs gives the total computational complexity:
FLOPSHADES = FFLOPSHADDSSSmiii
S.OASSADESSpp
(33)
T (H -LLOPsilee + FLOP FLOPSconal (cont) + FLOPSHADES-OR-
(34)
To quantify the computational benefit, we analyze the reduction in FLOPs:
APLOPS FL.OPs - FLOPSHADES
(35)
T (M H) FLOP Sniter FLOPSHADES-ORS
(36)
Main Savings
Roating Overheal
28

---

Published as a conference paper at ICLR 2026
Table 15: Additional per-token FLOPs introduced by the HADES selection mechanism.
HADES Operation
FLOPs/token
Residual computation
2d
Selection score projection
2(d M ( H 28) Cop E log E
Top-Q selection
Spectral bias
2H 2H
Delta modulation
Since the mixer cost FLOPStiter involves heavy O(N log .) operations while the routing head FLOPSHADES-Ops consists only lightweight linear projections, the savings term dominates the overhead (Main Savings >> Routing Overhead). This guarantees substantial net reduction in com- putational complexity. Consequently, by disregarding the negligible overhead, we can approximate the total FLOPs of HADES scaling proportionally with the filter ratio:
H
FLOOPSHADES
Samhu2
(37)
M
This approximation succinctly captures the efficiency gain obtained by activating only H out of M filters.
F.6 TRAINING COMPLEXITY ANALYSIS
To clearly illustrate the training dynamics of HADES, we present the loss convergence landscape and training time observed during training. The training was conducted under the experimental settings described in Appendix C Thanks its parameter reduction, in Fig. [T HADES exhibits faster training speed compared to both Mamba2 and Mambal, and the loss curve in Fig. Ii(a) shows stable and natural convergence throughout training.
Mambaz Mamba1 MADES
146
Step Time (seconds)
104,
1.00
96.565
00
3
20
Training Step
Model
(a) Loss convergence
(b) Training time
Figure 11: Comparison of training behavior across models. Training sequence length was 2048, parameter size 370M, average time per 100 training steps for one epoch.
G LIMITATION AND FUTURE WORKS
While this study introduces a novel perspective on Mambu2 by reinterpreting it as filter bank through the lens of GSP and proposes a new design methodology, there are some limitations. Although our design is inspired by GSP principles, we have not explicitly enforced spectral properties within the model. Instead, we adopt implicit design approach, where spectral characteristics are indirectly encouraged with slight modification of biases. Explicitly enforcing spectral properties could lead to overly rigid behavior, which may hinder model performance. Our current approach aims to maintain flexibility while subtly guiding the model toward desirable spectral behavior, For future work, we aim conduct theoretical analysis of the advantages of explicit spectral design and explore methods for biasing and filter selection that directly leverage these properties. Such an investigation could lead to more robust and interpretable state-space models.
29

---

Published as a conference paper at ICLR 2026
H MORE VISUALIZATIONS
In this section, we provide more visualizations on our method. We compare output difference
regarding 7-value in Fig. 12 following procedure in Appendix E.4 We applied Fourier transform
to filter outputs obtained from a randomly sampled sentence from the Pile dataset. Although the spectrum of filter outputs varies with different values of y, comparing the outputs with and without the bias consistently shows that the bias behaves as intended.
1.0
C35
0-15
1.0
0.35 Irout Scawence
Magnatudee
Iiput Sequence
Nomaa 1200 Magnitude
VIpu Sequence
Maenitude
400
400
Frequency
Frquuncyy
Frequency
(a) variaaiio
1.0
1.5
Spectrna Blas
win Spectral Blass
Nomalred Magritude
0.0
NormalDeed Songgiuudee
Input Sequerce
Inpur Sag Jence
0.0
Noraltzed
0.4
0.
100
-00
200
400
-40
-200
200
400
4000
-200
200
400
Frequency
Frequency
Frequency
(b) y= 0.15
Shared Riter Output inpur Sequence
Excert Rioor OutouE Spectial
Exjort Fittor Output
Warmalize Megnitati
0.8
Doggituuda
Fiput
Cocmmlaa Haantbude
wil Spociaal Soquence
0.6
0.6
Rorallzed
Frequency
frequency
(c) 7 0.35
Figure 12: (a) --variation on expert filters with spectral bias outputs on 13th layer. (b) and (c) Frequency spectrum analysis of filter outputs on 13 layer. Left: Shared filter outputs. Center: Expert filters with spectral bias outputs. Right: Expert filters without spectral bias outputs.
30


Reviews:
[{"review_id": "JL0w56j3pb", "reviewer": "Reviewer_kLjo", "summary": "This paper proposes HADES, a new architecture based on Mamba2 that is inspired by Graph Signal Processing (GSP). The authors frame Mamba2's multi-head structure as a filter bank operating on a line graph. The HADES model introduces a hierarchy of filters: shared filters for global, low-frequency information and expert filters for local, high-frequency details. A routing mechanism selects which expert filters to apply to a token based on its \"spectral residual,\" which is also used to modulate the delta parameter. The method is regularized using two auxiliary losses to encourage filter balance and diversity. The primary contribution is a model that achieves comparable performance to Mamba2 on several benchmarks while using only 58.9% of the parameters.", "questions": "1. The experiments use 8 shared filters and 8 expert filters (S=8, E=8). What was the rationale for this 1:1 ratio? Could you provide any ablation studies on the effect of changing this ratio (e.g., S=4, E=12 or S=12, E=4) on performance?\n\n2. A core motivation is that Mamba2's heads are unstructured. Did you perform a spectral analysis on the heads of a baseline Mamba2 model? it is possible that Mamba2 already learns a diverse set of low and high-pass filters, even without an explicit mechanism. Such a comparison would provide a stronger baseline.\n\n3. Given that the GSP framework is \"implicit\", could you elaborate on why this GSP-inspired router is superior to a standard MoE router that simply learns to route tokens based on a linear projection of the input ?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces an interesting conceptual link between Graph Signal Processing and Mamba2, re-framing multi head SSMs as filter banks. This GSP perspective motivates a novel routing mechanism based on a spectral residual which is a creative approach to token-adaptive computation.\n\n2. The authors conduct a thorough set of ablations on their proposed 370M parameter model, which helps validate the components of their design, such as the auxiliary losses and the hierarchical filter structure. The parameter reduction to 58.9% at this scale is notable. The in-depth analysis showing filter specialization (low-pass vs. high-pass) is also a quality contribution.\n\n3.The paper is clearly written, and the figures are helpful for understanding the architecture and its intended behavior. The authors provide a good explanation of their GSP-inspired framework.\n\n4. The work demonstrates a potential path to more parameter-efficient SSMs. The in-depth analysis showing filter specialization is a good step towards more interpretable models.", "weaknesses": "1. The paper's primary weakness is that all experiments are confined to a single, small 370M parameter model. The central claim of 58.9% parameter savings is not validated at larger scales (e.g., 1B+), where model dynamics and efficiency trade-offs are known to change. This severely limits the generality and impact of the findings.\n\n2. The paper's core premise of a GSP framework is not very strong. The authors admit that \"spectral properties are not explicitly enforced\" but rather \"indirectly encouraged\". The model does not perform operations in the spectral domain. This makes the GSP contribution feel more like a motivating analogy for a heuristic-based MoE router, rather than a principled application of GSP.\n\n3. The method is functionally a Mixture-of-Experts (MoE) design that routes tokens to different heads. However, the paper fails to provide a direct comparison to more standard (and simpler) MoE routing mechanisms. It is unclear if the added complexity of the \"spectral residual\" calculation and \"spectral bias\" modulation offers any real advantage over a simple top-k router on the base token representations.\n\n4.  The paper introduces several new and important hyperparameters, chief among them the ratio of shared (S) to expert (E) filters. The paper defaults to a 1:1 ratio (8 shared, 8 expert) without any ablation or justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NSdFSJ6XTY", "reviewer": "Reviewer_QPLB", "summary": "The paper analyzes multi-head SSMs and Mamba2 as a graph filter bank over a line graph, and introduces HADES. HADES routes per-token to a small set of expert filters, while shared filters are always applied to preserve global information. HADES adds two regularizers - (i) load-balance over expert scores and (ii) output-diversity across filters to prevent collapse. HADES reports competitive performance across 8 zero-shot tasks while using only 58.9% of the parameters, and demonstrate stronger passkey retrieval upto 16K context length.", "questions": "- For passkey, can the region order be randomized to evaluate the model's robustness?\n- When computing the running mean for the token router, is this strictly causal?\n- How might HADES compare to simpler head routing apporaches under the same budget?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The formalization of SSM heads into Graph Signal Processing's filter bank is clear and allows principled analysis about low-pass and adaptive behavior. \n- The routing/bias mechanism tied to \\Delta_{HADES} gives a minimal hook for content-adaptive dynamics which aligns with SSM parametrization.\n- Competitive results at a much lower parameter count regime and performance improvements in long-context tasks.\n- The ablation study and analysis are thorough.", "weaknesses": "- While the competence of HADES with respect to the reduced number of parameters seems promising, the architecture does seem to cause more FLOP overhead. Listing this analysis would strengthen the contributions of this work.\n- The spectral analysis (FFT) on hidden sequences from one layer may be confounded with the layer or the gamma value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GwLqX1U8i9", "reviewer": "Reviewer_neR9", "summary": "The paper introduces a novel SSM-based architecture that builds upon Mamba2. Hierarchical ADaptive filter bank for Efficient SSMs (HADES) is a Graph Signal Processing (GSP)-inspired framework that modifies portions of Mamba2, such that each head corresponds to a graph filter (here, the sequence of tokens is viewed as a line graph) and the multi-head SSM to a bank of filter whose outputs are aggregated. HADES introduces two kinds of filter: namely, a shared filters that is applied (always selected so not part of routing) to capture global content, and expert filters that are routed per token (similar to Mixture-of-Experts or Mixture-of-Heads). Here, the router routes based on the \"spectral residual\", defined as $x_t - mean(x_1,...,x_t)$, and $\\Delta_{t, base}$. In order to guarantee that all expert filters are used evenly, they add a load-balancing loss, similar to MoE designs, and a diversity loss that penalizes filter outputs' deviations from pairwise orthogonality.\n\nExperimentally, the authors show that HADES outperforms other architectures in this line of work, such as Mamba2 and DeltaNet, on 8 zero-shot language tasks (those standard in LM-eval harness and used in the Mamba2 paper for evaluation). When trained on 200B tokens on Pile, their 218M-parameter model outperforms 370M-parameter baseline models. Plus, there is no issue regarding latency and computation memory. Hades beats Mamba2 in long-context retrieval (pass-key retrieval). Finally, the paper offers some interpretability analyses. Frequency‑domain plots suggest shared filters act low‑pass while expert filters emphasize higher frequencies, particularly with the spectral bias on $\\Delta$, and expert‑selection heatmaps show task‑region‑specific specializations.", "questions": "For questions below are minor. Addressing the weaknesses (and suggestions therein) would help me re-evaluate this work.\n\n**Minor misc.**\n1. Eq. (1) in L88 should have $\\bar{A} h(t)$ instead of $\\bar{A} h(t-1)$, right? In continuous time one expects no shift and  $\\bar{A} h(t)$ is the one written for the Mamba paper.\n2. Top‑K selection is non‑differentiable; the paper does not state whether it uses straight‑through, Gumbel‑TopK, or train with continuous weights and harden at eval. Or does Top-Q selection mean something else than I am missing?\n3. I don't entirely understand the load-balance loss in Eq (15). Is this written correctly and precisely? What expectation are we taking and how do we square a vector?\n3. Appendix B.1 says: “For fair comparison, all models are trained... with 370M parameters...”, but I assume here HADES is 218M? It's written confusingly here. If you put the number of parameters and also FLOPs on the table, that would be helpful.\n4. L1025 calculation is wrong: 368,346,624−150,407,424=217,939,200.\n5. Can you also report training perplexity in Table 1?\n6. Can you report the model size in Table 6? I would interested in the latency of HADES when controlling for model size or memory, but here, the latency primarily seems to come from being smaller.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Originality: \n- Recasts multi-head Mamba2 as a graph filter bank on a line graph, connecting LTV SSMs to graph signal processing (GSP) and framing heads as node-variant graph filters. Introduces a novel architecture HADES, a hierarchical filter bank with (i) always-on shared filters and (ii) token-routed expert filters, selected via a spectral residual and $\\Delta$-modulation.\n- Their construction of expert filters creates more opportunity for modular/interpretable filters, as they are trained to be distinct from each other.\n\nQuality:\n- Strong efficiency–performance trade-off: ~59% of Mamba2’s parameters (≈218M vs. 370M) while matching or surpassing baselines on zero-shot reasoning and long-context passkey retrieval; reported ~1.37× speed and memory gains.\n- Their construction is not complicated and does not add much computational overhead when routing.\n- The ablation and sensitivity analyses make the empirical investigation more thorough, and show that each of their designs are necessary.\n\nClarity:\n- The presentation is clear for readers who are less familiar with SSM-based architecture, and the differences between Mamba2 and HADES are clear, both mathematically and visually. Background sections concisely link SSM kernels, convolutional views, and GSP operators, helping readers follow the reinterpretation.\n- The hyperparameters and experimental descriptions are quite thorough, but certain sections of the appendix are not properly linked to the main body so, at times, it may be a slightly confusing.\n\nSignificance:\n- While I don't find the connection entirely illuminating, the GSP lens offers a principled design template for structured, adaptive filtering in SSMs, which may be useful beyond HADES for designing and analyzing SSM-based models.\n- Similarly to the points made for originality, the paper demonstrates decent empirical performances and long-context understanding ability. The devised architecture may help future interpretability research as well.", "weaknesses": "While I enjoyed the paper's presentation and ideas overall, I think the major weakness of the paper is the strength of their empirical evidence. I will list this in two major axes:\n1. **Lack of scale**: Despite the thorough experiments in ablation, sensitivity, and multiple baselines, the paper only operates on the 200B-token Pile training of 370M-parameter models. Because we only get one data point in the (number of tokens trained, model scale), it is hard to know if the model will scale or holds in any other regime of training. If we have another model size (preferably on the larger side) and another training size (100B or 3-400B tokens), the authors would be able to make more robust conclusions. \n- Also, while I acknowledge that SSM-based works such as Mamba pretrain on Pile, this practice holds less value in 2025. We have DCLM, FineWeb, and Nemotron-CC, which are better natural language datasets, but to continue doing experiments on Pile dataset seems outdated.\n- Instead of the 218M HADES model, I would like to see a 370M (or FLOPs-equivalent) HADES model being compared to the baseline models to observe the real gap when controlling for FLOPs or parameters.\n- \"even with a drastically reduced model size of approximately 38.64% (143M) in the H = 8 setting, our model maintains performance comparable to the optimal hyperparameter configuration and even outperforms it on two tasks\" (L903-905): while the authors analyze their results with such framing, it makes me wonder if HADES does not scale well due to this result. If more FLOPs or more parameters lead to better performance (a necessary feature for language models), then this result seems unintuitive. \n2. **Randomness across seeds**: Table 1 and Table 3 do not account for randomness across training runs/seeds. While standard error is low, this only accounts for sampling variance from the dataset and not the variance in training. If possible, averaging evaluation accuracies across five identically trained models (different seeds) for Table 1/3 would make the results more trustworthy. \n- Did you also perform a hyperparameter sweep/tuning for the baseline models, or only perform hparam tuning for HADES?\n\nMy score is between 4 and 6, so for now, I will keep my score as 4 (marginally below the acceptance threshold). Once some questions and concerns are addressed, I am willing to increase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DYp0V4kcgW", "reviewer": "Reviewer_BvVa", "summary": "The paper leverages the framework of Graph Signal Processing (GSP) to interpret and enhance Mamba2, and proposes a hierarchical filter band model architecture, HADES. Each SSM head is interpreted as a graph filter over a line graph, and HADES leverages shared and expert filters for capturing global and local properties, respectively. The paper further designed two losses - load balance loss and diversity loss - to optimize the learning of the expert filters. Empirical results show that HADES reports comparable to better accuracy than baseline across language modeling, reasoning, and retrieval, without long context fine-tuning. Filter selection analysis and frequency response analysis further validate the behavior of the expert filters.", "questions": "1. How is the training speed and convergence behavior of HADES compared to baseline models? \n2. How are the gradients of the Top-K expert selection propagated?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The graph signal processing perspective on Mamba and ssm architecture is novel. Connecting the S4 and Mamba2 kernel to graph filters on a line graph and extending to time-varying filters makes the multi-heads as filter-banks interpretation explicit and operational. The shared plus expert filter design with spectral bias is a novel mechanism to impose functional diversity with small overhead. \n2. The architecture design (shared and expert filter with respective modulations) and loss design (load balance loss to prevent collapse of filter usage and diversity loss for decorrelating filter outputs) are well-motivated, clearly formulated, and empirically supported by in-depth analysis. \n3. The experimental design, especially the in-depth analysis into the behavior of the filters, provides interesting insight into the effectiveness and efficiency of the model, further validating the approach.", "weaknesses": "1. While the GSP-inspired perspective is intriguing, it mostly serves as an interpretation rather than a source of rigorous new theory. The idea of a ``line graph filter” for a time series is essentially equivalent to a standard 1-D convolution or recursive filter, which is a well-studied concept. The paper does not derive new analytical results (e.g. no formal theorem about stability or frequency response), and it doesn’t seem to leverage graph signal processing theory beyond the descriptive level.\n2. The HADES model introduces additional complexity over the base Mamba2. It splits the SSM heads into shared vs expert categories and requires a routing network per time step, along with new hyperparameters. \n3. The comparison only consists of baseline models (mamba1, Mamba2, and linear transformer variants). It would be interesting to see the performance comparison between HADES and Mamba2 models with additional comparable enhancements, such as mixture models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)