The Curious Case of In-Training Compression of State Space Models
Authors: Makram Chahine
Philipp Nazari
Daniela Rus
T. Konstantin Rusch
Venue: ICLR 2026 Poster (2026)
Keywords: State Space Models
Sequence modeling
Model compression
Dynamical Systems
Primary area: learning on time series and dynamical systems
TL;DR: We introduce a control-theoretic approach to compressing state space models during training, achieving faster optimization and higher performance than training smaller models directly.

Abstract: State Space Models (SSMs), developed to tackle long sequence modeling tasks efficiently, offer both parallelizable training and fast inference. At their core are recurrent dynamical systems that maintain a hidden state, with update costs scaling with the state dimension. A key design challenge is striking the right balance between maximizing expressivity and limiting this computational burden. Control theory, and more specifically Hankel singular value analysis, provides a potent framework for the measure of energy for each state, as well as the balanced truncation of the original system down to a smaller representation with performance guarantees. Leveraging the eigenvalue stability properties of Hankel matrices, we apply this lens to SSMs $\textit{during training}$, where only dimensions of high influence are identified and preserved. Our approach, CompreSSM, applies to Linear Time-Invariant SSMs such as Linear Recurrent Units, but is also extendable to selective models. Experiments show that in-training reduction significantly accelerates optimization while preserving expressivity, with compressed models retaining task-critical structure lost by models trained directly at smaller dimension. In other words, SSMs that begin large and shrink during training achieve computational efficiency while maintaining higher performance. Project code is available at https://github.com/camail-official/compressm.

Published as a conference paper at ICLR 2026
THE CURIOUS CASE OF IN-TRAINING COMPRESSION OF STATE SPACE MODELS
Makram Chahine CSAIL, MIT
Philipp Nazari
Max Planck Institute for Intelligent Systems & ETH Zürich & ELLIS Institute Tübingen
Cambridge, MA 02139, USA chahine@mie edu
philipp.
de
Daniela Rus
T. Konstantin Rusch
CSAIL,, MIT Cambridge, MA 02139, USA rus@csail mit. edu
ELLIS Institute Tübingen &
Max Planck Institute for Intelligent Systems & Tübingen AI Center &
Liquid Al
tkrusch@tue.cm www.eetteeeeeele......
ABSTRACT
State Space Models (SSMs), developed to tackle long sequence modeling tasks efficiently, offer both parallelizable training and fast inference. At their core are recurrent dynamical systems that maintain hidden state, with update costs scal- ing with the state dimension. A key design challenge is striking the right balance between maximizing expressivity and limiting this computational burden. Control theory, and more specifically Hankel singular value analysis, provides a potent framework for the measure of energy for each state, as well as the balanced trun- cation of the original system down to smaller representation with performance guarantees. Leveraging the eigenvalue stability properties of Hankel matrices, we apply this lens to SSMs during training, where only dimensions of high influ- ence are identified and preserved. Our approach, COMPRESSM, applies to Linear Time-Invariant SSMs such Linear Recurrent Units, but is also extendable selective models. Experiments show that in- in-training reduction significantly accel- erates optimization while preserving expressivity, with compressed models retain- ing task-critical structure lost by models trained directly at smaller dimension. other words, SSMs that begin large and shrink during training achieve computa- tional efficiency while maintaining higher performance. Project code is available at liggthuu... caaam//amaaleee caaammlllffffio -officiall compressm
INTRODUCTION
State Space Models (SSMs) (Gu et al. 2021 Hasani et al., 2022] Smith et al. 2022 Orvieto et al.,
2203 Rusch & Rus, 2025) have recently emerged as a powerful alternative established sequence models such as Recurrent Neural Networks (RNNs) and Transformers. They combine the paralleliz- able training efficiency of scaled dot-product attention (Vaswani 2017) with the computational and memory advantages of RNNs, enabling strong performance acToss large-scale language, vision, and audio modeling tasks (Gu & Dao 2024 Goel et al. 2022 Nguyen et al. 2022).
Despite their efficient structure and recent progress in hardware- implementations, current SSMs remain computationally intensive. While both memory and runtime scale with sequence length, the size of the SSM state further amplifies these costs. Reducing the state dimension therefore provides an effective strategy to simultaneously reduce memory usage and runtime. This can be achieved by leveraging techniques from structured compression. However, most existing approaches are commonly applied
a large model trained to completion and only compressed afterwards. Popular examples include knowledge distillation (Hinton et al.) 2015), post-training quantization (Jacob et al. 2018), low-rank factorization (Hu et al. 2022), and structured pruning (Li al. 2016). All of these methods typically require the costly upfront training of large network.

---

Published as a conference paper at ICLR 2026
In this article, address the issue of costly pre-training by introducing COMPRESSM, a principled in-training compression technique that effectively reduces the dimension of the SSM while largely preserving the expressive power of uncompressed models. We motivate approach by the control- theoretic origins SSMs (Kalman 1960). In particular, we draw on balanced truncation (Antoulas) 2005], classical Model Order Reduction (MOR) technique that approximates high-dimensional state space system with a now-dimensional one while retaining its essential input-output behavior. Observing that the dominant Hankel Singular Values (HSVs) of SSM are punk-rreerrinn during training. we propose truncating dimensions associated with small Hankel singular values once they fall below a predefined relative threshold.
Main contributions. In the subsequent sections, we will demonstrate the following features of COMPRESSM:
We rigorously justify the validity of our in-training compression approach by establishing means to identify and track the HSVs an SSM during training, and showing that dom- inant singular values HSVs can be safely truncated.
prnk-preserving. Thus, SSM dimensions associated with small
We show that COMPRESSM is broadly applicable, including to SSMs with structured state matrices such diagonal matrices, with simple extensions to Linear Time-Varyine Varying systems discussed.
We provide an extensive empirical evaluation demonstrating that COMPRESSM largely preserves the expressive power of uncompressed models.
COMPRESSM significantly accelerates training, while achieving similar or higher accuracy than larger uncompressed models by truncating large portions of the state early in training.
2 MATHEMATICAL PRELIMINARIES
2.1 DISCRETE LINEAR TIME INVARIANT SYSTEMS
Let be a discrete Linear Time-Invariat (LTI) system described by state equations:
h(k + 1) A hkk) + B akk), ykk C hkk) + D x(k),
hx(0) ho
(I)
where h E RD is the state, E IRP the input, and y E Re the output, with A E Ruxn B E Raxp, Ce R8xn, and E R2xp.
A more general class of systems are Linear Fime-Varyine (LTV), where the matrices A, D, are functions of time. Such systems become relevant the context selective SSMs, where the system matrices depend on the input. For now, we restrict to the LTI case the base framework. The LTV case discussed in Appendix E.
The LTI framework provides tractable and well-understood setting in which powerful tools such as Gramians and balanced truncation can be developed. Before introducing these concepts, we formalize the standard assumptions of stability, controllability, and observability, which ensure that the system is both well-posed and non-degenerate. For precise definitions of terms and detailed background presentation we refer the reader chapters 4, 5, and of Chen (1999).
Assumption 2.1. The system is stable, i.e all the eigenvalues of A are of amplitude less than I. Assumption 2.2. The pair (A, B) is controllable, i.e., the state can be steered from any initial state any final state in finite time.
Assumption 2.3. The pair (A, C) is observable, i.e., observing the oulput and the input a of the system for some finite time suffices to determine the initial state hg-
2. CONTROLLABILITY AND OBSERVABILITY GRAMIANS
The concepts of controllability and observability be captured quantitatively by matrix-valued energy measures called Gramians. These respectively encode how easily internal states can be excited by inputs observed from outputs.
2

---

Published as a conference paper at ICLR 2026
SSM block Rlyyers
Proyection
Encoder
Projection/ pan-linearityy Convolution
Discrete Linear dynamical systems
Product/ Non-lineaiiyy
Decoder
M
Input
Output
kelh aepam
mplice
Reduction Analysis
Balanced Truncation
System Replacement
P.O
Invariant Grrent ans ())
meilancing_Patriiee (atrlx((,t)
Hankel singular values .000
Ab, Cb Balance BysCT,A,B,C)
5 Br
Reduction, Order(a, tol)
Ar,Br,CCr Trancate(r.Ab,BB..Ch)
C-
Figure 1: Overview of the proposed balanced truncation pipeline. The method applies the level of the discrete linear dynamical systems inside SSM layers, independently of surrounding design choices such as projections, con-lincassties. convolutions, or skip connections. Each dynamical sys- tem is isolated, balanced via its controllability and observability Gramians, and truncated according Hankel singular values before being reinserted into the model.
Under assumptions 2.1 and 2.2 there exists unique symmetric positive definite solution P E Raxx to the discrete Lyapunov equation:
APA - P =0
P Σ ABB (
(2)
1=0
P is known as the discrete controllability Gramian. intuitively captures how much energy from
the input reach each state dimension over time. Large entries indicate states that easily influenced by inputs.
Under assumptions 2.1 and 2.3 there exists unique symmetric positive definite solution Q Raxx to the discrete Lyapunov equation:
A'QA-Q+C
Q (A 2=0
Q is known as the discrete observability Gramian. It similarly measures how much each state contributes to the outputs over time. Large entries correspond state directions that are easily observed through the outputs.
2.1.2 BALANCED REALIZATIONS
The notions of controllability and observability are key for establishing a diagonal balanced realiza- tion, in which the system is transformed that both Gramians equal and diagonal. This provides a natural coordinate system where each state dimension has well-defined "importance."
Definition 2.4 (State Space Realization). A discrete-time linear system G is fully characterized by its onputt map
g: (a(k)) + (k)))))
realization is any quadruple matrices (A, C, D) and state h(k) E R that realizes this input-output map via the dynamics given by Equation]] T Many different realizations can realize the same map. particular, if (A, D) is realization, then so is (T AT, AT B,, CT D) for any invertible T E Rox..
Definition 2.5 (Minimal/Balanced realizations). A realization is called minimal it is both control- lable and observable. The corresponding state dimension is called the order of the realization.
realization is said to be balanced if P = Q. this case we denote the common matrix by W, and refer to it simply the Gramian of the balanced system.
3

---

Published as a conference paper at ICLR 2026
Theorem 2.6 (Antoulas) (2005)), Any stable, minimal discrete LTI system admits bal- anced realization, in which the controllability and observability Gramians coincide as W diag(or) (i... σ)), with di (HSV). This diagonal balanced realization can be explicitly constructed.
2 on 0 called "Hankel singular values"
The HSVs can also be computed in decreasing order from non-balanced realizations via:
σ = sorty (Vspec(PQ)
(4)
The HSVs quantify the joint controllability and observability of each state. Large values indicate state directions that both strongly affect the output and are strongly influenced by the input, while small values correspond to weakly contributing states.
2.1.3 BALANCED TRUNCATION
Balanced truncation is MOR scheme leveraging the ordering of Hankel singular values to obtain lower-dimensional approximation of the system, while guaranteeing stability and error bounds.
Consider a stable minimal balanced realization (A, B,,, of G with Gramian W diag(or (r) = diag( 11 Σz) where Σ1 is diagonal of size r and E₂ of size - with the smallest entry in Σ, larger than the largest in Σ2. The state space matrices can be rewritten as:
A1,1
A1.2
B
B
C C= C2], B2
A
A2,2
with A1,1 E RRxx, Bi E TRMX,, and C₁ E Rexr.
It is well established that the reduced system & defined by A1,1 B1, C1, D is stable and
- 9 2 Σ o..
Balanced truncation thus provides principled way to reduce model order while controlling the approximation еπrоr in terms of discarded Hankel singular values. This makes it central tool for simplifying state space models while preserving their dominant dynamics.
2.2 SPECTRAL STABILITY HERMITIAN MATRICES
In practice, training SSMs with gradient descent modifies the learned state matrices incrementally. Understanding how the downstream Hankel singular values shift under such perturbations is there- fore critical to establish in in-training reduction protocols. For Hermitian matrices, Weyl's theorem provides a powerful tool.
Let W and W' be Hermitian matrices of size (i.e. symmetric for real-valued matrices), and let 5W = W W. Also, E [1, [1.....,n] n). let (WW) represent the i-th largest eigenvalue of W. The ordering 0, (W) ... 2 o, (W) can always be established the eigenvalues of Hermitian matrices are guaranteed to be real.
Theorem 2.7 (Weyl (1912)). Vi [1....,,i] n). o, (W) is Lipschite-comtinuuus the space of Her- mitian matrices with operator norm:
la:(W") o. (W)) < max (lo,(WW) = max((o((6W)),
i=....
In other words, each of the eigenvalues of W can at most fluctuate by the largest absolute eigenvalue of the perturbation 8W, providing a bound spectral variations under Hermitian perturbations. This bound will turn out to be crucial for our in-training reduction scheme (see Section 3.3
THE PROPOSED IN-TRAINING REDUCTION SCHEME
Our general pipeline (illustrated in Figure is designed be applicable to all types of SSMs surgically acts on the dynamical systems within SSM layers of models regardless of the choice of projections, non-linear activations, convolutions, skip connections, etc.
4

---

Published as a conference paper at ICLR 2026
To achieve significant gains in training time, we aim to reduce the model's hidden state dimension where possible early on in training. Typically, we attempt to reduce SSM state dimensions at snap- shots of the model obtained at fixed intervals at early stages of training (e.g. during learning rate warm-up). Section 3.3 provides justification for the validity of this approach.
3.1 COMPRESSM: THE ALGORITHM
At given training step, we proceed per block. For an input feature vector sequence Rxx,, where H is the inner dimension and I. the sequence length, common SSM blocks contain either single Multi-Input Multi-Output (MIMO) system that transforms the sequence E RR to y E R or H independent per-channel Single-Inpur Single-Output (SISO) systems that transform x² E g to y _ for E { H... J. In the latter case, we proceed per channel.
For given block (and possibly given channel index), the reduction algorithm works follows:
1. Extract the discrete linear system matrices A, B, C from the model weights. We denote by n the current order (rank) of the system.
2. Solve Equation 2 and Equation 3 (using Equation 14 if A is diagonal, is the case for many modern SSMs) to obtain the Gramians P and Q respectively.
3. Compute the Hankel singular values via Equation
4. Find the smallest rank such that the top-r singular values retain predetermined threshold T) E [o, 1] of the total energy,
(1 (
min
2
5. If the rank is smaller than a given fraction of the initial state dimension (i.e. the reduction is large enough to warrant the trouble), compute the balancing transformation matrix T from Theorem 2.6 Otherwise, leave the system unchanged.
6. Transform the original system its diagonal balanced realization,
(Ab, BB, C.) (T- AT, B,,TT
7. Truncate the balanced system down rank (with slight abuse of tensor slicing notation),
(,, C,) = (Aul: r,: r]. B&|: r. Cb(:;: r))
(10)
8. Replace the model weights for the dynamical system matrices. Based the architecture this might require diagonalizing the truncated system to ensure computational consistency.
(A, BA C+)
(11)
Note that the above algorithm not deliver improved small models if there is no clear correlation between state dimension and model performance. The fact that COMPRESSM applies an in-training compression scheme enables significant increase in performance per training time. Indeed, validate the speedup experimentally in Section 4.2 and provide an in-depth complexity and time reduction analysis in Section of the Appendix.
3.2 PRAGMATIC VARIANT
In ablations presented in Appendix C we observe that training can sustain successive moderate bal- anced truncations without divergence from the large-model upper-bound performance, until impact- ful HSVs removed. This observation motivates slight tweak to the COMPRESSM algorithm. Although one does not have access to the upper bound trained at the large dimension throughout, we track the validation metric during training and assume that, long no drop in performance occurs, the reduction does not cause any noticeable global performance drop.
In practice, the procedure is implemented with simple safeguard. At each fixed-fraction reduction step (10% per reduction is reasonable), we first save the current (pre-reduction) checkpoint. We then apply truncation, train the model for very small number of steps, and evaluate it the validation
5

---

Published as a conference paper at ICLR 2026
Haniol Singilar Vilues
Mao. Alexdute Eigenvoiue dH
Libely HSV Trajecuurio
Eaery Fastion Botzom
18
100
10p.
0.5
106
10"1
10300
Step
Step
Step
Figure 2: In-training per-step analysis of Hankel singular value dynamics for single LRU block with state dimension of 8 on the sMNIST dataset for the first 25k steps. The leftmost plot shows the raw HSVs (as a set). The middle-left plot depicts the maximum absolute eigenvalue of 5H as described in Section 333 The middle-right plot overlays the maximum variation bound as an error margin around each HSV, with each shade now representing highly probable path for specific state dimension obtained by step by step linear sum assignment solving. The rightmost plot shows the relative contribution of the bottom HSVs the total energy.
set. If validation performance continues to improve, we proceed to the next reduction. If perfor- mance degrades, we discard the reduced version and revert to the previously saved checkpoint, after which no further reductions are applied. Results with this approach are provided in Section B.2
This protocol ensures that model quality always remains close that of the unreduced baseline, without the need to explicitly tune the number of reductions or fixed tolerance level.
3.3 IN-TRAINING REDUCTION
The validity of our proposed in-training truncation relies on several non-trivial properties, which first motivate intuitively before formalizing. First, the method requires tracking how the relative importance of individual states evolves with training. Second, even if we can measure importance continuously, training dynamics may render initially insignificant dimensions crucial at later stages. Early truncation of such dimensions would therefore be undesirable. Consequently, it is necessary that the relative ordering of importance remains approximately stable, least for the dimensions with low initial contribution. Finally, since our balanced truncation approach relies on the energy contribution of each dimension relative to the total system energy, it desirable that the cumulative importance of the bottom-r dimensions does not increase substantially during training, Otherwise, dimensions could converge to regime of near-equal importance, making early truncation unjustified even if the ordering is preserved.
To leverage Hankel singular value analysis during training, we must first establish a protocol for tracking individual state importance as the system evolves under gradient updates. This existence of such a protocol is non-trivial and its development is central to this work.
Indeed, between gradient steps, model weights are updated according to the negative gradient of loss with respect to the parameters. the SSM level, this translates into the state matrices being incrementally updated such that discrete system described by (A, B, C) becomes different dynamical system (A', B', C"), where
(12) The omission of is deliberate for both notational simplicity. but also since it is often fixed as a skip layer and not leamned. Throughout this section, adopt the convention that plain symbols denote pre-update quantities, while primed symbols denote their post-update counterparts.
A' A B' B'=B+8B
C' C'
Now, using the expressions of the controllability and observability Gramians given respectively by Equation 2 and Equation 3. one can see that both Gramians are continuous with respect to the gradient perturbation the system matrices thus we can also consider,
p'= P OP, Q'
Q + 8Q.
(13)
where 8P P is some continuous function of (ōA, 8B), and similarly 5Q of (ōA, 8C).

---

Published as a conference paper at ICLR 2026
Also recall that the Hankel singular values can be obtained as the square root of the eigenvalues of PQ (Equation 4.. Generally, PQ need not be symmetric, but we use this form for computational efficiency. Noticing that PQ is similar to the symmetric positive definite matrix PAQP: we let H = (P*OPA) The matrix H's eigenvalues are exactly the Hankel singular values. In addition. by composition, H continuous with respect to the perturbations the system, so we write H' = H + & with HH continuous function of (6.A, 8 8B, (C).
Lemma 3.1 (Continuity of Hankel singular values under training updates). By application of Weyl's Theorem 2.7 the matrix H and its perturbation H', between gradient steps, each Hankel singular value can at most change by the largest absolute eigenvalue of 8 H' H.
While the previous argument establishes that Hankel singular values evolve continuously with gra- dient updates, the remaining conditions-stafee staability of relative ordering and low contribution of the bottom-r dimensions cannot
cannott be guaranteed theoretically. However, empirical evidence strongly suggests that standard training dynamics are favorable in practice.
We examine the case of a single LRU block trained on the sMNIST dataset with state dimension of for visual clarity in Figure 2 (and provide plots for larger models and datasets in Appendix [B.3). To keep track of the state dimension to HSV value correspondence, we overlay the maximum absolute top of each HSV. Empirically, this allows to robustly identify probable HSV trajectories as the continuity bound is consistently small enough to ensure each eigenvalue clearly isolated, with minimal bound overlaps and rare gradual HSV relative order crossings occurring (prediction of evolution established via linear sum assignment solution in such cases). Furthermore, the cumulative contribution of the bottom-r HSVs stabilizes rapidly. Indeed, after small initial number of steps, we observe that both the ordering of singular values remains constant, and dimensions of low importance seldom gain substantial relative energy during training.
eigenvalue of 8H.
In sum, these observations provide empirical justification for early in in-training truncation: dimen- sions identified as negligible at early stages typically remain so throughout training. Consequently, truncation decisions made during training rarely conflict with the final importance ranking, making our approach both effective and robust in practice.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
In order to empirically validate in-training balanced truncation, we train a linear recurrent unit (LRU) (Orvieto et al. 2023) on datasets of different complexity, ranging from sMNIST tasks from the long range arena (LRA) (Tay et al. 2020).
We use the same training pipeline as Rusch & Rus 2025 Walker et al. (0255) accounting for additional quirks of LRU training like a learning rate factor for the sequence mixing layer. The LRA dataloaders are borrowed from (Smith al. (2022). the hyperparameters largely taken from (Orvieto
et al. 2023 Table 10). We summarize them in Table 3
Reduction starts from the full model order reported in column 4 of Table 3 For all datasets but IMDB and SMNIST, we attempt four equidistant reduction steps during the warm-up period of the learning rate, which equals 10% of the total steps. Doing so ensures maximal speed up potential for the subsequent 90% of training, while also staying robust to large early training perturbations.
As sMNIST is trained without learning rate decay, we attempt truncation during all of training. For IMDB, include waiting stage and attempt reduction inside smaller time window. The details can be found in Appendix AA.3 Generally, reductions are only executed the reduced dimension is less than 95% of the current state dimension.
The baselines we compare against are non-reduced models trained all along with each block initial- ized at the average final order of the compressed ones allow for a fair comparison. To increase our baseline statistics and further establish correlations between state dimension and model perfor- mance, train further models with different state dimensions.
7

---

Published as a conference paper at ICLR 2026
Table Average final state dimension (mean + std) and Top-3 runs mean performance with/without reduction for LRU under different tolerances т.
Dataset
Metric
1.5.10
7 1 1.10-1
1.
5.10-
3-10-
2.10
67.4
100.=
6.1
CIFARIO
COMPRESSM
0.2
36.2011
1..0
85.8 0.1
86.0 = 0.2
86.1 0.2
Baseline
0.72
81.8 0.3
0.2
8.
819 1..0000
56.0 0.1
80.5 0.3
Stare dim
5..8334
81.8
108.8
6.5.4-6.8
167.6 167.6.5.77
213.8
256
ListOps
COMPRESSM
0.7
$1.8009
4822
47.5
49.2=03
47.1414
Roseline
4.4
46.3 0.5
49.4 1.8
49.2 + 0.7
48.2 2.1
47.6 +. 1.7
40.7 0.8
State dim COMPRESSM
19
1.
111.0
136.6 2.9
170.0
203.2 13.7
256
AAN
0.3
0.3
8T.2
87.9
Baseline
8.3
$7.9 0.3
87.8 0.2
87.8 1: 0.5
87.3 0.4
$7.4 0.5
87.3 4: 0.3
State dim COMPRESSM
2.30
119.6
136.8 +:
150. 1.2
165,0 1.3
192.0 0.0
192.0
0.4
Baseline
82.7
$3.5 0.1
83.7 +: 0.0
84.0 0.4
84.3 0.0
84.5 + 0.1
84.7 0.1
State dim
34.6 1.9
51.2 1.7
65.6 2.3
$1.2 1.6
105.0
129.8 5.2
256
Pathtinder
COMPRESSM
97.9
97.6
Baseline
9.2
97.9
98.0
98.1
98.2
98.11
98.3 0.1
== 4-10-2
2-10-2
1-10-2
5-10-a
f 2-10-3
1-10-a
* 0
State dim COMPRESSM
1.07
27.6 1.8
16.8
76.3 7,5
191.4
256
97.2
Buseline
92.6
96.0 0.2
95.9 4:
96.4 0.2
0.2
97.3
97.3
4.2 EMPIRICAL RESULTS
We repeat all experiments using five different random seeds and report mean top-3 well as top-1 performance. Table [ contains the top-3 results. The top-1 results can be found in the appendix (Table The state dimensions reported for multi-block models are the averages of the SSM orders per-block. accompanied by Figure which presents the performance visually and contains further non-reduced benchmark models.
For some datasets, for example AAN or Pathfinder, our baseline experiments reveal small cor- relation between state dimension and model performance, given the other model parameters taken from Orvieto et al. (2023) (see entries and 5 in Table [ or figures 4c and 4c On these datasets, COMPRESSM not deliver better performance for small models.
However, on datasets where state dimension does correlate with model performance, COMPRESSM improves small model performance. On CIFARIO. 10, for example, model performance almost stays constant as a function of reduction tolerance (and thus for different state dimensions), while the non- compressed counterparts exhibit approximately 10% performance drop (see entry in Table For Figure 3a). The sMNIST results paint similar picture (entry 5 in Table [ or Figure 44).
Similarly, ListOps, the performance of non-compressed models drops significantly for state di- mensions smaller than 120 (see Tfg While the compressed models perform on par with their uncompressed counterparts for larger state dimensions, smaller models outperform the baseline.
On Pathfinder, we can observe a similar trend (Figure 4c), where the unreduced model performance does not show strong correlation with the state dimension. Just for the smallest state dimension, the unreduced model sees a small drop in performance and is outperformed by the top-1 reduced model.
The IMDB results reveal the importance of the prerequisites mentioned in Section 3.1 (see Fig- Even after significantly reducing the parameters and increasing the droprate (see Ap- pendix A the unreduced models only learn for roughly 8k steps before overfitting. However, in order to in-training balanced truncation, the actual training phase needs to be long enough to allow for couple of reduction steps with large enough spacing so that the model can follow the training dynamics for a while without being pruned. Indeed, for non-aggressive pruning (that is, pruning with small tolerance r), the top reduced models often outperform the baseline.
ure
As mentioned previously, an advantage of COMPRESSM is that it comes with a training speedup; the state dimensions get pruned, training speeds up. Indeed, Figure 3b demonstrates this effect on CIFAR10. COMPRESSM at dimension 92 preserves nearly full baseline accuracy (85.77 (85.7% vs. 86.5%) and achieves 1.5x speedup. By comparison, training directly at dimension 92 is only
8

---

Published as a conference paper at ICLR 2026
marginally faster (1.6x speedup) yet markedly worse, reaching just 81.8% accuracy. A detailed breakdown of in-training speedup gains is provided in Appendix D
100
155
Armwacy
Avcuuacy
0384
0.54
1.8
Bos Tey-S
viRoduced
rideluoed
0.7
0.78
15
L1
12
1.
1.4
1.5
Mom SSM Order
Training Time Spontup
(a) Test accuracy vs. Final state dimension
(b) Test accuracy vs. Training speedup factor
Figure 3: Subfigure shows the performance of different models trained on CIFARIO function of the state dimension. Grey data indicates con-reduced models, and the shades of orange correspond reduced models, with tolerance decreasing with redness. The circles represents the top-3 mean, while the star corresponds to the top-1 model. Subfigure (b) shows top-3 performance as a function of training speedup with respect to the original large model training without reductions (n = 384 baseline). Marker diameter is proportional to the final model order (also annotated) and in-between models are omitted for visual decluttering.
4.3 COMPARISON TO OTHER MODEL ORDER REDUCTION TECHNIQUES
We compare COMPRESSM to other model order reduction approaches, namely reduction with Han- kel Nuclear Norm (HNN) regularization (Forgione et al., 2024 Schwerdtner al.) 2025), well as Knowledge Distillation (KD) (Hinton et al. 2015). The approaches are described in Appendix with implementation details provided as well. Results of the experiments summarized in Ta- ble 2
The former experiment reveals three clear findings: (i) HNN regularization is computationally pensive, slowing training by an order magnitude at least due to repeated Gramian-eigee -iinniann-egenvalueeee eval- uations (at every single gradient step); (ii) HNN-constrained models underperform unconstrained baselines even before reduction; and (iii) when targeting small final dimensions, COMPRESSM achieves higher accuracy while being over dramatically faster.
On the other hand, knowledge distillation performs on with COMPRESSM as long as the teacher has similar state dimension as the student. However, as the model further compressed, KD's performance drops significantly. while COMPRESSM is able to maintain performance. KD fur- thermore requires training the large teacher model to completion before training a smaller student model, as well as doing a forward pass through the teacher even when training the student. This can be substantially slower than COMPRESSM.
RELATED WORK
Model compression techniques in machine learning. The question of model compression has received considerable attention the literature (Deng al.] 2020 Zhu et al. 2024), leveraging various techniques (and mixtures thereof) such as pruning (Han et al. 2015 Frantar & Alistarh 2023), which removes redundant parameters to reduce network size; quantization (Xiao al. 2022
Gholami et al. 202I), which compresses models by lowering numerical precision; low-rank factor- ization (Lin et al. (2024), which exploits structure weight matrices to reduce dimensionality: and
knowledge distillation (Hinton al. 2015 Gou al. 2020). where smaller model learns to mimic a larger one.
In-training post-training paradigms. Techniques such quantization-ane aware training (Choi ct al., 2018)) Zhhang et al.) 2018). and dynamic pruning (Hoefler et al., 2021] Wimmer et al. 2022) perform in-training compression during the optimization process. In contrast, most methods like

---

Published as a conference paper at ICLR 2026
Table MOR techniques on SMNIST and CIFARIO. Top-3 mean test accuracies (%) and training speedup factors are reported. HNN Regularization is evaluated on sMNIST, and Knowledge Distil- lation evaluated on CIFARIO. n represents models' final order (rounded).
Method
Metric
13
n
28
11
47
n
76
n
148
191
256
Baseline
Acc. (9)
92.6
96.0
95.9
96.4
97.3
97.3
SMNIST
Speed
3.1
2.8
2.7
2.4
2.0
1.7
1.0
COMPRESSM
Acc. (9)
95.9
96.9
96.9
96.9
97.0
97.2
Speed
2.8
2.6
2.5
2.3
1.9
1.6
HNN Reg.
Acc. (%)
91.7
95.8
95.8
95.8
95.8
95.8
95.9
Speed 0.06
0.06
0.06
0.06
0.06
0.06
0.06
Method
Metric 57
n
93
126
1 161
n 214
327
384
Baseline
Acc. (%) Speed
81.8
83.7
84.2
86.5
CIFAR10
1.69
1.62
1.58
1.43
1.22
1.03
1.00
COMPRESSM
Acc. (%)
84.4
85.7
86.0
85.8
86.0
86.1
Speed
1.52
1.41
1.33
KD
Acc. (%)
79.4
83.5
84.4
85.3
87.0
Speed 0.55
0.52
0.61
0.51
0.49
0.45
Deep Compression (Han et al. 2015) follow a post-training paradigm, applying pruning or quanti- zation after convergence and relying on retraining or fine-tuning to recover accuracy.
Compression in SSMs. Work on compressing SSMs has primarily explored quantization. Post- training quantization has been applied stabilize SSM inference under 8-bit constraints (Abreu et al. 2024: Chiang et al.] (224). while quantization-awa -wwaee training has been used to maintain ac- curacy below bits (Zhao et al. 2025) and to improve robustness for deployment on specialized or analog hardware (Siegel et al. 2024 2025).
By contrast, control-theoretés MOR approaches have been applied to diagonal S4 layers, but only as a post-hoc step initialize retraining (Ezoe & Sato 2024). Elsewhere, regularizers based on the Hankel nuclear norm or modal & have been shown to encourage parsimonious state represen- tations during training, though without explicit intermediate truncation events, and with price to pay in terms of both per step compute time as well as optimal performance (Forgione et al., 2024] Schwerdtner et al., 2025). Finally, H4-ootimal reductions have been proposed as competitor to balanced truncation in offline SSM MOR settings (Sakamoto & Sato 2025). This approach is com- putationally expensive, requiring gradient complex optimization problem, without the solution guaranteeing stability of the reduced order system.
Though compression strategy, the authors in Yu et 2024) approach the problem of HSV efficiency from the perspective of initialization, aiming to avoid fast decaying singular values.
To the best of our knowledge, our contribution the first propose principled in-training model order reduction method applicable to broad spectrum of SSMs.
6 CONCLUSION
We propose COMPRESSM: framework for principled in-training compression of SSMs. Lever- aging control theory, we utilize Weyl's theorem track the evolution of HSVs during learning. Empirically, we observe that the relative contributions of individual state components remain highly stable. This allows us to safely apply balanced truncation early the training process.
COMPRESSM is put to the test with LRUs on a range of LRA tasks. Provided there is a correlation between state capacity and model performance, we verify empirically that compressed models out- perform their uncompressed counterparts while delivering better performance per unit of training time. In short, COMPRESSM enables performance of high capacity models at both the inference and training overheads of smaller counterparts.
Finally, we provide an implementation strategy for linear time-varying systems. In the selective case, propose averaging the dynamics over the input space before applying the reduction scheme. Future work will focus on extending COMPRESSM to linear attention models, e.g., Gated Linear
Attention Yang et al. 2023), Mamba2 (Dao & Gu 2024.. and Gated DeltaNet (Yang al. 2024).
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Philipp Nazari was supported by the Max Planck ETH Center for Learning Systems. This work was supported in part by the Hector Foundation, the Boeing Company, and the ONR Science of Autonomy program N00014-23-1-2354.
REFERENCES
Steven Abreu, Jens Egholm Pedersen, Kade Heckel, and Alessandro Pierro. Q-S5: Towards Quan- tized State Space Models, 2024.
Athanasios C. Antooulas. Approximation of Large-Scale Dynamical Systems, chapter 7. Balancing and Balanced Approximations, pp. 207-247. SIAM Advances in Design and Control, 2005. doi: 10. 001113771.7780099778887783888 URL htttht://pppussttts siam. wwtp:o/ggg//ooooooottttet//tottt 1.9780396718113
Chi-Tsong Chen. Linear System Theory and Design. Oxford University Press, 3 edition,
1999. ISBN 978-0-19-551177--6 ktCC8LLKOF (ww.LoOOO/IIiieeee system theryyyrntttoryrrrrnntttterrr
Hung-Yuch Yuch Chiang. Chi-Chih Chang, N. Frumkin, Kai-Chiang Wu. and Diana Marculescu. Quamba: a Post-Training Quantization Recipe for Selective State Space Models, 2024.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, P. Chuang, Vijayalakshmi Srinivasan, and K. Gopalakrishnan. PACT: Parameterized Clipping Activation for Quantized Neural Networks, 2018.
Tri Dao and Albert Gu. Transformers SSMs: Generalized Models and Efficient Algorithms through Structured State SpaceDuality. arXiv preprint 1xXiv:2405.2106 2024.
Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Al- bert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Des- jardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models, 2024. URL hww.:tttss//// /ttp///axvvvvv// wwwp/ba////......../////////
By Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model Compression and Hardware Acceleration for Neural Networks: Comprehensive Survey, 2020.
Haruka Ezoe and Kazuhiro Sato. Model Compression Method for S4 With Diagonal State Space Layers Using Balanced Truncation, 2024.
Marco Forgione, Manas Mejari, and Dario Piga. Model Order Reduction of Deep Structured State-Space Models: System-Theoreth Approach, 2024. / wwt.:orr/aabb abs/ 2403. 14833
Elias Frantar and Dan Alistarh. SparseGPT: Massive Language Models can be Accurately Pruned in One-Shot, 2023.
A. Gholami, Sehoon Kim, Zhen Dong, Z. Yao, Michael W. Mahoney, and K. Keutzer. A Survey of Quantization Methods for Efficient Neural Network Inference, 2021.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's Raw! audio Generation with State-Space Models. In International Conference on Machine Learning, pp. 7616-7633, PMLR, 2022.
Jianping Gou, B. Yu, S. Maybank, and D. Tao. Knowledge Distillation: a Survey, 2020.
Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces, wttp://wwww..//////22..2....
2024. URL /tt.at/s///arvv..oo////////22 00752
Albert Gu, Karan Goel, and Christopher Ré, Efficiently Modeling Long Sequences with Structured State Spaces. arXiv preprint arXiv:21 2021.

---

Published as a conference paper at ICLR 2026
Song Han, Huizi Mao, and W. Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding, 2015.
Ramin Hasani, Mathias Lechner, Tsun- Suun- Wang, Makram Chahine, Alexander Amini, and
Daniela Rus. Liquid Structural State-Space Models. arXiv preprint arXiv:2209. 12951, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. arXiv preprint arXiv:1503.00 02531, 2015.
T. Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in Deep Learning: Pruning and growth for Efficient Inference and Training in Neural Networks, 2021.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 1(2)3, 2022.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and Training of Neural Networks for Efficient Srrteere-rttimtii--OOo Only Inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704 2713, 2018.
Rudolph Emil Kalman. A new Approach to Linear Filtering and Prediction Problems, 1960.
Angelos Katharopoulos, Apoory Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5156-5165. PMLR, 13-18 Jul 2020. html]
URL https::/pprceeciingng
mlr. http://.........sss/////aaaaaapa
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning Filters for Efficient Convnets. arXiv preprint arXiv: 1608.0870000 2016.
Chi-Heng Lin, Shangqian Gao, J. Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. MoDeGPT: Modular Decomposition for Large Language Model Compression, 2024.
Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. $4nd: Modeling Images and Videos Multidimensional Signals with State Spaces. Advances in neural information processing systems, 35:2846-2861, 2022.
Antonio Orvicto, Samuel L Smith, Albert Gu, Anushan Fernando. Caglar Gulcehre, Razvan Pas- canu, and Soham De. Resurrecting Recurrent Neural Networks for Long Sequences. Interna- tional Conference on Machine Learning, pp. 26670 16670-26698. 26698. PMLR, 2023.
Konstantin Rusch and Daniela Rus. Oscillatory State-Space Models. In International Conference on Learning Representations, 2025.
Hiroki Sakamoto and Kazuhiro Sato. Compression Method for Deep Diagonal State-Space Model Based on H2 Optimal Reduction, 2025.
Paul Schwerdtner, Jules Berman, and Benjamin Peherstorfer. Hankel Singular Value Regulariza- tion for Highly Compressible State Space Models, 2025. URL /ttpss/////ax
httaaaaxvvvooo.. abs/
251 0. 22951
Sebastian Siegel, Ming-Jay Yang, and John Paul Strachan. IMSSA: Deploying Modern State-Space Models on Memristive In-Memory Compute Hardware, 2024.
Sebastian Siegel, Ming-Jay Yang, Younes Bouhadjar, Maxime Fabre, Emre Neftei, and John Paul Strachan. QS4D: Quantization-Awan Training for Efficient Hardware Deployment of Structured State-Space Sequential Models, 2025.
Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified State Space Layers for Sequence Modeling. arXiv preprint arXiv:2208.00 04933, 2022.
12

---

Published as a conference paper at ICLR 2026
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long-Range Arena: A Benchmark for Efficient Transformers. arXiv preprint arXiv:2011.04000 2020.
Inar Timiryasov and Jean-Loup Tastet. Baby Llama: Knowledge Distillation from an Ensemble of Teachers Trained on a Small Dataset with no Performance Penalty, 2023, URL https:
//axii..rg/abb//2333 02019
Vaswani. Attention All You Need. Advances Neural Information Processing Systems, 2017.
Benjamin Walker, Lingyi Yang, Nicola Muca Cirone, Cristopher Salvi, and Terry Lyons. Structured Linear CDEs: Maximally Expressive and Parallel-in-TIe Sequence Models, May 2025. URL
Hermann Von Weyl. Das Asymptotische Verteilungsgesetz der Eigenwerte Linearer Partieller Dif- ferentialgleichungen (mit einer Anwendung auf die Thcoric der Hohlrausstrahlung). Math- ematische Annalen, 71:441-479, 1912.
URL https: /aapss://ap...///
CorpusID: 12007782412
Paul Wimmer, Jens Mehnert, and A. Condurache. Dimensionality Reduced Training by Pruning and Freezing Parts of Deep Neural Network: Survey, 2022.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. SmoothQuant: Accurate and Efficient Tost-nrain Quantization for Large Language Models, 2022.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated Linear Attention Transformers with hardware-Efficiees Training. arXiv preprint arXiv:2312.06635, 2023.
Songlin Yang. Jan Kautz, and Ali Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv preprint arXiv:2412.066 2024.
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing Linear Trans- formers with the Delta Rule over Sequence Length, 2025. URL htt.ttss///xaiii.r///aat.t
24C6.064
Annan Yu, Michael W. Mahoney, and N. Benjamin Erichson. Hope for robust parattccriittnntnnnoiinnn of long-memory state space models, 2024. URL
Dongging Zhang. Jiaolong Yang, Dongqiangzi Ye, and G. Hua. LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks, 2018.
Leo Zhao, Tristan Torchet, M. Payvand, Laura Kriener, and Filippo Moro. Quantizing Small-Scale State- State-Space Models for Edge AI, 2025.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A Survey on Model Compression for Large Language Models. Transactions of the Association for Computational Linguistics, 12: 1556 1577, 11 2024. ISSN 2307-387X. doi: 10. 10.11622 162/116/cll..... a a.7044 URL httpps://ooi. Ezc/10.1162/tac. 00704
13

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
A.1 SOLVING THE LYAPUNOV EQUATIONS FOR DIAGONAL SSMS
For SSMs with diagonal state transition matrix A diag X ..... Ag which covers lot of SSMs used today such as LRU (Orvieto et al. 2023) and S5 (Smith et al., 2022 Equation and Equation admit simple, entry-wise closed-form solution:
[B P. 1 A.
(CTC) Qú 1 A,1
vis sn.
(14)
In the non-diagonal case, can either solve the Lyapunov equations by vectorization the argument put forward by brviiie et a (2023) and realize that every state transition matrix A E R can be diagonalized over C up small perturbation.
A.2 HYPERPARAMETER SELECTION
The hyperparameters used can be found in Table 3 For all but one LRA tasks we use the same ones as reported by (Orvieto et al., 2023 Table 10). The exception is IMDB, which we observed to overfit massively with the given hyperparameters, We mitigate this issue by increasing dropout and reducing the total number of layers.
Table 3: Hyperparameters for LRU experiments. h refers to the dimension the hidden state, n the state space dimension, B the batch size and aLR to the learning rate factor applied the sequence mixer (Orvieto et al. 2023).
Task
Depth
h
Steps
B
aLR
Weight Decay
Dropout
sMNIST
8
256
200k
50
0.1
CIFARIO 10
6
512
384
180k
50
0.25
0.05
0.1
ListOps
128
256
80k
32
0.5
0.05
IMDB
256
192
50k
32
0.1
0.05
0.1
AAN Pathfinder
128
256
100k:
64
0.5
0.05
0.1
6
192
256
500k
64
0.25
0.05
0.0
On LRA tasks, the learning rate warmed up from 10-7 10 for 10% of the total steps, before it is cosine-decayed back to 10-7 For sMNIST, the learning rate is fixed at 4 10-1 for the entirety of training.
A.3 REDUCTION DETAILS
We find that LRU overfits IMDB, even after reducing the number of parameters by 6 and doubling the dropout rate compared to Orvieto et al. (2023). Training LRU on this dataset, we furthermore observe an initial training period in which the loss plateaus. Just IMDB,
thus wait for an initial 1/ steps before doing the balanced truncation. Instead of doing 4 reduction steps until the end of warmup, we also just do 2 until 3/: steps in order to avoid entering the overfitting regime.
14

---

Published as a conference paper at ICLR 2026
B ADDITIONAL NUMERICAL RESULTS
This appendix expands upon the empirical findings presented in the main paper by providing more complete set of numerical results and supporting analyses. We aim to complete the view of how COMPRESSM behaves, and substantiate the assumptions underlying tion framework. We first report extended performance curves and dimension statistics across all benchmarks, results for the pragmatic variant of the algorithm, followed by additional evaluations of the dynamical behavior of Hankel singular values throughout training. Together, these results offer a holistic understanding of the stability, robustness, and practical effectiveness of the proposed method.
in-training reduc-
B.1 PERFORMANCE.
In Figure we provide the state dimension vs test performance plots for all datasets. These follow the same conventions and experimental setup as those described in the main text for Subfigure (a). The baselines again correspond to models that are trained from scratch at the with the mean of the reduced dimensions reached the various tolerance levels. Time gain plots are not provided, but similar gains are observed across all experiments, with an in depth discussion and analysis on the subject provided in Section of this appendix.
Tod Aetsius
01:0
0150
Text
Tip-S Skadlas
0.00
0.00
e-Bochaod
0150
61
128
0.79
Moan SSM Order
Mean SSM Order
(a) sMNIST
(b) CIFARIO
11:00
0.53
Ted Accuznicy
0.57
Thet
Text
0.03
Bost our Toy-
Tope ueas Shadthr
Basstre
Rollnsl
i-Bellood
15s
256
Mean SSM Order
Mear SSM Order
(c) AAN
(d) IMDB
0.32
Ttst Acctzttyy
Test Accitroeg
0.
8.30
0.49
Top-3 meas Thaddirr
Bastre e-Bichaodd
0.98
256
6.22
500
Mean SSM Order
Meac SSM Order
(e) Pathfinder
(l) ListOps
Figure 4: Test performance final state dimension for all our experiments. Stars correspond to best performance, circles to the mean the top-3 runs. Grey shapes correspond to non-reduced models, and the shades of orange reduced models, with tolerance decreasing with redness.
15

---

Published as a conference paper at ICLR 2026
Table Final state dimension (mean + std) and MAX performance reduction for LRU under different tolerances
Dutaset
Metric
1.5 1.5-10
1 1.100
7-10-
5.
7 3 30-00---
20100-
dim COMPRESSM
37.4
92.6
126.0
160.8
213.6
381
CIFARIO
85.8
86.1
85.9
86.2
Baseline
78.7
82.2
810
81.5
85.0
86.2
86,9
State dim COMPRESSM
56.8 +
41.8
100,8
135.4 4.5.4.66..8
167.6 +
213.8 + 28.0
256.0 0.0
ListOps
53.0
49.7
49.7
49.5
49.2
Boseline
43.8
46.9
50.8
49.7
61.1
50.0
50.7
State dim
53.6 1: 1.9
514 4 1.4
111.0 2.0
136 2.9
170.0 a
200
13.7
256
AAN
COMPRESSM
87.6
88.0
88.1
Baseline
87.8
88.2
88.1
88.2
87.9
88.1
87.5
State dim COMPRESSM
95.0 2.3
119.6 d: 2.2
136.8 1.9
150.4 1.2
165.0 = 1.3
192.0 0.0
192
IMDB
82.9
84.1
84.6
84.7
Baseline
82.9
83.6
83.7
84.3
51.1
84.7
81.7
State dim
316 1.6:110 1.9
51.2 1.7
65.6 2.3
81.2 1.6
105.0 2.1
129 5.2
206
Pathfinder
COMPRESSM
98.1
98.0
98.0
98.1
98.1
98.1
98.0
98.1
98.1
98.2
98.4
3-10-2
2-10-2
1.-1--11 1-10
5-10-3
7
2-10-3
1-10
Dim (+ std)
12.7 + 3.0
27.6 1.8
46.8 3.2
76.3 7.5
148.1 + 9.8
191 .4 4.7
256
SMNIST
COMPRESSM
96.9
07.2
97.6
Baseline
93.3
96.2
96.1
96.6
97.5
87.6
97.3
B.2 PRAGMATIC APPROACH PERFORMANCE
We set out to simplify the tuning of hyperparameters linked to the COMPRESSM protocol. We avoid having to decide on the non-intuitive energy threshold and the number of reductions to be scheduled priori. Instead, one selects: (i) the fraction of states to truncate at each reduction step, and (ii) an acceptable performance drop tolerance. We fix these to reasonable values: truncating a constant 10% of the current state dimension per reduction, with tolerance level of 1% of the pre-reduction accuracy after training for an additional 1% of the total steps. Results for the SMNIST and CIFARIO benchmarks are provided in Table
Table 5: Pragmatic COMPRESSM results with fixed 10% reduction fraction and 1% performance tolerance. Final state dimension (mean + std across all runs) and Top-3 mean test accuracy reported, consistent with Table
Dataset
Metric
Pragmatic
COMPRESSM
Baseline
State dim
121.3 + 89.8
191.4 4.7
256
sMNIST
Top-3 Acc. (%)
96.9 + 0.2
97.2 + 0.3
97.3 = 0.1
State dim
307.0 + 43.9
327.2 16.0
384
CIFAR10
Top-3 Acc. (%)
86.2 + 0.3
86.1 + 0.2
86.5 + 0.3
The pragmatic variant achieves competitive performance: 96.9% on sMNIST (vs. 97.2% for lolerance-based COMPRESSM However, the key trade-off lies in the predictability of compression. While tolerance-based COM- PRESSM yields consistent final dimensions (e.g.. 191.4 4.7 on sMNIST), the pragmatic approach exhibits high variance (121.3 + 89.8), with final dimensions ranging from 15 to 230 across seeds. This variability arises because the acceptance criterion depends on the validation metric trajectory. which is sensitive to initialization and optimization dynamics.
10-3) and 86.2% on CIFARIO (vs. 86.1% at 2- 10
Notably, some seeds achieve extreme compression (e.g., 15 states with 96.8% accuracy on sMNIST), while others reject reductions early and remain close to the full model size. This behavior suggests that the pragmatic approach effectively adapts to the "intrinsic compressibility" of each training run, though at the cost of less predictable outcomes. For applications where consistent model sizes required, the tolerance-bart based approach remains preferable; for exploratory compression where max- imal reduction is desired, the pragmatic variant offers simpler, hyperparameter-light alternative.
16

---

Published as a conference paper at ICLR 2026
B.3 EMPIRICAL IN-TRAINING HANKEL STABILITY
A central assumption underlying our reduction procedure is that the Hankel singular values (HSVs) of learned SSM evolve smoothly during training and, after initial transient, maintain stable relative ordering. This stability is what enables us to reliably identify and truncate low-energy modes without waiting for the model fully converge. In this section, we provide additional empirical evidence supporting this assumption across a wide range of datasets, model sizes, and architectural configurations.
Across all experiments-spaaneee SMNIST, IMDB, CIFARIO, and ListOps- wee we consistently ob- serve the same qualitative behavior. Immediately after initialization, the HSV spectrum typically undergoes brief reshaping phase in which the dominant modes separate from the rest. After this point (usually within the first few thousand training steps), the spectrum stabilizes and the relative ordering of HSVs becomes highly consistent. Larger modes drift slowly but retain their rank order, while smaller modes flatten and remain several orders of magnitude below the truncation threshold. This separation persists throughout training, even for deeper models and larger state dimensions.
These observations directly support the assumptions used in our analysis in the main text. First, they validate the claim that the HSV spectrum is bell-behaved and exhibits only mild temporal vari- ability once early training transients dissipate. Second, they confirm that the lower-energy portion of the spectrum remains largely inactive and can be removed with negligible impact on validation performance. Finally, they highlight that the qualitative structure of the HSVs is robust across tasks and model scales, lending practical reliability to in-training reductions.
Figures 21 22 23 and illustrate these trends for representative runs across datasets. In each plot, visualize the evolution of HSVs over the course of training, sampled at intervals on the order of thousands of steps. Because these intervals are much larger than the per-step analysis used in the main text, the exact perturbation continuity bounds become too noisy to compute meaningfully; nevertheless, we apply indices over time. In all cases, the dominant HSVs quickly settle into stable trajectories, while smaller HSVs decay toward near-zero values and form clear truncation region. This structure justifies the design of our reduction heuristics and further demonstrates that the behavior exploited by COMPRESSM is intrinsic to training dynamics rather than specific to single dataset architecture.
tracking to produce consistent alignment of HSV
17

---

Published as a conference paper at ICLR 2026
C COMPRESSM ABLATIONS
In this section present and discuss three ablations that justify the design choices adopted in COMPRESSM: first we illustrate the importance of the control theoretic approach of selecting the smallest HSVs for reduction the actual capacity of recovery of models is limited, second, look at the effect of increasing the number of reduction steps between the same initial and final state dimension, and last but not least, evaluate the effect of performing reductions at different phases of training.
All ablations in this section performed the SMNIST dataset, with single block LRU. The initial dimension considered 256 and the final reduced model dimension hardcoded to reach 32. references consist, like in the main text, in training the model without
The baselines that reduction with state dimensions 256 and 32. Similarly, results are obtained by averaging over ten random seeds.
C.I BALANCED TRUNCATION SANITY CHECK
The experimental setup consists running COMPRESSM training with equally spaced reductions (at the 5, 10, 15, 20k steps, total training for 200k steps). Each reduction keeps the same fraction = 0.395 the model, such that 256 x p² = 32. The ablation consists only in changing the selection scheme of the HSVs during reduction. We examine three variants: the correct balanced truncation approach removing the bottom HSVs, random selection of HSVs to remove, and an adversarial example removing the top HSVs each reduction step.
Hankel Selection Strategy Comparison
0.96
Validation Accuracy
0.90
0.55
0.80
Top
0.75
25000
50000
75000
100000
125000
150030
173000
2000000
Training Steps
Figure 5: Validation accuracy evolution during sMNIST training with three HSV selection schemes. Each setup averaged over ten seeds and the shaded regions around curves represent the standard deviation. The gray curves represent the non-reduced baselines with state dimension fixed to 32 and 256 (lighter and darker respectively). The yellow curve shows the correct reduction removing the bottom HSVs, the orange curve removes HSVs uniformly randomly, and the red curve removes the top HSVs. Dotted vertical lines show the steps at which reductions are performed.
Figure 5 shows the average model accuracy on the validation dataset during training, and Figure 6 depicts the HSVs evolution for a single seed across the three reduction schemes. Removing the top HSVs clearly incurs catastrophic performance damage. that is not recoverable even with 90% of subsequent training. These drops performance are present yet pronounced when removing random HSVs. In this case, the original gains gathered from training at larger state dimension are not completely lost, as the network is somewhat able to recover to perform slightly better than the baseline trained with dimension 32 from the start. In the case of proper bottom HSV truncation, there are no notable drops in performance in comparison to the 256 baseline, apart from a small departure the final reduction step. The performance observations evolution each case. Indeed, we observe that the correct balanced truncation maintains a majority concentration of large HSVs throughout training, all within single order of magnitude, maximizing
also backed up by the HSVs
18

---

Published as a conference paper at ICLR 2026
the per dimension contribution. Random truncation leads to sparser spread of HSVs with many small surviving HSVs trailing an order of magnitude or two behind the leading dimensions. Expres- sivity is most diluted in the adversarial case, where removing the top values leads to dynamics with only a small proportion of HSVs carrying training while the largest bulk remains several orders of magnitude behind.
Bottom HSVs
Ransiorn HSVs
Top HSVs
10
10
16'
Hackel Stagukk Valus
101
10
10'
107
10
107
10-11
10
10.
10-2
10-2
10-2
10000
20.000
20000
40000
50000
300000
20.000
200000
40000
200000
30000
20:000
30000
40000
50000
Training, Stege
Tralning Stepe
Truining Stepe
Figure 6: HSV evolution of a single seed for the three truncation schemes, From left to right we have the proper bottom HSV truncation, uniform random selection, and finally the adversarial top HSV truncation. Shades red to yellow track the probable HSV trajectories between reductions. Dotted vertical lines show the steps at which reductions performed. Note that all three plots share the same y axis limits for better HSV spread visualization. We show only the first 50k steps for better visualization of the HSV dynamics the values evolve little for the subsequent portion of training.
The first conclusion from this ablation is that the COMPRESSM approach is sane, not only from static dynamical system's perspective which an established theoretic result, but crucially in the context of SSM training. Indeed, this shows that correct balanced truncation is indispensable as models are not able to recover from improper reductions. The argument for HSV continuity during training made in Section 3.3 provides the thcoretical backing for this observation, as establish that prominent as well as weaker HSVs maintain their relative ordering to a large extent. Hence, the argument holds both ways, small HSVs be reduced without hindering long term training performance, but also, large HSVs cannot be removed without incurring performance losses, even with sustained post-reduction training.
C.2 NUMBER OF REDUCTIONS
The experimental setup consists in running COMPRESSM training with R equally spaced reduc- tions occuring during the first 10% of training (total training for 200k steps), where we consider R E {1, 4, 8, 16). Each individual reduction keeps the same fraction pr of the model, such that 256 x p = 32. The ablation studies the effect of smoother or more abrupt reductions on HSV evolution and performance.
Figure shows the average model accuracy on the validation dataset during training, and Figure depicts the HSVs evolution for single seed across only the first four variants for clarity of presenta- tion (1, 2, 4, and 8 reductions). The effect of increasing the number of reduction to go from an initial large state to the same final dimension seems provide some performance improvement although marginal. The yellow curve for the most incremental reductions is the only one not to suffer a small drop in performance reductions do suffer non-neglig drop yet are able to recover. Overall, the evidence suggests there might be slight performance advantage in phasing out the reductions instead of clumping larger reductions into fewer occurrences. The compromise is to be found between performance gain and compute time. This formalized and quantified in Section D The evolution of HSVs in each case corroborates the observation of minimal effect of R on the overall training procedure. Indeed, it is hard to discern any differences between the post-reduction HSVs in all cases considered, as all values seem to share similar distributions and populate the same order of magnitude.
the shared final reduction step of 20k steps. Intriguingly, the runs with
This experiment sheds some light on the dependency of training dynamics on incremental reduc- tions. Again, the HSV continuity analysis of Section 3 in the main text allows to understand that whether we remove all the bottom HSVs at once or phase them out more progressively, the global ordering and evolution of HSVs is little affected. The subsequent post-training HSVs appear
19

---

Published as a conference paper at ICLR 2026
Number of Reductions Comparison
0.97
Baselize
Baseline 206
0.96
Reduction
Reductions
Validation Accurace
0.96
Reductions
Redluctions
0.94
16 Reduuctions
0.93
My
0.92
0.91
0.90
0
25000 50000
750000
100000
125000
150030 173000
2000000
Training Steps
Figure 7: Validation accuracy evolution during sMNIST training with five increasing number of reductions between 256 and (shades from red for R to yellow for R = 16). Each setup is averaged over ten seeds and the shaded regions around curves represent the standard deviation. The gray curves represent the non-reduced baselines with state dimension fixed to 32 and 256 (lighter and darker respectively).
Reduction
2 Reductions
Reductions
8 Reductions
Hanke) Singular Valuos
10t
10"
101
10"1
20000
13
30000
10
40000
20000
40000
Training Steps
Training Steps
Training Steps
Training Steps
Figure& 8: HSV evolution of a single seed for 1, 2, 4, and 8 reduction steps (from left to right). Shades of red to yellow track the probable HSV trajectories. Dotted vertical lines show the steps at which reductions are performed. Note that all four plots share the same axis limits for better HSV spread visualization. We show only the first 50k steps for better visualization of the HSV dynamics as the values evolve little for the subsequent portion of training. We omit the 16 reductions for readability.
to behave very similarly. Incremental reductions might offer small performance benefits other layers of the network as well as the optimizer state are less perturbed. Based on the model architec- ture, overhead costs of reduction, and performance optimality constraints, a suitable balance can be found for the frequency of reductions.
C.3 REDUCTION WINDOW
The experimental setup consists in running COMPRESSM training with 4 equally spaced reductions at intervals of 5k steps, yet sliding the window during which the reductions are performed. Each reduction keeps the same fraction p = 0.595 of the model, such that 256 x pa = 32. We examine 5 windows: the scheme used in the main text applying reduction in the 0-20k window, then 25-45k, 50-70k, (00-120k and 170- step windows.
Figure 9 shows the average model accuracy on the validation dataset during training, and Figure 10 depicts the HSVs evolution for a single seed across only the first four variants for clarity of presenta- tion (0-20k, 25-45k. 50-70k, 100-150k steps). Here also, there appears to be conclusive evidence that early reductions lead to a larger drop in performance than later ones. Indeed, although the red curve (corresponding reduction in the ()-0kk window) appears to be slightly noisier than others,
20

---

Published as a conference paper at ICLR 2026
Reduction Timing Comparison
0.97
0.96
Validation Accuracy
0.95
0.94
0.93
Baseline
Beeline
0.92
M My
Wizdoww 0-200.
Wizddow 25k-45k
Wizadow Gilk-
0.91
Window 100k- 1akk
Window 160k-170
0.90
0
250000 50000
750000
100000
125000
150030
173000 200000
Training Steps
Figure 9: Validation accuracy evolution during sMNIST training with five increasingly late reduction windows (shades from red for 0-20k steps to yellow for 150-1 steps). Each setup is averaged over ten seeds and the shaded regions around curves represent the standard deviation. The vertical shaded regions delimit each reduction window with the corresponding color. The gray curves represent the non-reduced baselines with state dimension fixed to 32 and 256 (lighter and darker respectively).
pecially toward the end of training, its best performance validation accuracy does not underperform among the other reduced models. In fact, all reduced models reach practically the same maximum in the 170-200k end of training regime. This once more confirms that HSVs appear behave in continuous and relative rank preserving fashion, and largely justifies early reductions which lead to considerable training time gains as discussed and quantified in Section [ The noisiness observed in the performance curve for the earliest reduction is somewhat reproduced in the HSVs evolution plots, with handful of the smallest values proving to be relatively erratic. However, the larger pic- ture stays consistent with the observation that values maintain similar spread and lie in the same order magnitude regardless of reduction timing.
Window 0-20k
Window 25k-
Window 50k-70K
Window 100k-1200
Hankel Singalar Valuas
102
10a
10'
10
10
10'
10'
107
100
11P
10-1
10ml
10"
10-1
10"
50000
100200
150000 200000
10
50000 100000
I50000 200000
10
50000
10000C
150000 202000
50000 200000 150000 202000 Troining Stege
Triining Steps
Troining Steps
Training Stepe:
Figure 10: HSV evolution of single seed for reductions within the 0-20k, 25-45k, 50-70k, 100- 150k step windows (from left right). Shades of red yellow track the probable HSV trajectories. Dotted vertical lines show the steps at which reductions are performed. Note that all four plots share the same y axis limits for better HSV spread visualization. We omit the 150-170k steps window for readability.
Taken together, these results indicate that the precise timing of reductions has minimal impact on the final performance of the model. Even when reductions are performed very early-well before the model has partially converged-the HSV spectrum rapidly re-stabilizes, and the final validation accuracy matches that of runs where reductions occur much later. This aligns with our discussion in Section [3.3] where we showed that HSVs tend to evolve smoothly and preserve their relative ordering across training. Since delaying reductions does not offer any measurable performance benefit, applying them early remains the more effective choice in practice, as it preserves accuracy while providing substantially greater training-time savings.
21

---

Published as a conference paper at ICLR 2026
C.4 PRAGMATIC COMPRESSM.
Furthermore, we observe that reasonable balanced truncation practically does not hinder the upper bound performance, until it does. This observation can be used slightly tweak the COMPRESSM algorithm. Although one does not have access to the upper bound trained with the large dimension all along, we can keep track of the validation metric during training and assume that while there are no drops in performance, reduction will incur no noticeable global performance penalties. In practice, the procedure implemented with a simple safeguard. At each fixed fraction reduction step, we first save the current (pre-reduction) checkpoint. We then apply the reduction, train the model for a small number steps, and evaluate it on the validation set. validation performance continues to improve, we proceed to the next reduction. If performance degrades, we discard the reduced version and revert to the previously saved checkpoint, after which no further reductions are applied. This protocol ensures that model quality always remains close to that of the unreduced baseline without the need to explicitly predefine the number of reductions and a tolerance level in the absolute.
22

---

Published as a conference paper at ICLR 2026
D TRAINING COMPLEXITY, SCALABILITY AND TIMING ANALYSIS
This section provides a detailed analysis of the computational costs involved in COMPRESSM, in- cluding both empirical timing results and a formal model of the wall-clock speedup achieved through in-training reductions. Beyond timing, we also include a study of the computational complexity of the key operations involved in the reduction pipeline and discuss pragmatic implementation con- siderations, such as GPU/CPU placement and library-level optimizations. Together, these analyses give complete picture of where the dominant costs arise, which components scale favorably with reductions, and how to optimize end-to-end performance in practical implementations.
D.1 IN-TRAINING SPEEDUP ANALYSIS
We train a single-block LRU model on the sMNIST dataset with an initial state dimension of n 256 and a batch size of 50, such that at every 5000 gradient steps we artificially reduce the state dimension by removing 8 states. We record the wall-clock time of various operations during training on single NVIDIA RTX 6000 GPU, broken down as follows:
1. Gradient Steps: Mean time spent each gradient update during training.
2. Inference: Time spent on evaluation, divided into pure inference time and dataloading overhead,
3. Model Reduction Pipeline: Time spent Hankel singular value (HSV) computation, model balanced truncation with diagonalization and replacement, and optimizer reinitial- ization.
Figure 11 summarizes the evolution of these components as the model is progressively reduced.
In plot (a), the average gradient-step duration decreases sharply with the state dimension, dropping by nearly a factor of three between 256 and 8. This confirms that the computational cost of training scales strongly with the state size, and that the benefits of dimensionality reduction manifest immediately within the training loop.
Plot (b) separates evaluation time into dataloading and inference components. Dataloading remains effectively constant across dimensions, as expected, while inference time exhibits a moderate re- duction but at slower rate than the gradient updates. This highlights that the primary performance gains of model reduction occur in-training, rather than in post-training inference.
Plot (c) details the components of the model-reduction pipeline: computation of Hankel singular values, generation of the reduced model through balanced truncation and diagonalization, and op- timizer reinitialization. We observe that HSV computation dominates the total reduction time but decreases only slightly as the models shrink. Because dense matrix operations become unstable the GPU for dimensions above roughly 200, HSV computations are performed the CPU, introducing a constant overhead for transferring layers between devices. Consequently, the total reduction time does not scale down proportionally with the state dimension. Subsequent operations, including truncation, diagonalization, and parameter replacement, decrease substantially with model size, while optimizer reinitialization adds a small, constant overhead. It is important note that fixed JIT recompilation overhead of approximately 5 seconds is incurred at every reduction step, as new model shapes trigger re-compilation of the training graph- an overhead that dominates all other reduction costs and thus motivates minimizing the number of reductions performed.
Finally, plot (d) compares the relative time budgets for 5000 training steps across reduction stages. The breakdown clearly shows that the dominant gains arise within the gradient-step portion, which accounts for most of the overall runtime improvemen--aproximmeee
x2 at n 128 and x3 at = 8. In contrast, dataloading and pure inference times show only marginal improvement, while the reduction overhead remains nearly constant, dominated by CPU-bound operations. These results illustrate where computational savings primarily achieved and where residual overheads remain- demonstrating that in- tn-training reductions yield the most substantial efficiency gains by directly accelerating the optimization loop.
23

---

Published as a conference paper at ICLR 2026
(a) Batch Gradient Step
(b) Batch Evaluation
(c) Reduction Computation
x10
x 110
x10
Inference
3.0
2.0
Datalonding
2.5
1.8
Time
2.0
1.6
1.5
Model
0.5
Optimisot
1.0
256 224
192
160
128
96
64
32
256
224
192
96 State Dim
100
128
64
32
State Dim
State Dim
(d) Time Budget vs Model Size
Reduuciion Pipuline
Evaliaation
Grucikes
Tine (8]
15
256
224
192
160
128
96 64 32
State Dim
Figure 11: Timing analysis in-training reductions. (a) Batch srach-gradient time decreases sharply with state dimension. (b) Evaluation time decomposed into dataloading (constant) and inference (moderately improved). (c) Breakdown of reduction-related computation (HSV, model trunca- tion/diagonalization,, optimizer reset). (d) Relative time budgets over 5000 steps, showing that most gains arise from faster gradient updates, while evaluation and reduction overheads remain compara- tively small.
D.1.1 QUANTIFYING IN-TRAINING COMPRESSION SPEEDUP
We now formalize the wall-clock speedup obtained when performing in-training model reductions. Let denote the number of gradient steps in an epoch and E' denote the total number of training epochs. Suppose we perform R reductions at the ends of selected epochs. Let no be the initial state dimension and denote by nę the state dimension while training in the k-th reduction phase (so n the final dimension). Let Eg be the number of epochs run while the model has state dimension ni. with ER
Lk=0 Ex- F. Define the following per-dimension timing functions:
ttrain (n) time per gradient step at state dimension n,
(n) time per epoch for evaluation state dimension 72,
fanalysis (n) time to run the reduction analysis (HSV computation, trunca- tion/diagonallzzatin,, parameter replacement),
tjat fixed JIT recompilation overhead per reduction.
Total wall-clock time without reductions:
Thas: = E(s lerain (no) + teval(noo)..
Total wall-clock time with reductions:
R
R (aaalyyis (n:-1) + tta).
Tred
Σ
Ex (s terain (nw) + (nx))(
+
24

---

Published as a conference paper at ICLR 2026
Speedup:
Thase
Ted
Reduction speedup prediction example We instantiate the above formulas for the setup used in the experiments:
5000, =0
R ==4 4.
Reductions oceur at the ends of the first four epochs:
Fg = 1, E = 1, E2 = 1, Fg = 1, F4 = 336
State dimensions shrink from 256 to 96 via four reductions:
7gg = 256 216, 176, 136, 96].
Measured timings (from Fig.
7
256
216
176
136
96
Itrain (72)
3.4 10-3
2.4x 10-3
1.9 x 10
1.6x 10-3
1.4x 10
teval (TL) (s)
3.6 10-3
3.4x 10-3
3.3x10 0.64
3.2x 10-3
3.1x10-3
Uanalysis (1) (S)
0.80
0.68
0.60
0.55
We time the JIT recompilation overhead cost as
ljit ~ 5.0
Baseline (no reductions).
Tasse = 40 (5000- 3.4 *+ 3.6 10 680.1 s.
With reductions. Trainingtevaluationn only subtotal:
Trai(tr-en 298.6
Reduction overheads:
Ted(ov) (0.80 0.68 + 0.64 + 0.60) 4- 5.0 22.72
Total:
Tist 321.3 s.
Speedup:
680.1
2 2.1.
321.3
Interpretation. Under the empirically measured per-dimension timings from the experiments, per- forming four aggressive early reductions that bring the state dimension from 256 down 96 yields an overall wall-clock speedup of approximately 2.1 x for the full 40-epoch run. The calculation highlights the tension between per-step savings (which rapidly accumulate when many subsequent steps are executed at small n) and fixed reduction costs (CPU-bound analysis and JIT recompilation), illustrating how early reductions provide the most substantial gains while still incurring predictable overheads.
D.2 REDUCTION PIPELINE COMPLEXITY
In this subsection we study the complexity one application of the reduction pipeline (see Sec- tion 3.1).
Step 2: For diagonal transition matrix A, as the case for LRU, Equation says that
(BBT) ){
(CTC)s) Qu 1-
Ps Aod,
visisj
(15)
For B Ruxp computing BB takes O(n²p) FLOPS. Similarly. computing C C for C
Raxn takes O(n2g) qq) FLOPS. Thus, computing the Gramians P and Q requires Onn²p + 3n?) and
Onn2g 3n²) FLOPS, respectively.
25

---

Published as a conference paper at ICLR 2026
Step 3:
Computing the HSV form the spectrum of PQ requires computing the SVD of the product of the gramians, which requires Onz") FLOPS.
finding the smallest rank that maintains at least (1 - -) of the total energy requires com- puting the cumulative sum of the HSV, which is ((n) FLOPS.
Step 4:
Step 6: Transforming the system
(A,, BB. C.) = AT
(16)
to its balanced truncation costs On
Thus, the total cost of balanced truncation is O( + n + n n²p). Importantly, for LTI systems, the cost of balanced truncation is independent of the length of the input sequence.
To empirically validate these theoretical complexity bounds, we benchmark the key operations in the reduction pipeline for state dimensions ranging from 1024 to 1. Figure 12 shows the timing results for each dimension. The Gramian computation (Step 2) exhibits clear O(n²)) scaling, while the HSV computation via SVD (Step 3) and the balanced transformation (Step both display O(n²) behavior, consistent with our analysis. The full pipeline is dominated by the cubic operations, as expected from the total complexity of O( n²qq).
Importantly, these benchmarks confirm that the reduction overhead remains modest even for large state dimensions. For example, at seconds, while at = 512, it requires approximately 3 seconds. This overhead is negligible com- pared the cumulative training time saved through in-training reduction, as demonstrated above in Section D.I where models achieve significant speedups despite periodic reduction steps.
128, the full reduction pipeline completes in under 0.1
Reduction Pipeline Complexity
10
Pipciuu PQ (Stp 3)
10
Balanced Trundom (Sta 0)
(Step 2)
10-1
(ic2)
O(ec)
3
100
Tine
10-1
30
30
2
State Dimension (n)
Figure 12: Empirical scaling of reduction pipeline operations. The Gramian computation (Step 2) in red, the balanced transformation (Step 6) in dark orange, the HSV computation via SVD of PQ (Step 3) in bright orange and the full reduction pipeline in yellow. Gray reference lines show theoretical (n²) and O(n²) sealing for asymptotic behavior comparison.
26

---

Published as a conference paper at ICLR 2026
E SELECTIVE SSMS, SISO SYSTEMS AND MAMBA
In this section, we examine how COMPRESSM can be adapted to the broader landscape of selective state-space models, where the system dynamics depend explicitly on the input. We first outline practical strategies for handling such LTV/LPV architectures, emphasizing the computational chal- lenges that arise when the system matrices vary with the signal. We then clarify the key distinction between per-channel SISO formulations and fully coupled MIMO state-space models, as this struc- tural choice fundamentally shapes how state dimension influences model capacity and, consequently, how effective any reduction method can be. Finally, we present an in-depth case study on Mamba, demonstrating both the viability of our reduction framework in this selective, SISO setting and the specific limitations that mmaerg--highligttinn where COMPRESSM succeeds, where the underlying architecture constrains performance gains, and how these insights point toward future extensions to richer MIMO-based designs.
E.1
SELECTIVE SSMS HANDLING
Let be a discrete Linear Time- Varying (LTV) system described by state equations:
h(k + A h(k) + BN x(k), h(0) = ho
(17)
y(k C2 h(k) D2 (rk),
where the state matrices also depend on time.
Selective SSMs are built with dynamical systems that fall under special case of LTV systems, the linear parameter varying (LPV) framework, with the parameter being the layer input. Such systems are also referred to as Linear Input Varying (LIV) and their general case state equations are given by.
hnlk + 1) = ((r(k)) (k) + (xx())) (kk) y(k) ((=(k)) h (k) + ((xr(k) (k),
h()) = ho
(18)
(19)
For these systems, the controllability and observability Gramians are no longer stationary. In par- ticular, for each possible input æ E X, one would in principle need to solve a set of Lyapunov inequalities of the form
A A(x) P(xA - P(x) + B(x) B (a) S 0, A (z) (z) (=x) C (x) S 0,
(20)
(21)
so that the Gramians P and Q are input- depttdeeennent.
In practice, solving for fully input-dependent Gramians and applying the subsequent per input reduc- tion is clearly neither computationally tractable nor practical. A common simplification is to seek input-invariant Gramians P, Q that satisfy the inequalities for all æ E X:; this reduces the problem to an LTT-like Lyapunov condition over all inputs, which can still be expensive for high-dimensional X, when such a solution even exists. In practice this still too constraining.
A cheaper alternative is simply averaging the dynamics over the input space:
À
1 Σ A(x), X
B B(z), C= X
C(x)) #EX
(22)
The caveat is that the mean system may not be stable, controllable, observable; one may therefore need to regularize the mean matrices to satisfy these assumptions before applying a single global reduction based on the LTI approach. We study the case of Mamba (Gu & Dao 2024) and provide practical implementation as well as results in Section E.3 (with the caveat discussed in the following section E.2
E.2 STATES IN MIMO vs SISO SYSTEMS
Sequence models based space layers differ in how their hidden state is used map inputs to outputs. Indeed, some models employ a multi-input multi-oupput (MIMO) approach while others use a per channel single-input single-output (SISO) state-space structure.
27

---

Published as a conference paper at ICLR 2026
SISO per channel LTV systems:
Input sequence:
Output sequence:
h h A h B 2i
Vi.L
yk C h D
{
(htt1 A h B
11.E
3i.L
vk C h1 D a
hi+1 = A h B o
VK C h D xx
Figure 13: SISO state-space system applied independently to each input channel, with separate latent states per channel.
Input sequence:
Output sequence:
Single d-to-d MIMO LTV system:
Vi.L
hk+1 AA h4 + B₁ Z
FiL
27
yk = C2 h4 + D2 œb
Vı.r.
ViL
Figure 14: MIMO state-space system where single latent state evolves jointly for all input channels and produces all outputs.
In MIMO linear dynamical system, state vector of dimension n governs the evolution of an entire d-dimensional feature vector. The state therefore directly mediates interactions among all channels, and expressive capacity contributes jointly to all output degrees freedom. State size thus acts as meaningful global capacity measure: increasing n expands the shared dynamical subspace available to all features.
By contrast, many popular state-space architectures-incuuuiie several LTT models (e.g.. S4 (Gu et al. (02T). $5 (Smith al. 2022) and LTV models (e.g., Mamba (Gu & Dao 2024 Liquid-S4 (Hasani et al. 2022) variants) adopt SISO formulation. In these designs, each feature channel processed by independent ID state-space system, the latent state predicts the evolution of a single input-output mapping rather than the entire feature vector. We provide visualizations of these differences in approach in figures [ and [4 While per-channel SISO formulations scale efficiently due to parallclization, their per-channel factorization dilutes the expressive advantage normally associated with larger state dimensions. We observe that for LRA experiments, the state dimension hyperparameter does not seem to be main performance driver (more on this in Section E.3 Future work will extend implementation to experiments where such models do benefit from larger states (language tasks are perhaps more suited).
Not all LTV architectures limited SISO formulations. Several state-of-the-an designs, such as Griffin (De et al., al., 2024), employ MIMO state transitions, while others building on linear attention models (Katharopoulos al. 2020 like DeltaNet (Yang al. 2025) matrix-valued
28

---

Published as a conference paper at ICLR 2026
Shared preprocessing at timestep k:
base (2r)) proj (2x)
(xxx) proj B (2x)
((xx) proj (1%)
For each channel at timestep k:
Discretization:
Input sequence:
Output sequence:
A'(xx) W A base (Itx)
ViLz
1:L
A' A'(xx) exp( A (axs A*)
TiLL
B B'xxx) A (xxx)) ((xx)
Biz
= yi
SISO recurrence:
ViLL
h&+1 (xxx h1 + ('xx)
1/4 C(xu) h D ot
Figure 15: Vanilla S6 computation flow
dynamical updates. We consider extending reduction framework such systems exciting direction for future work.
E.3 MAMBA CASE STUDY
In this section, we take deep dive into applying COMPRESSM to Mamba. Our goals are threefold: (i) to demonstrate the feasibility of selective system handling, (ii) to show that the framework's ideas extend cleanly to more intricate initializations such as Mamba's S6 layer, and (iii) to better understand the limitations and structural constraints that arise.
E.3.1 MAMBA INITIALIZATION AND COMPUTATIONAL GRAPII
Reasoning about balanced truncation for Mamba is nontrivial because the per-channel SISO discrete LTV systems are coupled through shared projections and the input- dependent discretization. We begin by breaking down the computations inside the selective recurrent layer (S6), as illustrated by the diagram in Figure 15
Initialization and shared structure. Mamba instantiates dinne single-input single-output (STSO) channels that all depend on single sequence of projected features. At timestep k, the shared preprocessing block produces matrices A (xx), B(xx), and C(xx) that are reused by every channel. Each channel owns diagonal continuous-time drift matrix A' and scales the shared discretization through a learned vector W yiclding (((x)) W Buse (xx). The lifted discrete parameters are
(mx))) exp( A (m)))
B'(xx) (xx B(x)),
(*( C(xx).
The recurrence evolves according
h = ((xx)) ) h + BB'(x)) y (xx))) h + D'
Although channels are nominally independent, they all consume the same B(xx) and C(æy), which couple their effective dynamics through the shared projections. This coupling is central to the diffi- culty of applying balanced truncation.
29

---

Published as a conference paper at ICLR 2026
Why balanced truncation is challenging.
Classical balanced truncation assumes a time-invariant LTI system with fixed (A, B, C. Mamba violates these assumptions in several ways:
(1) A' ( B' (æx), and C* (*xxx) vary with inputs at every timestep, making the controllability and observability Gramians detendeepnn;nt;
(ii (xx) and C(æs) are shared across channels, so balancing each channel independently would break the shared projections shown in the first figure;
(iii) the selective-scan CUDA kernels rely hard-coded loop bounds tied to the full state di- mension, meaning that even if the desired reduced rank were known, the runtime cost would not change unless the kernels become rank-aware.
These factors make a naive application of balanced truncation both mathematically awkward and computationally inefficient.
E.3.2 PRACTICAL REDUCTION WORKFLOW
To enable COMPRESSM with Mamba, we opt for a pragmatic implementation that relies on the recipe described and illustrated in Figure
Mean LTI surrogates. During calibration, each channel i accumulates running averages over inputs Bi, C, and A These define stationary proxy
BA,
which approximates the expected discrete dynamics of channel i. This surrogate temporarily de- couples the channels, enabling per-channel Gramian computation without destroying the shared projection structure.
Per-channel balancing and rank selection. From each surrogate we form controllability and observability Gramians P and Q'. Their product yields the Hankel singular values of, which determine the retained dimensionality r². We compute the balancing transform T and its inverse (T+)-2 that simultaneously diagonalize the Gramians p and Q'
Applying (T À'T's and truncating to the first r rows/columns produces A Similarly,
B = B'
C = ČT
key implementation detail that we store both T and (T')-1, This allows to reapply the same rotation-am waw--anndddtttteee fresh on-the-fly projections B (xx) and C(æx) at runtime. These cached transforms anchor the reduced subspace consistently across all timesteps.
Runtime reuse of the stored transforms. At execution time, Mamba still produces ((xx) and C'æg)) at every timestep. For each channel i, we retrieve the cached T and (Ti)- and compute
B' (mxx (T') ((xx), ((ax),
(*rx) C(a.)T'
We then keep only the first r² coordinates before feeding the selective recurrence. The vector of ranks r (r" ........................ Ama)) is passed the selective-scan CUDA kernels, which terminate their inner loops at r² for channel i. This eliminates all work on pruned coordinates and makes the runtime cost proportional to reduced dimension.
Because the transforms are stored persistently and applied to both B(xx) and C(æx), every timestep automatically consistent with the balanced dynamies of A'. Meanwhile, the kernels never ex- pend computation beyond the retained modes. These mechanisms dedpittt in Figure coo-preseee Mamba's expressive shared projections while enabling efficient selective reductions.
Overall, the workflow turns the highly coupled Mamba initialization into collection of tractable mean systems, applies balanced truncation per channel, and enforces those ranks consistently throughout training and inference. Crucially, this is achieved without ever baking static reduced weights into the model: the reduction lives in the transforms and their runtime application, not in frozen parameter tensors.
30

---

Published as a conference paper at ICLR 2026
Shared preprocessing at timestep k:
base (2r) proj (2x)
( proj Ø (zt)
((xx) proj (1%)
For each channel at timestep k:
Discretization + balancing:
A((m) = W hase (2kg
A((xx) exp A (ax)) AA')
Input sequence:
Output sequence:
B'(xx) = (ax) (T*) (2xx) (xx))
Vi-L
('(x)) C(xz)) T"
Truncation:
01:L
I 1
ViL
yI-L
('xx)) = A (<xx))) (xx))r)).r
(xxx)) Bd (xx))
(xx) = (z))))
State-reduced SISO recurrence: h A'(xx) h + (xxx) x v (ax) hi + D' #
Invariant balancing and rank:
Rddu x daw x dayn
T E Rddee
A = T AT
Figure 16: COMPRESSM S6 computation flow
E.3.3 EXPERIMENTAL EVALUATION:
0081
Test Accurscy
0.00
0.50
0.58
Best cus
Ronelita
0.57
128
SSM Order
Figure 17: Performance of Mamba on CIFAR on five different random seeds. Grey markers corre- spond to non-reduced models. Stars correspond to best runs, circles the top-3 mean.
31

---

Published as a conference paper at ICLR 2026
We train Mamba IMDB to showcase COMPRESSM for LTV systems (see Section E.1 The results can be found in Figure 18
Figure 18a shows there is only weak correlation between performance and state dimension in this setup, which we believe is due to the single-input-sine -onti-ooou-snglooupuuuuuu (SISO) nature of Mamba. This observation is confirmed CIFAR, performance is stable for state dimensions ranging from 128 to 8. Nevertheless, larger state-size does seem to help stabilize training, as indicated in Figure [8b
Because of this weak correlation, only evaluate COMPRESSM on IMDB to confirm the in- training speedup also holds for LTV systems.
We test the performance of COMPRESSM in this setting by compressing with means the model retains at least 99.99% of its energy at every reduction step. The final model has an average state dimension of 12, starting from 128. This shows that the HSV spectrum is very tail-heavy, with most energy in just a handful of dimensions. This explains why random dropping of HSV (the red markers in Figure markers), though at a higher variance.
0.001. This
performs competitively with balanced truncation (the orange
Even though the correlation between state dimension and performance is not very strong, COM- PRESSM yields competitive performance on Mamba, at a lower variance, especially when account- ing for outliers. However, we believe the approach for LTV systems should be refined in later works to account for heavy-tailed distributions of HSV.
0.55
0.55
01:0
30p3 rava
1bss Accurmry
Bewdlin
Aceuracy
Randin Figy
0.41
0.83
6.80
0.52
0.7
Medkas
676-
Baclie
0.83
r-Hledooed
Handion TBV
Muan SSM Order
Mea SSM Order
(a) Top Performance
(b) Median Performance
0.53
Tost Aetmnay
8.00
0.32
BAL
8,00
Test
0.79
0.79
rillwaaned Bundies BSV
-iettoed Reaskon BBV
0.78
125
6.7
16
Moan SSM Order
Mewn SSM Omier
(c) Top Performance
(d) Median Performance
Figure 18: Performance of Mamba on IMDB on five random seeds. Grey markers correspond to non-reduced models, orange is a model reduced using т 0.001. Red corresponds to random dropping of HSV. Plots the left column show top-3 and top-1 performance (circle and star), plots in the right column mean performance. The first row considers all five random seeds, while the second row drops the best and the worst run.
Beyond maintaining competitive performance, our implementation achieves significant training time speedups for LTV systems. Figure demmonstates that a model initialized with state dimension 128 and progressively reduced to 14 using 0.001 trains in approximately the same time as model that starts and remains at dimension 16. This represents substantial speedup of ~ 4x compared impressive than those reached
to the full 12s-dimensional baseline. These speedups with LRU, though that can down to multiple factors as the codebases use different libraries (we I LRU in JAX and Mamba in PyTorch).
32

---

Published as a conference paper at ICLR 2026
Test Aceuacy
Topa Beadine
r-Redueed
Training Time Speodup
Figure 19: Training time speedup for Mamba IMDB. The orange marker shows the CoM- PRESSM-reduced model (r 0.001, starting from dimension 128 and reducing to ~ 14). The gray markers represent the baseline runs. Data point size is proportional to final state dimension.
33

---

Published as a conference paper at ICLR 2026
F COMPRESSM VERSUS OTHER COMPRESSION TECHNIQUES
In this section we pit COMPRESSM against other popular compression techniques. For SSMs, the paradigm of HSV nuclear norm regularization has established itself recently as an promising avenue. We discuss its shortcomings and make a case for the superiority of our framework both in terms of performance as well training time. Similar arguments are also made about the student-teacher distillation paradigm. We remind the reader who might have ventured thus far that the main appeal of COMPRESSM that it does not require training the full capacity model to completion before applying any reduction scheme, a point we hope to make crystal clear case for in the experiments below.
F.1 HANKEL NUCLEAR NORM REGULARIZATION
We investigate the differences in approach between COMPRESSM and spectral regularization tech- niques presented in the works of Forgione et al. (2024) which attempt both Hankel nuclear norm and modal regularization on the LRU model, and the contribution of Schhwertterr al. (2025) which performs Hankel nuclear norm regularization on MIMO SSMs parametrized via scaled rotation ma- trices.
We shall discuss Hankel norm regularization only & regularization, although extremely cheap to compute, does have any proper theoretical links to input-output mapping behavior apart from intuition on the frequency of dynamics discussed in |Forgione et al. (2024).
F.1.1 DIFFERENTIABLE HANKEL NUCLEAR NORM REGULARIZZATION
We regularize the LTI dynamics inside each SSM layer by penalizing the nuclear norm of the product of the controllability and observability Gramians, Po and Qo. This encourages the system have rapidly decaying Hankel singular values, promoting low effective state dimension.
For diagonal Kate-transition matrix As diag(Xe) the Gramians admit closed-form solutions. The discrete-time controllability and observability Lyapunov equations,
AoP Po
Bg BJ = 0,
(23)
A QoAs Qo
C/ Co 0,
(24)
have elementwise solutions
(Po) (Bo )m
(25)
20,20.3
(C Colis
(Qe)y = 1- -e
(26)
The Hankel singular values follow the same definition as the main text (Eq, 4):
( = sortt (pspec(PoQo)
(27)
and we denote the k-th one by a% (0).
The nuclear norm penalty is therefore
PaQoll = Σ (0).
(28)
Am
Differentiability.
All operations needed to compute (0) are differentiable:
1. The SSM parameters (Ag, Bo, Co) are differentiable by design.
2. The diagonal Lyapunov solutions involve only multiplication and division.
3. The eigenvalues of PoQ9 are differentiable almost everywhere.
4. The square root and summation operations preserve differentiability.
34

---

Published as a conference paper at ICLR 2026
Thus the regularized objective,
m1 > 1=
Creg (0) Lrack
(0),
(29)
fully compatible with automatic differentiation. Backpropagation through the entire computation,
θ + B, + (Po. Qo) + PoQo + o(0),
is supported natively in JAX.
Numerical Stability,
During training we apply the following stabilizations:
Add a small diagonal shift eI to Pe and Qa-
Clamp eigenvalues of PoQe to be non-negative before applying V-
Clip denominators )ı avoid division near zero.
F.1.2 PERFORMANCE AND COMPUTATIONAL COST
We use the sMNIST dataset with single LRU block trained with an initial state dimension of 256 as experimental setup. For ten different initial seeds, we train the model to completion (200k) with Hankel nuclear norm regularization as discussed in Section F.I.I with & = 1, = 0.1 in Equation 29 (following values used in Forgione et al. (2024)), then reduce to the values obtained for various tolerance levels with COMPRESSM presented in the main text in Table [ More precisely, we apply balanced truncation down to state dimension 191, 148. 76, 47, 28, and 13, and evaluate the test performance of the reduced models. provides the test performance and mean batch gradient step speed with respect to training at state dimension 256 without any regularization terms.
The results highlight three central limitations of relying on Hankel Nuclear Norm (HNN) regular- ization for producing compact state-space models:
1. HNN regularization is computationally prohibitive. Training must occur at the full state dimension with the regularizer applied at every step. Because evaluating the Hankel nuclear norm requires computing the eigenvalues of PoQo at each gradient update, training is roughly 16x slower than unconstrained optimization in our experiments.
2. HNN-regularized models exhibit constrained performance. The regularization forces rapidly decaying Hankel singular values, which reduces effective model capacity and leads to measurable accuracy degradation compared to unconstrained training-consiste with findings in Forgione et al. (2024). Even before any reduction, the HNN-trained model fails to reach baseline accuracy.
3. HNN is dramatically less efficient than COMPRESSM for obtaining low-dimensional models. For example, achieving a final dimension of 28 yields 96.9% accuracy with COM- PRESSM versus only 95.8% when training the full 256-dimensional model with HNN and reducing afterward. Moreover, COMPRESSM reaches this result with 46x effective speed advantage. since reductions are performed carly and training proceeds at much smaller state sizes.
Taken together, these points show that while HNN-regularized models are compressible, the combi- nation of high computational cost and constrained accuracy makes them impractical for achieving high- regh-performanc -iw-eninnsional SSMs. In contrast, COMPRESSM provides a more effective and scalable alternative, combining strong accuracy, efficient training, and substantial compression.
F.2 KNOWLEDGE DISTILLATION
We furthermore compare COMPRESSM to knowledge distillation Hinton et al. (2015) CIFARIO, where the teacher has state dimension of 384 and the students have the same state dimension as the final, in-training compressed models (see Table
35

---

Published as a conference paper at ICLR 2026
Table 6: Top-3 mean SMNIST test performance (accuracy %) and batch gradient step speedup for different final state dimensions (rounded mean for COMPRESSM) with ten different random seeds. Results compare COMPRESSM, Baseline, and Hankel Nuclear Norm (HNN) regularization.
Method
Metric
Final State Dimension
13
28
47
76
148
191
256
Accuracy (%) Speed
92.6
96.0
95.9
96.4
97.3
97.3
97.3
Baseline
3.1
2.8
2.7
2.4
2.0
1.7
1.0
Accuracy (%)
95.9
96.9
96.9
96.9
97.0
97.2
COMPRESSM
Speed
2.8
2.6
2.5
2.3
1.9
1.6
Accuracy (%)
91.7
95.8
95.8
95.8
95.8
95.8
95.9
HNN Regularization
Speed
0.06
0.06
0.06
0.06
0.06
0.06
0.06
The effective knowledge distillation loss for classification C is a superposition of the cross-entropy loss between the student and the targets and the Kullback-Leibler divergence between the teacher and the student with temperature scaling:
C= - -1-1a) HVV ((2)) + aT (xu.lo((a//))ooo/////
H denotes the standard cross-entropy loss, !/ represents the ground truth labels, 2, and 2; are the logits of the student and teacher respectively, o(-) is the softmax function, T is the temperature parameter, and a is balancing weight. We use the standard parameters
2 and a 0.5
Timiryasov & Tastet 2023).
Our experiments reveal that knowledge distillation performs better the closer the state dimension of the student is to the one of the teacher. Distilled models perform roughly par with COMPRESSM the state dimension of the student is similar to the one of the teacher. However, our in-training reduced models able maintain superior performance also for heavily reduced models, while knowledge distillation suffers from a clear drop-off.
Phase Teacher Training
Phase Student Training
Step k: Optimize
Step k: Fwd Pass Large Teacher
Step k: Optimize
Large Teacher
Small Student
Start (2 =
Wall Clock Time
End (fyswer))
Figure 20: Wall clock time for training student from a teacher via knowledge distillation. In the first phase, one trains large teacher. In the second phase, its knowledge is distilled to smaller student. Every forward step the second phase requires pass through both the student and the teacher.
We furthermore evaluate the total time takes to obtain the final, distilled model. Knowledge distillation not only requires first training a high-dimensional teacher, but also doing forward pass through the teacher while training the student to obtain the logits (see Figure 20). Consequently, even after the teacher has been trained completion, the training speedup small students models have compared to larger ones is mitigated.
36

---

Published as a conference paper at ICLR 2026
Table 7: Top-3 mean CIFARIO test performance (accuracy %) and batch gradient step speedup for different final state dimensions (rounded mean for COMPRESSM) with ten different random seeds. Results compare COMPRESSM, Baseline, and Knowledge Distillation.
Method
Metric
Final State Dimension
57
93
126
161
214
327
384
Accuracy (%) Speed
78.2
81.8
83.7
84.2
84.9
86.0
86.5
Baseline
1.69
1.62
1.53
1.43
1.22
1.03
1.0
Accuracy (%)
84.4
85.7
86.0
85.8
86.0
86.1
COMPRESSM
Speed x
1.58
1.52
1.41
1.33
1.17
1.03
Accuracy (%)
79.4
83.5
84.4
85.3
86.0
87.0
Knowledge Distillation
Speed x
0.55
0.52
0.61
0.51
0.49
0.45
37

---

Published as a conference paper at ICLR 2026
Hankel Singniar Values
Likely Teajectories
Energy Fraction Bottom r
10
6 (10%)
10P
0.8
13 (207)
(30%)
0
26 (10%)
(50%)
0.4
10P
10a
0.0
50000
100000
150000
100000
(50000
50000
100000
150000
Step
Step
Step
Figure 21: Single LRU block with state dimension of 64 on the s.NIST dataset.
Hankel Singular Values
Likely HSV Teaeectories
Energy Fraction Bottom
1.0
6 (1:9%
(20%)
(30%)
0.5
r (4066
Block
(5006)
0.00
20000
40000
20000
40000
40000
Step
Stepp
Step
Figure 22: Single LRU block with state dimension of 64 on the IMDB dataset.
38

---

Published as a conference paper at ICLR 2026
Singila Value
Liely Tesjectriin
Euenyy Fowtinn Buttom
(10%)
11 (206)
17 (306)
10
23 (410))
Biok
50000
1000000
04000
100301
100000
3000
100EC0
19000
(009)
11 (200)
Blooks
25 (506))
0.4
50001
193881
150000
14000
100301
198000
T10ECO
19880
(108)
0.5-
17
20 (106)
11.
Block
25 (3066)
30030
11 (206)
17 (306)
20 (400)
(306))
117
(10%)
11 (200)
17 (306)
(506))
S4/600
1000000
154000
100000
190000
100000
20000
100000
1,0000
(10%)
(206))
23 (403)
0.5
28 (50K)
0.4
56080
100000
154000
50000
100000
150000
50000
100000
15000
Skop
Stpp
Slapp
Figure 23: Six LRU blocks with state dimension of 57 on the CIFARIO dataset.
39

---

Published as a conference paper at ICLR 2026
Hariel Sugule Velues
Tilely HSV Trolstorios
Praction
r- 6 (105)
[805)
[309])
(505)
Bonk
08
29 (5056)
04
16090
23391
13023
20000
40301
1.0
(205)
(9056
(00%)
Barkk
10000
23391
3400
23031
200.0
40301
K000
6 (100))
(205)
0.
(0056)
Back
28 (905)
0.4
50000
1903)
42003
1.0
(10%)
(208))
(908)
(90%)
Birkk
022
50000
2300
2000
1983)
0000
200.0
42300
0.000
8000
6 (106)
(209)
86
(403))
Beck
28 (90%)
0.
1902)
20000
1.5
(1166)
(2076)
(9076)
(9076)
02
18
20009
40030
00000
50000
23000
80000 Skep
63630
30000
20008
42500
0o/0c
80000
Step
Skep
Figure 24: Six LRU blocks with state dimension of 56 the ListOps dataset.
40


Reviews:
[{"review_id": "YXVuC1qrjE", "reviewer": "Reviewer_qcBm", "summary": "The paper introduces CompreSSM, a generalized in-training compression framework for state space models to accelerate training while maintaining accuracy. The method uses Hankel singular value analysis to identify and prune low-energy state dimensions during training (inspired by Model Order Reduction and balanced truncation from control theory). The authors show that the rank ordering of Hankel singular values remains relatively stable throughout training. The authors use six datasets to provide empirical evidence for its ability to maintain or even improve accuracy as compared to a less efficient baseline training.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The authors provide good theoretical grounding for CompreSSM, bringing concepts from Hankel singular value analysis and truncation; they address a very relevant topic with a well-outlined algorithm.\n\nAuthors provide reasonable justification of the in-training truncation and its validity. Analysis of Hankel singular value trajectories supports their analysis.\n\nThe authors provide convincing results across six different datasets and five random seeds, while describing the experimental setup in great detail facilitating reproducibility.", "weaknesses": "If I understand correctly, empirical evidence is centered only around LRU, without other Deep SSMs or so-called \"selective\" (input dependent) architectures. \n\nThe authors should provide additional analysis or detail on the computational complexity of the Algorithm (3.1) and overhead/clock time of each reduction step, more details into the recovery of the model after each reduction step (mentioned in 4.2 as part of the results, but not analyzed as far as I could tell) and how this translated into end-to-end training cost and efficiency on GPUs versus the vanilla method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iZBNW9hETQ", "reviewer": "Reviewer_p3jH", "summary": "This paper proposes a method for compressing State Space Models (SSMs) during training rather than after training. The key innovation is using balanced truncation from control theory to identify and remove unimportant state dimensions while the model is being trained. The authors leverage Hankel Singular Values (HSVs) to measure the importance of each state dimension. Based on this information, they truncate low-importance dimensions in training, achieving computational speedups while maintaining or improving model performance compared to models trained directly at smaller dimensions.", "questions": "1. Step 5 of the algorithm (Section 3.1): What fraction values work well in practice? The paper doesn't provide specific guidance\n2. Can you provide examples where there's no clear correlation between state dimension and model performance? I cannot think of such applications\n3. Are you evaluating Sequential MNIST (SMNIST)? The paper mentions MNIST results but doesn't clarify if it's SMNIST\n4. How does the computational cost of computing Gramians scale with state dimension?\n5. Have you considered inverting the x-axis in Figure 3 to show \"training time speedup\" instead of \"normalized training time\"? This would make the Pareto front point to the top-right, which is more intuitive", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel application of control theory to SSM compression during training, with good theoretical grounding using Hankel Singular Values and Weyl's theorem to justify the approach\n- The method is architecture-agnostic and works with different SSM variants\n- Empirical validation showing that dominant HSVs are rank-preserving during training\n- Achieves both training speedup over full dimension and better performance than smaller dimensions\n- Well-written paper with clear mathematical exposition", "weaknesses": "1. Only tested on LRU architecture, not on other SSMs like Mamba, S4, etc.\n2. No comparison with other compression techniques like knowledge distillation, quantization, or pruning\n3. Method requires correlation between state dimension and model performance (acknowledged in the paper)\n4. Needs sufficient training steps between reduction steps for model recovery (like pruning and QAT)\n5. Missing ablation studies on key hyperparameters (reduction frequency, threshold values)\n6. No analysis of what information is preserved/lost during truncation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9cMnhIy31f", "reviewer": "Reviewer_vMHR", "summary": "The authors propose a complexity reduction strategy to accelerate SSM training. The idea is to remove non-essential dimensions of the state space by looking at the eigenvalues of the Hankel matrix during training. The pruned models perform better than a baseline where the dimensionality of the state space is fixed from the start.", "questions": "- Is this the first time Hankel-based regularisation techniques are applied to SSMs during training? What is the difference between the proposed method and more straightforward strategies, such as a simple $L_2$ regularisation of the matrices?\n- How is the *normalised training time* mentioned in the figures defined?\n- Would it make sense to include a simple alternative regularisation method as a second baseline?\n- Is the truncation irreversible? Did you observe suboptimality issues associated with the greedy nature of the proposed method?\n- Training complexity gains can be achieved if the reduction is irreversible and happens early enough. How is the latter compatible with the observation that there should be *\"enough time in between successive reduction steps for the model to recover from pruning\"*? How is the trade-off handled in practice?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea reminds me of successful techniques for imposing sparsity when learning NNs.\n- SSMs have become popular as surrogates of transformers. The possibility of compressing them during training in a principled way may help find a trade-off between computational costs and flexibility.", "weaknesses": "- While reducing the complexity of the trained model, the approach does not seem to imply complexity gains during the training phase.\n- The analysis focuses on linear and time-invariant models, which are often outperformed by transformers on several tasks. It is unclear which parts of the idea can be applied to models with input-dependent matrices, such as Mamba-2.\n- The method involves several hyperparameters, which may be expensive to tune.\n- Experiments do not include alternative regularisation approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vwE406xClB", "reviewer": "Reviewer_ELEM", "summary": "The paper proposes an in-training compression technique of state-space models. It is based on application of the balanced truncation method adaptively during training. Empirical results on the long-range arena tasks show that the proposed method improves the model in certain cases.", "questions": "1. You choose a fixed $\\tau$ throughout. If I am given a task, how would I go for a $\\tau$ that achieves a time--accuracy balance that I expect? I think this is a crucial question, and especially given that you do not want to do any additional hyperparameter tuning (otherwise, why not just setting $\\tau = 0$ and training only one model?).\n2. Relatedly, have you experimented with having a scheduling for $\\tau$, instead of fixing it throughout the training stage?\n3. The \"Baseline\" in Table 1 is not well-described in the section. How did you train the \"Baseline\" model? Is it trained by fixing $n$ to be the rounded average \"State dim\" for all LTI systems in an SSM and train it from scratch?\n4. Controllability and observability are not generally assumed in SSMs. How would you reconcile the fact that these two assumptions may not generally hold?\n5. Hankel singular values are very sensitive to initialization and parameterization. For example, this work (https://openreview.net/forum?id=RZwtbg3qYD) shows that without careful initialization, you will start with fast-decaying singular values, and without careful parameterization, LTI systems are driven to low-degree ones during training. These are mainly due to the diagonal elements of $\\mathbf{A}$ that are far apart from the imaginary axis. Is it possible to use that as a criterion for \"when to truncate\"?\n6. Have you tried other ROM techniques other than balanced truncation? For example, those based on $\\mathcal{H}_2$ optimization, moment-matching, or rational approximation?\n7. You mentioned that the work can be adapted to time-variance (or selective) case; however, for selective SSMs, e.g., Mamba, the parameterization is completely different, and $\\mathbf{B}$, $\\mathbf{C}$, and even $\\Delta$ are computed via more sophisticated mechanisms. This would make compression harder. Do you have a simple approach to that? If not, then better avoid overclaiming.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Accelerating the training speed of SSMs is an important task, and the proposed method shows the benefits of the method over directly training a smaller model from scratch.\n* The method is grounded in rigorous analysis of LTI systems and the balanced truncation method from the ROM literature, which has been shown reliability.", "weaknesses": "* The idea proposed in the paper is very incremental. It looks like a naive application of the balanced truncation method at the training stage. The Weyl-type perturbation analysis is not new either. The proposed method can be totally viewed as a multi-stage fine-tuning process, where post-training compression and fine-tuning are applied alternately.\n* The paper proposes an in-training compression method, but there is only one experiment that shows the training-time benefit in Figure 3b on a specific dataset. More importantly, reducing the latent dimension of SSMs does not significantly reduce the training speed (at least on LRA). That said, Figure 3b is visually misleading because the left end starts from somewhere between 0.55 and 0.6 instead of 0. More benefits would come from test time, especially if you plan to do it recurrently, but this is not the advantage of the proposed method over post-training compression.\n* The accuracy of the proposed method is comparison with the baseline is also not compelling enough. Among the LRA tasks, only CIFAR and ListOps have shown some significant benefits. Also note that for ListOps, it is known that a small state dimension is totally capable of achieving SOTA performance. (For S4D, the SOTA performance comes from $n = 4$ and works well even when $n = 1$.)\n* The paper would benefit from a discussion of the post-compression weights, and how it lands in the training loss landscape. This is a more interesting problem than the Weyl-type perturbation analysis. Note that the model is optimized via gradient descent; although compressing the system does not change its impulse response a lot, the parameters can be totally changed. This introduces more subtlety in CompreSSM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)