Hankel Singular Value Regularization for Highly Compressible State Space Models
Authors: Paul Schwerdtner
Jules Berman
Benjamin Peherstorfer
Venue: NeurIPS 2025 poster ()
Keywords: state space models
sequence modeling
model reduction
model compression
Primary area: deep_learning
TL;DR: We regularize the Hankel singular value distribution when training state space models for high compressibility.

Abstract: Deep neural networks using state space models as layers are well suited for long-range sequence tasks but can be challenging to compress after training.
We use that regularizing the sum of Hankel singular values of state space models leads to a fast decay of these singular values and thus to compressible models. To make the proposed Hankel singular value regularization scalable, we develop an algorithm to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that we use in our state space model parametrization. Experiments on Long Range Arena benchmarks demonstrate that the regularized state space layers are up to 10$\times$ more compressible than standard state space layers while maintaining high accuracy.

Hankel Singular Value Regularization for Highly Compressible State Space Models
Paul Schwerdtner
Jules Berman
Courant Institute of Mathematical Sciences
Courant Institute of Mathematical Sciences New York University
New York University
New York, NY 10012
New York, NY 10012
paul. hhall.sshherdtzer@nyy...om
jmb1174@nyu. edu
Benjamin Peherstorfer
Courant Institute of Mathematical Sciences
New York University
New York, NY 10012
pehersto@cims myy.edu
Abstract
Deep neural networks using state space models as layers are well suited for long- range sequence tasks but can be challenging compress after training. We use that regularizing the sum of Hankel singular values of state space models leads to a fast decay of these singular values and thus to compressible models. proposed Hankel singular value regularization scalable, we develop an algorithm to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that use in our state space model parametrization. Experiments on Long Range Arena benchmarks demonstrate that the regularized state space layers are up to 10x more compressible than standard state space layers while maintaining high accuracy.
make the
Introduction
1.1
Compressing state space models
As deep neural networks (DNNs) get bigger, compression via quantization [27, 32], pruning [64, 39, 62], and distillation [29, 43] becomes ever more important [9, 65, 19]. In this work, we focus on compressing neural networks for long-sequence modeling. In particular, we focus on DNNs with state space models (SSMs) as layers [21, 22, 24, 15), which have been shown to achieve state-of-the-art accuracy while reducing training and inference costs compared to transformer models [21, 53, 59]. Significant progress was necessary make SSM layers competitive for long-sequence tasks [22, 15]. After the initial success of S4 layers [22], big step forward were S5 layers [53], which contain linear time-inven invarianr systems that have diagonal system matrices and reach state of the art performance while balancing expressivity with training and inference costs. Additionally, initialization of SSMs has been identified as being critical for training success, for which HiPPO matrices are now commonly used [22]. In this work, we build on DNNs with SSM layers that are well suited for long song-sequence tasks and aim to further reduce inference costs by training neural networks such that the SSM layers are highly compressible: The critical quantity in SSMs that controls inference costs is the order n, which the dimension of the internal state [26]. Therefore, instead of training DNNs with general SSMs
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

as layers with order n. we regularize the training to determine SSMs of order n but such that at the same time SSMs with lower order < n exist that mimic the sequence-to-see -e-euuenee map of the individual SSM layers. Thus, in our approach, training is performed with a larger order to provide the opportunity for exploring large parameter space for high expressivity while enabling compression as postpproessiing step. Our proposed regularization is founded on well-established and rigorous system- saleemtthoortic results [1].
1.2 System-theoretic perspective on SSM compressibility
A classical seseem-hheorete way [1] of describing sequence-to-sequeee map {ux (uk))= + {yx l induced by a linear time-in invariant dynamical system (as used in S5 [53]) via the convolution with the impulse response h& E Rexm
yk
1 k = 2,,,,,,
where the input us E R and output y E IR at time step kt are of dimension and p, respectively. Note that the common design choice deep state space models is to set m = p. This convolution
leads to the Hankel operator H E ly that acts on sequences and maps {ux ) k== to {yx k= The Hankel operator can be explicitly described by the blocks H4j = hg-i-j for i. E 0, 1, 2, 3... and is linear and bounded. Importantly the number of non-zero singular values of Hankel operator is finite and gives the McMillan degree, which is the minimal order n necessary for an SSM to deseribe the map (ux)e = fyı j [1]. Analogous definitions hold for finite-length sequences n, < oc, in which case the finite Hankel operator is obtained by Hy hg-i-5 for i. = ...... n2, While clearly a system of order n, exists describe the map (ux k- + {y, o the goal is finding a system with order n << n4 that achieves the same mapping or at least good approximation of it.
The key for compressibility are the singular values of the Hankel operator: if the number of non-zero singular values 01..... on > 0. then there exists system of order 1nx
that maps [ux]2=0 + { with error 0 yxlle, S 2|/uille, Eisr+1 dg Thus, the quicker the singular values of the Hankel operator decay, the more compressible system is, which is relevant for, e.g.. model reduction [50, 5, 36).
When training SSM layers with standard optimization, however, one only prescribes order and then the optimizer can distribute the Hankel singular values will, which typically leads to systems that are only poorly compressible. In Figure 1, we show the decay of the Hankel singular values of systems learned for the Long Range Arena (LRA) [59] image benchmark. The Hankel singular values decay slowly and thus there cannot exist much smaller system with << that achieves good approximation of the sequence- toaeceett--eeeee sequenee map of the original state space models of order
regularization
Energy
Truncated
regulanization no regularizaiin
coordinates
coordinates
102
layer
layer 4
balancing transformation
transforatt
reduced order Inference
accuraty
80
state truncation
60
decty
20
10P
HSV regularization
lest accuneey d
100
Hankel
10"
slow 1SV deeny
80
60
10"
layer
40
layer
102
byer
60
80
90
128
192
256
320
384
truncation ratio
Figure 1: We propose regularize the Hankel singular values of SSMs that they become compressible. Left: Regularizing with the Hankel singular values during training leads to SSMs with fast Hankel singular value (HSV) decay. Middle: SSMs with fast HSV decay have many low-energy states that only contribute little to the layer output. Right: Compressing the SSM by truncating only such low-energy states changes the corresponding cequence-to-sequence map insignificantly and retains the overall accuracy. Without our regularization, HSVs decay slowly and compression leads to an accuracy deterioration.

---

n. The results in Figure I are in agreement with other attempts of compressing SSMs such as [13], which show that on standard LRA benchmarks the Hankel singular values of the SSM layers also decay slowly and thus the systems are not well compressible when trained with default procedures.
1.3 Literature review
Models with state space layers DNNs with state space models as layers have recently been made popular starting with the introduction and utilization of the high-order polynomial projection operators (HiPPO) framework in [22]. which was applied as structured state space sequence model (S4) to efficiently model long sequences in [23] and outperformed the state of the art in the Long Range Arena (LRA) benchmark [59]. This was a major step forward as other architectures such as recurrent neural networks (RNNs) [2. 12, 51, 8] and transformers [60] and memory efficient variants of transformer layers [10, 33, 35, 4, 25, 61] achieved poorer performance at the time on long-range sequence tasks in [59]. After the initial introduction of S4, simplified SSM layer (S5) was introduced in [53]. in which S4 was streamlined from its original single-input single-output (SISO) and convolutional formulation The time-domain formulation, which leverages a parallel scan for computational efficiency, was an important contribution as it facilitates more variations such as time-vane aaee-nayingg and input-dependent SSM operators, which are explored independently in Liquid-S4 [28].
multi-input multi-output (MIMO) time-domain formulation.
Large scale SSMs are explored in Mamba-St S [21], which introduces selective SSMs that allow for efficient filtering and context compression, and H3 [15], which is extended further in [49]. The new
class of structured SSMs has found wide range of applications such as audio generation [17] and vision tasks [44, 31].
Even methodological contributions to the layer architecture are still made such as recent state-free cransferfuncction-aaee based implementation [46], novel parameterization schemes [63], a reformulation of RNNs in the SSM framework [45], or bidirectional extensions of Mamba [30].
Compression and distillation of SSMs Given that costs and the number of parameters grow with n, the order of the SSM, there has been investigations about how to keep n small without sacrificing expressivity. While pruning and post-training compression is extensively studied for transformer architectures [43, 7, 58], model compression for SSM and mixed models has only recently started to gain traction [16, 42, 56). The work [26] develops system and control-inspired criteria for state- pruning over multiple layers. In contrast, we propose to regularize the Hankel singular values already during training, which is approach from classical system and control theory for system identification [48, 55, 20] and has been pursued in the context of deep networks with state space model layers in [14]. Our work combines the ideas of [14] and [26]: we leverage Hankel singular value regularization as in [14] and multi-layer state pruning as in [26]. Combining Hankel singular value regularization with multi-layer state pruning enables layer-dependent rank adaptation, which is important when networks get deeper as necessary for the benchmarks we consider. In contrast, the work [14] uses the same rank for all layers. Moreover, we consider a block- structurdd real-var val-va parametrization of stable systems, for which we propose novel algorithmic methods for the efficient Hankel singular value computation compared to the diagonal matrices and corresponding algorithms considered in [14]. Combining the layer-adaptive rank, the block olock-diagonal structure, and our novel algorithms for computing singular values allows us to demonstrate scalability on Long Range Arena benchmarks, thereby highlighting relevance to language modeling tasks beyond the physics problems considered in [14]. Directly applying general model reduction methods to SSMs is attempted in [13] but the method in [13] requires retraining a smaller model after the reduction, which can be costly step that our approach avoids. Additionally, there is distillation that aims to learn a smaller model from a larger teacher model; see, e.g., [29]. Distillation in the context of SSMs is explored in [43].
1.4 Our approach and summary of contributions
We propose a training procedure that allows efficiently regularizing the Hankel singular values of the SSM layers that the singular values decay favorably for compression. Our contributions are:
Connecting SSM layers to system theory for deriving conditions under which layers are compress- ible.
3

---

Regularizing Hankel singular values to nudge the optimizer to seek models that can be well compressed with standard tools from system theory such as balanced truncation, while maintaining accurate sequence-tor toreeett----euncee mappings. To this end, we prove the differentiabillty of nuclear norm regularizer constructed from the Hankel singular values.
For efficient training, developing an algorithm that enables the computation of so-called gramians, which are needed evaluate the Hankel singular values of SSMs, with our imposed block structure, that reduces the computational cost from Onn²) to ( ) in the state dimension n..
Demonstrating that applying our method can lead to up to 10x improvement in accuracy for strongly compressed models on Long Range Arena benchmarks.
2 Hankel singular value regularization (HSVR)
We propose to regularize the training of neural-network models with SSM layers so that the SSMs have a fast Hankel singular value decay which means they can be compressed efficiently. The key to regularizing the Hankel singular values is ensuring that the regularizer building on them is differentiable and that computing the singular values is efficient during the training iterations, for which we introduce scalable parametrization and a scalable algorithm.
2.1 Parametrization of SSMs for scalable computation of Hankel singular values
Time-discrete SSMs and their Hankel singular values time-invariant dynamical system,
Each SSM layer consists of a linear
0k+1 Axx + Bus
2O RR"
(I)
yk Cr + Dus:
where A E Rnxx, B E RTxp, C E Rex", and D E RPxP, are the system, input, output, and feedthrough matrices, respectively, and in- and in. are the input signal and the system output, respectively. The controllability gramian P and observability gramian Q of size n x n of the system (I) carry information about the singular values of the underlying Hankel operator H: For stable, controllable, and observable systems (terms defined in the Appendix A), the non-zero Hankel singular values σi, ...... of the system (I) are the square-roots of the eigenvalues of the product PQ, (Hl) VA,(PQ), where A; denotes the i-th eigenvalue of its matrix argument. The gramians can be computed as the solutions to the discrete Lyapunov equations
APA - BB =0,
(2)
A A - + C
(3)
straightforward solution these Lyapumov equations can be computed by vectorizing both equa- tions, which yields
vec (P) - C A-1) 1) vec (eee(vec(BB BB vec(Q) - (A 8 A - 11) vec (C))
(4)
(5)
where vec denotes the vectorization operator that stacks all columns of its matrix argument. However, computing P and Q using (4) scales as Onn)). The Bartels-Stewan algorithm [3] can be used to solve general Lyapunov equations in Onn²) using generalized eigen-decompositions.
Parametrization via rotation matrices We now parametrize discrete dynamical systems (1) via scaled rotation matrices. Scaled rotation matrices lead to favorable properties such as making it easy enforce stability. Furthermore, the rotation block structure allows to derive algorithms to compute solutions of the Lyapunov equations (2)-(3) with costs that scale as Onn)) (instead of (nn)) as the standard -netell-Steward algorithm).. Note that we are not claiming we are the first to recognize the benefits of parametrizing SSMs with rotation matrices as a discrete-time alternative the HiPPO framework (see, e.g., [11]) but we show how they are useful for efficiently computing Hankel singular values for regularization.

---

In the following, we set n/2 and only consider the case m = to ease the notation burden. At layer l, have system of the form (1) with matrices
Ai
e1 B
A
0
A2(p2
2
(
0
B(O)
el
B
2
(6)
:
(o) aa))))
B
0
el
where [1, and each block in A is a rotation matrix
(9) sin(a cos(a;
((0)
(. 0 - (/()
cos( az sin(a;
E R2x2
B
E
IR
2x(x1111
(7)
b/2
which has the complex conjugate eigenvalues P (2) of size p x has no special structure. In our experiments, we only use diagonal feed-through matrices ( of size p pt which is standard also in other works [53].
cos((a{)) iisin(a?")) The output matrix (
We can summarize the parameters of the SSM in layer & in the parameter vector (()) g2atn(p-1)+pptz which contains vectorized form (+ (()), b111 :....... b2 c(0) ((),, Notice that the number of SSM-parameters per layer, i... the dimension of (( grows linearly in the order n the SSM, which is the same as using diagonal system matrices as in the S5 SSM layers [53].
The parametrization given by rotation matrices is universal in the sense that is dense in the space of linear lime-invariant systems of order n, which is shown in the following. Recall that systems with the same impulse response are equivalent in the sense that they describe the same sequence-to-see to-seuencce map.
For any linear uime-invarian system of order n there exists an infinitesimal perturba- tion such that the sequence-to-sed to-suuenee map {u I== + 1ec of the perturbed model can be described by an SSM with matrices of the form (7) with a. p R", where ñ n.
Proposition 1.
Our proof, which present in the Appendix A.1, uses [34, Chapter 2.2] to break up any Jordan blocks with infinitesimal perturbations, exploits the fact that controllable systems are dense in the space of n-dimensional systems [54, Proposition 3.3.12], and finally uses the block Schur decomposition to bring A into the desired form and applies Givens rotations to establish the first column of B as repeated standard-basis vectors.
Enforcing stability Stability of SSMs parametrized via rotation matrices can be achieved by enforcing that the entries of the p vector remain below one for all layers. Thus, for example, it is sufficient to threshold the entries of p with tanh, which we use in our implementation. Moreover, we scale the parameters a using tanh to be contained in the interval [0, TT. With this parametrization, eigenvalues inside the unit-disk and thus have a simple
we can attain any pair of complex-conjugan and differentiable parametrization of stable systems.
2.2 Scalable training procedure
Leveraging block-diagonal structure for computing singular values Recall that if we want regularize the distribution of the Hankel singular values, then we have to compute them at least once in each gradient-descemt step during training. The standard algorithm for computing Hankel singular values is solving the Lyapunov equations (2) and (3) with the Bartels-Stewart algorithm [3], which incurs costs that scale as O(n²) with the system order n. We now introduce an algorithm that leverages the block-diagonal structure (6) of our system parametrization and achieves a cost scaling of On??). We describe the algorithm for the control Lyapunov equation (2); the treatment of (3) analogous.
Plugging our parametrization (6) into the control Lyapunov equation (2) leads to
A ...
Pı
P12
Pio
A
P.: P12 ... Pio
B
B T B;
0
Aa
Pl
Pa ... Pz.
0
A T
Pla Pm ... Pzu
+
B;
0,
P P. "
:
P
B,
Bg
09
5

---

where each P.j E RR222 This decomposes into q Lyapunov equations
A.P;A,"- Ps: + B, = O,
for E {I,...,,)) 2) and a(9 - 1)/2 Sylvester equations (note that the solution P of (2) is symmetric)
A,P Psj + BBB
(8)
for i E (I...., 2 and j E (i+ 1,....... 4) which can be solved similarly by vectorizing as in (4), which is efficient for small 2-by-2 blocks. As each block Piş can be solved independently, solving for P can be efficiently parallelized. The overall costs scale as Oga²) and thus as (n because q = n/2. Note that the computational costs of the regularizer is independent of the sequence length
After computing the gramians P and Q. the Hankel singular values are the singular values of the product PQ, which can be computed using standard algorithms. Note that the block structure in the system matrix A does not imply a block structure in P and Q. This is again because the Hankel operator does not consider the subsystems separately but considers their interaction as well. For a block structure in P and Q, the matrices B and C would have to be block-diagonal as well, which limits the expressivity of the SSM, and thus of the corresponding neural-network model.
Fast time integration with associative scan A major ingredient for making SSMs scalable is using parallel scans for computing output sequences. For diagonal matrices A, parallel scan version has been introduced in [53], but it critically depends on A to be diagonal matrix to keep the costs in O(n). The work [11] I] builds on rotation matrices as well and proposes a parallel scan version that explicitly calculates products of the blocks. In contrast, now show that an analogous associative scan operation exists for our parametrization via rotation matrices that avoids having to compute products of the blocks explicitly.
For an associative binary operator *, i.c. an operator such that (b)b)== + a (b* c). and a sequence of elements (d1,..., I, the scan operation returns (a1, (a1 *02), (01 * 02 * an,)] }}. In [53], the authors consider the associative binary operation
(Cax*, CC)) (C"xa Cn)) - (C"x", C"), (X, x), (Y, y)) + YY y),
where the matrix-matrix product Y X can be computed in On)). when X and Y are diagonal matrices. This is the key why SSMs with diagonal matrices in [53] scalable. The sequence, on which the scan operates, is then initialized as
(9) It is easy to verify that the scan with over leads to the state sequence of (1) by noting that the scan output elements can be written s, (A', Ei A Bux). The second element of each tuple s; exactly the state a, for a discrete system (1). Here A denotes the i-fold matrix product of A with itself.
a = [(A, Bu1). Bu2), (A, Bun,
For our cilutton-basse parametrization, can define a similar associative binary operation
(TR,, R9, RR",, (TR., IR9 R")
I TR29 R"),
(10)
(1), x (2) x (8)),
3) + ((x(1) (1) O (1), (x (2) + A(y) y 2x)
(3) + ))
where @ denotes the Hadamard product and A(, .) is formed as in (6) for its vector-valued arguments. We can then initialize the sean sequence for each layer with
b [(p, a, Bu,), (p, a, Bug).... (p, a. Bu,,)) )}-
We can verify that scan with * over leads the state sequence because the sean output elements can be written as 8; = a)) (A a)) Bux). Here we use the fact that for sealars D. 8, we have that
cos( sin (
sin
cos(7)
sin(7)
cos(B +7)
sin ( + +7)
cos [3] y/
sin(7) cos(7)
xy
sin +7)
cos + +7)
such that we can use that A(x, A) compute the product of all blocks the diagonal as in [1]]. Using our parallel scan operation, the costs of generating an output sequence of length n, scale as ((ogg(n)))). assuming n, processors which is the same scaling when using diagonal matrices as in [53].
A(x @ y. 3 7) in (10). This avoids having to

---

2.3 Regularizing Hankel singular values during training
Differentiable regularizers involving Hankel singular values Building on the efficient compu- tation of Hankel singular values from the previous section, we now develop a regularizer R that depends on the Hankel singular values all layers. We stress that building regularizers based on the Hankel singular values is standard practice in systems and control theory [48, 55, 20] and has been proposed in the work [14] for deep state-space models for the first time; see Section 1.3 for an in-depth comparison to these works.
First, we show a new result that even though the individual Hankel singular values are not differen- tiable with respect to the entries of the system matrices (6) (i.e., the network parameters), the sum of the Hankel singular values is differentiable. We follow similar arguments as used in [52, Proposition 3.7], which uses that different branches of singular value curves that intersect each other and form
non-simple singular value still add up smoothly locally. Denote with ((1)) = [0 singular values of the system at layer E.
] the
Proposition Given an asymptotically stable matrix A, as well as B and C such that the pairs (A. B) and (A,C) are controllable and observable, respectively, let P and Q be the solutions (2) and (3), respectively. Then the sum of Hankel singular values Vim1 03 of PQ depends smoothly on A, B, and C.
For proof, see Appendix A.2.
Regularizing the Hankel nuclear norm Nuclear norn regularization, i.e., penalizing the sum of singular values of matrix, to encourage singular values to decay rapidly, common practice in machine learning: see, e.g., [20, 14] for uses cases in reducing systems. With the tools we just developed. we can now regularize sums of Hankel singular values of SSM layers for a fast Hankel singular value decay. In particular, the Hankel nuclear norm of system is the sum of its Hankel singular values
For this, we introduce the regularizer
R.(o
(1)
(11)
which is added to the loss during training. Recall that Proposition 2 in combination with our parametrization guarantees that R. is differentiable with respect to the neural-network parameters.
2.4 Compressing the trained models (post-processing)
Compressing regularized SSMs with balanced truncation After having trained a DNN with SSM layers with regularized Hankel singular values, we can apply off-the-shelf model reduction methods to compress the SSM layers. We use balanced truncation to compute the compressed (reduced) systems, because it is well studied and developed and the reduced system inherits favorable properties such as stability from the original, full system. We use the standard square-root method [6, Chapter 6.2] to compute the balanced truncation SSM with reduced state dimension. For this, in each layer we first compute the final controllability and observability gramians P and Q by solving (2) and (3). After that, we compute the singular value decomposition ΦΣ svd R), where SST Q and RR P are Cholesky decompositions of Q and P. Then can define the projection matrices V = S
TT. t E.prr and W R V ,,:r Exprr, where for a matrix M, the expression M..r denotes the first r columns of M and sr.er denotes the upper left r-dimensional block of M. The reduced r-dimensional SSM is then obtained as [WT AV, W B, B).
For balanced truncation, the error incurred in the sequence-to-sequence map {un + Hu of the compressed system bounded as
0 Vxllez S 2ulee
Σ oi
(12)
and thus controlled by the sum of truncated Hankel singular values. Thus, the bound (12) provides a viable criterion for choosing the compression order r in the different layers; see next paragraph.
7

---

1000
100
1001
aucuracy
60
40
40
600
20
Lo005on 010000 T18C
Landoom 110000
61
1001
Emincctionn mili
cnuncation mitio (b) sMNIST dataset
truncation ratio 56 (c) IMDB dalaset
(x) .CIFARIO (gragscale)
accumity
60
Mildoin Chance TiN
Fandam TICIOS THNC
40
truncasion ratio
cuncattion ratio % (c) PATII-X X datiset
PATHI datazel
Figure 2: Regularizing Hankel singular values leads highly compressible SSMs while maintaining accuracy.
Balancing the reduced order of compressed SSMs across all layers The Hankel singular values give us guidance to which order r < n to compress the regularized SSMs: We define the energy of the SSM layer as e Σニ= of, which is the sum all Hankel singular values of the SSM at layer e. We then prescribe criterion such as retaining 99% of all energy, which means truncating at order that e 0.99 for eç_ = E of
We stress that there is one-to-one correspondence to the error incurred in the sequence sequence map, because of the bound (12) satisfied by models compressed with balanced truncation. Notice that we prescribe the same energy criterion (e.g.. 99%) for all layers l = ......l L but that the corresponding compression order .'. .' can be different for each layer.
Alternatively, we can prescribe total budget r. = - (2) of state dimensions, where p()) ((0) are the state dimensions of the SSMs corresponding to layers & = 1...., L. Given total budget re. we can then distribute the state dimensions across the L..... layers such that the same amount of energy is preserved in each layer. For this, we use bisection algorithm that is described in detail the Appendix in Section C.
Diagonalizing compressed systems The reduced systems are balanced but not necessarily di- agonal or block-diagonal, which essential for an efficient application the associative sean operations; see Section 2.2. We diagonalize the compressed SSMs using eigenvalue decompo- sition: Let A, W AV be the reduced system matrix. Then decomposition TA,T-1 = Ar, where A, E Rexr is diagonal. An equivalent diagonal system to [W AV. w B, CV, D] is then given by nalized system might be complex-valued like the systems [53]; however, since the eigenvalues will appear in complex-conjugate pairs, a real-valued input sequence will be mapped to real-valued output sequence, which also used [53].
compute eigenvalue
AVT,T- ' B, CVT. DJ. The diago-
Results
Benchmarks We demonstrate our HSVR approach on five sequence classification examples. The first example consists of the 32x32 CIFAR- 10 images [37] that are converted to grayscale, flattened into 1,024-length sequences, and normalized zero mean and unit variance across the entire dataset. It includes 50,000 training, and 10,000 test samples and has ten target classes. The second example also sequentialized image classification task and consists of the 28x28 grayscale MNIST [38] images, where again each image is flattened into sequence of 784 scalar values. The goal is to predict the depicted written digits correctly. The third task uses the IMDB sentiment dataset [41], where movie reviews are represented as sequences of one-hot encoded characters with 129 possible values, padded to a maximum length of 4,096. The goal to classify each review as positive or negative; the dataset includes 25,000 training and 25,000 test examples. Finally, consider the

---

to
Fd2 RSVK joirs
Hankel singnite vilue
[+2, po IEg.
HSVR (OUIS) HSVR
(-d.
100
200
300
50
100
100
150
indcx
index.
index (c) IMDB datuset
(0) (grayscale)
(b) SMNIST danases
10a
rog.
Hunkel singular value
HSVI (N0m)
HSVK (wun)
200),
Cob, HSVR (MUP)
Endl no
Eeb RSVR (oins
10x
10"
100
150
200
2500
50
100
150
200
250
index
PATEE datnee
(e) PATTE-X dataset
Figure 3: Our HSVR approach trains SSMs that have favorably Hankel singular value decay for compression.
Table 1: Test accuracies for different methods for different truncation ratios.
Method
sCIFAR (grayscale)
sMNIST
IMDB
trunc. ratio
60%
70%
80%
90%
60%
70%
80%
90%
60%
70%
80%
90%
LAST [26]
62.93
36.66
17.35
11.19
95.11
89.17
62.37
27.67
88.48
85.05
80.26
57.08
global [26]
28.91
13.62
11.12
10.47
91.67
83.32
52.52
21.94
88.28
87.70
83.75
63,80
uniform [26]
58.90
34.45
19.18
12.67
97.74
79.20
44.38
23.20
82.44
77.34
64.79
53.22
no reg.
71.28
41.98
21.14
9.84
91.32
13.35
11.05
10.55
71.45
71.04
51.32
50.00
HSVR (ours)
81.84
81.75
81.37
51.08
99.45
99.22
98.90
86.95
87.26
87.16
86.97
86,40
Method ratio
PATH
PATH-X
60%
70%
80%
90%
60%
70%
80%
90%
LAST [26]
50.51
50.11
49.96
50.25
50.33
49.16
49.53
49. 49.64
global [26]
49.16
50.15
50.16
49.37
49.50
50.93
49.14
50.61
uniform [26]
50.32
49.78
49.84
50.16
49.74
50.23
49.70
50.47
reg.
60.21
53.48
50.35
50.20
56.09
50.39
50.16
50.14
HSVR (ours)
65.94
63.64
50.50
49.27
87.74
82.82
54.02
51.97
PATH and PATH-X datasets, which consist of the flattened pathfinder images [40], which consists of two points and a set of paths. The classifier must determine whether the two points are connected by the paths. The flattened PATH images have sequence length of 1,024 and flattened PATH-X images have a sequence length 16,384a. We denote the examples by SCIFAR (grayscale). sMNIST, IMDB, PATH, and PATH-X. The examples sCIFAR (grayscale), IMDB, PATH, and PATH-X are also part of the Long Range Arena (LRA) benchmark [59] collection.
Setup We select the state, input, and output dimensions of our SSMs according to the setup in [53]. In particular, for sCIFAR 10 (grayscale). n = m = = 128 for sMNIST, n = 192, m = P 256 for IMDB, n 256 = p = 192 for PATH. and n 256, m = p 128 for PATH-X. As in [53] for sCIFAR 10 (grayscale) IMDB, PATH, and PATH-X, we use 6 SSM layers and for sMNIST we use 4 layers. The remainder of the model architecture, which we describe alongside the training parameters in the Appendix in Section B, is also the same in [53]. In all examples, we use HSVR with the Hankel nuclear norm regularizer (11), even though other regularizers based on the Hankel singular values could be used, which remains future work. One notable difference compared to [53, 26] is that we only use unidirectional associative scans, whereas [53, 26] scan bidirectionally for SCIFAR (grayscale) and IMDB. This means that they apply the associative scan to the given and the reversed sequence and double the dimension of the output matrices of the SSMs to merge both sequences into one output sequence.
state dimension
384, and input and output dimensions m = p 512

---

SSMs frained with our HSVR are highly compressible In Figure 2, we show the model accuracy on the given test data, as we increase the truncation ratio X. which the maximum allowed average reduced state dimension across all layers. For an original state dimension n, the maximum allowed average reduced state dimension is ( -X -x). For our comparison to [26], we extracted the accuracies reported in Figures 2 and 6 in [26]. For HSVR, we show the median over three training runs initialized with different random seeds; standard deviations are reported in the Appendix in Section D. The results in Figure 2 clearly demonstrate the effectiveness of our proposed HSVR. The accuracy of the full SSM model is retained for truncation ratios of 80% for SCIFAR (grayscale) and sMNIST and even for over 90% truncation ratio for the IMDB dataset. Even for the more challenging PATH and PATH-X X datasets, we can observe a higher test accuracy for larger truncation ratios compared to other methods. Without regularized training, the accuracy drops much earlier, which is also the case for LAST-based pruning of [26], which also follows the S5 architecture in its experiment setup. The results in Figure provide further evidence that HSVR leads to favorable Hankel singular value decay compared to unregularized training. Note that only for the PATH dataset our HSVR regularizer did not lead to significant difference in the HSV distribution when comparing with unregularized training; especially when comparing with the HSV distributions for the other datasets in Figure 3, where a clear gap appears between regularized and unregularized training. Moreover, on the PATH dataset, we achieved the smallest test accuracy improvement. This again emphasizes that the HSV distribution is key when considering the compressibility of SSMs.
HSVR achieves higher compression than previous methods In Table 1, we conduct a comparison to all pruning methods proposed in [26] as well as training our models with no regularization. Overall, Table again demonstrates the benefits of HSV regularized training. Our HSVR approach outperforms all other compression methods over wide range of compression ratios and accuracy ranges; the only exception being at very low compression ratios, which are of less interest in most cases. For example, we maintain accuracy of around to 99% with a compression ratio of 80% in the sMNIST data set, while compressing unregularized SSMs leads to an accuracy drop to almost 10%. Notably, with our regularization we can maintain a high accuracy on the challenging PATH-X dataset even at compression ratios above 60%, where prior methods collapse to random performance.
4 Conclusions, limitations, and impact statement
Conclusions We demonstrated that regularizing the Hankel singular values of the SSMs is key for compression. While the individual Hankel singular values not differentiable, their sum is, which is all that is needed for obtaining a differentiable regularizer. A key aspect is that we developed algorithm that can efficiently compute the Hankel singular values to keep training costs low. Experiments with standard LRA benchmark examples demonstrate that we can compress models by up to 90% while maintaining acceptable accuracy, An implementation is provided at https: //gi thub. com/ Al Algopaul/ httppa//haanklrreeeeeee
Limitations (a) Models need to be trained with the regularizer achieve compressibility, which means that our compression approach is not applicable to pre- pre-trained models without our regularizer. Because is known that linear equivalence transformations cannot change the Hankel singular values, it remains future work to find nonlinear transformations to achieve compressibility also for pre-trained models. (b) By regularizing the Hankel singular values and compressing with gystem-theoreti tools such balanced truncation, we are restricted to linear compression. This is reasonable as the SSMs are linear in the state too but there can exist more efficient nonlinear compressions. Rigorous nonlinear compressions for dynamical systems are an active research direction in systems and control theory [47] and it remains future work to develop corresponding regularizers for SSM layers and neural-network models. (c) We focus on SSMs that are linear time-invariant systems; however, using time-varying system matrices can increase expressivity of the corresponding netwolknetwork models without increasing parameter count and they are explored in MAMBA architecture [21]. It remains future work to extend our approach to systems with time-varying system matrices.
Impact statement sion approach.
We are not expecting negative societal impacts that are specific to our compres-
10

---

Acknowledgements
The authors have been partially funded by the Air Force Office of Scientific Research (AFOSR), USA, award FA9550-24-1-0327.
References
[1] Athanasios C. Antoulas. Approximation of Large-Scale Dynamical Systems, volume 6 of Adv. Des. Control. SIAM, Philadelphia, 2005.
[2] Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks.
In International conference on machine learning, pages 1120-1128. PMLR, 2016.
[3] R. H. Bartels and G. W. Stewart. Solution of the matrix equation AX + XB = C. ACM Commun. Comput. Algebra, 15(9):820-822 1972.
[4] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-doeument transformer. arXiv preprint arXiv:2004.05/550 2020.
[5] Peter Benner, Serkan Gugerein, and Karen Willcox. A survey of projection- based model reduction methods for parametric dynamical systems. SIAM Review, 57(4):483-531, 2015.
[6] Peter Benner, Mario Ohlberger, Albert Cohen, and Karen Willcox. Model reduction and approximation: theory and algorithms. SIAM, 2017.
[7] Akhiad Bercovich, Tomer Ronen, Talor Abramovich, Nir Ailon, Nave Assaf, Mohammad Dabbah, Ido Galil, Amnon Geifman, Yonatan Geifman, Izhak Golan, et al. Puzzle: Distillation- based nas for inference-optimized llms. arXiv preprint arXiv:24/1.19146. 2024.
[8] Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi. Antisymmetricra:: A dynamical system view recurrent neural networks. arXiv preprint arXiv: 1902.0969, 2019.
[9] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35(1):126-136, 2018.
[10] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.1477 14794, 2020.
[11] Rares Dolga, Kai Biegun, Jake Cunningham, and David Barber. RoIRNN: Modelling long sequences with rotations. In Next Generation of Sequence Modeling Architectures, International Conference of Machine Learning (ICML), 2025.
[12] Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Mahoney. Lipschitz recurrent neural networks. arXiv preprint arXiv:2006. /2070, 2020.
[13] Haruka Ezoe and Kazuhiro Sato. Model compression method for s4 with diagonal state space layers using balanced truncation. IEEE Access, 2024.
[14] Marco Forgione, Manas Mejari, and Dario Piga. Model order reduction of deep structured state-space models: A system-
wystem-theoretic theoretic approach. In 2024 IEEE 63rd Conference on Decision IEEE, December 2024.
and Control (CDC), page
[15] Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry Hungry HiPPOs: Towards language modeling with state space models. arXiv preprint arXiv:2212. 14052, 2022.
[16] Tamer Ghattas, Michael Hassid, and Roy Schwartz. On pruning state-space llms. arXiv preprint arXiv:2502. /8886, 2025.
[17] Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-space models. In International conference on machine learning, pages 7616-7633. PMLR, 2022.

---

[18] Gene H Golub and Charles F Van Loan. Matrix Computations. Johns Hopkins Studies in the Mathematical Sciences. Johns Hopkins University Press, Baltimore, MD, 4 edition. 2013.
[19] Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Viston, 1996)1179-1189, Jun 2021.
[20] Pawan Goyal, Benjamin Peherstorfer, and Peter Benner. Rank-minimizing and structured model inference. SIAM Journal on Scientific Computing, 4673A1A187--11992,, 2024.
[21] Albert Gu and Tri Dao. MAMBA: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.0005 2023.
[22] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474-1487. 2020.
[23] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00390 2022.
[24] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-ting models with linear state space layers. Advances in neural information processing systems, $4:572-585, 2021.
[25] Ankit Gupta and Jonathan Berant. Gmat: Global memory augmentation for transformers. arXiv preprint arXiv:2006. 03274, 2020.
[26] Minseon Gwak, Seongrok Moon, Joohwan Ko, and PooGyeon Park. Layer-adaptive state pruning for deep state space models. Advances in Neural Information Processing Systems, 37:10613-10645, 2024.
[27] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning. trained quantization and huffman coding. arXiv: Computer Vision and Pattern Recognition, 2015.
2015.
[28] Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. arXiv preprint arXiv:2209.1244 2022.
[29] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 150..0233..,,,.....
[30] Sukjun Hwang, Aakash Sunil Lahoti, Ratish Puduppully, Tri Dao, and Albert Gu. Hydra: Bidirectional state space models through generalized matrix mixers. Advances in Neural Information Processing Systems, 37:110876-110008 2024.
[31] Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. In European Conference Computer Vision, pages 87-104. Springer, 2022.
[32] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-arihmmetioonly inference. In Proceedings of the IEEE conference on computer vision and pallern recognition, pages 2704-2713, 2018.
[33] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156-5165. PMLR, 2020.
[34] Tosio Kato. Perturbation Theory for Linear Operators. Springer Berlin Heidelberg, 1995.
[35] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.044:
[36] Boris Kramer, Benjamin Peherstorfer, and Karen E. Willcox. Learning nonlinear reduced models from data with operator inference. Annual Review of Fluid Mechanics, 56 Volume 56, 2024):521-548, 2024.
12

---

[37] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009.
[38] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(1 011:2278-2224 1998.
[39] Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-basee pruning. arXiv preprint arXiv:2010.07611, 2020.
[40] Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Information Processing Systems, volume 31. Curran Associates, Inc., 2018
and R. Garnett, editors, Advances in Neural
[41] Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150. Association for Computational Linguistics, 2011.
[42] J Pablo Muñoz, Jinjie Yuan, and Nilesh Jain. Mamba-shedder: Post-transformer compression for efficient selective structured state space models. arXiv preprint arXiv:2501. 2025.
[43] Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. Advances in Neural Informa- tion Processing Systems, 37:41076 411-411222 102, 2024.
[44] Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. Advances in neural information processing systems, 35:2846-2880 2022.
[45] Antonio Orvieto, Samuel Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pages 26670-26698, PMLR, 2023.
[46] Rom N Parnichkun, Stefano Massaroli, Alessandro Moro, Jimmy TH Smith, Ramin Hasani, Mathias Lechner, Qi An, Christopher Ré, Hajime Asama, Stefano Ermon, al. State-free infer- ence of state-space models: The transfer function approach. arXiv preprint arXiv:2405.06144 2024.
[47] B. Peherstorfer. Breaking the Kolmogorov barrier with nonlinear model reduction. Notices of the American Mathematical Society, 69:725-733, 2022.
[48] Gianluigi Pillonetto, Tianshi Chen, Alessandro Chiuso, Giuseppe De Nicolao, and Lennart Ljung. Regularized linear system identification using atomic, nuclear and kernel-based norms: The role of the stability constraint. Automatica, 69:137-149, 2016.
[49] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pages 28043-28078. PMLR, 2023.
[50] G. Rozza, D. B. Huynh, and A. T. Patera. Reduced basis approximation and a posteriori error estimation for affinely parametrized elliptic coercive partial differential equations. Archives of Computational Methods in Engineering, (5(3):229-273 Sep 2008
[51] T Konstantin Rusch and Siddhartha Mishra. Unicornn: A recurrent model for learning very long time dependencies. In International Conference on Machine Learning, pages 9168-9178 PMLR, 2021.
[52] Paul Schwerdtner and Matthias Voigt. SOBMOR: Structured optimization-based model order reduction. SIAM J. Sci. Comput., 45(2):5502-A529) 2023.
13

---

[53] Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations. 2023.
[54] Eduardo D. Sontag. Mathematical Control Theory. Springer New York, 1998.
[55] Yue Sun, Samet Oymak, and Maryam Fazel. System identification via nuclear norm regulariza- tion. arXiv, 2203. 16673, 2022.
[56] Ali Taghibakhshi, Sharath Turuvekere Sreenivas, Saurav Muralidharan, Marcin Chochowski, Yashaswi Karnati, Raviraj Joshi, Ameya Sunil Mahabaleshwarkar, Zijia Chen. Yoshi Suhara, Oluwatobi Olabiyi, al. Efficient hybrid language model compression through group-aware ssm pruning. arXiv preprint 0rXiv:2504.11400 2025.
[57] Masayuki Tanaka. Weighted sigmoid gate unit for an activation function of deep neural network. Pattern Recognition Letters, 135:354 -594, 2020.
[58] Shengkun Tang, Oliver Sicberling, Eldar Kurtic, Zhiqiang Shen, and Dan Alistarh. Darwinlm: Evolutionary structured pruning of large language models. arXiv preprint arXiv:2502.07780, 2025.
[59] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.00 1.04006, 2020.
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[61] Sinong Wang, Belinda Li. Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 2020.
[62] Kaixin Xu, Zhe Wang, Xue Geng, Min Wu, Xiaoli Li, and Weisi Lin. Efficient joint optimization of layer-adaptive weight pruning in deep neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 17447-17457, 2023.
[63] Annan Yu, Michael W. Mahoney, and N. Benjamin Erichson. HOPE for a robust parameter- ization of long-memory state space models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.net, 2025.
[64] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint 1rXiv:1710.01878. 2017.
[65] Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12:1556-1577, 11 2024.
A Proofs
For our proofs, the following notions for LTI systems are helpful. An LTI system is controllable if
the controllability matrix C [ AB A²B A" n-1 B), i.e. rank(C) n. Moreover, the
system is observable if the observability matrix rank, i.c. rank(O) n.
[C (CA)T
(CA has full
The system is asymptotically stable if all eigenvalues of lie strictly inside the unit circle.
A.1 Parametrization with rotation matrices is dense
We present a proof for Proposition (1), which is restated here for convenience.
Proposition. For any linear such that the sequence-te so-sequence map {uz k== + {yr 1 of the perturbed model can be described by an SSM with matrices of the form (7) with a, p 12" where ñ n.
system of order n there exists an infinitesimal perturbation
14

---

Proof. Let the LTI system be given by [A, C. DJ. As we allow for infinitesimal perturbations, we can assume without loss of generality (w.lo.g), that A is diagonalizable [34, Chapter 2] and that the pair (A, B) )) is controllable [54, Proposition 3.3.12]. For any nonsingular T E R the impulse response of [A, B,C, D) and AT
CT, D) coincide. We will now construct an invertible T that brings [A, B. C, D into a form attainable by (7). By [18, Theorem 7.4.1] we can find an orthogonal I1 such that T AT is in block upper triangular form with either 1-by-1 or 2-by-2 blocks the diagonal, which contain the real and complex conjugate cigenvalues of A. respectively. Moreover, by [18, Theorem 7. 7.1.6] and since we assume diagonalizabilly, we can find a nonsingular T2 such that (T 22) A(T, is block diagonal. W.l.o.g, we can assume that each 2-by-2 block has the form
by
the 2-by-2 blocks contain complex conjugate eigenvalues. The final equivalence transformation apply a block diagonal transformation that brings the first column of (T) B into the desired form. For this, let
[bi,
b1.2
2
2
(T)T2) B
b121
where y is the number of blocks in (T)T,) A(T)T:) and b,, is either in IR IR² depending on the corresponding block dimension in matrix such that that T3 -I contains either 1/b,,1 when b,, E R or a scaled Givens rotation G when b E 2 that is constructed such that ,Gbb, = [1, 0] Note that we can construct such a rotation for b E ?R or have a finite value 1 1/b for b;, IR. since we have assumed controllability.
Now we construct T3 as block diagonal
Moreover, we have that (TTT2T3) A((,TTTT)) = (T) (TT,T)) AT))
W.l.o.g. let us assume A(T,T 22) are sorted such that a partitioning
for convenience that the blocks in (T (T)
A 0
4 (TT A
can be chosen such that A1 is diagonal and A₂ contains all 2-by-2 blocks. Thus, the system T AT,T-BB.CT B,CT, DJ with T T,T,Ta, where T-B.
B is of the form
b
:
b
b +1,2
T-B=
0
bg.+2.2
1
bg +4,+3.2
0
1
b 2222.2
0
by +22..2
where 2 E RR.m and and (2 are the number of real and the number of pairs of complex-conjugate eigenvalues in A, respectively. The final step to reach the form (7) is to add a zero row in 1 after each row in {l,..., 41} and zero column in CT after each column in (1,... ......1.. }. The system T AT by choosing P: = A, and a; O. when A, a 1-by-1 block and by setting A = |Ael, a; = (Xx: /R(()))))) when A, is a 2-by-2 block. Here X; denotes one of the complex conjugate eigenvalues of A. Since we only applied equivalence transformations and the states we add in our final step do not change the impulse response (they are uncontrollable and unobservable) the impulse response remains the same. Moreover, since the number of blocks in À is less than or equal to n, we can parametrize the system with p [PI,-. ... Pal and = (01,....) Qn where ñ <
matrix is formed from
15

---

A.2 Proof that sums of Hankel singular values are differentiable
We present a proof for Proposition (2), which is restated here for convenience.
Proposition. Given an asymptotically stable matrix A, as well as B and C such that the pairs (A. B) and (A,C) are controllable and observable, respectively, let P and Q be the solutions (2) and (3), respectively. Then the sum of Hankel singular values Di=1 0g depends smoothly on A, B, and C.
Proof. Note that A is asymptotically stable, the matrix (A @ A - I) has no zero eigenvalues, such that the inverse in (4) exists and P depends smoothly A and B.
An analogous argument establishes the smooth dependency of Q on A and C.
If all eigenvalues of PQ are simple, the result is automatic.
For non simple eigenvalues of PQ, we can use [34, Theorem 6.8] to define set of smooth functions representing the repeated eigenvalues, such that their sum differentiable as well. Note that controllability and observability of A, B, C' ensures that PQ has no zero eigenvalues.
D
B Experimental setup
For the model architecture, closely follow the setup proposed in [53] for comparability. In particular, we use the same number of layers and SSM dimensions. Moreover, as in [53], the SSM layers in between linear encoder and decoder layers, that map the dimension of the sequence (I in all our experiments) to the SSM input dimension, and the SSM output dimension to the number of classes for classífication, respectively. In after each sequence layer, we apply the same nonlinearity as [53], which is a weighted sigmoid gated unit [57]. It transforms the SSM output yt such that ÿ gelu(y)) @ sigmoid W gelu()))), with a learnable matrix W. As in [53] final output is mean- pooaeed to compress the output of the last SSM layer along the sequence length dimension to enable softmax classification of the given sequence. Each sequence layer is preceded with nonialloomaiiziiinn While in [53] two different learning rates for the SSM parameters and the other parameters are used, we use a single learning rate for all parameters. As in [53], we do not apply weight-decay to the SSM parameters (except for the feedthrough matrix D as it is not affected by our HSV regularization).
Table 2 we show the parameters used to generate our results. The parameters for dropout, weight-decay and regularization magnitude (the scalar by which we multiply our regularizer (I1)) are found via grid-search. Note that the regularization magnitude is small because it includes sum of all the HSVs in all the different layers. Since we implement our regularizer by adding it to the softmax cross coos-yyy loss we use during training, this must be scaled, appropriately.
With this setup, in our flax/nnx implementation, it takes around two hours to train an sMNIST model, around four hours for training an IMDB model, and around six hours to train an SCIFAR model on a single H100 GPU. This is slightly higher than the train times reported in [26]: which we attribute to our slow train data input pipeline. Parameters are initialized from standard normal distributions unless stated otherwise. The SSM parameters p are initialized using Gaussian distributions with mean 1.5, and standard deviation 0.25, which yields to an eigenvalue distribution similar to that of HiPPO matrices after discretization. Note that, as stated in Section 2.1, p is subsequently thresholded using a tanh nonlincarity. The matrices B and C in each state space layer are initialized with pero-mean Gaussian distributions, with standard deviation 1/Vn² 2 + m² and 1/ 1/vn2 n22 + p², respectively.
C Postprocessing details
C.1 Bisection algorithm for selecting ranks
We present our algorithm for selecting the ranks in the different layers in Algorithm 1. Given a total budget rt, distributes the state dimensions across the = I,..., L layers such that the same amount of energy is preserved in each layer. It terminates once prescribed tolerance or maximum number of iterations is reached. In our experiments, we set the tolerance to e 10-8 and the maximum number of iterations to Rmax = 100.
16

---

Table 2: Hyperparameter setup in our experiments. Depth denotes the number of sequence layers, LR the learning rate, WD the weight decay, and reg. mag. the scalar by which we multiply our regularizer (11).
depth
π1
p
m
dropout
LR
batch dim.
epochs
WD
reg. mag.
SCIFAR
6
384
512
0.2
0.001
50
200
0.3
0.00002
SMNIST
128
128
0.1
0.001
50
250
0.1
0.00001
IMDB
6
192
256
0.1
0.001
50
35
0.1
0.001
PATH
6
256
192
0.1
0.001
100
0.03
0.00000001
PATH-X
6
256
128
0.0
0.0001
16
30
0.0
0.00000001
Algorithm Bisection method for reduced state dimension determination
Require: sorted HSVs of the different SSMs [Σ, Ez...., Eel. target order r, tolerance c, maximum
number of iterations mama
Ensure: truncation order for each layer
1: Normalize each HSV vector such that Σm=1 Σj
for all Σ1, Σ,, Σg
2: Set 7/min 0, Tiax
3: Set = (Tmin + 7max)/ 3/2
4: Set k
5: Compute f as mean of Jargmin E1 > 7), argmin(E2 > > c and < Pmmax do
argmin (E, argmin.. >1)]
6: while
7: Set = (Ymin + 7mmax) //2
8: Set
++11
9:
if f > r then
Set Tmax =7
11:
else
Set Tmin =
13:
end if
14:
end while
15:
Return reduced orders jargmin(Z] > 7), argmin Eg > 7),.... argmin Eg 7)]
D Extra Results
Table 4 evaluates the computational cost of adding regularizer the SSM training procedure. When adding our regularizer with our block-wise Lyapunov-solver, the training-time only increases slightly, while the naive Lyapunov-solver leads to prohibitively high training-times. We report the increase in train time relative to unregularized training. In Figure 4, we compare our approach to all methods proposed in [26].
In Table 3, we demonstrate the runtime speed up that is obtained during inference at different truncation ratios.
In Table 5, we report median and standard deviation of the test accuracies across three different training runs, in which the models are initialized with different random seeds. Note that, importantly, for truncation ratios, where the accuracy of the original model retained (until around 80% for SCIFAR (grayscale) and sMNIST), the standard deviation is low, and it only increases after that threshold. This because after losing approximation accuracy of original SSM layers, the sequence-tor sequence maps change in different ways across the different runs, which has different impact on the test accuracy.
We also compare our HSVR approach to simple é norm regularization of the diagonal blocks to justify the computational overhead incurred when computing the Hankel singular values during
Table 3: Inference runtime ratios (SCIFAR) at different truncation ratios
trunc. ratio
50%
60%
70%
80%
90%
runtime ratio
0.66
0.57
0.51
0.45
0.40
17

---

Table 4: Relative runtimes measured over one training epoch. We report "-" when training is impossible due excessive resource consumption. Remember that the costs of naive Lyapunov solver scale as (nn)) Training time is measured in and extra run for one epoch to ensure the same hardware is used. Experiments are carried out on a single H100 GPU.
regularizer none HSVR (blocked) HSVR (naive)
SCIFAR (grayscale)
SMNIST
IMDB
1x
1x
1x
1.59x
1.12x
1.15x
27.3x
Table 5: Median and standard deviation of test accuracies [%] for HSVR.
quantity trunc. ratio
sCIFAR (grayscule)
sMNIST
IMDB
60%
70%
80%
90%
60%
70%
80%
90%
60%
70%
80%
90%
median
81.53
81.74
27.72
99.28
98.95
60.58
87.32
87.47
87.40
85.62
dev.
0.20
0.43
1.22
2.95
0.02
0.22
16.75
0.13
0.13
0.15
0.87
optimization. We compare our result to fine parameter sweep for the 4--norm regularization magnitude on the CIFAR, MNIST, and IMDB datasets in Tables 6, 7, and 8, respectively, and observe that for almost truncation ratios, our approach outperforms the simple l regularization, which in line with the system theoretical results for balanced truncation.
In Figure 4, we show a comparison of our method to all methods proposed in [26].
trunc. ratio
é,, 10-6
e,, 10-3
E1, 10-4
e,, 10
6,, 10-1
E1, 10-1
HSVR
50%
40.17
29.18
63.02
65.88
78.86
72.53
82.19
60%
38.05
28.09
57.53
61.34
77.15
69.57
81.84
70%
35.35
24.94
13.56
55.79
74.84
64.95
81.75
80%
12.28
21.92
09.95
45.64
69.90
52.67
81.37
90%
10.76
09.71
10.76
30.80
55.54
31.84
51.08
Table 6: CIFAR: Test accuracies (%) in comparison to LI regularization
18

---

trunc. ratio
(1, 10
f1, 10-5
L1, 10
l1, 10
f, 10-
1, 10
HSVR
50%
50.68
54.75
56.17
52.27
50.02
59.16
87.25
60%
51.39
55.15
50.93
50.94
50.02
56.61
87.26
70%
52.67
51.04
50.02
50.78
50.57
61.97
87.16
80%
51.74
50.99
50.06
51.66
50.16
58.72
86.97
90%
51.23
50.14
50.14
49.60
50.22
51.94
86.40
Table 7: IMDB: Test accuracies (%) in comparison to & regularization
trunc. ratio
E,, 10-6
f 10-5
(,, 10-1
L,, 10-3
L,, 10-1
&,, 10-1
HSVR
50%
58.58
66.94
98.99
99.01
99.00
95.88
99.29
60%
39.57
46.28
97.85
98.54
98.28
84.92
99.45
70%
13.94
18.88
86.50
97.53
97.09
76.81
99.22
80%
12.03
11.03
14.78
82.56
90.12
55.09
98.90
90%
11.12
10.62
9.51
13.61
37.03
11.72
86.95
Table 8: MNIST: Test accuracies (%) in comparison to l regularization
(ouns)
*6 Pruning LAST [26]
Proming unitom (26]
Pruning plvbal [66
100
100
100
60
accuracy
60
40
80
20
random chance 1UC
Cindain himoo 10E6
40
50
60
70
80
00
100
50
60
70
80
90
100
50
60
70
90
100
(x) .CIFARIO (grayscale)
(b) aMNIST dataeet
(c) IMDB dataset
80
60
chance
Tandom
truncasion maño PATH datasel
truncationn rañio & PATH datneet
Figure 4: Comparison of HSVR to methods in [26]
19

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the abstract we claim that regularizing the Hankel singular value distribution leads more compressible models, which we motivate theoretically by considering the role of Hankel singular values in model compression and we demonstrate experimental results that support our claims, In particular the statement: Up to 10x more compressible models found in Table 1, in which at truncation ratio of 80% our HSVR model retains an accuracy close to 100%, whereas the unregularized truncated model has an accuracy of around 10%.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We added the limitations Section 4.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
20

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[Yes]
Justification:
We provide proofs to Propositions 1 and 2 in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We clearly describe the experimental setup in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
21

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our jax implementation is available at www. github. www.ttu..ooooo..eeeeooeooe.oetee Con/AAgopaul/
hankelreg.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https:/nnip) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. ctc/public/guus
SoodeSubmissionPP.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify training details in Section B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For our proposed method (HSVR) we report the median of the test accuracies across three different the main text and present the standard deviations for each truncation ratio in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:
[Yes]
Justification: We provide explanation of hardware setup and execution times in Section B.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///nuripppp www.uris..cppppllli cc/public////ccccpcccccccccccccc Coublic/Etten Eetthttthhieeeeee csGuidelines"
Answer: [Yes]
Justification: We have reviewed the Code of Ethics and have ensured that the research conducted in the paper conforms with in every aspect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have stated in our conclusion that we are not expecting negative socictal impacts that specific to compression approach.
Guidelines:
23

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do anticipate such risks in our work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly credit the datasets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasttssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We clearly explain the implementation details of our layer and regularizer and a jax implementation is available at www
www.gi github. com A1 www.g//pppaaalllaaae// haaneeleg..
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The method does involve LLMs as any components.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //tp:r///.eu...eeeeeee cc/Coonferences/2255// /LLM) for what should or should be described.
26


Reviews:
[{"review_id": "XAYl0wF2ir", "reviewer": "Reviewer_1ZYF", "summary": "This paper proposes Hankel Singular Value Regularization (HSVR) for compressing SSM layers. The regularization leads to a decay in the singular values of the Hankel matrix and effectively reduces the dimensionality of the SSM layer. The authors develop an algorithm to compute Hankel singular values during training using block-diagonal rotation matrices, and demonstrate compression improvements on a subset of tasks from the Long Range Arena benchmark.", "questions": "- Have the authors considered how their method compares to a simple sparsity regularization over diagonal SSMs?\n- Can the authors please address the issue of the practical tradeoff of using the method suggested? Specifically discussing the overhead of the Hankel computation and the performance speedup.", "limitations": "Please see points and concerns raised above.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n- Important Problem: The paper addresses the relevant issue of sequential model compression, which is crucial as these models scale to larger applications.\n- Novel Theoretical Approach: The connection between Hankel singular values and SSM compressibility is an interesting theoretical topic with ties to classical system theory.\n- Technical Soundness: The manuscript appears technically sound with rigorous mathematical development. Specifically, the contribution of an efficient algorithm for computing Hankel singular values.\n- Clear Presentation: The paper is generally well-written with clear exposition of the methodology and theoretical background.\n\n\n### Major Weaknesses\n1. Missing Critical Baseline Comparison - The most significant weakness is the lack of comparison to L1 regularization on diagonal SSMs. This represents a fundamental gap in the experimental evaluation. The most obvious comparison is to diagonal SSMs which have been studied and showed to achieve comparable performance to the original SSM architecture (Diagonal State Spaces are as Effective as Structured State Spaces; Gupta et al. NeurIPS 2022).\n\nThe fact that this comparison is missing raises fundamental concerns regarding the relevance of the suggested method. L1 regularization is the most natural and obvious approach for encouraging sparsity/compressibility in diagonal systems, and the paper provides no justification for why the proposed complex rotation-based parametrization and Hankel singular value computation is necessary when a simple L1 penalty on diagonal elements could achieve similar compression goals.\nThe authors must either:\n- Demonstrate that L1 regularization on diagonal SSMs fails to achieve comparable compression\n- Provide theoretical arguments for why their approach is fundamentally superior\n- Acknowledge this as a major limitation\n\nWithout this comparison, the contribution appears over-engineered for the problem at hand.\n\n2. Practical considerations and applicability: \n- Computational overhead: The paper does not include any analysis of training time costs from computing Hankel singular values during optimization. Is this method practical?\n- Gains from compression: What is the practical application the suggested method should lead to? The authors could attempt to tie the Hankel regularization to the problem of finding the minimal degree necessary to solve a problem to a desired accuracy. As it stands, without any additional context, it is not clear if the suggested method brings any benefit without analyzing the overhead incurred by the regularization with the speedup gained by the compression.", "was_revised": true, "final_justification": "The authors have satisfied my requests and provided further experiments supporting their claims.", "rebuttal": "> The most significant weakness is the lack of comparison to L1 regularization on diagonal SSMs.\n- Experiments: We now conducted experiments with L1 regularization; see tables with new results below. In all experiments, our approach outperforms compression with L1 regularization. For high compression of 90%, the accuracy of the model using L1 regularization is consistently 20 percentage points below what our HSVR models achieve. See tables below.\n\nCIFAR:\n\n| trunc. ratio | L1 1e-5 | L1 1e-3 | L1 1e-1 | HSVR |\n| --- | --- | --- | --- | --- |\n| 50% | 29.18 | 65.88 | 72.53 | 82.19 |\n| 60% | 28.09 | 61.34 | 69.57 | 81.84 |\n| 70% | 24.94 | 55.79 | 64.95 | 81.75 |\n| 80% | 21.92 | 45.64 | 52.67 | 81.37 |\n| 90% | 9.71 | 30.80 | 31.84 | 51.08 |\n\nImdb:\n\n| trunc. ratio | L1 1e-5 | L1 1e-3 | L1 1e-1 | HSVR |\n| --- | --- | --- | --- | --- |\n| 50% | 54.75 | 52.27 | 59.16 | 87.25 |\n| 60% | 55.15 | 50.94 | 56.61 | 87.26 |\n| 70% | 51.04 | 50.78 | 61.97 | 87.16 |\n| 80% | 50.99 | 51.66 | 58.72 | 86.97 |\n| 90% | 50.14 | 49.60 | 51.94 | 86.40 |\n\nMNIST:\n\n| trunc. ratio | L1 1e-5 | L1 1e-3 | L1 1e-1 | HSVR |\n| --- | --- | --- | --- | --- |\n| 50% | 66.94 | 99.01 | 95.88 | 99.29 |\n| 60% | 46.28 | 98.54 | 84.92 | 99.45 |\n| 70% | 18.88 | 97.53 | 76.81 | 99.22 |\n| 80% | 11.03 | 82.56 | 55.09 | 98.90 |\n| 90% | 10.62 | 13.61 | 11.72 | 86.95 |\n- Theory: From system theory we know that fast decaying Hankel singular values are determining whether an LTI system is compressible or not (line 50 in the paper). This means that while L1 regularization can work too, our approach is backed up by the strong mathematical foundation of system theory. In particular, balanced truncation (which we use to compress our HSVR models) comes with the strong guarantee given by the error bound in (12). In contrast, regularization with L1 regularization and subsequent pruning/truncation can be related to ‘modal truncation’ in the system theory, which again is a viable approach but does not explicitly try to keep the error of the input-output (seq-to-seq) map low as eq (12).\n- We will add the new experiments to the paper, should this paper get accepted.\n> No analysis of the training time computational overhead. “Is this method practical?”\n- In Table 3 in the appendix we show that HSVR with the proposed solver incurs costs of 1.12x to 1.59x of the costs of having no regularization, which indicates that the training overhead is manageable.\n- We stress that the training overhead is so low because of the algorithms that we developed in this work (Section 2.2). Off-the-shelf techniques are orders of magnitude slower (as we also report in Table 3).\n> No demonstration of gains from compression\n- First, we note that compression alone already reduces storage requirements. This can be useful if networks are meant to be embedded in large applications.\n- Second, there is an inference runtime speedup because the evaluation becomes cheaper when there are fewer parameters. We conducted new experiments and we can report runtime improvements in the appended table, which shows the runtime ratio alongside the retained accuracy for the sCIFAR example. For example, for the 80% truncation, we maintain an accuracy of 81.37 but reduce the inference runtime by 55%. We will add these new results as well as for the other examples on inference runtime speedups in the paper, if it gets accepted.\n\n| trunc. ratio | HSVR (accuracy) | HSVR (runtime ratio) |\n| --- | --- | --- |\n| 50% | 82.19 | 0.66 |\n| 60% | 81.84 | 0.57 |\n| 70% | 81.75 | 0.51 |\n| 80% | 81.37 | 0.45 |\n| 90% | 51.08 | 0.40 |"}, {"review_id": "m9lMUuDH4R", "reviewer": "Reviewer_3qrK", "summary": "This work (HSVR) discussed Hankel singular value regularization (HSVR) for state space models (SSM) in order to make SSMs more compressible. The author first introduced a regularizer based on the Hankel singular value, which aims to make the model more compressible while balancing the task accuracy. The authors studied how to efficiently compute HSVR under the rotation matrices parameterized SSMs as previously proposed in the HiPPO framework. Showing the final output error bound is upper limited by terms involving summation of Hankel singular values across layers, the authors discussed how to compress with balanced truncation across different layers in a single model. Empirical experiments were conducted on several long-range arena tasks and showed that the HSRV method outperformed a major previous SSM regularization method and no regularization baselines at different truncation ratios.", "questions": "Questions:\n\n1. The weight decay rate selected in Table 2 is quite different from the LAST paper. Were the LAST results in this work using the same weight decay as the HSVR method? How does it interact with HSVR for different weight decay?\n\n2. Line 251, what is $\\u_k$? Should it be input $x_k$?. And for completeness, could you provide proof for equation 12? \n\n3. While applying tanh on the vector \\rou could enforce stability, does it hurt the performance? Is there an ablation study supporting the benefits?", "limitations": "Yes: the authors have adequately addressed the limitations and potential negative societal impact of their work", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Deducted the formulation for computing the Hankel singular values, exploiting the block-diagonal structure, which brings down the complexity from $n^3$ to $n^2$.\n\n2. The work discussed the compatibility and complexity of the proposed methods in the SSM paradigm, including the regularizer, associative scan, and post-training compression.\n\n3. Heuristic results on limited tasks show a promising truncation-performance trade-off, superior to one major baseline. \n\nWeaknesses:\n\n1. The author did not compare to the work [1], which studies the SSM initialization and training stability with the Hankel operator. The authors also developed a parameterization scheme that utilizes Markov parameters within Hankel operators. Considering the close relationship between this work and [1], a detailed comparison in theory and experiments should be included. If the authors could articulate the difference and provide an experimental comparison, I would consider raising the score for originality and overall score.\n\n2.. LAST covered more experiments from the long range arena. It would be nice to also include them in the benchmarks, especially for Pathfinder and Path-X. Otherwise, there are in total 3 tasks from LRA that have been empirically examined, which is quite lacking. From the training hours provided in the LAST paper, the training resources needed for Pathfinder and PATH-X are on the same level as sCIFAR. \n\n3. The method is a model pruning/ compression method. But the author did not show real inference speed up or theoretical FLOPs numbers.\n\n\n[1] HOPE for a robust parameterization of long-memory state space models, Yu et al. 2025 ICLR https://openreview.net/pdf?id=RZwtbg3qYD    on arxiv in 2024 https://arxiv.org/pdf/2405.13975", "was_revised": true, "final_justification": "This work proposed a different parameterization schema as [1], which resulted in a better compression rate than [1], despite the fact that  they both inspect SSM through the Hankel operator. Considering the significance of these 2 works, I am raising the score by 1 point. The authors should include a clear comparison and citation of [1] in the final version if upon acceptance.\n\nThanks to the author for the additional experimental result. However, similar to Reviewer exo8’s comment, the experimental evaluation is rather limited compared to works alike.", "rebuttal": "> The author did not compare to the work (Yu et al. 2025 ICLR), which studies the SSM initialization and training stability with the Hankel operator. [...]  Considering the close relationship between this work and (Yu et al. 2025 ICLR), a detailed comparison in theory and experiments should be included. [...]\n- We thank the reviewer for sharing this interesting work with us. The work cited by the reviewer introduces a new parametrization of SSMs which achieves compression of 3x of the parameters (from $3n^2$ to $n^2$) of a regular S4D/S4 model. However, we stress that we achieve compression rates of 10x (1/10 of the parameters of the uncompressed model) with our approach. We achieve these higher rates because we not only use a favorable parametrization as (Yu et al. 2025 ICLR) but additionally introduce a regularization that nudges the optimization to find compressible models.  \n- We will cite this work and add this comparison to our paper, if our work gets accepted.\n\n> It would be nice to also include [...] Pathfinder and Path-X.\n- We are adding results for the Path example to the paper. We attain for example 77.46% accuracy for 50% compression ratio which outperforms the best result in [Gwak, 2024] (LAST), which is at 56.45% for the same compression ratio. Notice that in this example all methods achieve lower accuracy. Our method can shine in situations when accuracy is high, in which case we show that we can compress more than other methods (see Table 1 in the paper). \n\n| trunc. ratio | LAST | Global H-inf | Uniform H-inf | HSVR (ours) |\n| --- | --- | --- | --- | --- |\n| 50% | 56.45 | 50.10 | 55.61 | 77.46 |\n| 60% | 50.51 | 49.16 | 50.32 | 65.93 |\n| 70% | 50.11 | 50.15 | 49.78 | 63.64 |\n| 80% | 49.96 | 50.16 | 49.84 | 50.50 |\n| 90% | 50.25 | 49.37 | 50.16 | 49.28 |\n\n> [...] author did not show real inference speed up or theoretical FLOPs numbers.\n- We conducted new experiments and we can report runtime improvements in the appended table, which shows the runtime ratio alongside the retained accuracy for the sCIFAR example. For example, for the 80% truncation, we maintain an accuracy of 81.37 but reduce the inference runtime by 55%. We will add these new results as well as for the other examples on inference runtime speedups in the paper, if it gets accepted.\n\n| trunc. ratio | HSVR (accuracy) | HSVR (runtime ratio) |\n| --- | --- | --- |\n| 50.00 | 82.19 | 0.66 |\n| 60.00 | 81.84 | 0.57 |\n| 70.00 | 81.75 | 0.51 |\n| 80.00 | 81.37 | 0.45 |\n| 90.00 | 51.08 | 0.40 |\n\n> Were the LAST results in this work using the same weight decay as the HSVR method? How does it interact with HSVR for different weight decay?\n- We have extracted the LAST results directly from the LAST paper so the setup is exactly the same and was not modified. In our experiments, using weight-decay prevented the model from overfitting.\n\n> Line 251, what is u_k? Should it be input x_k?\n- We denote with u_k the input and with x_k the state. The seq-to-seq map goes from input u_k to output y_k, where x_k is the internal state. It is correctly written on line 251.\n\n> And for completeness, could you provide proof for equation 12?\n- Proving the error bound of balanced truncation is involved, but we will include a reference to Section 7.2.1 in A.C. Antoulas, Approximation of Large-Scale Dynamical Systems, SIAM, 200, which shows a proof. \n\n> While applying tanh on the vector \\rou could enforce stability, does it hurt the performance? Is there an ablation study supporting the benefits?\n- Training without using tanh here is very unstable because the optimizer easily lands in regions where the \\rho becomes numerically infinite and the optimization cannot recover. Thus, an ablation study shows very poor performance without tanh."}, {"review_id": "bsTo82aaHR", "reviewer": "Reviewer_exo8", "summary": "This paper proposes Hankel singular value regularization (HSVR) and a post-training optimization for state space models (SSMs) parameterized with block rotation matrices, inspired primarily by the balanced truncation in the model order reduction literature. HSVR induces trained systems to have fast-decaying Hankel singular values (HSVs), meaning that the energy flow is concentrated in a small number of states, by regularizing the sum of HSVs. After training, the method performs balanced truncation, eliminating low-energy states. The block rotation matrix parameterization is introduced to enable efficient computation of HSVs during training. The method is validated on selected tasks from the Long Range Arena (LRA) benchmark.", "questions": "1. **(Block rotation parameterization)** Why is the rotational parameterization necessary for HSVR? In other words, why didn’t the authors simply parameterize the SSMs as diagonal continuous-time SSMs with discretization? While the rotational parameterization does have the benefit of directly ensuring a stable discrete-time SSM, isn’t it also possible that diagonal SSMs can compute HSVs efficiently in $O(n^2)$? It is thus unclear whether the claimed advantage of efficient HSV computation justifies the use of the rotational SSMs. An ablation study or a comparative analysis of the computational cost between diagonal and rotational SSMs would help clarify this point.\n2. **(Balanced truncation)** If the system is not yet balanced, isn’t it appropriate to define and compute the HSVs as $\\sqrt{\\lambda_i(PQ)}$, not as $\\sigma_i(PQ)$? However, the in-training regularization process before the balanced transformation is described using $\\sigma_i(PQ)$ (e.g., Line 122, Line 182). Was this the actual implementation of HSV computation during training? Moreover, do the compressed SSMs have different parameters from the uncompressed SSMs, applied by balanced transformation?\n3. **(Regularization)** In the absence of additional bias, how can one ensure that regularizing nuclear norm (penalizing the sum of HSVs) promotes fast decay in HSVs rather than uniformly reducing all HSVs? The claim that ”Nuclear norm regularization, i.e., penalizing the sum of singular values of a matrix, to encourage singular values to decay rapidly, is common practice in machine learning.” (Lines 230-231) would benefit from citation of relevant references in the machine learning literature.\n4. **(Experimental verification)** Can the proposed method be applied to longer sequence modeling tasks? Specifically, while results are shown on sMNIST and selected LRA tasks (sCIFAR10, IMDB), can the result be extended to the full LRA benchmark or other long sequence datasets?\n5. **(Correctness and consistency of equations)** Many **Lyapunov, Sylvester, and vectorized equations** throughout the paper appear to be written wrong or deviate from standard formulations in terms of sign (e.g., Equations (2)-(5), Lines 174-176). Moreover, the notation for $A(\\text{angle}, \\text{radius})$ or $A(\\text{radius}, \\text{angle})$ should be unified across Sections 2.1 and 2.2 for consistency.", "limitations": "Limitations are already addressed in the paper ((a) inapplicability to pretrained models, (b) linear compression, and (c) time-invariant models).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- **Strength 1**: HSVR is grounded in the well-established balanced truncation framework, empirically demonstrating effective compression of trained SSMs.\n- **Strength 2**: The block rotation matrix parameterization introduces a novel class of SSMs, which can be efficiently implemented via associative scan by leveraging the mathematical properties of rotation matrices.\n- **Weakness 1**: While diagonal SSMs are the dominant choice in recent literature, the necessity of using block rotation matrix parametrization in conjunction with HSVR remains unclear. A comparison or ablation study on different parameterization methods would strengthen the claim.\n- **Weakness 2**: The scope of experimental validation is limited. Only a subset of the LRA dataset was used, and there is a lack of experiments on tasks with longer sequences.\n- **Weakness 3**:  Several equations throughout the paper (e.g., Lyapunov, Sylvester, and vectorized forms) raise concerns about the correctness and clarity of mathematical presentation.", "was_revised": true, "final_justification": "My initial Weak Accept score was based primarily on the contribution and empirical effectiveness of HSVR. Although the authors have made a careful rebuttal and offered more details on the new system parameterization and experiments, I remain unconvinced that the proposed parameterization is necessary. Moreover, the experimental additions are partial. Thus, I maintain my Weak Accept.", "rebuttal": "> While diagonal SSMs are the dominant choice in recent literature, the necessity of using block rotation matrix parametrization in conjunction with HSVR remains unclear. A comparison or ablation study on different parameterization methods would strengthen the claim.\n- The purpose of the block structure is to stay in real arithmetic. What we show is that in this parametrization, we can efficiently compute Hankel singular values as well. We also stress that the block rotation matrix parametrization is used for training purposes only. But once trained and compressed, we transform the system back into a diagonal system. \n> Only a subset of the LRA dataset was used, and there is a lack of experiments on tasks with longer sequences. [...] Can the proposed method be applied to longer sequence modeling tasks?\n- We demonstrate our approach on a range of examples and achieve in all experiments significant compression, outperforming other approaches. We agree that even more experiments could make our argument stronger. We therefore also computed the Path test case (see table below) We attain for example 77.46% accuracy for 50% compression ratio which outperforms the best result in [Gwak, 2024] (LAST), which is at 56.45% for the same compression ratio. Notice that in this example all methods achieve lower accuracy. Our method can shine in situations when accuracy is high, in which case we show that we can compress more than other methods (see Table 1 in the paper). \n\n| trunc. ratio | LAST | Global H-inf | Uniform H-inf | HSVR (ours) |\n| --- | --- | --- | --- | --- |\n| 50.00 | 56.45 | 50.10 | 55.61 | 77.46 |\n| 60.00 | 50.51 | 49.16 | 50.32 | 65.93 |\n| 70.00 | 50.11 | 50.15 | 49.78 | 63.64 |\n| 80.00 | 49.96 | 50.16 | 49.84 | 50.50 |\n| 90.00 | 50.25 | 49.37 | 50.16 | 49.28 |\n\n- All experiments that we conducted indicate that we achieve favorable compression ratios while maintaining accuracy. Thus, while there are more experiments we could conduct, we believe the current ones and the new additional one provide a strong indication that our approach is meaningful - additionally to the strong theoretical foundation provided by system theory.\n\n> If the system is not yet balanced, isn’t it appropriate to define and compute the HSVs as \\sqrt{\\lambda_i(PQ)) instead of \\sigma_i(PQ). [...]\n- Yes, the reviewer is correct, sqrt(lambda(PQ))=sigma(PQ) only when PQ is normal, which is not in general the case. We are computing sqrt(lambda(PQ)) in our code and we will adapt the lines 122, 182, accordingly. We will update the text, if this work gets accepted.\n\n> [...] how can one ensure that regularizing nuclear norm (penalizing the sum of HSVs) promotes fast decay in HSVs rather than uniformly reducing all HSVs?\n- The nuclear norm is the best convex approximation of the rank function (Fazel, PhD thesis, 2002). The idea is similar to using the \\ell_1 norm instead of the \\ell_0 norm in sparse recovery. The nuclear norm is used, for example, for matrix completion in the famous paper by Candes and Recht (FoCM, 2009), which is essentially about solving the “netflix problem” in machine learning. \n\n> Many Lyapunov, Sylvester, and vectorized equations throughout the paper appear to be written wrong or deviate from standard formulations in terms of sign (e.g., Equations (2)-(5), Lines 174-176).  [...] notation for A(angle, radius) and A(radius, angle) should be unified across Sections 2.1 and 2.2 for consistency.\n- Yes, thank you very much for catching this. We will update the Lyapunov equations to APA^T - P + BB^T = 0 and  A^T Q A - Q + C^T C=0. We will unify the notation to A(radius, angle) for consistency."}, {"review_id": "j2rAwqYNc9", "reviewer": "Reviewer_1JxP", "summary": "The paper introduces Hankel Singular Value Regularization to improve the compressibility of state-space models. \nThe proposed regularization encourages rapid decay of Hankel singular values, making the SSM layers more compressible. \nThe authors also develop an efficient algorithm to compute this regularization during training. \nExperimental evaluations demonstrate significant improvements in compressibility compared to unregularized models.", "questions": "How would applying your proposed truncation ratio affect performance specifically in architectures with relatively small SSM components, such as LRUs (Orvieto et al., 2023) or the Mamba architecture (Gu et al., 2023)?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 1, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* Clearly written and well structured.\n* Comprehensive experiments that validate the effectiveness of the proposed method.\n* Offers a simple and intuitive solution.\n\n**Weaknesses**\n* The motivation for this research appears weak. Specifically, the portion of SSM parameters in contemporary architectures like Mamba (Gu et al., 2023) and other non-selective SSM variants such as LRUs (Orvieto et al., 2023) is minimal relative to the total network parameters. For instance, Mamba explicitly states most parameters reside in linear projections rather than the SSM component, which is relatively small (with hidden dimensions typically set at 16). Thus, the actual practical gain from compressing only the SSM parts may be limited.\n* The paper would greatly benefit from demonstrating significant model-size reductions in realistic, widely-used architectures.", "was_revised": true, "final_justification": "Final justification for score:\n\nAs noted in the rebuttal thread, I remain unconvinced by the practical usability of the method. The paper targets compression of SSM components, but in realistic architectures like Mamba, these components represent a small fraction of total parameters—e.g., <4%.\n\nFurthermore, their regularization approach assumes time-invariant dynamics, which does not hold for Mamba's selective (time-varying) SSMs.\n\nThe response did not resolve these concerns. I see potential in the direction but believe stronger evidence is needed for real-world impact.", "rebuttal": "> [...] the portion of SSM parameters in contemporary architectures like Mamba (Gu et al., 2023) and other non-selective SSM variants such as LRUs (Orvieto et al., 2023) is minimal relative to the total network parameters. [...] Thus, the actual practical gain from compressing only the SSM parts may be limited.\n- The reviewer brings up a good point: there are other parameters in network architectures that our approach does not compress. We agree with this statement. However, we also would like to point out that for example Mamba-2 (Dao, Gu, 2024) uses a state-space dimension of 256, which is similar to the sizes we use. Furthermore, also the paper that the reviewer cites (Orvieto et al., 2023; Table 10) considers models with state-space dimension in the 100s. Given these numbers from the literature and that we also use state-space dimensions of similar size, we do believe that compressing the SSM parameters can be beneficial also for realistic models. \n- In any case, if this work gets accepted, we will add a clear statement to the limitation section that our approach compresses the SSM parameters but that many models have other parameters that we do not compress.\n> The paper would greatly benefit from demonstrating significant model-size reductions in realistic, widely-used architectures.\n- We consider standard benchmark problems for long-range tasks and show compression ratios of 10x while maintaining more than 90% of the accuracy. While these are not architectures with as many parameters as, e.g., Mamba-2, we see similar dimensions of state-space models in other architectures (see previous comments) and thus believe our results indicate that our approach is promising."}, {"review_id": "zJxMQ4umZo", "reviewer": "Reviewer_6j3S", "summary": "This is a strong paper overall, clearly written, dense with results (both mathematical and empirical).\n\nThe major contribution is to develop a new regularization approach that improves the compressibility of deep state space models (SSMs). The method achieves a 10x improvement in compressibility (compared to a baseline with no regularization).  These SSMs have become increasingly popular recently, so this work has good potential applied impact.", "questions": "One question re: novelty: there is a relevant paper by Forgione et al 2024 (\"Model order reduction of deep structured state-space models: A system-theoretic approach\") - if this isn't by the same authors, there should be some discussion of this paper.\n\nIn the abstract, the authors claim \"HSVR state space layers are up to 10× more compressible than standard state space layers trained without regularization\" - a more informative statement here would be to compare against the SOTA compression approach, rather than an approach with no regularization.", "limitations": "adequately addressed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Repeating myself from above -\n\nThis is a strong paper overall, clearly written, dense with results (both mathematical and empirical).\n\nThe proposed method achieves a 10x improvement in compressibility.  These SSMs have become increasingly popular recently, so this work has good potential applied impact.\n\nNo major weaknesses noted.", "was_revised": true, "final_justification": "I'll stick with my original score.  I remain enthusiastic about this paper.", "rebuttal": "> There is a relevant paper by Forgione et al 2024 (\"Model order reduction of deep structured state-space models: A system-theoretic approach\") - if this isn't by the same authors, there should be some discussion of this paper.\n- We thank the reviewer for bringing this interesting work to our attention. Indeed the authors also follow a system-theoretic approach and regularize Hankel singular values, which is a natural idea when balanced truncation is used for compression. The main methodological difference to our work is that they use a complex-valued diagonal parameterization, whereas we build on real block-diagonal parametrization. Our block-diagonal parametrization has advantages in terms of stability (see Section 2.1) but requires a more careful approach for computing Hankel singular values efficiently (Section 2.2). We also believe that our approach achieves better compression rates because we propose an energy-bisection approach (Section 2.4) for determining the reduced rank in each layer considering the entire DNN while Forgione et al. choose the same reduced ranks in each layer. In any case, we completely agree with the reviewer that this paper is very relevant and shares the idea of regularizing the Hankel singular values. We will certainly cite this paper, update the Introduction and contributions of our paper, and provide a detailed discussion in our paper, if it gets accepted.\n\n> In the abstract, the authors claim \"HSVR state space layers are up to 10x more compressible than standard state space layers trained without regularization\" - a more informative statement here would be to compare against the SOTA compression approach, rather than an approach with no regularization.\n- We will extend the abstract and state that we maintain high accuracy of 99% even at high compression rates such as 80% where other compression methods have a loss of accuracy to 60% (LAST)."}]

Author rebuttal:

Meta-review: The authors present a method for regularizing deep SSMs with time-invariant dynamics to use lower dimensional hidden states. Their method is based on well-established principles for model order reduction based on the Hankel singular values, but they show how those singular values can be computed efficiently for SSMs with block diagonal dynamics matrices. The paper addresses an interesting and important topic, the presentation is clear, and the results are compelling.

Some reviewers leaned towards rejecting the paper on the grounds that it: 1) did not originally compare against L1 regularized diagonal dynamics matrices (1ZYF); 2) does not address linear time-varying layers like Mamba (1JxP); and 3) may not be as impactful in models where SSM layers constitute a smaller fraction of total parameter count (1JxP). I think the authors did a good job of rebutting the first criticism, and while I think the 2nd and 3rd points are valid, the strengths of this paper outweigh those limitations. Indeed, three other reviewers found this paper to be a significant contribution worthy of acceptance, and I agree with them.

Minor comments: 
- Lines 35-38 imply that $u_k \in \mathbb{R}^p$, but you also seem to use $m$ for the input dimension. I would suggest introducing the inputs as $m$-dimensional and then noting (as you do later) that you assume $m=p$ for analysis, since that is the common design choice.
- Reviewer exo8 made a reasonable point that the block diagonal parameterization of a real-valued A matrix looks equivalent to a diagonal complex-valued A matrix since the eigenvalues come in complex conjugate pairs. Since readers will likely be familiar with the complex diagonal parameterization in S5, I would suggest making this correspondence more clear. 
-  Reviewer 6j3S mentioned an important related work of Forgione et al '24. Please cite and discuss the relationship to your work in the final manuscript.
Decision: Accept (poster)