Layer-Adaptive State Pruning for Deep State Space Models
Authors: Minseon Gwak
Seongrok Moon
Joohwan Ko
PooGyeon Park
Venue: NeurIPS 2024 poster (2024)
Keywords: Sequence model
model order reduction
model compression
network pruning
system norm
long range arena
Primary area: other
TL;DR: We propose a structured layer-adaptive pruning method for deep state space models.

Abstract: Due to the lack of state dimension optimization methods, deep state space models (SSMs) have sacrificed model capacity, training search space, or stability to alleviate computational costs caused by high state dimensions. In this work, we provide a structured pruning method for SSMs, Layer-Adaptive STate pruning (LAST), which reduces the state dimension of each layer in minimizing model-level output energy loss by extending modal truncation for a single system. LAST scores are evaluated using the $\mathcal{H}_{\infty}$ norms of subsystems and layer-wise energy normalization. The scores serve as global pruning criteria, enabling cross-layer comparison of states and layer-adaptive pruning. Across various sequence benchmarks, LAST optimizes previous SSMs, revealing the redundancy and compressibility of their state spaces. Notably, we demonstrate that, on average, pruning 33\% of states still maintains performance with 0.52\% accuracy loss in multi-input multi-output SSMs without retraining. Code is available at https://github.com/msgwak/LAST.

Layer-Adaptive State Pruning for Deep State Space Models
Minseon Gwak Seongrok Moon', Joohwan Ko', PooGyeon Park DDeparmeent of Electrical Engineering, POSTECH Department of Computer Science, University of Massachusetts Amherst (mi nseon25, s srmoon, ppg) @postech. ac. kr, joohwanks&cs umass.
Abstract
Due the lack of state dimension optimization methods, deep state space models (SSMs) have sacrificed model capacity, training search space, or stability to allevi- ate computational costs caused by high state dimensions. In this work, we provide a structured pruning method for SSMs, Layer-Adaptive STate pruning (LAST), which reduces the state dimension of each layer in minimizing model-level output energy loss by extending modal truncation for single system. LAST scores are evaluated using the Hos norms of subsystems and layer-wise energy normalization. The scores serve global pruning criteria, enabling cross-layer comparison of states and layer-adaptive pruning. Across various sequence benchmarks, LAST op- timizes previous SSMs, revealing the redundancy and compressibility of their state spaces. Notably, we demonstrate that, on average, pruning 33% of states still main- tains performance with 0.52% accuracy loss in multi-ing input multi-output SSMs with- out retraining. Code is available at https: / //g gith.b. com homm/mwwwkk / LAST.
Introduction
Deep state space models (SSMs) have proven effective in modeling sequential data by optimally compressing input history internal states [Gu et al., 2020, 2021, 2022b, Gu and Dao, 2023, Zhang et al., 2023, Parnichkun et al., 2024]. Given their modeling capabilities, ensuring the feasibility and stability of SSMs during training has become a crucial research focus for achieving efficient learning without divergence. Leveraging the knowledge founded in linear system theory [Kailath, parameterization [Gu
1980], various advancements have emerged, including et al., 2022a], general system architecture [Smith et al., 2023], and efficiency improvements via frequency-doma operations, utilizing the fast Fourier transform and the transfer functions systems [Gu et al., 2022b.a, Zhang et al., 2023, Parnichkun al., 2024].
One of the main computation and memory contributors of SSMs is the state dimension n. Since the initial proposal of SSMs, multiple single-input single-output Amulti-SISO, architecture has been employed for scalable and efficient training Gu et [2022b,a], Gu and Dao [2023], Zhang et al. [2023], Parnichkun et al. [2024]. In this architecture, rather than directly learning an n-dimensional system, -imenso-dimeniional SISO systems are trained in parallel and then integrated through channel-mixing layer. Within this structure, Gupta et al. [2022] presented that diagonal systems can achieve matching performance to nondiagonal systems. Gu et al. [2022a] introduced a stability- guaranteed model, where the diagonal systems are trained to satisfy the necessary and sufficient stability condition.
Instead of utilizing multiple SISO systems in parallel, Smith et al. [2023] adopted a multi- input multi-output (MIMO) architecture, where the enhanced information usage through a MIMO system.
"corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

LAST score
c
B
36 (z,² 22))
A(1)
((1)
C
2 Noe (x) 00
Pruned SSMs
Sorted by
Ha score
Layer
(22)
(2)
((2)
Energy normalization
Sorted by Hip score
Layer 2
Cross-layer comparison
Figure 1: Illustration of LAST for two layers. Matrices are divided by lines on per-state basis, and subsystems are sorted in descending order by their Ho norms. LAST scores are obtained by normalizing each 2o norm by the sum of all Hoc norms in a layer when the states with lower Hoc norms are excluded. Since LAST scores correlate with model-leve output energy loss, we prune all parameters corresponding to states with low LAST scores.
This architecture provides high performance with much smaller state dimensions than equivalent block systems in multi-SISO layers. For instance, in the Path-X X task that involves the longest tested sequences, this architecture showed ttateoof-the-an performance [Smith et al., 2023, Parnichkun et al., 2024]. However, both architectures lack optimization methods for state dimensions, leading to inefficiencies when the model is over aver-parameterizee for the task.
Recently, Parnichkun et al. [2024] parameterized transfer functions of SISO systems and proposed state-free inference. However, this approach indirectly trains the poles of the transfer functions, resulting restrictive search space or stability being guaranteed only at initialization.
Focusing on the tuabiitt-guaranteed diagonal SSMs, we develop and verify a layer-adaptive model order reduction (MOR) method for SSMs to identify the least significant states subsystems in terms of their impact on task performance. Inspired by layer-adaptive neural network pruning [Evci al., 2020, Lee et al., 2021, Xu al., 2023] and extending the traditional MOR for single system [Green and Limebeer, 2012], we propose Layer-Adaptin scores for learned states are evaluated and used as global pruning criteria. LAST scores measure the relative maximum frequency-domain gain of each subsystem when subsystems with lower scores are excluded, as illustrated in Figure LAST prunes insignificant subsystems to achieve desired compression level, reducing unnecessary computational and memory costs while bounding the output distortion by the Hop norms of the pruned subsystems.
STate pruning (LAST), where importance
We validate the insignificant state identification performance of LAST on long-range sequences, including Long Range Arena (LRA) [Tay al., 2021] and Speech Command [Warden, 2018] benchmarks. Our results present that previous SSMs have great compressibility. demonstrating that pruning 33% (26.25%) of the trained states resulted in only 0.52% (0.32%) of accuracy loss MIMO models) on average, including the non conpcoppres cases.
models (in
Background
2.1 Stability of state space models
DT SSM is stable if all poles, roots of a denominator, of its transfer function lie within the unit circle. However, it is challenging to train systems to ensure stability at every step. One approach for this issue is to confine the search space to sufficient stable region [Zhang et al., 2023], as illustrated in Figure 5 for second-order linear time-invariant (LTT) system. Due to the restricted search space, training under this condition can limit model performance [Parnichkun et al., 2024]. Another approach is initializing the system the center of the stable region, marked in Figure 5, referred

---

to as zero initialization in [Parnichkun et al., 2024]. While this approach mitigates the performance limitation, the stability guaranteed only at initialization.
In contrast, diagonal SSMs [Gupta et al., 2022, Gu et al., 2022a, Smith et al., 2023] directly pa- rameterize the system poles, enabling the model explore all expressible systems that possess stability-satiifying poles. Thus, all our derivations are based on the diagonal SSMs to leverage the guaranteed stability, which allows for the application of various system analysis techniques. Detailed explanations on the stability regions are provided in Appendix A.I.
2.2 Diagonal state space models
Architectures. Diagonal SSMs consist of an encoder that increases the number of input channels to h, L SSM layers, and a decoder for the downstream task. Each SSM layer can be designed with either multi-SISO or MIMO architecture. In the multi-SISC architecture [Gu et al., 2022a], independent systems are trained for each input channel, with total of / n, th-hrrder SISO systems being learned in layer. A fully connected layer is then used to mix features from different channels. In contrast, the MIMO architecture [Smith et al., 2023] employs an 7.m th-order MIMO system within each layer, handling b-dimensional input and output signals. As noted in Smith et al. [2023], SISO systems in layer can be represented as one MIMO system, where specific states are assigned to each input channel. Therefore, we describe SSM layers using MIMO expressions, defining the effective total state dimension for an SSM layer by n, where n = ngh for a multi-SISO layer and n = 7mm for MIMO layer.
Parameterization.
The learnable parameters in a diagonal SSM layer with state dimension n CT system matrices A C"**, B E Caxh, C Chx", D Raxh, where A E C"x* is diagonal matrix and complex-valued matrices consist of elements that form conjugate pairs to handle real-valued signals [Gu et al., 2022a]. In the diagonal structure, each subsystem is discretized by applying different timescales from A E IR" to process discrete sequences. By zero-order hold (ZOH) discretization [Chen, 1984], discretized LTT diagonal system Σ + y in a layer fa Σ) can be represented as follows:
Xk+1 = Axe: + Bug
yk =k= = Cx + Du..
where A = AA and B = A B are the discretized system matrices, X E C is a state vector, u2 E I2'h is an input signal, and yx E I is an output signal. The stability the discretized system can be achieved by ensuring the stability of the CT parameters with Hurwitz parameterization [Gu et al., 2022a], as derived in Appendix A.2. Finally, nonlinear activation function o(-) is applied to the output of the linear system, i.e., fo (ug; Σ) (E(ue))) introducing nonlinearity to the SSM.
2.3 Hox norms of systems
In robust control, the Hos norm is widely used to minimize the worst-case gain from disturbances to outputs, ensuring stability and performance under system uncertainty [Qin and Sun, 2023, Zheng al., 2023]. In this work, we use the H.s norm to measure the divergence between the original and approximated systems.
For DT LTT system E E:ur y with the transfer function matrix (TFM) G, the Hoc norm of the system is defined by
||
sup 7 (GG())))
@c(0.2m)
where 전 denotes the maximum singular value of a matrix. In robust control design, the Hoo norm frequently minimized to design controllers that ensure the system performs optimally under disturbance.
In this work, utilize the following important property of the Ho norm, that is, the energy of the output signal ||y 22 can be bounded with the squared Hos norm and the energy of the input signal | i.e.
1yl3 < |
(2)
(See Appendix B. for derivation). other words, the Hos norm of system measures the maximum gain of the system, which is useful in assessing the energy loss caused by pruning.
3

---

3 LAST: Layer-adaptive state pruning for SSMs
We propose structured pruning for SSMs with per-state pruning granularity, where all parameters associated with the identified insignificant state are pruned. Although pruning is implemented by masking, we represent pruned systems with their effective remaining states and parameters.
In Section 3.1, we derive a local pruning criterion by evaluating the layer-level energy loss for single SSM layer, which consists of MIMO system followed by nonlinear activation. In Section 3.2, we extend this to a global pruning criterion by assessing the model-level energy loss when considering multiple stacked SSM layers.
3.1 Hoc scores as local pruning criteria
From DT system Σ (A, B, C), suppose we prune the ith subsystem E, (Ag, B, C2) correspond- ing to the ith state x,, leaving the ith state-pruned system Σ-,. Specifically, the state-pruned system can be written as follows:
A ; = diag(
A
A1-1, A--1X "
An):
Σ
B ; B
T
B B
B ...
1,
m
-_ C
C
C.1
C,
where A diag(A1. Am). B
= I ... ............... B, with C=
Ior B,
Cl and Ch Our objective is to minimize the layer-level output energy loss, defined as the squared Eg distortion in the output signal, incurred by the system approximation through state pruning. The optimization is formalized by
minimize PCS
Iu: - fo Ep:E1))1
subject 1
where S = [1,... n) } is the set of state indices in the full system, P is the set of pruned state indices, and indicates the required level of model reduction.
Using the properties of diagonal systems and the Hoo norm in Equation (2), the energy loss can be bounded follows:
(u; fo (u: - Σ GG, |ul
SEP
where -P = == S \ P and G; is the TFM of Σ, (See Appendix B.2 for proof). Therefore, we can reduce a system by pruning subsystems with small Hos norms, minimizing the upper bound in Equation (3). This result shows that, even in the presence of nonlinearity, pruning for a single layer can performed similarly to modal truncation [Green and Limebeer, 2012]. As the stability is guaranteed by Hurwitz parameterization [Gu et al., 2022a], the Hoo norm of subsystem is evaluated as follows:
C. |G 1 A
Hence, the importance of x; can be defined by the squared H. norm of E, with a minor optimization of computational efficiency for rank-1 matrix C,B, as follows:
| B
Hoo (x;; Σ)
(1 1x,))
where Hox (x,; refers to the Hoc score of x;, and we prioritize pruning states with lower scores. This can also be simplified with |2 = 1 when B is fixed while C is trained. Moreover, when two matrices are used for bidirectional SSMs, || |/2l2 can be substituted as the average for the two matrices, i.e., |2 = (IIC 12 + | where ||C| is for forward direction and ||C]] is for backward direction.
The H.x score can be used as a local pruning criterion once the target pruning ratio for each layer is determined. However, this approach has limitations, as it requires a heuristic to determine the pruning ratio for each layer and applies the same amount of pruning without considering layer-specific characteristics.

---

3.2 LAST scores as global pruning criteria
To extend the local pruning criterion global pruning criterion, we now consider the model-leve output energy loss incurred by pruning L layers. In the following description, superscripts indicate layer indices from
for signals, systems, and state index sets.
Following the notation in Lee et al. [2021], Xu et al. [2023], the output of kth layer is obtained by recursively applying the preceding systems and activation functions follows:
Jo (u )( x(2))) = (I
(Sa( ())) (
Our objective is to minimize the model-level output energy loss follows:
minimize CS0)
((1t)) - Ilfo (
fo
subject to El IPO) 2 R.
where 2 == =00 -P)) and R represents the total required level of model reduction across all layers.
Similar to Lee et al. [2021], we consider greedy iterative optimization, where we decide the next pruning state s. (1) by optimizing the following problem for every step:
minimize (E(1,-- (e)
(i; 2(1:1))
where (1; 2g )( = (l 55 (()))) fo fo(") 5(1--1) (1)))((11))
1/ t denotes the
step index, and - (0(1)
with s c s indicating the set of remaining states at t step. The
5
objective function in Equation (7) represents the model-level output energy loss when pruning
single subsystem in one layer among layers pruned to different extents. By the proof provided in the Appendix B.3, the objective function can be upper-bounded by
Ji S4
II
(K) 5:th 1
Therefore, the upper bound for subsystem correlates with the ratio of the squared Hoo norm of the subsystem to the squared Hzo norm of the remaining system for layer l. The other terms, except for the ratio, in Equation (8) are common across all layers and can, therefore, be excluded from the cross-layer importance score calculation.
Although initially considered an iterative optimization determine the next pruning state, the important scores for all states in all layers can be computed with a few steps, as each score is independently determined based the trained parameters. For efficient evaluation of the scores, we sort the subsystems in each layer in descending order with their Ho0 norms in advance, s.t., Hox (x{))0)) E())= Hoo (x(") E for i < j. Finally, define the LAST score for x!" follows:
LAST (xx)),10)) = Hox (x: (1) E(0)
(9)
Σ Hoc (x)) 2()
(1 -(C--110
- (1)
B (1) 2
| 1 (1 (1-12
(10)
Similar to the local pruning criterion, Equation (10) can be modified for the case of using fixed B" or bidirectional SSMs. The LAST score for each state reveals the contribution of the subsystem to the model output by assessing the relative gain within the remaining system a layer, thereby indicating the significance of the subsystem. We refer to this relative metric calculation as energy normalization, which adjusts the state importance from different layers to comparable scale, enabling a cross-layer comparison of the states from different layers. In this way, states with lower LAST scores are selected from the overall states, and layer-adaptive pruning is performed according to the desired model- compression rate.
5

---

4 Related works
Model order reduction. In linear system theory, MOR methods have been extensively researched to approximate high-dimensional systems in engineering applications, such as VLSI [Antoulas and Sorensen, 2001], power systems [Li and White, 1999]. and various systems that employ spatial discretization [Jones and Kerrigan, 2010, Curtain and Zwart, 2012, Penzl, 2006]. Using the Hox norm to characterize a stable system, modal truncation [Green and Limebeer, 2012] removes states from diagonal realization for minimal Hos norm distortion of the system. Balanced truncation [Khalil et al., 1996, Safonov and Chiang, 1988] transforms a given system into a form, not necessarily diagonal. where all states are controllable and observable, then truncates the transformed system. Due to its superior approximation quality, balanced truncation has been developed for various systems and conditions [Petreczky et al., 2013, Besselink et al., 2014, Cheng et al., 2019]. However, transformation into non-diagonal systems is not applicable to current diagonal SSMs, where diagonal parameterization is necessary for computational efficiency and stability. In our work, per-state pruning granularity operates similarly to modal truncation. Compared to traditional MOR, which is reducing single linear system, we extend it to multi-system reduction, where nonlinear functions are also involved, which have not been addressed in traditional system theory.
Layer-adaptive neural network pruning. Using magnitude as pruning criterion, previous works have demonstrated the superiority of layer-adaptive pruning. where layers have different pruning ratios [Morcos et al., 2019, Han et al., 2015, Mocanu et al., 2018, Evci et al., 2020, Lee et al., 2021, Xu et al., 2023], compared to uniform pruning [Zhu and Gupta, 2017, Gale et al., 2019]. In Han et al. [2015], Mocanu et al. [2018], Evci et al. [2020], layer-adaptive pruning was achieved by setting specific magnitude threshold or target pruning ratio for each layer. In Morcos et al. [2019]. Lee et al. [2021], layer-adaptive pruning was performed using global pruning criterion and simultaneously comparing scores from different layers under the target pruning ratio. Specifically. Lee et al. [2021] proposed a global pruning criterion designed from the Frobenius norm- norm-based upper bound of the worst-case & distortion caused by pruning one layer while fixing the other. Xu et al. [2023] advanced this approach into joint optimization for the sum of filtered layer-wise worst-case distortion over pruning ratios. Inspired by Lee et al. [2021]. we provide the first global pruning criterion for SSMs, where non-magnitude-basee -aaed criterion is essential due the different transfer functions of SSMs compared to other neural networks. Lastly, we provide a missing design motivation for the squaring operation in score evaluation in Lee et al. [2021] by offering a clear rationale based on signal energy.
5 Experiments
Table presents the average performance of pruning methods for 10 tasks and models. Further experimental details are explained below.
Models and tasks. Experiments were con-
Table I: Average pruning ratio and accuracy loss for all tasks. Values in parentheses are evaluated by excluding non-compressible cases.
ducted with single A6000 48GB or RTX 3090 24GB GPU. We verify our method on S4D (S4D-
LegS) [Gu et al., 2022a] and S5 [Smith et al., 2023] models, which are multi-SISO and MIMO SSMs, respectively. Although our main motiva- tion was to reduce the state dimension of MIMO models, we also investigated the compressibility of multi-SISO models and the applicability of LAST them.
Avg. prun. ratio (Compressible only)
Avg. accuracy loss 4
Model
(Compressible only)
Uniform
25.00 25.00 (33.33)
0.39 (0.52)
S4D
Global
1.16 (1.55)
LAST
25.00 (33.33)
0.32 (0.42)
Uniform
33.00 (36.67)
4.32 (4.80)
S5
Global
33.00 (36.67)
7.51 (8.3) 35)
LAST
33.00 (36.67)
0.52 (0.58)
The models were reproduced with three seeds according to the reported configurations [Gu et al., 2022a, Smith et al., 2023] for the six tasks in LRA benchmark [Tay et al., 2021], the raw speech clas- sification task using Speech Commands dataset [Warden, 2018], and pixel-level image classification tasks using MNIST and CIFARIO datasets. We evaluated the performance of the full (unpruned) and one-shot pruned models while freezing other parameters not involved with SSM layers. See Appendix C for more experimental details.
The unique transfer functions of SSMs require the state pruning granularity and Hos norm-based pruning criteria, not simple magnitude- based pruning criteria, validated in Appendix D.
Baselines.

---

Table 2: Accuracy of pruned models on LRA tasks. LAST is evaluated at the maximum tested pruning ratio with less than 1% accuracy loss, and other methods were evaluated for the same pruning ratios.
sps
Text
Rctrievan (4,000)
Image (1,024)
Pathfindes (1.024)
Path-X (16,384)
(2,048)
(4.096
Avg.
Model
Prun.
Acc.
Prun.
Acc.
Prun.
Acc.
Prun.
Acc.
Prun.
Acc.
Prun.
Acc.
Acc.
Full model
0%
56.42
0%
86.40
0%
90.46
0%
77.02
0%
87.94
0%
88.07
81
Uniform No
10%
55.82
80%
86.02
60%
89.87
0%
77.02
10%
87,59
0%
88.07
80.73
54D
Global Mi
10%
49.95
80%
86.20
60%
89.84
0%
77.02
10%
87.20
0%
88.07
79.71
LAST
10%
56.27
80%
85,95
60%
89.46
0%
77.02
10%
87,83
0%
88.07
80.77
Full model
0%
61.48
0%
88.88
0%
91.20
0%
87.30
0%
95.15
0%
98.41
87.09
Uniform Ro
0%
61.45 41.48
60%
82.49
50%
90.29
30%
86.45
30%
71.38
30%
90,90
75.50
S5
Global Hise
0%
61.48
60%
88.56
50%
90.93
30%
87.04
30%
57.20
30%
69.21
75.74
LAST
0%
61.48
60%
88
50%
90.42
30%
86.34
30%
94.45
30%
97.95
86.53
Here, we compare LAST with two pruning methods: utilizes the local pruning criterion, H.s score, and applies the same pruning ratio each layer. Global Hos employs Hoo score as global criterion, serving as the ablation of the energy normalization used in LAST. Moreover, we present random state pruning results to demonstrate the effectiveness of developed local and global pruning criteria in identifying insignificant states. After one-shot pruning, evaluate whether these methods appropriately identify significant and insignificant states by measuring accuracy without retraining.
Uniform Hos and Global Ho Uniform Hos
Pruning ratios. For models pruned by Global Ho or LAST, which apply layer-adaptive pruning ratios, the reported pruning ratios indicate the average pruning ratios across all layers. We compare Uniform Ho and layer-adaptive pruning methods, Global Hoo and LAST, by setting the same desired compression rate. The tested pruning ratios were 10%, 20%. ... 90%, and 100%, where a pruning ratio of 100% indicates the extreme case leaving only one pair of complex-conjugate subsystems in each layer.
5.1 Long range arena
The LRA benchmark [Tay et al., 2021] has been used evaluate the ability capture long-range context from sequences with lengths ranging from 1,024 to 16,384. Table 2 shows the accuracy of models when each model is compressed to the maximum pruning ratio which LAST achieved an accuracy loss below 1 % for LRA tasks.
Without any retraining, LAST outperformed other methods, achieving the average accuracy loss of 0.56% (0.67%) for the average compression rate of 33.3% (40.0%), with (without) the non- compressible cases. Since the complexity and state dimension differ across tasks, achievable com- pression rate varied: for the most compressible case Text, 80% compression on S4D resulted in less than 1% loss in accuracy, while for the least compressible case Li stOps, where the state dimension was initially set to 16 for S5 layers, even 10% compression led to large performance degradation.
Figure 2 shows the accuracies of S5 models at different pruning ratios, including randomly pruned models. For Pathfinder and Path-X -X tasks, LAST consistently outperformed Uniform H.x and Global Ho.. and the accuracy of Global Hoo significantly dropped at high pruning ratios in these cases.
Moreover, we observed that the performance of Uniform Hos was comparable to LAST at low pruning ratios, whereas high pruning ratios, its performance became inferior LAST. We hypothesize that this was because the number of significant states in layer was considerably lower than the number of original states. That is, if Uniform Hos pruning begins pruning beyond the lowest proportion of insignificant states in any layer, it can subsequently cause great accuracy degradation. See Appendix E.2 for full results in LRA tasks.
5.2 Raw speech classification
The inductive bias and CT parameterization of SSMs enable 1) encoding raw speech without requiring feature engineering using methods such as short-time Fourier transform and 2) adapting to changes in sampling rate [Gu et al., 2022b, Goel al., 2022, Gu et al., 2022a, Smith et al., 2023].
7

---

ListOps
Text
Retrieval
90%
Accuracy
Accurday rasss mas
Accuracy
80%
ceuracy Toss Tine
Accuracy
80%
xeeoraey TME"
40%
70%
20%
60%
Kandom guess
Rakdto 8 Burker
10
200
150
100
50
250
200
150
100
50
State dimension
State dimension
State dimension
Image
Pathfinder
Path-X
100%
100%
80%
Accuracy
Accuracy
Accuracy loss 10% Ino
Accuracy
Kaaurssy 1833 106
60%
80%
40%
60%
20%
Rar 010
150
State dimension
State dimension
State dimension
Random
Uniform Ma.
Global M.
LAST
Figure 2: Efficiency-accuracy trade-off curves of pruned S5 models for tasks in LRA benchmark. LAST maintained accuracy better than other methods, Uniform Hoe and Global Hoc (LAST without energy normalization), demonstrating its superior ability to identify insignificant states.
Table 10 presents that these properties remained consistent after pruning, as pruned models maintained their performance on raw speech and flexibly processed to sequences at different sampling rates by adjusting the learned timescales according to the sampling shifts, similarly to Gu et al. [2022b]. See Figure for more results for Speech Command task.
5.3 Pixel-level image classification
We applied pruning to tasks that classify sequenced images, including sequential MNIST (SMNTST), permuted sequential MNIST (PsMNIST), and sequential CIFAR (sCIFAR), where SCIFAR is the colored version of Image task LRA. LAST consistently exhibited the smallest accuracy loss on average. See Appendix E.1 for results on the pixel-level classification tasks.
5.4 Analysis
5.4.1
Ablation study on energy normalization
We conduct an ablation study on the energy normalization of LAST by Global Ho.. which is LAST without using energy normalization. LAST normalizes the differences in layer-wise signal amplification, enabling the cross-layer comparison of states on common scale. Figure 3 shows the effect of the normalization S5 models for Path-X task. In Layers and 6, the overall Hos scores were relatively lower than other layers except Layer 1. Global H.s directly used Has scores, resulting in excessive pruning in Layers 5 and 6 from the pruning ratio 40%. However, LAST adjusted the scores by accounting for the low total energy transmission of the layers, making their states less prioritized in pruning. This led to different accuracy loss of methods as shown in Figure 2.
Moreover, energy normalization, which excludes pruned subsystems and normalizes accordingly. expands the range of high scores compared to normalizing without exclusions. In the case of Layer 1, this effect results in greater differences between LAST scores, making the scores distinguishable and pruning decisions easier. As a result, Layer was identified have more insignificant scores compared to other layers, leading to the removal of large number of states. In conclusion, energy normalization was critical in the pruning process, ensuring a robust cross-layer comparison and preserving the model performance.
8

---

Global M
LAST
107
10-1
Importance
Importance
10-2
10
Layer
Layer
Layer Layer
Layer Layer
10-5
Layer
Layer
10
Layer
Layer
Layer
Layer
50
100
50
100
Path-X (Layer 6)
Sorted stalte index
Sorted state index
State dimension
State dimension
200
200
50-
25
100
1.00
10:05
Re
-25
Pruning ratin 0%
Layer
Layer
Layer 3
Layar
Layer 5
Layer
Layer 1
Layer
Layer 3
Layer
Layer
Layer
-50
Pruning ratio 20%
Pruning 40%
Prun. 0%
Prun: 60%
Prun. 0%
Pnun, 60%
-75
Pruning rate 80%
Pruning
Prun. 20%
Prun. 20%
80%
Prun. 40%
Prun. 40%
-6
Figure 3: (Top) Evaluated state importance score and (Bottom) remaining state dimensions in S5 model for Path-X task. The state indices are sorted by Hoc scores, evaluated once for each conjugate pair.
Figure 4: Remaining poles in
A (6) of an S5 model for Path-X task.
5.4.2 Compressibility of models
We considered S4D, multi-SISO model, as the equivalent block diagonal MIMO model and applied the same pruning methods. While per-state structured pruning can completely remove state parameters, we implemented masking following the common practice in neural network pruning experiments. This approach allowed us to prune without compromising the parallelism of the multi-SISO model.
We observed that although the effective state dimension is larger in multi-SISO models, the average pruning ratio that does not result in severe accuracy loss was smaller in multi-SISO models (25%) compared to MIMO (33% models. This is likely because, in multi-SISO, specific states are assigned specific channels, meaning that each state is given certain role. Consequently, pruning a single state can result in greater loss. Additionally, this characteristic resulted in each subsystem exhibiting significantly low Hoo norm.
Discussion
Toward efficient training with guaranteed stability. Relying guaranteed stability, previous diagonal SSMs have used as many state dimensions as possible due to the challenge of optimizing state dimensions for the task. Although excessive states are initially used, efficient and stable learning can be achieved if insignificant states are pruned under well-planned pruning schedule.
Given this objective, first proposed SSM pruning method that adaptively reduces the order of multiple systems within a deep layered structure with nonlinearity. We derived a local pruning criterion considering the nonlinearity and layer-level output energy loss, applying the criterion in the Uniform H.S and Global H.30 methods, which can also be viewed as independently applying traditional MOR to systems in each layer. However, we empirically verified that our method can be more robust than locally applying MOR, particularly when there are significant differences in the Hox norm scale the proportion of important states across layers. As demonstrated in our application of the proposed method in multi-SISO models, this approach can be applied alongside parallelism.
Which states are pruned? Lessons for future work. We investigated the pruned states, which have been judged insignificant for the task, presenting some insightful observations for future work. known that Re(A,) controls the decay rate and Im()) controls the oscillating frequencies of dynamics [Gu et al., 2022a, Chen, 1984]. As the Hog norm is computed with these values, large

---

[Re(A,) (fast decaying mode) and large [lm[ pruned, as shown in Figure
dynamics) were more prone to be
Based on the insignificant pole characteristics, future work might explore new training strategies for SSMs, e.g., making poles constrained avoid having the insignificant characteristics. Moreover, this provides a conjecture for the empirical effectiveness of the bookk-dagoonal initialization in S5 [Smith al., 2023], suggesting that the initialization performed well because resulted in fewer large (..H) Even using the block-diagonal initialization, we found that previous models tend to have very large [Im[A:H]. e.g., over 1,000, which also could be addressed in future work.
Limitations. This paper has the following limitations. Although we explored the pruning criterion for SSMs, questions about when and how often to prune SSMs remained unresolved. Additionally, our proposed method was verified on a specific set of tasks, where both mult-SISO and MIMO models have been evaluated in previous work, and the adaptation to other tasks remains be investigated. However, we believe that our work opens opportunities to utilize the full capacity of MIMO SSMs by making them as compact as possible, not sacrificing their capacity, search space, or stability.
Acknowledgments and Disclosure of Funding
This research was supported by the Basic Science Research Program through the National Re- search Foundation of Korea (NRF) funded by the Ministry Science, ICT, and Future Planning 20220RAA222055009 This work was supported by Samsung Electronics Co., Ltd. (I0201211- 08100-01).
References
A. C. Antoulas and C. Sorensen. Approximation of large-scale dynamical systems: An overview. International Journal of Applied Mathematics and Computer Science, 11(5):0933-12)) 2001.
B. Besselink, N. van de Wouw, J. M. Scherpen, and H. Nijmeijer. Model reduction for nonlinear systems by incremental balanced truncation. IEEE Transactions on Automatic Control, 59(1)) 2739 2753, 2014.
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, VanderPlas, S. Wanderman-Miine et al. Jax: composable transformations of python+ numpy programs. 2018.
C.-T. Chen. Linear system theory and design. Saunders college publishing, 1984.
H. Cheng, M. Zhang, and J. Q. Shi. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
X. Cheng, J. M. Scherpen, and B. Besselink. Balanced truncation networked linear passive systems. Automatica, 104: 2019.
R. F. Curtain and H. Zwart. An introduction to infinite-dimensional linear systems theory, volume 21.
Springer Science Business Media, 2012.
U. Evei, T. Gale, J. Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. In International conference machine learning, pages 2943-2952. PMLR, 2020.
T. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.094 1902.09574. 2019.
K. Goel, A. Gu, C. Donahue, and C. Ré. It's raw! audio generation with state-space models. In International Conference on Machine Learning, pages 7616-7633. PMLR, 2022.
M. Green and J. Limebeer. Linear robust control. Courier Corporation, 2012.
A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint 20X00:2312.00752. 2023.
10

---

A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474-1487, 2020.
A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572-585, 2021.
A. Gu, K. Goel, A. Gupta, and C. Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Prucessing Systems, 35:35971-35983, 2022a.
A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. In The International Conference on Learning Representations, 2022b.
A. Gupta, A. Gu, and J. Berant. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 2112299222299 2022.
S. Han, J. Pool, Tran, and W. Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Y. He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 160.:0860008 2016.
R. A. Horn and C. R. Johnson. Matrix analysis. Cambridge university press, 2012.
B. L. Jones and E. C. Kerrigan, When is the discretization of spatially distributed system good enough for control? Automatica, 46(9):1462 1468, 2010.
T. Kailath, Linear systems, volume 156. Prentice-Hae Englewood Cliffs, NJ, 1980.
I. Khalil, J. Doyle, and K. Glover. Robust and optimal control. Prentice hall, 1996.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
J. Lee, S. Park, Mo, S. Ahn, and Shin. Layer-adaptive sparsity for the magnitude- based pruning. In The International Conference on Learning Representations, 2021.
J.-R. Li and J. White. Efficient model reduction of interconnect via approximate system gramians. In 1999 IEEE/ACM International Conference on Computer-Aided Design. Digest of Technical Papers (Cat. No. 99C113055), pages 380-383. IEEE, 1999.
D. Linsley, J. Kim, V. Veerabadran, C. Windolf, and T. Serre. Learning long-range spatial dependen- cies with horizontal gated recurrent units. Advances in neural information processing systems, 31, 2018.
A. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics:
Human language technologies, pages 142-150, 150, 2011.
D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 911):3383, 2018.
A. Morcos, Yu, M. Paganiii, and Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. Advances in neural information processing systems, 32, 2019.
N. Nangia and S. R. Bowman, Listops: A diagnostic dataset for latent tree learning. arXiv preprint arXiv:/8044 06028, 2018.
B. Neyshabur, R. Tomioka, and N. Srebro. Norm-based capacity control neural networks. In Conference learning theory, pages 1376-1401. PMLR, 2015.

---

R. N. Parnichkun, S. Massaroli, A. Moro, J. T. H. Smith, R. Hasani, M. Lechner, Q. An, C. Ré, H. Asama, S. Ermon, T. Suzuki, A. Yamashita, and M. Poli. State-free inference of state-space models: The transfer function approach, 2024.
T. Penzl. Algorithms for model reduction of large dynamical systems. Linear algebra and its applications, 415(2-3):322-33 343, 2006.
M. Petreczky, R. Wisniewski, and J. Leth. Balanced truncation for linear switched systems. Nonlinear Analysis: Hybrid Systems, 10:4-20 20, 2013.
Y. Qin and Z. Sun. Observer-based asynchronous event-triggered robust Ho adaptive switching control for nonlinear industrial cyber physical systems under data injection attacks. International Journal of Control, Automation and Systems, 21(7):2175-2182, 2023.
D. R. Radev, P. Muthukrishnan, and V. Qazvinian. The ACI Anthology network corpus. In M.-Y. Kan and S. Teufel, editors, Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries, pages 51,1, Suntec City, Singapore, Aug. 2009. Association for Computational Linguistics. URL https: /aclaannnolooyyy org/ /WO 9-09-9900 3607.
S. Rush and S. Karamcheti. The annotated s4. In Blog Track at ICLR, 2022.
M. Safonov and R. Chiang. A schur method for balanced model reduction. In 1988 American Control Conference, pages 1036-1040. IEEE, 1988.
J. T. Smith, A. Warrington, and S. Linderman. Simplified state space layers for sequence modeling. In The International Conference on Learning Representations, 2023.
Y. Tay, M. Dehghani, S. Abnar, Y. Shen, Bahri, Pham, J. Rao, L. Yang. S. Ruder, and D. Metzler. Long range arena A benchmark for efficient transformers. In The International Conference on Learning Representations, 2021.
Warden. Speech commands: dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.0322 2018.
K. Xu, Z. Wang, X. Geng, M. Wu, X. Li, and W. Lin. Efficient joint optimization of layer-adaptive weight pruning in deep neural networks. In Proceedings the IEEE/CVF International Conference Computer Vision, pages 17447-17457, 2023.
M. Zhang, Saab, M. Poli, Dao, K. Goel, and Ré. Effectively modeling time series with simple discrete state spaces. The International Conference on Learning Representations, 2023.
Zheng, W. Shi, K. Wu, and S. Jiang. Robust Hoo and guaranteed cost filtering for ts fuzzy systems with multipath quantizations. International Journal of Control, Automation and Systems, 21(2): 671-683. 2023.
M. Zhu and S. Gupta. To prune, not prune: exploring the efficacy of pruning for model compression. Xixiv preprint arXiv: 171000187,, 2017.
12

---

Supplementary Material
Contents
Introduction
2
Background
2.1
Stability of state space models
2.2 Diagonal state space models
2.3 Hec norms of systems..
3 LAST: Layer-adaptive state pruning for SSMs
3.1
Hos scores as local pruning criteria
3.2
LAST scores as global pruning criteria
4
Related works
5
Experiments
5.1
Long range arena
5.2
Raw speech classification
5.3
Pixel-level image classification
5.4
Analysis
8
5.4.1
Ablation study on energy normalization
5.4.2
Compressibility of models
Discussion
A Stability of state space models
15
A.1 Indirect pole training
15
A.1.1 Sufficient stability condition
15
A.1.2 Stability guaranteed only at initialization
15
A.2 Direct pole training
16
B Proofs
16
B.1 System norm property
16
B.2 Bounded layer-level output energy loss
16
B.3 Bounded model-level output energy loss
17
Experimental details
18
C.I Tasks
19
13

---

C.2 Hyperparameters
20
Validation of pruning granularity and criterion
20
D.I State pruning granularity
20
D.2 Comparison with magnitude pruning
21
Full results
22
E.1 Pixel-level image classification
22
E.2 Long range arena
23
E.3 Speech command
26
14

---

A Stability of state space models
A.1 Indirect pole training
The rational transfer function of an nth-order system can be defined by:
b12 + b,
(x) ho +
1 1012-14..
following the notation in Parnichkun al. [2024]. For second-order system, the characteristic function that determines stability is
a(2) = 22 0= 02-
For DT system to stable, the poles the transfer function should be within the unit circle. This can be checked through the Schur-Cohn test [Kailath, 1980], which provides the stable region for the second-order system with the characteristic function a(2) as:
2
and (1 + az)² - at 0,
as shown in Figure 5.
A.1.1 Sufficient stability condition
In models where the poles are trained indirectly, stability be ensured by applying sufficient constraints for stable poles during training. Montel's constraint [Horn and Johnson, 2012] serves as sufficient stability condition by restricting the coefficients follows:
|as1 S 1.
For the second-order case, Montel's constraint is
|aوl + |a2l
<
This defines sufficient stable region shown Figure However, as highlighted Parnichkun et al. [2024], this search space restriction can confine the model performance.
A.1.2 Stability guaranteed only initialization
As an alternative, zero initialization [Parnichkun al., 2024] initializes the system at the center point of the stable region. Thus, the initial coefficients of zero initialization for second-order system are
61 and 02 = O,
as marked in Figure 5. However, this does not guarantee stability subsequent training, which potentially causes states diverge and makes training infeasible.
Search space (n = 2)
0.5
S
-0.5
Stable
Montel's is cones,
Zero init.
-2
2
Q1
Figure 5: Search space in the two-dimensional coefficient space for stability.
15

---

A.2 Direct pole training
For models like diagonal SSMs that train poles as parameters, it possible directly control them to satisfy stability conditions. For example, in the case of CT SSMs, ensuring that the system is Hurwitz, i.., Re(A.) < 0 for i S, guarantees stability. CT SSM stable, the ZOH-discretizer SSM is also stable, i.e., [ < for S, since
[ = |e^:Ai|
(e Hmm(Ar4.)))
Ro(Ar
11
(11)
where the inequality holds since Re(A:) < 0 and A; > 0 for stable CT SSM.
B
Proofs
B.1
System norm property
The transfer function matrix G of a system 2 u is defined by Y = GU, where U and Y are the Z-transforms of u and y. The energy of the output signal y bounded with the Hoc norm the system as follows:
|1 = |
(12)
|GUI2
1
2n
((())))(())))))3))
2x
1 2m
2x
s sup 72 (G((E))
[|(e)0))1220
DE (0.2m
2T
|| |u
(13)
where we use Parseval's 's theorem, i.e., |[vl[ = |2 for signal v and the Z-transform 2, in (12) and (13).
B.2 Bounded layer-level output energy loss
We first show that the TFM of a diagonal system is the sum of the TFMs of subsystems. By applying the Z-transform to (1) (For simplicity, the feed-fon forward matrix D is excluded.), we have
<x(x) (x + BU(2),
Y()) CX(2),
where X, U. and are the Z-transforms of x, u, and y, respectively. We can combine equations by Y()) = (z)) where
G(z) C(2I, - -B B
(14)
C,B.
(15)
and the decomposition (15) holds since the considered system is a diagonal system. Similarly, by applying the Z-transform to subsystem 2, we can derive its TFM G; as follows:
C,B.
G= 2 X
(16)
16

---

Substituting (15) with (16) shows that the TFM of the diagonal system is the sum of TFMs of all subsystems as follows:
G(2) Σ G.(z).
(17)
Now, we consider the layer-layer output energy loss caused by pruning states in P, i... reducing Σ into Epp. In previous SSMs, GELU [Hendrycks and Gimpel, 2016] has been widely used as the activation function. Using the 1-Lipschitzness of GELU, TFM decomposition (17), and Ho norm property (2), the layer-level energy loss is upper bounded as follows:
Ilfo (u:E) fo E-)) ([(()) (())(l
[E(u) -
I-Lipshhitzness
|GU P
Parseval's theorem,
Linearity of Z-transform
11 1 G
V (17)
XEP
|G,U
Triangle inequality
REP
(u)
Parseval's 's theorem
iEP
$ ||G 1 ||u1|2
>
(2)
KEP
This inequality builds upon the approach in Neyshabur al. [2015], Lee et al. [2021], adapting for diagonal SSMs and deriving bounds using signal and system norms. In the above derivation, we can analyze on a subsystem basis by utilizing the diagonal structure. Parseval's theorem [Green and Limebeer, 2012] allows to switch between the time and frequency domains with energy equivalence. Even in the presence of nonlinear functions and signal distortion analysis, we achieved result similar to modal truncation [Green and Limebeer, 2012], where the Ho norm distortion of an LTI system is bounded by the sum of the H. norms of the truncated LTI systems.
B.3 Bounded model-level output energy loss
We show that the model-level output energy loss in Equation (8) is bounded with Hox norms of subsystems:
Ji )((:1)) <
II
G (k) 5
2
So (u where (i; 2 5((:1)) = [o (u)) (000)))1)))
From Lth layer to 1 + 1th layer, we can keep bounding the output energy of formal layers using the I-Lipschitzness of (-) and Hov norm property in Equation (2) as follows:
fa (u (1) ; - fo (u("), 5(1:-11) (1))
-0+1-11
8 (0)"
/i2ggln sfy (w 3((1:(=1))) 0 (L)
(1) (11-11 (01
111+1L-111 12
5:0h 103
(1))
(): S (((1)-11) - 2uti (u (1); 5: 5:1:111) 1)
2010
5:(1+1:--11)
<
s(
5000
2
(1)
[o(u [o(1
"[ :((1:1111] - (o(1 (1( ((1--1) 1)
s(1)
s(L
((1) (1)
17

---

2
5 (Io(u )()
(Io(u ((
s(t)
(0)
k=++1
Since 2:11
can apply the bounded layer-level energy loss property in Equation (4):
2
II
(k)
(fo(u(); (fo("();
1)) ( (1"))((1-)))))
(fo(u();
()))))))))
S(A)
km1+1
$61 (Io(0);(((t11) |
G s.00
(k)
[()) s
k=|+1
L II k=1+1
2
$ G
(Gt)
(k)
Then we can add an auxiliary term (0) and use the 1- i-Lipschitzeess property since (0) = 0 holds for GELU activation a:
G
(1)
G (k) S(())
(u 1)) ||
km
L
(1)
G
(R)
|0-511 11-11 (Jo (u (1)) -((=-))))) - -0(0 1
s(k)
k=++1
<
G
(8)
II
G (R) S(
| (fo 55 (((--1)))
Again, the output of / - 1th layer can be bounded using the Hoc norm property, and keeping the procedures to the first layer proves the statement as follows:
G 9
G
(R)
Σ 11 (fo (u
10 (11 211
k=
2 (o(1 (,01))
G
(k) (
G
(1-1) 5/
1)
1-1
(())
(R)
s(K))
5
(
2
G (R) S(())
m
G
C Experimental details
Our experiments were conducted with JAX [Bradbury et al., 2018] on a single A6000 48GB GPU or RTX 3090 24GB GPU. We reproduced S4D models using the implementations from Rush and Karamcheti [2022] and S5 models from Smith et al. [2023] We used bidirectional SSMs for all
Phttps://gi www..h......o...cooooooo...o.oa http:/..cm//sss/mmmttteeeees Phttps:/ httpt/ggttt///t/////////ttttthtt com/ Limm//diddrmmala///55
18

---

tasks except SMNIST and PSMNIST tasks. Following S5, we implemented bidirectional S4D models to have C matrices for reverse convolution. For inference, we used Vandermonde product and convolution kernel schemes for S4D and parallel scans for S5 models.
C.1 Tasks
The following lists ten tasks where we tested our proposed method, along with the specified resources and the time taken for model training for each task.
SMNIST: 10-way classification task with flattened MNIST images, each having sequence length of 784. Original images are for handwritten digits. It took 30 minutes train an S5 model with an RTX 3090 24GB GPU.
PSMNTST: 10-way classification task with flattened and fixed-order permuted MNIST images, each having a sequence length of 784. Original images are for handwritten digits. It took hour to train an S5 model with an RTX 3090 24GB GPU.
SCIFAR: 10-way classification task with flattened CIFAR-10 images [Krizhevsky et al., 2009], each having a sequence length of 1,024 for each R, G, B channel. The dataset includes 45,000 training. 5,000 validation, and 10,000 test sequences. It took 7 hours to train an S4D model or an S5 model with an A6000 48GB GPU.
ListOps. 10-way classification task with longer variations of ListOps data [Nangia and Bowman, 2018], each having a maximum sequence length of 2048 for a single channel. The task is solving nested mathematical operations applied to numbers in the range of 0-9 to derive final result. One-hot vectors for 17 values, including operators, enclosers of operators, and numbers, are concatenated. The dataset includes 96,000 training, 2,000 validation, and 2,000 test sequences. took 2 hours to train an S4D model or an S5 model with an RTX 3090 24GB GPU.
Text: 2- 2-way byte-level text classification task with IMDB review data [Maas al., 2011], each having a maximum sequence length of 4,096 for a single channel. The task is classifying the sentiment of a review. One-hot vectors for 129 characters are concatenated. The dataset includes 25,000 training and 25,000 test sequences. It took 2.5 hours to train an S4D model and 1.5 hours to train an S5 model with an RTX 3090 24GB GPU.
2-way byte-level document retrieval task with ACL Anthology Network document data [Radev et al., 2009], each having a maximum sequence length of 4,000 for a single channel. The task is classifying if two documents are linked by equivalent citations. One-hot vectors for 97 characters are concatenated for each document. The dataset includes 147,086 training. 18,090 validation, and 17,437 test sequence pairs. It took 15.5 hours to train an S4D model with A6000 48GB GPU and 6 hours to train an S5 model with an RTX 3090 24GB GPU.
Ret ritrien eval: rivall
10-way classification task with flattened CIFAR- 10 images [Krizhevsky et al., 2009], each having a sequence length of 1,024 for a single channel. It took 9.5 hours to train an S4D model and 7.5 hours to train S5 model with an RTX 3090 24GB GPU.
Image:
Pa Pathfe fethfinneer inder: 2-way classification task with flattened Pathfinder challenge images [Linsley et al., 2018), each having a sequence length of 1,024 for single channel. Original images for points with connecting or distracting paths. The dataset includes 160,000 training, 20,000 validation, and 20, 000 test sequences. It took 14 hours to train S4D model and 11 hours train an S5 model with an RTX 3090 24GB GPU.
Path-x: 2-way classification task with flattened scaled Pathfinder challenge images [Linsley et al., 2018], each having a sequence length of 16,384 for single channel. Original images are for points with connecting or distracting paths. Original images are for points and connecting or distracting paths. It took 3 days to train an S4D model and I day to train an S5 model with an A6000 48GB GPU.
Speech Command: 35-way classification task with 1-second word-speaking audio recording data [Warden, 2018), each having a sequence length of 16,000 for a single channel. For the varying sampling frequency tests, the data was downsampled from 16kHz to 8kHz. The dataset includes 24,482 training, 5,246 validation, and 5,247 test sequences. took 21 hours to train an S4D model and 8 hours to train an S5 model with an RTX 3090 24GB GPU.
19

---

C.2 Hyperparameters
We followed the hyperparameters in [Gu al., 2022a, Smith al., 2023]. For Path-X task, was challenging to train S4D models with the original learning rate of 0.0005, thus we changed it to 0.001.
Table 3: Training configurations of S4D models for all tested tasks. nt: state dimension of each SISO system. LN: layer normalization, BN: batch normalization, Pre: pre-normalization D: dropout. LR:
learning rate. B: batch size. E: epochs. WD: weight decay. t: The value is changed from the original
release [Gu et al., 2022a] for training feasibility.
Task
L
h
R2
Norm
Pre
D
LR
B
WD
(Amin Amax)
SCIFAR
6
512
64
LN
Falsc
0.1
0.01
50
200
0.05
(0.001, 0.1)
TistOps
8
128
64
BN
False
0
0.01
50
40
0.05
(0.001, 0.1)
Text
6
256
64
BN
True
0
0.01
16
32
0.05
(0.001, 0.1)
Retrieval
6
256
64
BN
Truc
0
0.01
64
20
0.05
(0.001,
Image
6
512
64
IN
False
0.1
0.01
50
200
0.05
(0.001,0.1)
Pathfinder
6
256
64
BN
True
0
0.004
64
200
0.03
(0.001,0.1)
Path-X
6
256
64
BN
Truc
0
0.001
32
50
0.05
(0.001,0.01)
Speech
6
64
BN
True
0.01
40
0.05
(0.001,0.1)
Table 4: Training configurations of $5 models for all tested tasks. All models used batch normalization, pre-normalization, and Amaz = 0.1. Rm: state dimension of a MIMO system. J/: number of blocks for block initialization of A. D: dropout. LR: learning rate. SSM LR: learning rate for SSM
parameters, B: batch size. E: epochs. WD: weight decay.
Task
fmm
D
LR
SSM L.R
B
E
WD
Amin
SMNIST
96
128
0.1
0.008
0.002
50
150
0.01
0.001
PSMNTST
128
128
0.15
0.004
0.001
50
150
0.01
0.001
sCIFAR
512
384
0.1
0.0045
0.001
50
250
0.07
0.001
ListOps
8
128
16
8
0
0.003
0.001
50
40
0.07
0.001
Text
6
256
192
12
0.1
0.004
0.001
50
35
0.07
0,001
Retrieval
6
128
256
16
0
0.002
0.001
32
20
0.05
0,001
Image
6
512
384
3
0.1
0.005
0.001
50
250
0.07
0.001
Pathfinder
6
192
256
8
0.05
0.005
0.0009
64
200
0.07
0.001
Path-X
6
128
256
16
0
0.002
0.0006
32
75
0.05
0.001
Speech
96
128
16
0.1
0.008
0.002
16
40
0.04
0.001
D Validation of pruning granularity and criterion
D.1 State pruning granularity
As in channel pruning [He et al., 2017], state pruning is named based on its granularity of pruning, that is, all parameters associated with insignificant states are pruned at once. For instance, the parameters X from A, the row vector B; from B, and the column vector C, from C are pruned when the state i identified an insignificant state.
To explicitly demonstrate the necessity of state pruning in SSMs, we compared the performance of unstructured random pruning and structured random state pruning using S5 models. For unstructured random pruning, we pruned randomly selected elements from the system matrices, obtaining the results in Table 5.
Despite the similar number of parameters being pruned, the model suffered a significant performance degradation, with average accuracy loss of 59.92%, in the case of unstructured random pruning.
20

---

Table Average pruning ratio and accuracy loss for all tasks. Values in parentheses are evaluated by excluding con-compressible cases.
Method
Average pruning ratio
Average accuracy loss
Unstructured random
33,00 (36.67)
59.92 (66.58)
Structured random
33.00 (36.67)
29.53 (32.82)
This is because unstructured pruning can alter the learned dynamics in all subsystems. In contrast, state pruning maintains the functionality of unpruned subsystems, leading to less performance degradation. This highlights the importance of considering the structure and mechanism of the model when applying pruning techniques.
D.2 Comparison with magnitude pruning
Magnitudes and L, norms of parameters are simple but effective pruning criteria to obtain efficient neural networks [Cheng al., 2024]. Given the necessity of state pruning granularity in SSMs, set the pruning granularity to state pruning and then compared the significant state identification abilities of the magnitude and H. pruning methods. extend Table we define magnitude state pruning methods as follows:
Uniform Magnitude. Every layer is uniformly pruned to have the same pruning ratio with the importance of each state as X B, ||| While any Lp norm can be used, we present the results using the L2 norm as an example.
Global Magnitude. The same state importance criterion as in Uniform Magnitude is used, but the comparison group is extended from intra-layer inter-layer, ensuring that the pruning ratio is met globally for the entire network.
LAMP. This method employs a criterion of Σツ B| C2 adapted from Lee et al. [2021], w² which originally used Σ ES as a criterion for a real-valued weight parameter W. The state indices the denominator are assumed to be ordered based on their evaluation using the basic magnitude criterion similar to LAST.
|A B C.I.
Extending the S5 model part in Table 1, Table 6 reports that, at the same pruning ratio, LAST and other H.o pruning methods significantly outperform magnitude pruning methods by showing less accuracy loss, which implies that Hox pruning methods can better distinguish significant and insignificant states. This performance gap and suitability can be explained with the unique transfer function of SSMs, which is defined in the frequency domain as in Equation (14) for the whole system and Equation (16) for subsystem. Specifically, the importance of A; is evaluated by (1 - [A:]) in Hox pruning methods, while magnitude pruning methods evaluate |Aa]-
21

---

Table Average pruning ratio and accuracy loss in S5 models for all tasks. Values in parentheses are evaluated by excluding non-compressible cases.
Method
Average pruning ratio Average accuracy loss |
State importance
Random
33.00 (36.67)
29.53 (32.82)
Uniform magnitude
33.00 (36.67)
22.03 (24.48)
M B |
Global magnitude
33.00 (36.67)
17.49 (19.43)
A B |
Ad B I
LAMP
33.00 (36.67)
18.07 (20.07)
[ "|.
Uniform 3o.
33.00 (36.67)
4.32 (4.80)
[1-1X:
Global Hoc
33.00 (36.67)
7.51 (8.35)
|CC, 9 [B
(1 1X.)
|c 2 IB.1
LAST
33.00 (36.67)
0.52 (0.58)
[10 11,1 (1-12;1)*
Egs
E
Full results
E.1
Pixel-level image classification
Table 7 highlights the results evaluated at the maximum pruning ratio where the accuracy loss of LAST was less than 1%. Figure shows the accuracy at all tested pruning ratios.
As shown in Table 7 and Figure 6, both S4D and S5 had great compressibility. In Table 7, the state dimension of S4D indicates the average n, of SISO systems, while in Figure refers to the average effective state dimension n across layers.
SCIFAR
Accuracy
60%
40%
30K
20K
10K
State dimension
SMNIST
PSMNIST
SCIFAR
100%
100%
Accuracy
Aecooae aty UUS RAE
Accuracy
80%
Ateuaayy fear yo12 NRS
Accuracy
10% MAR
60%
60%
60%
40%
40%
40%
20%
20%
75
30
State dimension
State dimension
State dimension
Random
Uniform M.
Global N.
Figure 6: Efficiency-accuracy trade-off curves of pruned (Upper) S4D (Lower) S5 models for pixel- level image classification tasks. LAST obtained more efficient models that maintain performance compared to Uniform Hoo, which was observed more stably and consistently than Global H.s (LAST w/o score normalization).
22

---

Table Accuracy of pruned models on pixel-level image classification tasks. LAST is evaluated the maximum tested pruning ratio with less than I % accuracy loss, and other methods were evaluated for the same pruning ratios.
Task
Model
Method
Prun.
State dim.
Accuracy
S4D
Full model
0%
128
99.55 0.02
SMNIST (784)
Uniform Ho.0
50%
64
99.26 + 0.15
S5
Global H.
50%
64
98.75 + 0.24
LAST
50%
64
99.01 0.57
S4D
Full model
0%
128
98.39 + 0.06
PSMNIST (784)
Uniform Ho
30%
90
96.32 + 0.19
S5
Global Hox
30%
90
93.59 3.82
LAST
30%
90
98.09
0.30
Full model
0%
64
83.97 + 0.30
Uniform Hoc
30%
45
83.10 + 0.35
S4D
Global Hoo
30%
45
83.02 0.46
LAST
30%
45
83.21 0.33
sCIFAR (1,024)
Full model
0%
384
88.52 + 0.29
Uniform Ho.
30%
269
87.37 + 0.85
S5
Global H.x
30%
269
87.22 0.05
LAST
30%
269
87.53 0.41
E.2
Long range arena
To evaluate the practical efficiency resulting from LAST, we implemented pruning by removal, in addition to pruning by masking implementation, by transferring selected significant parameters to smaller-dimensionaa model. Table 8 presents the average evaluation step speed and peak GPU memory usage of pruned S5 models for an NVIDIA RTX 3090 GPU. Reducing the state dimension improved efficiency in both computational and memory costs, with the degree of efficiency depending on the channel size per task.
Table 8: Efficiency improvement in computational and memory costs in S5 models.
sist
Text
Retrieval
Tmage
Pathfinder
Path-X
Pruning ratio
0%%
60%
50%
30%
30%
30%
Inference speed
1.0x
1.6x
1.7x
1.2x
1.1x
1.3x
GPU memory usage
1.0x
0.9x
0.6x
1.0x
0.8
0.8x
Table 9 highlights the results evaluated at the maximum pruning ratio where the accuracy loss of LAST was less than 1%. Figure shows the accuracy at all tested pruning ratios. Table 9, the state dimension of S4D indicates the average n, of SISO systems, while in Figure 7, it refers to the average effective state dimension n across layers.
In ListOps task, where the initial state dimension was small, the S5 models were uncompressible. For Text task, both S4D and S5 models showed the highest compressibility among all tasks, followed by Retrieval task.
In Image task, S4D models were uncompressible since the Hoo scores were significantly low and fell below the precision threshold of the floating-poin representation, making the comparison in local pruning and sorting required for LAST score calculation impossible.
23

---

ListOps
Text
Retrieval
60%
90%
Accuracy
4.0%
mecorssy
Accuracy
80%
accorsty
Accuracy
aeceal dey roso 1015 MMME
70%
20%
60%
Rarddon garss Wme
50%
Par M003
Ramrdam
8K
6K
15K
10K
SK
15K
10K
State dimension
State dimension
State dimension
Image
Pathfinder
Path-X
80%
Accuracy
60%
Accuracy loss 10% line
Accuracy
aecorssy NoN 10s 1one
Accuracy
80%
Xcci7/sey 1AN 10% TAY
70%
70%
4.0%
60%
60%
50%
50%
Wen/ee gu
20K
10K
State dimension
State dimension
State dimension
Random
Uniform Ma
Global Me
LAST
Figure 7: Efficiency-accurae trade-off curves of pruned S4D models for LRA tasks. LAST obtained more efficient models that maintain performance compared to Uniform Hoo, which was observed more stably and consistently than Global Hox (LAST w/o score normalization).
Notably, the state dimensions of S5 models were able to reduce by 30% in both the Pathfinder and Path-X tasks. The ability to maintain performance in Pat h-X X highlights the effectiveness of the MIMO structure of S5.
24

---

Table 9: Accuracy of pruned models for LRA tasks. LAST is evaluated at the maximum tested pruning ratio with less than 1% accuracy loss, and other methods were evaluated for the same pruning ratios.
Task
Model
Method
Prun.
State dim.
Accuracy
Full model
0%
64
56.42 + 0.02
Uniform Ho
10%
58
53.82 0.81
S4D
Global Hoc
10%
58
49.95 7.32
LAST
10%
58
56.27 + 0.70
ListÖps (2,048)
Full model
0%
16
61.48 + 0.24
Uniform Ho
0%
16
61.48 0.24
S5
Global Hoc
0%
16
61.48 + 0.24
LAST
0%
16
61.48 + 0.24
Full model
0%
64
86.40 + 0.21
Uniform Hos
80%
13
86.02 0.32
S4D
Global H.
80%
13
86.20 + 0.25
LAST
80%
13
85.95 + 0.26
Text (4,096)
Full model
0%
192
88.88 0.10
Uniform Ho
60%
77
82.49 3.07
S5
Global Hox
60%
77
88.56 + 0.30
LAST 60%
77
88.52 0.20
Full model
0%
64
90.46 0.18
Uniform oo
60%
26
89.87 0.79
S4D
Global Hox
60%
26
89.84 + 0.82
LAST 60%
26
89.46 0.58
Retrieval (4,000)
Full model
0%
256
91.20 0.16
Uniform Ho
50%
128
90.29 = 0.30
S5
Global Hoc
50%
128
90.93 0.34
LAST
50%
128
90.42 0.64
Full model
0%
64
77.02 0.91
Uniform Ho
0%
64
77.02 + 0.91
S4D
Global Hoc
0%
64
77.02 0.91
LAST
0%
64
77.02 0.91
Image (1,024)
Full model
0%
256
87.30 + 0.41
Uniform Hos
30%
179
86.45 0.32
Global Hoc
30%
179
87.04 0.26
LAST
30%
179
86.34 = 0.37
Full model
0%
64
87.94 + 0.70
Uniform Hos
10%
58
87.59 0.58
S4D
Global Hos
10%
58
87.20 0.21
LAST
10%
58
87.83 + 0.66
Pathfinder (1,024)
Full model
0%
256
95.15 0.22
Uniform Hos
30%
179
71.38 11.64
S5
Global Hoc
30%
179
57.20 + 9.45
LAST 30%
179
94.45 + 0.42
Full model
0%
64
88.07 1.17
Uniform Hoo
10%
64
88.07 1.17
S4D
Global H.
10%
64
88.07 + 1.17
LAST 10%
64
88.07 + 1.17
Path-X (16,384)
Full model
0%
256
98.41 0.12
S5 Uniform Ho
30%
179
90.90 = 2.05
Global H. 30%
179
69.21 + 20.57
LAST
30%
179
97.95 0.22
25

---

E.3
Speech command
Table 10 highlights the results evaluated at the maximum pruming ratio where the accuracy loss of LAST was less than 1%. Figure 8 shows the accuracy at all tested pruning ratios. In Table 10, the state dimension of S4D indicates the average n, of SISO systems, while in Figure 8, it refers the average effective state dimension across layers.
Table 10: Accuracy of pruned models Speech Command task. LAST is evaluated at the maximum tested pruning ratio with less than 1% accuracy loss, and other methods were evaluated for the same pruning ratios.
Model
Method
Prun.
State dim.
Accuracy (16kHz)
Accuracy (8kHz)
Full model
0%
64
94.69 + 0.10
91.23 + 0.93
Uniform H.
10%
58
94.36 0.10
90.56 0.90
S4D
Global Hox
10%
58
94.37 = 0.09
90.66 0.99
LAST
10%
58
94.61 = 0.07
90.83 + 0.98
Full model
0%
128
96.43 + 0.10
94.26 0.31
Uniform Ho
20%
102
96.20 0.11
94.00 0.18
S5
Global Hox
20%
102
96.21 + 0.07
93.91 + 0.28
LAST
20%
102
96.31 + 0.14
94.11 + 0.36
Speech Command
Speech Command
80%
AECuRRE 1053 1018
80%
1106
Accuracy
Accuracy
60%
40%
Random
Random
Uniform N.
Unifrrm N.
20%
Global N.
LAST
LAST
0%
State dimeenion
State dimension
Figure 8: Efficiency-accuracy tradeoff curves pruned (Left) S4D (Right) S5 models for Speech Command task. LAST obtained more efficient models that maintain performance compared to Uniform Hoo which was observed more stably and consistently than Global Ho (LAST w/o score normalization).
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction reflect the contribution of extending modal truncation to multi-system approximation for deep state space models. In addition, it accurately presents the average result values from all datasets used in the experiment.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes)
Justification:
The limitation of this paper discussed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer: [Yes]
Justification: This paper clearly states the assumption of a stable diagonal system before derivations, and this assumption is satisfied in the experiments through Hurwitz parameteri- zation. The proof for the upper bound the objective function in the proposed method is provided in Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose the final numerical form of our proposed scores in Equation (6) and Equation (10), and provide the simulation codes by external link.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
28

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: This paper clearly specifies the sources of the existing data and code used, and the external link includes the environment settings necessary for using the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: /ttpss//nipppoooo cc/
pub c guides CodeeSuomi ss Moooollicoo cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips.... cc/ publico iublicc// ht//guideee iess/ee//// Ccce/oodSuuaaa ubmi ss i imPollccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
The full training details are provided in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For all experiments, the standard deviation of test accuracies for the three seeds is shown in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computer resources used for model reproduction and the corresponding training time for each task are provided in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eeuurpeeeeeee/eee/e wwtpppbbicc/ttiieeuueeeeeeeeoeee
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics. The research con- ducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
30

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
This paper properly credits the datasets and backbone codes used. The licenses of the datasets (Apache-2.0) license for LRA dataset) and codes (MIT licenses for backbones) are properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
31

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers thoode com/ dat a ssets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
This paper makes the used code available as an asset through the external link.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
32

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "J8L6Jkd36x", "reviewer": "Reviewer_c1Mv", "summary": "Inspired by the traditional layer-adaptive neural network pruning, this paper develops and verifies a layer-adaptive model order reduction (MOR) method to reduce the state dimension in DSSM models. The proposed method reveals state importance and prunes insignificant subsystems for a desired compression level, reducing unnecessary computations while bounding the output energy loss.", "questions": "As mentioned in the weakness, \n\n1) please use some statistics to show the importance of state dimension?\n\n2) if the state dimension is significant, can you provide more related works on reducing the state dimension, maybe those related works are not from the perspective of pruning to reduce the state dimension, but i think it is meaningful to discuss them.", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Proposes to use layer-adaptive pruning methods in new areas DSSM models. \n\n2. Extensive experiments among 10 tasks show the robustness and effectiveness of the methods. Besides, it conducts ablation studies on hyper-parameters in Appendix.", "weaknesses": "Since I am not an expert in DSSMs area, I just doubt the impact of the state dimension to the computational cost, it might be better to provide some statistics to show the impact. Besides, if this state dimension is very significant, do we have other some related works but not pruning to reduce the state dimension? Can you elaborate more on related works on reducing state dimension?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's assessment that the paper contributes to new areas of DSSMs and the experiments are thorough to show the robustness and effectiveness of the proposed method. The reviewer's concerns were the practical impacts and related works of reducing state dimension.\n\n> Please use some statistics to show the importance of state dimension.\n\nWe present the average evaluation step speed and peak GPU memory usage when the state dimension is reduced to the maximum pruning ratio that resulted in less than 1% accuracy loss using our proposed method. The evaluation was conducted by implementing an actual decrease in model size on an NVIDIA RTX 3090. While there is some variance depending on the channel size of each model per task, reducing the state dimension improves efficiency in both computational and memory costs.\n\n|  | ListOps | Text | Retrieval | Image | Pathfinder | Path-X |\n| - | - | - | - | - | - | - |\n| Pruning ratio | 0% | 60% | 50% | 30% | 30% | 30% |\n| Inference speed $\\uparrow$ | 1.0$\\times$ | 1.6$\\times$ | 1.7$\\times$ | 1.2$\\times$ | 1.1$\\times$ | 1.3$\\times$ |\n| GPU memory usage $\\downarrow$ | 1.0$\\times$ | 0.9$\\times$ | 0.6$\\times$ | 1.0$\\times$ | 0.8$\\times$ | 0.8$\\times$ |\n\n> If the state dimension is significant, can you provide more related works on reducing the state dimension, maybe those related works are not from the perspective of pruning to reduce the state dimension, but I think it is meaningful to discuss them.\n\nTo alleviate the issue of large state dimensions, related research has evolved with two primary approaches: architectural and algorithmic.\n\n**1. Architectural approaches**\n\nOne architectural approach is modulating states for parallelized inference. Specifically, an $h\\cdot n_{SISO}$-th order system is modulated by $h$ systems, each with $n_{SISO}$ states, where $h$ is the number of channels. This means each $n_{SISO}$-th order system handles information from a specific channel. To address the inability to integrate information across channels, a channel mixing layer is introduced. This is referred to as, in our paper, the multi-SISO architecture (Gu et al. [2021], Gu et al. [2022a]), allowing for training $h\\cdot n_{SISO}$ effective state spaces with reduced computational burden by parallelizing into $n_{SISO}$, thus enabling the development of many subsequent SSMs (Gupta et al. [2022], Gu et al. [2022b]).\n\nIn comparison, another architecture proposed by Smith et al. [2023], referred to as MIMO architecture, uses $n_{MIMO}\\ll h\\cdot n_{SISO}$ states to determine the output for a channel based on information from all input channels, making the mixing layer unnecessary.  This parameterization allows for high performance with much smaller state dimensions than equivalent block systems in multi-SISO layers. For instance, in the Path-X task, which involves the longest sequences among LRA tasks, this model shows state-of-the-art performance. However, both architectures lack optimization methods for model size, leading to computational inefficiencies when the task does not require such a high model capacity.\n\n- **Relation to the pruning approach**\n\n    Our state pruning method can be applied to all these architectures, effectively reducing the model size to fit task or resource requirements while preserving the performance. It identifies and adaptively prunes the states with the least impact on performance based on layer-wise energy transmission. In multi-SISO architectures, since each state is assigned to a specific channel, the difference in importance among states may not be significant. However, our method still shows superior retention compared to other methods. In MIMO architectures, the difference in state importance becomes more pronounced, and the accuracy loss caused by pruning varies significantly depending on how a method evaluates the state importance. Consequently, our method effectively preserves the most significant states and demonstrates superior performance compared to other pruning methods, as shown in the table in Response to Reviewer y832.\n\n**2. Algorithmic approaches**\n\nOne recent study has shifted from learning the element of decomposition of the transfer function in DSSMs to directly learning the coefficients of the transfer function (Parnichkun et al. [2024]). The motivation behind this approach is also to address the computational load caused by large state dimensions. The primary contribution of this approach is making computations even independent of the state dimension. However, as discussed in Appendix A, this approach has a weakness of only guaranteeing stability at initialization, unlike DSSMs, which can consistently guarantee stability due to their direct parameterization for the poles of the transfer function. Therefore, as a branch of DSSM research, our work contributes to enhancing the efficiency of DSSMs, which is dependent on the state dimension, by minimizing the state dimension.\n\n---\n\nA. Gu et al. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In *Advances in neural information processing systems*, 2021.\n\nA. Gu et al. Efficiently modeling long sequences with structured state spaces. In *The International Conference on Learning Representations*, 2022a.\n\nA. Gupta et al. Diagonal state spaces are as effective as structured state spaces. In *Advances in Neural Information Processing Systems*, 2022.\n\nA. Gu et al. On the parameterization and initialization of diagonal state space models. In *Advances in Neural Information Processing Systems*, 2022b.\n\nJ. T. Smith et al. Simplified state space layers for sequence modeling. In *The International Conference on Learning Representations*, 2023.\n\nR. N. Parnichkun et al. State-free inference of state-space models: The transfer function approach, In *International Conference on Machine Learning*, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "km213wmtMr", "reviewer": "Reviewer_y832", "summary": "In this paper, the authors propose Layer-Adaptive $\\mathcal{H}{\\infty}$ STate pruning (LAST), a deep state space model (SSM) pruning method that optimizes the state dimension of a deep diagonal SSM in terms of model-level energy loss. Experimental results on different tasks demonstrate that the proposed method performs better than Uniform $\\mathcal{H}{\\infty}$ and Global $\\mathcal{H}{\\infty}$ methods.", "questions": "Could the structured weight pruning methods on the traditional DNNs be applied to the pruning of deep state space models?", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a novel method to prune deep state space models. Experimental results on different tasks demonstrate the effectiveness of the proposed methods.", "weaknesses": "In the experiments, the work only compares the proposed method with Uniform $\\mathcal{H}{\\infty}$ and Global $\\mathcal{H}{\\infty}$ methods. It would be good if more weight pruning methods could be applied to deep state space models for fair comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad the reviewer appreciated the novelty and effectiveness of the proposed method. As the reviewer suggested, we compare our method to three magnitude pruning methods that use magnitude-based criteria from traditional DNNs and state pruning granularity as in the proposed method.\n\nState pruning is structured pruning, where all training parameters corresponding to the insignificant states are pruned based on a state importance metric. In traditional DNN, such as MLP, the transfer function of a layer is a weight matrix, $f(u)=Wu$, where each element $W_{ij}$ is a weight parameter. This allows evaluating the training parameters by their magnitude, $|W_{ij}|$, as the magnitude controls the input's influence on the output. Applying this to SSMs could involve unstructured magnitude pruning, which masks the high-magnitude elements of each system matrix according to the pruning ratio.\n\nHowever, as discussed in Response to Reviewer xVFN, there is a fundamental disparity between unstructured and state pruning methods in SSMs, making it unfair to compare them. Thus, we define magnitude-based state pruning methods for SSMs and evaluate their **ability to distinguish significant and insignificant states**, thereby preserving overall model performance.\n\n- **Uniform Magnitude**\n\n    Every layer is pruned to have an identical pruning ratio by evaluating the importance of each state $i$ as $|\\overline{\\lambda}_i|||\\mathbf{\\overline{B}}_i||||\\mathbf{C}_i||$. The pruning ratio is applied uniformly across all layers. While any $L_p$ norm can be used, we present the results using the $L_2$ norm as an example.\n    \n- **Global Magnitude**\n\n    The same state importance metric as in Uniform Magnitude is used, but the comparison group is extended from intra-layer to inter-layer, ensuring the pruning ratio is met globally across the entire network.\n\n- **Layer-Adaptive Magnitude-based Pruning (LAMP)**\n\n    This method employs a metric of $\\frac{|\\overline{\\lambda}_i|^2||\\mathbf{\\overline{B}}_i||^2||\\mathbf{C}_i||^2}{\\sum\\_{j\\geq i}|\\overline{\\lambda}_j|^2||\\mathbf{\\overline{B}}_j||^2||\\mathbf{C}_j||^2}$ adapted from Lee et al. [2021], which originally used $\\frac{W_i^2}{\\sum\\_{j\\geq i} W_j^2}$ for a real-valued weight parameter $W$. The state indices in the denominator are assumed to be ordered based on their evaluation using the basic magnitude criteria.\n\nExtending Table 1, we report newly conducted JAX experiments with 3 different seeds on an NVIDIA RTX 3090 as follows:\n\n| Method | Average pruning ratio | Average accuracy loss $\\downarrow$ |  State importance |\n| - | - | - | - |\n| Random | 33.00 (36.67) | 29.53 (32.82) | - |\n| Uniform magnitude | 33.00 (36.67) | 22.03 (24.48) | $\\vert\\overline{\\lambda}_i\\vert\\Vert\\mathbf{\\overline{B}}_i\\Vert\\Vert\\mathbf{C}_i\\Vert$ |\n| Global magnitude | 33.00 (36.67) | 17.49 (19.43) | $\\vert\\overline{\\lambda}_i\\vert\\Vert\\mathbf{\\overline{B}}_i\\Vert\\Vert\\mathbf{C}_i\\Vert$ |\n| LAMP | 33.00 (36.67) | 18.07 (20.07) | $\\frac{\\vert\\overline{\\lambda}_i\\vert^2\\Vert\\mathbf{\\overline{B}}_i\\Vert^2\\Vert\\mathbf{C}_i\\Vert^2}{\\sum\\_{j\\geq i}\\vert\\overline{\\lambda}_j\\vert^2\\Vert\\mathbf{\\overline{B}}_j\\Vert^2\\Vert\\mathbf{C}_j\\Vert^2}$ |\n| Uniform $\\mathcal{H}\\_{\\infty}$ | 33.00 (36.67) | 4.32 (4.80) | $\\frac{\\Vert\\mathbf{C}_i\\Vert^2\\Vert\\mathbf{\\overline{B}}_i\\Vert^2}{(1-\\vert\\overline{\\lambda}_i\\vert)^2}$ |\n| Global $\\mathcal{H}\\_{\\infty}$ | 33.00 (36.67) | 7.51 (8.35) | $\\frac{\\Vert\\mathbf{C}_i\\Vert^2\\Vert\\mathbf{\\overline{B}}_i\\Vert^2}{(1-\\vert\\overline{\\lambda}_i\\vert)^2}$ |\n| LAST | 33.00 (36.67) | **0.52 (0.58)** | $\\frac{\\frac{\\Vert\\mathbf{C}_i\\Vert^2\\Vert\\mathbf{\\overline{B}}_i\\Vert^2}{(1-\\vert\\overline{\\lambda}_i\\vert)^2}}{\\sum\\_{j\\geq i}\\frac{\\Vert\\mathbf{C}_j\\Vert^2\\Vert\\mathbf{\\overline{B}}_j\\Vert^2}{(1-\\vert\\overline{\\lambda}_j\\vert)^2}}$ |\n\nThe table lists the accuracy loss of pruning methods for S5 models (Smith et al. [2023]), averaged across all 10 tasks. Values in parentheses exclude non-compressible tasks with zero pruning ratios and accuracy loss.\n\nAt the same pruning ratio, LAST significantly outperforms other methods by exhibiting the least accuracy loss. \n\nRandom state pruning results show that all methods can identify insignificant states, but all $\\mathcal{H}\\_{\\infty}$ pruning methods surpass magnitude pruning methods. This implies the suitability of $\\mathcal{H}\\_{\\infty}$ pruning for SSMs, which can be explained by the transfer functions of SSMs. In an SSM layer, the multiplicative mapping from input to output in the frequency domain is given by $\\mathbf{Y}(z)=\\mathbf{C}(zI-\\mathbf{\\overline{\\Lambda}})^{-1}\\mathbf{\\overline{B}}\\mathbf{U}(z)$, where $\\mathbf{U}$ and $\\mathbf{Y}$ are the $\\mathcal{Z}$-transforms of input and output sequences $\\mathbf{u}$ and $\\mathbf{y}$, and $\\mathbf{C}(zI-\\mathbf{\\overline{\\Lambda}})^{-1}\\mathbf{\\overline{B}}$ is the transfer function matrix. \n\nThe importance of training parameters in SSMs can be evaluated by their influence in the frequency domain. Using the $\\mathcal{H}\\_{\\infty}$ norm, which measures the maximum gain of the transfer function in the frequency domain, $\\mathcal{H}\\_{\\infty}$ pruning optimizes the system by **minimizing the maximum gain alteration**, leading to better performance than magnitude pruning. Specifically, $\\overline{\\lambda}_i$'s importance in $\\overline{\\mathbf{\\Lambda}}$ is measured by $(1-|\\overline{\\lambda}_i|)^{-1}$ in $\\mathcal{H}\\_{\\infty}$ pruning, while magnitude pruning measures $|\\overline{\\lambda}_i|$. Additionally, LAST normalizes this importance by the total energy transmission of the layer, preventing extensive pruning of high energy-conveying layers and enabling stable global comparison and pruning.\n\n---\n\nJ. T. Smith et al. Simplified state space layers for sequence modeling. In *The International Conference on Learning Representations*, 2023.\n\nJ. Lee et al. Layer-adaptive sparsity for the magnitude-based pruning. In *The International Conference on Learning Representations*, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "29jQXxNcFW", "reviewer": "Reviewer_xVFN", "summary": "This submission proposed to conduct pruning on Deep State Space Model (DSSM). Basically, it formulates the output distortion (energy loss) after pruning, the derive that the importance of state is related to $H_{\\inf}$ norm, which is used as the pruning criteria. Besides, it proposes to use a greedy optimization by pruning a (entire) subsystem (with the remaining parts intact) to derive the importance of a subsystem (LAST), leading to pruning ratio for this subsystem.", "questions": "1. What is the difference between pruning states and parameters?", "limitations": "N.A.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. It is interesting to see pruning in DSSMs and the leading method is coupled with the structure.", "weaknesses": "1. Since it is a pioneer work, more comparison method should be used, such as magnitude pruning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and suggestions based on their insightful paper summary. We appreciate their assessment that our method, which is closely coupled with the structure of DSSMs, is particularly interesting.\n\n### **Weakness:**\n> Since it is a pioneer work, more comparison method should be used, such as magnitude pruning.\n\nWe appreciate the reviewer's suggestion to include more comparison methods, such as magnitude pruning. As a common point, this has been thoroughly discussed in Response to Reviewer y832. In summary, we have newly defined magnitude-based state pruning methods and presented the extended experiment results to show clearer effectiveness and contributions. Notably, LAST outperformed other methods by a large margin, and all $\\mathcal{H}\\_{\\infty}$ pruning methods consistently showed superior results over magnitude pruning methods.\n\n### **Question:**\n> What is the difference between pruning states and parameters?\n\nAs in channel pruning (He et al. [2017]), the term of state pruning represents its pruning granularity. That is, the difference arises in the unit of parameters being pruned at once, and the act of pruning training parameters follows the same approach as traditional pruning methods. In a DSSM layer, how the input and output are associated with a state is represented by the training parameters, which include the continuous-time system matrices $\\mathbf{\\Lambda}$, $\\mathbf{B}$, and $\\mathbf{C}$, which are discretized into $\\overline{\\mathbf{\\Lambda}}$, $\\overline{\\mathbf{B}}$, and $\\mathbf{C}$. The implementation of state pruning incorporates the identification of insignificant states and masking all parameters corresponding to the states, i.e., masking $\\lambda_i$ from $\\mathbf{\\Lambda}$, the row vector $\\mathbf{B}_i$ from $\\mathbf{B}$, and the column vector $\\mathbf{C}_i$ from $\\mathbf{C}$. As compared in Section 4, this concept originates from model reduction studies for single SSMs, and LAST extends this to the deep-layered architecture with nonlinear functions.\n\nTo explicitly demonstrate the necessity of state pruning in SSMs, we compared the performance of unstructured random pruning and (structured) random state pruning using the same experimental setup in Response to Reviewer y832. For unstructured random pruning, we pruned randomly selected elements from the system matrices, obtaining the following results:\n\n| Method | Average pruning ratio | Average accuracy loss $\\downarrow$ |\n| - | - | - |\n| Unstructured random | 33.00 (36.67) | 59.92 (66.58) |\n| Structured random | 33.00 (36.67) | **29.53 (32.82)** |\n\nDespite the same amount of parameters being removed, it can be observed that the model completely loses its performance when pruning is performed in an unstructured manner.\n\nThis is because unstructured pruning can disrupt the integrity of the model's learned dynamics by altering subsystems into significantly different ones. In contrast, state pruning maintains the functionality of subsystems, leading to less performance degradation. This highlights the importance of considering the structure and relationships within the model when applying pruning techniques.\n\n---\n\nY. He et al. Channel pruning for accelerating very deep neural networks. In *The IEEE International Conference on Computer Vision*, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers and ACs for their efforts in reviewing our paper. We are glad that the reviewers found the proposed method novel and noted that the experiments support its effectiveness. We sincerely appreciate the insightful and constructive feedback, and we have carefully responded to all comments with individual replies.

**Highlights of rebuttal**
- Clearer descriptions of the motivation to reduce state dimension, with comparisons to related works
- Deeper explanations for why the proposed state importance metric works well for state space models
- Extended experiments with magnitude pruning
- Detailed comparison between unstructured pruning and state pruning
- Demonstrations showing the computational cost reduction by LAST

Meta-review: Thanks for your submission to NeuIPS.

This paper was borderline, leaning to accept, prior to the rebuttal and discussion period.  There were questions/concerns about the state dimension, along with requests for some additional experiments.  The authors provided a very thorough and useful rebuttal, addressing these issues satisfactorily.  In general, the reviewers agreed (with possibly some lingering questions still at the end), but ultimately all three reviewers indicated that they were OK accepting the paper.  As a result I am happy to recommend acceptance of this paper.

Please do add the additional results from the rebuttal into the final manuscript.
Decision: Accept (poster)