Parameter-Efficient Fine-Tuning of State Space Models
Authors: Kevin Galim
Wonjun Kang
Yuchen Zeng
Hyung Il Koo
Kangwook Lee
Venue: ICML 2025 poster (2025)
Keywords: parameter-efficient fine-tuning
state space model
mamba
lora
Primary area: general_machine_learning->transfer_multitask_and_metalearning

Abstract: Deep State Space Models (SSMs), such as Mamba (Gu & Dao, 2024), have become powerful tools for language modeling, offering high performance and linear scalability with sequence length. However, the application of parameter-efficient fine-tuning (PEFT) methods to SSM-based models remains largely underexplored. We start by investigating two fundamental questions on existing PEFT methods: (i) How do they perform on SSM-based models? (ii) Which parameters should they target for optimal results? Our analysis shows that LoRA and its variants consistently outperform all other PEFT methods. While LoRA is effective for linear projection matrices, it fails on SSM modules—yet still outperforms other methods applicable to SSMs, indicating their limitations. This underscores the need for a specialized SSM tuning approach. To address this, we propose Sparse Dimension Tuning (SDT), a PEFT method tailored for SSM modules. Combining SDT for SSMs with LoRA for linear projection matrices, we achieve state-of-the-art performance across extensive experiments.

Parameter-Efficiet
Fine- Tuning of State Space Models
Kevin Galim
Wonjun Kang" 12
Yuchen Zeng"
Hyung Il Koo
Kangwook Lee
Abstract
subquadratic time complexity. Efficient attention alterna- tives often rely on State Space Models (SSMs) or their variants (Gu al.. 2021: 2022b:a; Gu & Dao, 2024), which are akin to linear RNNs, maintaining hidden states of fixed size for sequential processing. S4 (Gu al., 2022b;a) over- comes RNNs' parallel training limitations by constraining parameter structures, enabling a convolutional form for effi- cient parallel computation. S6 (Gu & Dao, 2024) improves this with input-dependent parameters, enabling selective fo- cus on relevant information per token. Building on S6 with linear projection matrices (analogous to the Feed-Forward Networks in Transformer layers), Mamba-I (Gu & Dao, 2024) emerged as prominent SSM-based model. Mamba-I was later extended to Mamba-ll (Dao & Gu, 2024), with both models achieving Transformer-level performance in language modeling and gaining widespread recognition.
Deep State Space Models (SSMs), such as Mamba (Gu& & Dao, 2024), have become powerful tools for language modeling, offering high perfor- mance and linear scalability with sequence length. However, the application of parameter-efficien fine-tuning (PEFT) methods to SSM-based mod- els remains largely underexplored. We start by in- vestigating two fundamental questions on existing PEFT methods: (i) How do they perform on SSM- based models? (ii) Which parameters should they target for optimal results? Our analysis shows that LoRA and its variants consistently outper- form all other PEFT methods. While LoRA is effective for linear projection matrices, fails on SSM modules-yet still outperforms other meth- ods applicable to SSMs, indicating their limita- tions. This underscores the need for a specialized SSM tuning approach. To address this, we pro- pose Sparse Dimension Tuning (SDT), PEFT method tailored for SSM modules. Combining SDT for SSMs with LoRA for linear projection matrices, we achieve -fate-of-the-art performance across extensive experiments.
As SSMs gain popularity, performing parameter--ffiiceent fine-tune Ane-ttning (PEFT) pretrained models for downstream tasks is crucial, since full fine-tuning is costly and ineffi- cient. Numerous PEFT methods (Houlsby et al., 2019; Hu et al., 2021; He et al., 2021; Li & Liang, 2021; Lester et al., 2021; Zaken et al., 2022: Liu et al., 2021; 2022; Houlsby et al., 2019) have been developed, achieving notable success on Transformer models. The most popular PEFT meth- ods fall into three categories: (i) input-nnecttion methods, which add sequences to the model's 's main input (Lester et al., 2021) or prepend tokens to the intermediate inputs at each layer (Li & Liang, 2021): (ii) aechiirctuureeehhnneemeee methods, which adjust the model architecture. For example, Houlsby et al. (2019) added layers between Transformer lay- ers, while Additional scan (Yoshimura al., 2025) expands state dimensions in the SSM module; (iii) weight-tuning methods, which directly modify existing model weights. Notable weight-tuning approaches include BitFit (Zaken et al., 2022), which updates only bias terms, and LoRA (Hu et al., 2021), which modifies weight matrices through low- rank updates, along with its variants such as DoRA (Liu et al., 2024) and LoRA+ (Hayou et al., 2024). For simplicity, we denote LoRA and its variants as LoRA".
1. Introduction
In the past few years, Large Language Models (LLMS) such as ChatGPT (Achiam et al., 2023; Brown et al., 2020) have achieved groundbreaking performance and are now widely used in daily life. While many models rely on the Trans- former architecture (Vaswani et al., 2017), its quadratic time complexity due to the attention mechanism poses challenges for long sequences. To address this, alternative architec- tures such as linear attention (Katharopoulos et al., 2020), RWKV (Peng et al., 2023), RetNet (Sun et al., 2023), and Mamba (Gu& & Dao, 2024) have been developed, offering
'Equal contribution. Authors listed in alphabetical or- der. FFuriosaAl "Seoul National University Universsiyy of Wisconsin-Madison. Correspondence to: Kangwook Lee <kang- look.lee edsc.edu>.
Despite the success that existing PEFT methods have achieved in adapting Transformer-based models, their effi- cacy in adapting SSM-based models remains largely under- explored, leaving many interesting questions open.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Parameter- Efficiet Fine-Tuning of State Space Models
Benchmarking PEFT Methods
Better PEFT Method on SSM
Froeen
Trakable
LORA
LoKA
Sporse Dimemiion Turting
LofA
Core Pansetariids
Prompt Tuning
N
LottA
Prefox TuniinTnnnng
Sec.
Sec. 4
Sec.
Additional-Scam
Table
Convid
Convie
ConvZd
LoKA
LoRA
W
Other PEFT Methods
LoRA (LinProj)
LoRA (LinProj) LaRA (SSM)
LoRA (LinProj) SDT (SSM)
Figure 1. A visual guide to PEFT methods in SSM-based models: benchmarking and innovation. We compare various existing PEFT approaches SSM-based models, demonstrating that LoRA applied to linear projection matrices outperforms all other methods. However, extending LoRA to SSM modules fails to yield further improvements. To address this, we propose Sparse Dimension Tuning (SDT), which achieves statc-of-the-an performance SSM-based models when combined with LoRA for linear projection matrices.
1.
Do existing popular PEFT methods remain effective for SSM-based models?
can be considered equivalent to training dimensions set to zero. SDT selectively updates channels and fine-tunes specific dimensions within them, as illustrated in Fig. 1.
If applicable, what is the optimal way to integrate these methods into SSM-based models, and which parameters should be updated?
Demonstrating Effectiveness of SDT. Through extensive experiments, we demonstrate that integrating SDT into SSM-based models, combined with applying LoRA" to their linear projection matrices, achieves state-of-the-art fine-tuning performance.
3. If not, can we design specialized variants tailored to SSMs that yield superior performance?
Our main contributions to address these questions are:
Comprehensive Benchmarking of PEFT Methods. We benchmark six widely used PEFT methods across three categories on diverse tasks, including natural language understanding, generation, and computer vision. We evaluate these methods on both SSM-based models (i.e., Mamba) and a hybrid model (i.e., Jamba (Lieber et al., 2025)), which consists of both Transformer layers and Mamba layers. Our results show that LoRA* consistently outperforms all other PEFT methods both SSM-based and hybrid models. However, its effectiveness is limited to linear projection matrices, as further tuning of SSM modules does not improve performance. Notably, other methods applicable to SSM modules perform worse than LoRA*, further underscoring the need for a specialized approach to tuning SSM modules.
The roadmap of our paper is illustrated in Fig. 1. Our code is available at ht https:: / thub. com com/furiiii Ptrriosa-al.. sam cem-peft.
2. Related Works
Concurrent Works of PEFT SSMs. Several concur- studies (Halloran al., 2024; Yoshimura al., 2025; Kang al., 2025) have investigated PEFT methods for SSM-based models. Halloran et al. (2024) studied both in-context learning and parameter-efficient fine-tuning, with an orthogonal focus on analyzing Mamba's 's stability under mnixed-precision training using Lyapunov exponents, Kang et al. (2025) introduced state-based PEFT methods and proposed State-offset Tuning, solety focusing fine-tuning Mamba's S6 blocks. Yoshimura et al. (2025) benchmarked multiple PEFT approaches cpproaches-iicluuding including established methods and new method called Additional-scan (which adds a trainable state dimension to the SSM module), plus par- tial tuning (fine-tuning only a subset of parameters) introduced MambaPEFT through PEFT search strategies. While Yoshimura al. (2025) solely focused on Mamba-I, providing an in-depth study of that particular architecture, our work investigates broader class of SSM-based models including deep S4, Mamba-I, Jamba in the main body, as well as Mamba-Il presented Sec. C.2 and E.2, aiming to offer general insights on how to effectively tune SSMs rather than focusing on a single variant.
Introducing Sparse Dimension Tuning (SDT) for SSM Modules. To develop an effective method for tuning SSM modules, we conduct a theoretical analysis to under- stand the roles different parameters. This analysis moti- vates the Sparse Dimension Tuning and Praning (SDT-P) method, which improves efficiency by freezing and prun- ing certain channel and state dimensions while training only the remaining ones. We establish theoretical guar- antees for its effectiveness in SSM-based models when combined with LoRA applied to linear projection matrices. We then simplify SDT-P into Sparse Dimension Tuning (SDT) by omitting explicit pruning, as pruned dimensions
2

---

Parameter-Efficiet Fine-TTunin of State Space Models
Sparse Tuning. Several studies have explored sparse pa- rameter selection in fine-tuning (Song al., 2024) and skill localization (Panigrahi et al., 2023). Song et al. (2024) showed that sparse tuning is an effective PEFT method, link- ing the low intrinsic dimensionality of pre-trained models to the proportion of parameters needing updates. They propose selecting optimal fine-tuning parameters based on gradient magnitudes. We enable sparse tuning for SSM by apply- ing sparsity across entire dimensions (channel and state) rather than specific neurons. Panigrahi et al. (2023) focused on identifying neurons responsible for specific downstream tasks by fully fine-tuning the model and computing neuron masks to minimize task loss. While effective for skill local- ization, this method is computationally expensive and not optimized for parameter-efficient fine-tuning.
tion, and residual connection from input to output. Let + represent the element wwine product, and S4(.) denote the S4 mechanism, where each channel's output follows (3) with its convolutional kernel K (d) To facilitate theoretical analysis, certain subtle details-suue as activation functions- -ay differ slightly from those in previous studies (Gu et al., 2022b:a). We define the deep S4 layer as:
y: = ReLU(W S4,( 22) + 3 + u G xt), (4)
where W E R and B E RR. represent the linear pro- jection matrix and bias, respectively, and u E R is the coefficient of the residual connection. Trainable parameters include SSM parameters (A(d), B(d), Cka), (4)) across D channels with A()) being diagonal, well as linear layer 3W and residual connection
In Sec. A, we provide a more detailed discussion of related work on SSMs and PEFT.
Selective State Space Models (S6). All SSMs mentioned above exhibit linear time invariance (LTI), meaning their dynamics remain constant over time. A key limitation of LTI SSMs is their fixed dynamies, hindering selec- tive context extraction and input-dependent state transi- tions. S6 (Gu & Dao, 2024) addresses this by making parameters input-dependent. At each time step t, given input Dt E RDD, S6 introduces input-dependent step sizes A, (A! (1) ..... (4."....,..)))) (D))T E R?, input transition vectors B, E H and output mapping vectors C, E IR via linear projection:
3. Preliminaries
3.1. State Space Models
Discrete-Time SSMs. The initial SSM is derived from a specific continuous system that maps a one-dimensional function or signal xt) R to v(t) E I via an H-- dimensional latent state h(t) E R² as described in (1). In this formulation, input transition vector B E # indicates the input's impact on the state of the system, state matrix A E #x characterizes the sys- tem's internal state dynamics, and the output mapping vector C E IIIH relates the state to the output y(t).
A. softplus(Waas Ba), B, = WDE., C. Woz.,
where the diagonal state matrices A ...., A (D) remain input-independent. The weight WA E DDD is fac- torized as WA = WA. Wa,t with WA.1 E RDDR, Wa.t E RR to reduce computation (Wang et al., 2021; 2023a). Trainable parameters in S6 include (d) across D channels, WA.,. WA.1 and BA for computing A., and WB. We E HxD for computing B,, C,. Discretization follows: A exp(A (d) A(d), ). (d) A(4) B. Unlike (d)
h((t)= Ah(t)+ + Bu((t)
h Ahr-1 + BrG,
(1)
(2)
y(t) = Ch(()
y1 = Chh
K = (CB, CAB.
CA' B),
(3)
= (z), 21) K
To handle discrete inputs, the continuous parameters (A, B) are discretized into A, B) using a learnable step size A E R. A common discretization rule, the zero-order hold, de- fines A = AA), B- (AA) (exp( -) AB. The discrete-time SSM, given in (2), enables efficient in- ference via long convolution described in (3). For multi- channel inputs y E RD, separate SSMs are used per channel, with superscript (d) indicating channel-specific parameters when needed.
$4, where (d) varies per channel, S6's variation on ((d) stems from the scalar A Additionally. S6 shares C, for all channels at each time step while $4 assigns distinct (14) to each channel.
Mamba & Jamba. Similar to the Transformer block, which consists of attention and linear layers, the Mamba-I block proposed by Gu & Dao (2024) features an S6 mod- ule, a point-wise ID causal convolution layer (Convld) for token mixing, linear layers- including input (Win) and output (Wou) projection layers and gated MLP. Mamba- II (Dao & Gu, 2024) further simplifies the state matrix A to be a scalar. Building on Mamba-I, Jamba (Lieber et al., 2025) introduces a hybrid architecture that integrates both Transformer blocks and Mamba blocks, leveraging the strengths of both to enhance performance. This paper
Structured State Space Sequence Model (S4). S4, in- troduced by Gu et al. (2022b), is an early application of SSMs in deep learning, featuring a diagonal state matrix A. To introduce non-linearity and cross-channel mixing, S4 integrates a wosi linear layer, activation func-
Note that B, C are vectors; we bold capitals for consis- tency with prior work (Gu et al,, 2022b; Gu & Dao, 2024).
3

---

Parameter-Efficien Fine- Fine-Tuning of State Space Models
focuses Mamba-I (referred as Mamba in this paper) and Jamba, deferring Mamba-Il discussions the appendix.
LoRA by applying different learning rates to W and W enabling more effective feature learning. In contrast, Bit- Fit (Zaken et al., 2022) updates only the bias terms, offering lightweight and highly parameter-efficien alternative.
3.2. Parameter-Rffficint Fine-Tuning
Input-Injection Methods. Input-injection methods, such as prompt tuning (Lester et al., 2021) and prefix-tuning (Li & Liang, 2021), enhance the model's input by injecting specialized sequences. Prompt tuning prepends set of trainable embeddings P E DDM to the original input
4. Benchmarking PEFT Methods on SSM-based Models
In this section, we examine the effectiveness of popular PEFT methods when applied naively to SSM-hased models, specifically Mamba and Jamba.
X DXN, forming the concatenated sequence X X Prefix-tuning (Li & Liang, 2021) instead injects learnable vectors into the key and value matrices of each attention layer. For a Transformer layer, it prepends prefix states pK pV E ERLXD to the original projections:
4.1. Experiment Setup
We evaluate PEFT methods across three categories: input- injection, archiectur--uuhanemmnn, and weight-tuning. For input-injection methods, we use prompt tuning (Lester et al., 2021) and tuning employs an overparameteriized MLP for stable op- timization. For architecture-enhancome clude Additional troduces and fine-tunes newly added state dimensions in SSM modules. For weight-tuning, we consider BitFit (Za- ken et al., 2022) and LoRA*, including LoRA (Hu et al., 2021) and DoRA (Liu al., 2024), while LoRA+ (Hayou et al., 2024) is deferred to Sec. E.2. BitFit fine-tunes the bias terms of Convid and WA,1-
K K]. V - VV.
where K and V are the key and value matrices derived from the input. We note that prefix-tuning is functionally equivalent to prepending soft tokens to the imput at each attention layer and discarding the corresponding outputs associated with the prepended tokens. This view simpli- fies adaptation to SSMs, which lack explicit key and query projections. Yoshimura et al. (2025) also adopt this imple- mentation, though they refer to it as affix-tuning
(Li & Liang, 2021), where prefix-
methods, we in- (Yoshimura et al., 2025), which in-
Sechiteture--Fnhancemeee Methods. These methods modify the model's internal structure to introduce tunable components. In the context of SSMs, one example is Additional-scan (Yoshimura et al., 2025), which expands the state dimensions within the SSM block and fine-tunes only the added parameters, leaving the original weights untouched.
We use six datasets spanning different domains: GLUE for natural language understanding (Wang et al., 2019), DART for RDF-to-text generation (Nan et al., 2021), SAM- Sum (Gliwa et al., 2019) for summarization, Spider for text-to-SQL generation (Yu al., 2018), and two vi- sion datasets CIFAR-10 (Krizhevsky et al., 2009) and CelebA (Liu al., 2015), with the vision datasets pro- cessed by cropping, resizing, and flattening pixel values into space-separated numerical sentences. Details are in Sec. B. Prefix-tuning requires significantly more parame- ters than other PEFT methods due to its per-layer MIP for projecting fixed sequences into soft tokens. For all meth- ods-except prefix-tuning and the special case of LoRA and DoRA when applied to both linear projection layers-we limit trainable parameters to below 1 for Mamba and be- low 0.15% for Jamba. For Jamba, all PEFT methods are applied to Mamba layers, while Transformer layers remain frozen to isolate performance effects. See more details in Sec. C.I.
Weight-Tuning Tuning Methods. Notable weight- meth- ods include LoRA (Hu et al., 2021) and its variants (Liu et al., 2024; Hayou et al., 2024), as well as BitFit (Zaken et al., 2022). LoRA (Hu et al., 2021) fine-tunes a model by introducing low-rank updates to its weight matrices. Given weight matrix Wo DxD, LoRA updates it as follows:
W W. + W,W,
with W E DDR W. E RRxD and R < D being the rank. Only W. and W are trained, reducing the num- ber of trainable parameters from 2 to 2RD. Weight- Decomposed Low-Rank Adaptation (DoRA) (Liu et al., 2024) improves upon LoRA by decomposing the weight matrix into two components: magnitude (m e (R and direction (W, Wr), leading to the formulation
4.2. Results
Table I summarizes the benchmarking results. Detailed results for GLUE and Spider subtasks appear in See. C.2. We analyze the results from three key perspectives below.
Wo+ + W W,
W = m IWo + W.W,!
This additional parameter m enhances both training capac- ity and stability. LoRA+ (Hayou et al., 2024) modifies
4

---

Parameter-Eficient Fine- Fine-Tuning of State Space Models
Major Target Module
GLUE
DART
SAMSum
Spider
CIFAR-10
CelebA
Model
Method
Avg. Score
METEOR
BLEU
R1
R2
RL
Acc.
Acc.
Acc.
Prompt Tuning
Other
63.8
66.2
39.8
50.1
25.6
41.6
43.6
30.4
82.5
Prefix-Tuning
SSM
68.6
66.6
42.5
50.6
26.5
42.1
39.7
41.0
86.5
BitFit
Both
76.8
67.0
43.7
50.3
25.7
41.9
48.4
44.4
86.9
SSM
76.9
68.8
48.0
50.4
26.0
41.8
55.0
52.3
87.0
LoRA
LinProj
81.2
70.9
49.5
50.9
27.0
42.3
57.5
61.0
87.0
Mamba
Both
80.3
70.2
52.2
50.7
26.8
42.4
57.0
58.4
SSM
77.9
68.3
47.3
48.1
24.2
39.6
55.3
44.5
87.1
DoRA
LinProj
81.1
70.7
51.6
51.0
26.9
42.8
60.7
57.6
86.7
Both
80.8
70.8
51.4
51.3
27.2
43.0
58.1
58.2
89.8
Additional-Sean
SSM
62.4
60.6
15.8
37.6
17.5
30.9
26.9
32.2
86.0
Full Fine-Tuning
Both
80.5
71.0
51.8
51.2
27.3
42.9
66.2
60.0
89.4
Prompt Tuning
Other
73.3
54.1
6.3
54.7
31.8
46.8
74.9
40.9
85.6
Prefix- Tuning
SSM
56.9
59.6
14.4
11.5
1.8
10.4
0.3
29.9
82.2
BitFit
Other
75.2
59.2
14.8
54.7
31.9
47.0
73.7
45.6
86.3
Jamba
LoRA
LinProj
73.9
68.9
37.8
54.6
32.3
46.8
59.7
89.0
DoRA
LinProj
71,4
68.1
28.8
55.2
32.2
47.3
70.9
58.6
89.0
Additional-Scam
SSM
68.3
63.3
20.1
53.4
30.5
45.6
69.3
50.6
Table Benchmarking popular Parameter- Parameter-Efffcieett Fine-Tuning (PEFT) methods on Mamba (Gu Dao, 2024) and Jamba (Lieber et al., 2025) across six real-world datasets. R1/R2/RL stand for ROUGE-1/2/1 We evaluate PEFT applied to different target modules: SSM module only, linear projection matrices (LinProj) only, both, or other components such as embedding layer. For both Mamba and Jamba, all methods use fewer than 1% and 0.15% of parameters, respectively. except when the target module for LoRA DoRA is set "Both" or when prefix-tuning is applied. Comprehensive hyperparameter tuning was performed for all methods. Bold values indicate the best performance for each model (Mamba and Jamba) separately, while underlined values denote the second-hest performance for each task, excluding full fine-tuning. Key findings include: (i) among PEFT methods applied to SSM modules, LoRA" outperforms others, (ii) for all PEFT methods, LoRA* achieves the best performance, (iii) applying LoRA" to linear projections yields results comparable to applying it to both linear projections and SSM modules, while outperforming its application solely to SSM modules, and (iv) input-injection methods (i.e., prompt tuning and reffix-tunng),..o are generally ineffective.
Superiority of LORA".
The most prominent finding is that LoRA* consistently outperforms other PEFT methods (e.g., prompt tuning, prefix-tuning BitFit, additional-scat regardless of the target module.
nation of both. Note that S6 in Mamba and Jamba includes fine-grained parameters like x x_proj (Wn. Wo. ) and dt dt_proo (Wa,t), which were already explored by Yoshimura et al. (2025) on Mamba. We defer a deeper dis- cussion of them to Sec. C.4 and focus on the key question here: Is applying LoRA" to SSM modules necessary for performance gains? By narrowing our scope, we aim clar- ify LoRA"'s impact across major components (e.g.. SSM modules, linear projection matrices) rather than all specific parameters.
Finding: Across all target modules, LoRA" surpasses existing PEFT methods in performance.
Even when restricted to SSM modules, LoRA' still outper- forms all other PEFT baselines applied to the same target.
We evaluate LoRA"'s performance on linear projections us- ing Wi.. Wos. and both combined. Since the performance of different combinations linear projections is consistent across datasets (see Sec. C.4.), we only report the results for LoRA* applied to Wi, in Table For SSM modules, apply LoRA" weight matrices, including those for the input-dependent step size A. For state transition matrices A, we treat their diagonal structures as vectors, concatenate them across channels to form a matrix, and apply LoRA* Table summarizes results for the best-performing configu- rations (see Sec. C.2 for full results). Based on these results, we present the following finding:
Limitations of Input-Injection Methods. Input-injection methods like prefix-tuning are ineffective for based models (Table 1), as their expressiveness reduces to tun- ing only the initial hidden state (Proposition 1). Formal statement, proof and empirical verification are in See. C.3.
Optimal Application of LoRA' in SSM-based Models. Table shows that LoRA* outperforms all other PEFT meth- ods most scenarios. From our results, we explore the optimal layers for applying LoRA* in SSM-based models: the SSM module, the linear projection matrices, or a combi-
5

---

Parameter-Efficiet Fine- Fine-Tuning of State Space Models
the expressive power of certain SSM parameters (i.e., WB. Wc, and Wa.t), aligning with our empirical observation that tuning only the linear projections already performs well. However, key limitation of tuning only the linear pro- jection matrices remains: such tuning lacks the expressive power to affect the state matrix A, which is an essential operations. Therefore,
Finding: For LoRA": Tuning on SSMs is less effective than tuning linear projection matrices, with the latter performing comparably tuning both.
Detailed experiments, including LoRA" on different linear projection matrices and additional evaluations of Mamba-Il, are presented in Sec. C.2. These experiments reinforce the finding that LoRA* is highly effective for linear projections but less suitable for SSM modules.
parameter for sequence-lo-sequenee tuning the SSM modules is still necessary. Existing PEFT methods fall short in effectively tuning SSM modules: (i) alternative methods underperform compared to LoRA* on SSM modules, and (ii) applying LoRA' to SSM modules does not improve performance beyond applying it to lin- ear projections alone. These findings highlight a gap in current PEFT techniques for SSM modules, leading to an importantca question: Is there a more effective strategy for fine-tuning SSM modules?
To further elucidate this concept, we present the following lemma, which examines a simplified model architecture consisting of S6 with two linear input projection matrices at each layer. We demonstrate that fine-tuning one input projec- tion matrix encompasses the expressivity of fine-tuning the parameters WB. B. Wc, and Wa.t- Consider an S6 model with two input projection matrices Win.1. Win Dx the first affects how internal parameters depend on the in- put, while the second governs the input passed directly into the S6 module. Under this setup, the output VN can be expressed as:
5. Sparse Dimension Tuning
This section aims to develop an algorithm for tuning SSM modules. In doing so, start by first analyzing the roles of different parameters, outlined in Lemma 2. This analy- sis motivates us to classify channels and state dimensions into three categories: (i) zero, (ii) trainable, and (iii) frozen, leading to the development of the Sparse Dimension Tuning and Pruning (SDT-P) method. We then establish theoret- ical guarantees for applying SDT-P to SSM modules and LoRA to linear projection matrices (Theorem 1). Finally, simplify SDDT- into Sparse Dimension Tuning (SDT) by omitting pruning, as pruned parameters can be effectively considered as being trained to zero. This simplified version serves as the primary method used in our experiments.
Cepardeeeeeee. CN C(WEIRA)
depeeder deponden TT. Bo(Wao
60 NN
II A(Wn
Warme)
best per
deomeee.dgeettrreiiei... ofter prrpetios
When Wm.1 dard architecture with a single input projection fol- lowed by an S6 layer. For simplicity, we let GA = O. Then the full model is parameterized by (L (D) I: WB, Wc, Wa.t. Wa.,; Win.1. Wmn).. As- sume none of the parameters are zero and > 2H + R, where Ri is the rank of WA,J W Wa.t-
W m2 this reduces to stan-
5.1. Understanding Key Parameters in S4 Modules
Lemma 1 (Expressivity of Fine- TineTTnnng Projection Matri- ces). Consider two models with the architecture described above. Let:
Problem Setting. Inspired by the work by Zeng & Lee (2024), we analyze the expressive power of S4 parameters using similar framework. We assume well-performing target model and frozen model (pretrained random) and aim to update the frozen model efficiently to match the target. Following Zeng & Lee (2024), we assume the frozen model has a capacity at least as large as the target model. This assumption ensures analytical tractability and is reasonable, as frozen models are typically overparameterized in practice. Both models are S4 with hidden dimensions H, (target) and H H. (frozen). Assuming all hidden dimensions active (i.e., all parameters are non-zero), we define their dynamics using discretized parameters (AB.C) C):
A target model, f parameterized by (fA* W B W.. WA, W Win,, Win);;
dd=
A frozen model fo parameterized by (1) ((A+(d)) D WB.
Wc. WA,,. W Win.. 2
The two models share {Aa(d) D Ja=i p. W and W in while differing in WR. Wc. Wa,,, and Wo.1- Then, there ex- ists an updated projection matrix Win such that the frozen model matches the oulpul of the target model without updat- ing WB, WC, Wa, for any input sequence, ie.,
(Target model) f((x)= = =" C.A. B..mm m==
(A:(0)))))) 2 WB. Wo, WA, W... WHn Win)) = f (A00)) 12. Wi. We W W - Wit,, Wha))
fo(x), = =E" m=1 CoA m- "Bow
(Frozen model)
We expand on this discussion in Sec. C.4, where we present both theoretical proofs and empirical validation. The lemma shows that tuning the linear projection matrix can match
where diag A.), E RH. diag An), dia)(An), Bo An), Co E R This formulation shows that the S4 module remains unchanged even if the state dimensions are permuted.
6

---

Parameter-Effiiien Fine- Fine-Tuning of State Space Models
Parameter Efficiency Analysis on S4. We analyze the parameter efficiency of updating frozen S4 module after discretizing its parameters (Ao, Bo, Co) to match the func- tionality of a target S4 module with discretized parameters (A., B., C.). Based on this setup, we present the follow- ing result characterizing the minimum number of parameters that must be tuned for functional equivalence.
dimensions into three groups: pruned, frozen, and trainable. Then, the state dimensions of each trainable channel are also categorized as pruned, frozen, or trainable. This hi- erarchical selection ensures that updates are applied only when necessary, while pruned dimensions are discarded and frozen dimensions remain unchanged.
Dimension Selection Algorithm. To enable this struc- tured tuning process, we first introduce our dimension selec- tion algorithm. The algorithm starts with a warmup epoch, where the SSM modules are updated using a subset of the dataset for one epoch. After this warmup, we classify chan- nel dimensions based on the magniuude of the state matrix A: dimensions with small magnitude are pruned (set to zero), those with significant changes are marked train- able, and the rest remain frozen. Next, we apply the same classification to state dimensions, but only within the train- able channels. The detailed pseudo-code is in Sec. D.4.
Lemma 2 (Minimal Parameter Adjustment for S4 Fine-Tum Tun- ing). Assume all hidden dimensions of the target model f* are non-zero, i.e., all elements of (iig(A.)) @ B_ G C2 are non-zero. To update frozen model so such that it becomes functionally equivalent the target model fr, the minimum number of tunable parameters is:
eliminatiin recianiiant dimeusioons
min A.B.C [ding(A) (A) @ B C
I (11-11):11
(5)
atigning remaining doncarions with model
+ A A.U...... A.1, + B o C Jun.
B. 0.
Parameter Update Scheme. Once the channel and state dimensions are selected, we determine how to update the parameters. (S4) For S4, Gu et al. (2022a) showed that tuning C alone is as effective as tuning both B and C. Therefore, we always freeze B and update only A and C. Specifically, an entry in A or C is trainable if and only if both its channel and state dimensions are trainable. If either the channel or state dimension is pruned, the entry is pruned as well. All other entries remain frozen. (S6) For S6, where parameters are input-dependent. we update A, WB. and Wc instead. Since WB and Wc operate across channels, we categorize their updates based only on channel dimensions- we do not update individual state dimensions differently for each channel. Based on this categorization, we mark the corresponding columns of WB and Wc trainable, frozen, or pruned accordingly.
subject to
(A, E A(P B.. CoP): ; is permutation matrix).
Note that the search space consists of all possible S4 param- eterizations that can be obtained by permuting the hidden dimensions of the frozen model. Proofs and further details are provided in Sec. D.1. This result highlights three dis- tinct roles of the state dimensions. First, any dimensions that do not contribute to the target function (represented by the first term in (5)) are effectively zero and can be pruned. These correspond to state dimensions larger than those of the target model after permutation, indicating that redundant information can be directly removed to eliminate its impact. Second, among the remaining dimensions, alignment is nec- essary for those that do not already match the target. The state matrix plays a crucial role in sequence modeling by capturing dependencies between tokens at different po- sitions. To achieve functional equivalence (as represented by the second term in (5)), A must be aligned. Notably, dimensions that are already aligned with the target require no updates. These two insights motivate our Sparse Dimen- sion Tuning and Pruning (SDT-P) method, which classifies hidden dimensions into three categories: (i) zero, (ii) frozen (already aligned), and (iii) trainable. Finally, the third term in (5) indicates that the expressive power of B and C is essentially equivalent, meaning that tuning either one is equivalent to updating both.
The dimension selection algorithm and parameter updates together form the SDT-P method for tuning SSM modules. Next, we provide theoretical guarantees for applying SDT-P to SSM modules and LoRA* to linear projection matrices.
5.3. Expressive Power of SDT-P Combined with LoRA
Our analysis focuses on simplified SSM-based models, where each layer consists of an SSM module followed by linear projection matrices with residual connections. We refer to this structure as deep SSM layer: i) a deep S4 layer consists of an S4 module followed by linear projections; ii) a deep S6 layer follows the same structure but replace S4 with S6. A deep S4 model is composed of deep S4 layers, while deep S6 model consists of deep S6 layers. The detailed formulation of deep S4 layers is provided in Sec. 3, and deep S6 layer follows the same structure with S4 replaced by S6. The following theorem highlights the expressive power of SDT-P on updating SSM modules, where each layer uses a single type of SSM module (S4 or S6) followed by linear projections.
5.2. Sparse Dimension Tuning and Pruning (SDT-P)
Building on Lemma 2, we introduce SDT-P, the precursor to Sparse Dimension Tuning (SDT). SDT-P updates param- eters selectively based on the role of each state dimension. In the multi-channel case, we first categorize the channel
7

---

Parameter-Efficiet Fine- Fine-Tuning of State Space Models
Theorem 1 (Expressive Power of SDT-P with LoRA on Simplified SSM-based Models). Assume all layers use lin- ear activations. Let fo be a frozen deep S4 or 56 model with
5.4. Sparse Dimension Tuning (SDT): A Pruning-Free Alternative
While SDT-P classifies channels and states into three cate- gories, we simplify our approach by omitting pruning and categorizing parameters as either trainable or frozen. We refer to this simplified method as Sparse Dimension Tun- ing (SDT). This reduces the number of hyperparameters, as pruned parameters are effectively equivalent to being trained to zero. The resulting dimension selection approach is outlined in the pseudo-code (Alg. 1), which corresponds to the update scheme illustrated in Fig. 1. Experiments will show that this simplification remains effective.
L layers, each containing H hidden states per channel. Let I be a smaller target model of the same type (S4 or S6), with no residual connections, L* < L layers, and I < H hidden states per channel. Then, there exists a set of param- eter updates to fo satisfying the following conditions such that for any finite-length input sequence X = (21,--.;;22) with In E X C RD, where X is bounded, the resulting model J satisfies (
1. (SDT-P on SSM) In each SSM module, update at most [DL channels. Within each updated channel. fine- tune at most H* hidden states and set the rest 10 zero.
Overhead Analysis. We assess the computational over- head of applying SDT with LoRA (for linear projection matrices) versus LoRA alone with Table 2 summarizing the results. Although SDT involves an additional dimension selection stage, Table shows that this incurs minimal extra cost. Furthermore, with the same parameter budget, SDT for SSM modules combined with LoRA on linear projections runs faster than LoRA alone, since LoRA introduces extra matrix multiplications between two low-rank matrices for the SSM modules, whereas SDT does not. In Sec. D.6, we detail the experimental settings and present a memory us- age analysis showing that SDT also consumes less memory during fine-tuning for the same reason.
2. (LoRA' on Linear Projections) Apply mank-[T.TTr up- dates to each linear projection matrix.
3. (Minimal Additional Updates) Update only the residual connections, per-layer biases, and the final-layer oulpul projection matrix.
For proof and details, refer to Sec. D.2 and D.3. This theo- rem shows that larger pretrained model can be fine-tuned into any smaller model of the same architecture by applying SDT-P to SSM modules and LoRA" to linear projection matrices. Moreover, for less complex tasks, where the target model has fewer layers (I'*) and hidden states (H*), the re- quired number of trainable channels and hidden states also decreases. This aligns with the theoretical analysis of LoRA by Zeng & Lee (2024), which demonstrates that larger pre- trained models require fewer learnable parameters (i.e., a lower-rank update) during fine-tuning, especially for sim- pler tasks. While theorem assumes linear activations, no residual connections in the target model, and full fine- tuning of the last-layer projection matrix, our findings have broader implications. As our experimental results in Sec. 6 will show, these insights generalize beyond these theoretical constraints.
Stage
Method
Mamba-130M
Mambu-1.48
Jamba--Mini-22
Dim. Selection LoRA & SDT
16.5 3.9
85.8 + 5.3
163.9 + 10.2
Training (per spoch)
LoRA
410.9
2060.0 135.0
3427.5 1. 185.0
LORA & SDT
330,0 + 77.5
1697,5 4 87.5
3065 0 :: 232.5
Table 2. PEFT combining SDT with LoRA is more efficient than LoRA alone when the same number of trainable param- eters are used. Shown are dimension selection and per-epoch training times (s) for Mamba and Jamba models.
6. Experimental Studies of SDT
Algorithm 1 Dimension Selection Algorithm of SDT
In this section, we evaluate the performance of SDT in tuning SSM modules, comparing it to LoRA*, the best existing PEFT method for fine-tuning SSM modules, as shown in Sec. 4. Our experiments reveal the key result:
Input: A small subset of dataset DD, warmup epochs F, number of layers L, total channels D, total states H, channel freeze ratio a, state freeze ratio |
** Wa rmap epochs
/
Perform full update on SSM modules using D for E epochs
Finding: SDT outperforms LoRA* on SSM modules.
for = 1 to L do
/* Unfreeze dimensions
6.1. Synthetic Experiments on Deep S4 Models
Sort channels D based on changes of A
(d)
This experiment validates our theoretical guarantees un- der broader conditions, including residual connections and ReI.U activations in both models, without fully fine-tuning the last-layer projection matrix. See Sec. E. for details.
Freeze the bottom [ channels, denoted by Ty
for d E D' do
Sort state dimensions by the changes in || A Freeze the bottom [H[I state dimensions at the d-th
channel
(Experimental Setup) We employ a regression setting to validate our theoretical results. We randomly ini-
8

---

Parameter-Efficien Fine- Fine-Tuning of State Space Models
tialize two models: one-layer deep S4 model as the target and a four-layer deep S4 model as the frozen model. LoRA is applied to linear projection matrices, while different methods are tested on the SSM module to assess their effectiveness. The goal is to update the frozen model to match the target model's functionality. We generate an input se- quence X of length 200 and dimension 64, with values uniformly drawn from integers between 0 and 9. This input is then processed through the target model to obtain the corresponding out- puts. These input-output pairs are used to train the frozen model over 500 it-
LinProj
56
GLUE
BLEU
DART
MET.
CelebA
Acc.
R1
SAMSum
R2
RL
Spider
Acc.
LoRA
80.8
51.0
70.2
88.6
51.6
28.2
43.2
83.5
LORA
SDT
81.1
51.5
70.5
88.6
51.7
28.1
43.4
84.5
DORA
80.1
51.2
70.4
88.4
51.8
28.0
43,4
8388
DORA
SDT
78.2
51.5
70.8
88.6
52.1
28.3
43.7
85.1
Table 3. Performance comparison between SDT and LoRA on pretrained Mamba models. Bold numbers indicate the best per- formance for each task. We use Mamba- 130M to compare the performance of SDT and LoRA on GL.E (Wang et al., 2019), DART (Nan et al., 2021), and CelehA (I.iu et al., 2015) bench- marks. For all other datasets, we employ Mamba-1.45 We report only the best setting out of three for each method. We observe that SDT outperforms LoRA' on updating SSM modules on Mamba.
Frozen LoRA
MSE
SDT
40 Trainable Parameters (%)
20
Figure 2. SDT surpasses LoRA tuning S4 within deep S4 mod- els when LoRA is applied to lin- car projection matrices in syn- thetic experiments.
nificantly fewer parameters than those in the Mamba model, fine-tuning them yields limited performance improvements. Nevertheless, results on GLUE (Table 4) validate the effec- tiveness of our method. See Table 22 for more results.
erations using the Mean Squared Error (MSE) loss. (Re- sults) Figure 2 shows the MSE, averaged across all tokens, plotted against the number of trainable parameters for differ- ent methods on SSM modules. SDT achieves significantly lower MSE than LoRA on SSM modules, demonstrating its effectiveness in updating SSM modules.
LinPrij
S6
RTE
MRPC
CoLA
SST-2
ONLI
OQP
MNLI
Avg.
DoRA
65.7
77.8
93.9
77.8
67.8
85.4
67.9
DORA
SDT
67.1
71.5
7.5
94.2
79.6
72.7
85.5
69.2
Table 4. Performance comparison between SDT and DoRA on pretrained Jamba models. Bold numbers indicate the best per- formance for each task. We use Jamba-Tiny-3199 performance of SDT and DoRA on the GLUE (Wang et al., 2019) benchmark. We report only the best setting out of three for each method, We observe that SDT outperforms DoRA on updating SSM modules on Jamba.
compare the
6.2. Real- Experiments on Pretrained Models
Lastly, we conduct experiments evaluate our approach on pretrained models, including Mamba and Jamba with differ- ent model sizes. We consider five datasets: GLUE, DART, SAMSum, Spider, and CelebA. For these experiments, we split the datasets into three parts: train, validation, and test, different from benchmarking experiments. We combine our proposed SDT with LoRA* and evaluate in three different settings against three pure LoRA* settings. In SDT, 99% of channels are frozen, and we adjust state freeze ratios. For the pure LoRA* settings, we apply LoRA* to different parameter sets, selecting ranks to ensure all settings have comparable parameter budget for fair comparison. Residual connections and biases are frozen and learning rates are independently selected via a small grid search over data subsets. See Sec. E.2 for further details.
7. Discussion
In this paper, we study PEFT methods applied to SSM-based models. Our evaluation of existing PEFT methods provides valuable insights and guidelines for future researchers to parameter-efficiently fine-tune SSM- based models for other domains. Moreover, we take initial step in establishing a theoretical framework for studying PEFT methods SSM- based models. Additionally, we introduce SDT, a PEFT method specifically tailored to SSM modules, demonstrating superior performance compared to existing approaches.
Limitations & Future Works. The theoretical guaran- tees for SDT are restricted to linear activations and require full fine-tuning of the last layer. Nonetheless, our exper- iments show that SDT performs well in practice despite these constraints. Addressing these theoretical limitations or developing new PEFT methods applicable to broader scenarios is a promising future direction. Additionally, our theory shows that modifying subset of channels and states is sufficient but does not guide optimal selection. Our ap- proach, based on a warmup stage and parameter magnitude, might not be optimal. Future research could explore the impact of channel/state selection and improve dimension selection algorithms.
Mamba. The experimental results of Mamba are reported in Table 3, showing that applying SDT on SSM modules outperforms pure LoRA*, even when 99% of the channels are frozen. This underscores the effectiveness of SDT on fine-tuning SSM modules.
Jamba. We extend our experiments to Jamba, applying all tested methods exclusively to its Mamba layers. No- tably. the performance gain on Jamba is smaller compared to Mamba. This is because we freeze all Transformer layers to isolate the effect of Mamba layers for a fair evaluation. Additionally, since the Mamba layers in Jamba contain sig-

---

Parameter-Efficicee Fine- Fine-Tuning of State Space Models
Impact Statement
In Advances in Neural Information Processing Systems, volume 33, pp. 1474-1487, 2020.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and Ré, C. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in Neural Information Processing Systems, volume 34, pp. 572-585, 2021.
Acknowledgment
The works is supported by NSF Award CAREER Award CCF-2339978, Amazon Research Award, and grant from FuriosaAI.
NSF
Gu, A., Goel, K., Gupta, A., and Ré, C. On the parameteri- zation and initialization of diagonal state space models. In Advances in Neural Information Processing Systems, volume 35, pp. 35971-35983, 2022a.
References
Gu, A., Goel, K., and Re, C. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022b.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 technical report. arXiv preprint arXiv:2303. 087744 2023.
Gupta, A., Gu, A., and Berant, J. Diagonal state spaces are effective as structured state spaces. Advances in Neural Information Processing Systems, 35:22982-2298 222994, 2022.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
Halloran, J. T., Gulati, M., and Roysdon, P. F. Mamba state- space models can be strong downstream learners. arXiv preprint arXiv:2406. 00209 2024.
Dao. T. and Gu. A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning, 2024.
Hayou, S., Ghosh, N., and Yu, B. Lora+ efficient low rank adaptation of large models. In Proceedings of the 4 sst International Conference on Machine Learning, pp. 17783-17806, 2024.
Dinh, T., Zeng. Y., Zhang, R., Lin, Z., Gira, M., Rajput, yong Sohn, J., Papailiopoulos, D., and Lee, K. LIFT: ianguage-interfaced fine-tuning for non-language ma- chine learning tasks. In Advances in Neural Information Processing Systems, 2022.
He, J., Zhou, C., Ma, X., Towards unified view of parameter-efficien trans- fer learning. In International Conference on Learning Representations, 2021.
T., and Neubig,
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M.. and Gelly, S. Parameter-efficient transfer learning for NLP. International Conference on Machine Learning, pp. 2790-2799 2019.
Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Re, C. Hungry hungry hippos: Towards language modeling with state space models. In International Con- ference on Learning Representations, 2022.
Giannou, A., Rajput, S., and Papailiopoulos, D. The expres-
Hu, E. J., Wallis, P., Allen-Zhu, 7., Li, Y., Wang, S., Wang, L., Chen, W., et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learn- ing Representations, 2021.
sive power of tuning only the normalization layers. In The Thirty Sixth Annual Conference on Learning Theory,
pp. 4130 4131, 2023.
Gliwa, B., Mochol, I.. Biesek, M., and Wawer, A. SAM- Sum corpus: A human-annotated dialogue dataset for abstractive summarization. EMNLP-IJCNLP 2019, pp. 70, 2019.
Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.P.., Bing, L., Xu, X., Poria, S., and Lee, R. L.MM-adapters adapter family for parameter-efffcient fine-tuning of large lan- guage models. In Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5254 5276, 2023.
Gu, A. and Dao, T. Mamba: Linear-time sequence model- ing with selective state spaces. In First Conference on Language Modeling, 2024.
Jang, U., Lee, J. D., and Ryu, E. K. LoRA training in the ntk regime has no spurious local minima. In International Conference on Machine Learning, 2024.
Gu, A., Dao, T., Ermon, S., Rudra, A., and Ré, C. Hippo: Recurrent memory with optimal polynomial projections.
10

---

Parameter-Effcieet Fine- Fine-Tuning of State Space Models
Kang, W., Galim, K., Zeng, Y., Lee, M., Koo, H. I., and Cho, N. I. State-offset tuning: State-based parameter- efficient fine-tuning for state space models. arXiv preprint (7Xiv:5503.03499, 2025.
Computational Linguisties: Human Language Technolo- gies, pp. 232-447, 2021.
Oymak, S., Rawat, A. S., Soltanolkotabi, M., and Thram- poulidis, C. On the role of attention in prompt-tuning. In International Conference on Machine Learning, pp. 26724-26768, 2023.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pp. 5156-5165, 2020.
Panigrahi, A., Saunshi, N., Zhao, H., and Arora, S. Task- Specific skill localization in fine-tuned language models. International Conference on Machine Learning, pp. 27011-27033, 2023.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 3045-3059, 2021.
Park, J., Park, J., Xiong, Z., Lee, N., Cho, J., Oymak, S., Lee, K., and Papailiopoulos, D. Can Mamba learn how to learn? a comparative study on in-context learning tasks. In International Conference on Machine Learning, pp. 39793-39812, 2024.
Li, X. L. and Liang, P. Prefix-Tuning Optimizing Continu- ous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4582-4597, 2021.
Peng. B., Alcaide, E.,, Anthony, Q G., Albalak, A., Arcad- inho, S., Biderman, S., Cao, H., Cheng, X., Chung, M. N., Derczynski, L., et al. RWKV: Reinventing RNNs for the transformer era. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi- gos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev- Shwartz, S., et al. Jamba: Hybrid reanformer-mant language models. In The Thirteenth International Con- ference on Learning Representations, 2025.
Petrov, A., Torr, P. H., and Bibi, A. When do prompting and prefix-tuning work? a theory of capabilities and limitations. In International Conference on Learning Representations, 2024.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight- decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning, vol- ume 235, pp. 32100-32121, 2024.
Scholak, T., Schucher, N., and Bahdanau, D. PICARD: Parsing incrementally for constrained auto-regressive de- coding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9895-9901, 2021.
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang. Z., and Tang, J. GPT Understands, Too. arXiv:2103.1l 10385, 2021.
Song. W., Li, Z., Zhang, L., Zhao, H., and Du, B. Sparse is enough in fine-tuning pre-trained large language models. Proceedings of the 41st International Conference on Machine Learning, pp. 11-115,,,,, 2024.
Liu, X., Ji, K., Fu, Y., Tam, W., Du, Z.,, Yang, Z., and Tang, P-Tuning: Prompt Tuning Can Be Comparable to Fine- tuning Across Scales and Tasks. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), pp. 61-68, 2022.
Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:230700861/ 2023.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730- 3738, 2015.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t.., and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Nan, L., Radev, D., Zhang, R., Rau, A., Sivaprasad, A., Hsich, C., Tang, X., Vyas, A., Verma, N., Krishna, P., et al. DART: Open-Domain Structured Data Record to Text Generation. Proceedings of the 2021 Conference of the North American Chapter of the Association for
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.

---

Parameter-Effiiiet Fine-Tuning of State Space Models
Wang, H., Agarwal, S., and Papailiopoulos, D. Pufferfish: Communication-efficieee models at no extra cost. In Proceedings of Machine Learning and Systems, volume 3, pp. 365-386 2021.
Wang, H., Agarwal, S., Tanaka, Y., Xing, E., Papailiopoulos, D., et al. Cuttlefish: Low-rank model training without all the tuning. Proceedings of Machine Learning and Systems, 5, 2023a.
Wang, Y., Chauhan, J., Wang, W., and Hsieh, C.-J. Univer- sality and limitations of prompt tuning. In Advances in Neural Information Processing Systems, 2023b.
Yoshimura, M., Hayashi, T., and Maeda, MambaPEFT:
Exploring parameter-cfficient fine-tuning for mamba. In The Thirteenth International Conference on Learning Representations, 2025.
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, Yao, Q., Roman, S., et al. Spider: A large-scale scale human-labeled dataset for complex and cross- domain semantic parsing and text-to-sql task. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 391 1-3921, 2018.
Zaken, E. B., Goldberg, Y., and Ravfogel, S. BitFit: Sim- ple parameter-efficient fine-tuning for transformer-basee masked models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1-9, 2022.
Zeng, Y. and Lee, K. The expressive power of low-rank adaptation. In International Conference on Learning Representations, 2024.
12

---

Efrimittr-Effiieett Fine- Fine-Tuning of State Space Models
Appendix
A Additional Related Works
14
A.1 Additional Related Works on SSMs
14
A.2 Additional Related Works on PEFT
14
Details of Datasets
15
Details of Sec. 4: Benchmarking PEFT Methods on SSM-based Models
16
C.1 Experiment Setup
16
C.2 Extended Results Benchmarking Existing PEFT Methods
16
C.3 Limitations of Applying input-injection Methods on SSMs
17
C.4 Optimal Application LoRA* in SSM-based Models
22
D Details of Sec. 5: SDT
24
D.1 Understanding the Roles State Matrix A, Input Transition Vector B, and Output Mapping Vector for Single Channel S4 Modules
24
D.2 Extension to Deep S4 Models
27
D.3 Extension to S6
29
D.4 Sparse Dimension Tuning and Pruning (SDT-P)
30
D.5 Extension to S5
31
D.6 Memory Usage and Runtime Analysis of SDT
31
Expanded Sec. Evaluation of SDT
33
E.1 Experiments Deep $4 Models
33
E.2 Experiments on Mamba-I1, II, Jamba, and LoRA+
34
13

---

Parameter-Effiient Fine-Tuning of State Space Models
A. Additional Related Works
A.I. Additional Related Works on SSMs
Linear State-Space Layers (LSSL) represent one of the earliest SSM layers utilized in deep learning, functioning as continuous-time, recurrent, and convolutional models (Gu et al., 2021). LSSL employs HiPPO theory (Gu et al., 2020) to initialize the state matrix A, enabling the capture of long dependencies. However, LSSL is computationally expensive, limiting its practical application. Gu et al. (2022b) introduced Structured State Space Models (S4), which optimize computation efficiency by employing structured state matrix A. Gupta et al. (2022) proposed DSS, which simplifies the model by using diagonal matrix for A and empirically demonstrated that it suffices to achieve performance comparable to S4. Further, Gu et al. (2022a) provided a theoretical explanation for the effectiveness of the diagonal state matrix A in DSS and introduced S4D, which offers various initialization methods for A. Subsequently, the diagonal structure of the state matrix A has been adopted in follow-up methods (Gu & Dao, 2024). Despite differences optimization algorithms, we refer to S4 and its close variants, including DSS and S4D, collectively as S4. This terminology encompasses models that maintain the standard discrete-time SSM form with diagonal state matrix.
Despite of the remarkable performance of SSMs on certain tasks of sequence modeling, SSMs still showed worse perfor- mance than Transformers on language modeling. Fu et al. (2022) transitioned from synthetic language modeling tasks to real language modeling tasks with SSMs. They proposed H3, which is inspired by Linear Attention (Katharopoulos et al., 2020), introducing both diagonal SSM and shift SSM. Recently, Mamba (Gu & Dao, 2024; Dao & Gu, 2024) escaped from linear time invariance (L.TI) modeling by introducing input-dependent terms and achieved better performance than Transformers on language modeling. Furthermore, several hybrid models (Lieber al., 2025; Park et al., 2024) tried to exploit the advantages of both SSMs and Transformers.
A.2. Additional Related Works on PEFT
In this section, we provide a more detailed description of the baseline methods.
LoRA (Hu et al., 2021). LoRA (Low-Rank Adaptation) focuses on fine-tuning large models by freezing pretrained parameters and injecting trainable low-rank matrices into each layer of the Transformer architecture. The intuition behind using low-rank matrices comes from linear algebra, where a large matrix can be closely approximated by the product of two smaller matrices. The number of trainable parameters can be controlled with the rank of the low-rank matrices. LoRA also uses scaling parameter (LoRA alpha) for the weight matrices to control the balance of the original model weights and LoRA weights during training. After fine-tuning, LoRA weights can be merged with the original model weights, introducing no additional inference overhead.
Prompt Tuning (Lester et al., 2021). Prompt tuning freezes all model weights and prepends a trainable soft prompt to the input prompt. The soft prompt consists of trainable virtual tokens, which are continuous. At inference time, prompt tuning introduces an inference overhead based the number of virtual tokens used.
Prefix-Tuning (Li & Liang, 2021). Prefix-tuning also prepends trainable tokens to the input like prompt tuning but injects prepends trainable embeddings to the attention's 's
separate prefixes in every layer. For each Transformer layer, K and matrix. The authors have found that directly training these prefixes can lead to unstable training, so they propose to over peeaparameterize them with a large MLP to increase training stability. After training, the MLP can be dropped. Like prompt tuingg prefix-tuning introduces inference overhead, scaling with the number of trainable embeddings.
BitFit (Zaken et al., 2022). BitFit is a simple but effective PEFT method that freezes all model weights except the bias terms, consequently greatly reducing the number of trainable parameters. As no additional parameters are added, no inference overhead occurs.
Theoretical Understanding of PEFT. Numerous efforts have been made to theoretically understand existing PEFT methods. For input-injection methods, Wang et al. (2023b), Petrov et al. (2024), and Oymak et al. (2023) have theoretically analyzed the effectiveness and limitations of prompt tuning and prefix-tuning for Transformer-basee based models. For LoRA, Zeng& & Lee (2024) explored its expressive power by demonstrating that even a randomly initialized model can be adapted to match any smaller target model using LoRA. Some of our theoretical analysis draws upon the framework established by
14

---

Parameter-Efficien Fine-Tuning of State Space Models
Zeng & Lee (2024). Jang et al. (2024) conducted theoretical exploration of LoRA within the neural tangent kernel (NTK)
regime.
B. Details of Datasets
In this paper, we consider six datasets across three domains: (i) Natural Language Understanding (NLU), represented by
GLUE (Wang et al., 2019); (ii) Natural Language Generation (NLG), including SAMSum (Gliwa et al., 2019), Spider (Yu al., 2018) and DART (Nan al.. 2021); and (iii) Computer Vision (CV), represented by CIFAR-10 (Krizhevsky et al.. 2009) and CelebA (Liu et al., 2015).
GLUE (Wang et al., 2019). The GLUE (General Language Understanding Evaluation) benchmark is collection of datasets used for training, evaluating, and analyzing natural language understanding models across a range of diverse tasks, The benchmark includes nine sentence- or sentence- pair language understanding tasks that require various features of understanding, such as sentiment analysis, linguistic acceptability, semantic textual similarity, and question answering. We use seven datasets from the GLUE benchmark (RTE, MRPC, CoLA, SST-2, QNLI, QQP, MNLI) where the model has to choose between two or three (for MNLI) different choices for the respective task. Except for CoLA, we evaluate all used datasets with the accuracy metrie. For CoLA, Matthews correlation is employed.
SAMSum (Gliwa et al., 2019). SAMSum is dataset for dialogue summarization research, comprising approximately 16,000 synthetic text conversations with accompanying summaries. Created by Hnglish-fluent linguists, these exchanges simulate real-world digital communications across various topics and styles. The conversations range from informal to formal, incorporating elements like slang and emoticons to reflect authentic messaging patterns. Each dialogue is paired with a concise, third-person summary, capturing its essential content. This structure makes SAMSum particularly useful for developing and evaluating automated summarization systems capable of processing conversational text.
Spider (Yu et al., 2018). Spider is a lcrge-scale, complex, and cross-domain semantic parsing and text-to-SQL dataset. contains about 10,000 annotated SQL queries, distributed across 200+ databases, each with multiple tables. We follow Scholak et al. (2021) and use about 7,000 examples for training and about 1,000 examples for validation, where we ignore sequences longer than 1536 tokens. The dataset consists of English question and SQL query pairs, which cover wide range of SQL operations including SELECT, WHERE, COUNT, GROUP BY, ORDER BY, JOIN, and more. Given an English question and an SQL database scheme, the task for the model is to translate the English question into an appropriate SQL statement. Evaluation is performed via accuracy where the output is considered as correct if the model's predicted SQL query and the included GT SQL query give the same result when exccuted on the database. The dataset additionally categorizes each query into easy (25%), medium (40%), hard (20%), and extra hard (15%) based on the complexity of the required SQL statement. For evaluation, we report the execution accuracy of all categories.
DART (Nan et al., 2021). The DART (DAta Record to Text) benchmark is a large-scale, structured dataset designed for RDF-to-text (Resource Description Framework-to-ter generation with 80,000+ instances. The DART benchmark is composed of a collection of structured data triples and corresponding text summaries which are organized into different categories. The task of the DART benchmark is to generate natural language summaries that correctly represent the given structured data inputs. DART is typically evaluated with METEOR and BLEU.
CIFAR-10 (Krizhevsky et al., 2009). The CIFAR-10 (Canadian Institute For Advanced Research) dataset is a collection of images that are commonly used to train machine learning and computer vision algorithms. It is one of the most widely used datasets for image classification. The CIFAR-10 dataset contains 60,000 (50,000 for training, 10,000 for validation) 32x32 color images in 10 different classes. The 10 different classes are: airplane, car, bird, cat, deer, dog, frog, horse, ship, and truck. There are 6,000 images of each class. For training, we center crop each image to 24 24 pixels and flatten each image to string, with total of 24x24xx words, where each word is a number between 0-255 representing the respective pixel value. Although CIFAR- 10 is a dataset for computer vision, previous work (Dinh et al., 2022) showed that Transformers can be adapted to the vision domain from the language domain. In our work, we extend this investigation to SSMs, examining their ability to perform on vision data.
CelebA (Liu et al., 2015). The CelebA (CelebFaces Attributes) dataset is an extensive collection of more than 200,000 celebrity images, each tagged with 40 attributes. This dataset is notable for its diversity, volume, and comprehensive
15

---

Parameter-Effiient Fine-Tuning of State Space Models
Dataset
Size (Train)
Size (Val)
Size (Test)
Max. seq. len.
#Epochs
Mamba Size
Jamba Size
Metries
RTE
1992
498
277
291
10
130M
319M
Accuracy
MRPC
2934
734
408
105
10
130M
319M
Accuracy
CoLA
6840
1711
1043
47
10
130M
319M
Matthews corr.
GLUB
SST-2
53879
13470
872
68
10
130M
319M
Accuracy
QNLI
83794
20949
5463
602
10
130M
319M
Accuracy
QQP
291076
72770
40430
316
130M
319M
Accuracy
MNI.I
314161
78541
19647
425
130M
319M
Accuracy
Spider
5543
1375
1034
1412
10
1.4B,
2.8B
52B
Accuracy
SAMSum
14732
818
819
1174
10
1.4B
52B
ROUGE
DART
62659
2768
5097
491
10
130M
52B
METEOR, BLEU
CIFAR-10
40000
10000
10000
1730
130M
319M
Accuracy
CelebA
162770
19867
19962
12614
130M
319M
Accuracy
Table 5. Datasets and models for experiments. For each dataset, we report the number of training, validation, and test samples, maximum sequence length, training epochs, model size, and evaluation metric used.
annotations, encompassing 10,177 distinct identities, 202,599 facial images, and annotations of five landmark points with 40 binary attributes per image. The dataset, which includes images with varied poses and complex backgrounds, is essential resource for tasks in computer vision such as face recognition, attribute analysis, and detection, well as facial landmark localization, and it offers significant utility in face editing and synthesis.
The dataset characteristics, including our train, validation and test set sizes, sequence lengths, and number of epochs, are summarized in Table 5.
C. Details of Sec. 4: Benchmarking PEFT Methods on SSM-based Models
In this section, provide comprchensive experimental setup, proofs and further discussion of theoretical results, and more detailed experimental outcomes.
C.1. Experiment Setup
For each dataset, choose the model size depending how challenging the dataset is and perform small grid search for one epoch on subset of the data (1K-2k instances) with learning rates {4 x 10-1, 10-1, x 10-1, ...,, x 10-5) to find the optimal learning rate of each PEFT method. We only report the validation metric of the best epoch during training (early stopping) in our results. We fine-tune pretrained Mamba and Jamba models with Adam with a linear learning rate decay schedule. For LoRA we set rank to 8, alpha 8, and dropout to 0.1 for experiments. For evaluating NLG tasks, we employ beam search with five beams and maximum beam length of 1024.
C.2. Extended Results on Benchmarking Existing PEFT Methods
Mamba-I. We present comprehensive fine-tuning results for the GLUE benchmark (Wang et al., 2019), DART dataset (Nan al.. 2021), SAMSum dataset (Gliwa al., 2019), Spider dataset (Yu al., 2018), and CIFAR-10 (Krizhevsky et al., 2009) in Table 6, Table 7, Table 8, Table 9, and Table 10 respectively. These experimental results encompass various LoRA implementations (on different weight matrices and modules) and provide more fine- tne-grained results across all subtasks.
Mamba-II. Table 11 and Table 12 present the benchmark results of LoRA and full fine-tuning across different layers of Mamba-II. We follow the same experimental setup used for Mamba-l and demonstrate that, on Mamba-II, our conclusion holds: LoRA is more effective on linear projection layers than on SSM modules.
16

---

Parameter-Eficient Fine-Tuning of State Space Models
Jamba. Table 13 presents the benchmark results of LoRA and full fine-tuning across different layers of Jamba. Our findings demonstrate that, Jamba, LoRA is more effective linear projection layers than on SSM modules, which aligns with our conclusion on Mamba.
Layer
Method
Params (%)
RTE
MRPC
CoLA
SST-2
QNLI
QQP
MNLI
Arg.
Pretrained
0.00
16.9
67.9
0.0
52.1
50.5
36.8
32.3
41.0
Full
100.00
71.1
80.6
63.2
92.2
87.4
87.9
80.8
80.5
All
All
LoRA
1.92
69.9
61.4
91.9
88.4
87.6
80.2
Prompt Tuning
16 tokens
56.0
71.6
12.0
89.4
76.8
79.6
61.5
63.8
Prompt
Prefix-Tuniin
loken (no MI.P)
0.03
67.5
75.7
43.4
91.5
83.4
35.6
68.6
Bius
Ba, Convid
BitFit
0.06
69.5
80.4
54.7
92.0
86.2
85.3
77.2
77.9
All
LoRA
1.02
70.0
82.4
57.7
93.3
88.7
88.7
82.5
80.5
W
LoRA
0.34
70.4
82.1
37.4
91.7
88.3
87.7
81.2
79.8
Linear Projection Matrices
Wn.:
LoRA
0.34
70.0
82.4
58.1
92.4
87.3
87.3
80.4
79.7
WR. WIn..
LoRA
0.68
70.4
84.3
62.4
92.5
88.6
88.3
81.7
81.2
Wa..
LoRA
0.34
70.4
82.8
60.6
92.4
88.4
87.7
81.5
80.5
Full
4.31
69.7
78.9
39.1
91.5
88.1
87.5
80.5
79.3
All
LoRA
0.92
66.1
78.7
37.8
90.8
87.8
86.9
79.8
78.3
A
Full
0.46
68.2
82.1
54.2
90.9
86.4
87.9
79.4
78.4
Full
2.28
69.7
77.0
55.8
91.4
85.4
85.0
76.8
77.3
S6
Wo. Wo, Wa.
LoRA
0.69
67.9
78.9
48.8
91.4
86.9
85.8
78.6
76.9
Full
1.40
66.1
75.2
56.7
91.1
86.2
87.1
78.5
77.3
Watt
LoRA
0.23
67.1
79.9
55.1
90.9
52.7
86.6
78.7
73.0
Convid
Full
0.14
68.2
78.4
57.9
91.1
86.0
86.0
78.0
77.9
Others
D. LayerNarm
Full
0.04
65.3
79.2
40.3
91.1
83.9
86.0
67.0
73.3
Table 6. Full benchmark results on the GLUE (Wang et al., 2019) benchmark using Mamba-I 130M. We report accuracy (1) for RTE, MRPC, SST-2, QNLI, QQP, and MNLI tasks. CoLA performance measured using Matthews Correlation Coefficient (t). each Mamba block, W and Wi.,: are input projections that preprocess the input for the SSM modules and the gating branch, respectively. ou denotes the output projection after the gating mechanism. WB and We are weight matrices for computing input-dependent B, and C.. Wa.1 and WA, represent down and up projections of low-rank weight matrices in the linear layer computing input-dependent step size A.. BA represents the bias in this linear layer. D denotes the weight of residual connections.
C.3. Limitations of Applying Input-injection Methods on SSMs
We start by introducing the necessary notations. Denote the space of S4 mechanisms with D channels as Fs4 Let Hg (hg (1) h/2² . () ) E H represent the initial hidden state, and X (x1, 22, -.. .NN) RDN denote the input sequence. The output of the S4 mechanism is represented f(X; Ho). Furthermore, for d-th channel, let state transition matrix (d) = diag (af .....af
(bI,- ... bT)). where d = 1,..., D. For any vector u E R", we use U:y E Ri- to denote the subvector of v containing elements from i E N+ to j E N+, where i < j. Similarly, for any matrix M E |R we use Massiiia to denote the submatrix containing rows i E N+ to ji E N+ and columns i2 E N+ to j2 E N where in < Jj. i2 < j2.
.az (d()) and input transition vector B
Proposition 1 (Expressivity of Prefix- Prefix-Tuning on SSMs). Let f E Fst.D be an S4 mechanism. Consider prefix-tuning that prepends sequence P= (pi,...,
..h....,PM,, .Ps) E DDD to the inpul sequence X- = (21, 22,,. ... nN E R RDDN, For any prefix P E IR DxM, there exists an initial hidden state H E HxD such that the output of S4 after prefix-tuiing and that after initial state tuning are identical, i.e., f(x; Hi) = =[PP X); H.).H... for all X E RDDN (d)
a. d(d)) + 0 and II (1)) + for all channels d = I....,. D. Then the converse (i.e., for any Ho E HxD there exists a Pa E DDDM such that S(PP" X]; Ho) f(X; H)) for all X D holds if and only if M > H.
Furthermore, assume that Hostejen(aa
Proof of Proposition 1. Given that operations in S4 are independent across all channels, we can, without loss of generality, consider the case where the number of channels D Consequently, we can simplify our notation: the initial hidden states
17

---

Carameter-Effiiiee Fine- Fine-Tuning of State Space Models
Layer
Method
Purams
METEOR
BLEU
All
All
LoKA
1.32
71.0
29.5
DoRA
2,02
70.9
51.4
Pronage Tuning
Prompt
Profix: Tuning
64 tokeas
22,89
00.6
22.5
Blas
Aa. Convld
Bilk
0.06
67.0
63.7
LoKA
1.2
71.2
49.2
DoRA
1.00
71.2
50.5
WALE
LoRA
0.34
20.3
45.9
DuRA
0.57
70.8
69.9
Linear Projection Manrices
W
DoKA
0.37
7022
69.1
48.3
We.. We.s
LaKA
DoRA
0.88
70.9
49.5
W..
DoRA
LaRA
0.96
0.34
70.7
47.0
66.0
Full
4.31
70.3
68.7
50.5
DoRA
0.35
70.2
50.0
Full
0.46
09.3
48.1
Full
50.0
Wa. We. Wa.
LeRA
0.89
63.8
68.0
DeRA
0.89
68.3
47.3
Full
1,40
09.6
47.2
Wa.1
LaRA
0.23
47.0
DoRA
0.26
63.4
66.3
Cony.e
Full
0.34
68.6
47.9
Layertior
Table 7. Full benchmark results on the DART (Nan et al., 2021) benchmark using Mamba-I 130M. We report METEOR (1) and BLEU (f) scores. In each Mamha block, Wn.: and Wm.: are input projections that preprocess the input for SSM modules the gating branch, respectively. W. denotes the output projection after the gating mechanism. WB and We are weight matrices for computing input-dependen B, and C.. Wa. and Wa.t represent down and up projections of low-rank weight matrices in the linear layer computing depun-den step size A.. AA represents the bias in this linear layer. D denotes the weight of residual connections.
Layer
Method # Params (%)
R1
R2
RL
All
All
Full
100.00
51.2
27.3
42.9
LaRA
50.8
42,7
Prompt Tuning
64 tokens
0.01
50.1
25.6
41.6
Prompt
Prefix-Tuning
64 tokens
12.81
50.6
26.5
42.1
Bias
3a. Coavid
Bifiit
0.03
30.3
25.7
11.9
LoRA
0.51
50.8
12.8
W..
LaRA
0.17
49.8
25.4
41.2
I Linear Projection Matrices
W
LoRA
0.17
50.0
26.1
41.7
W... Wik.
LoRA
30.9
12.3
W..
LoRA
0.17
49.9
25.1
11.5
Full
4.46
51.1
26.9
42.2
All
LoRA
30.5
A
Full
0.23
30.1
25.9
11.7
Full
2.29
50.5
26.0
11.8
Wn. We. Wa
LoRA
30.4
Waat
Full
1.85
30.3
25.7
11.6
LoRA
0.12
50.2
25.4
41.3
convid
Full
30.1
Others
D. LeyesNorm
Full
0.02
49.6
21.8
11.1
Table 8. Full benchmark results on the SAMSum (Gliwa et al., 2019) benchmark using Mamba-1 1.4B. R1, R2, and RL represent ROUGE-1 (t), ROUGE-2 (t), and ROUGE-L (t), respectively. In each Mamba block, Winz and Wi., input projections that preprocess the input for SSM modules and the gating branch, respectively. W.. denotes the output projection after the gating mechanism. WB and We weight matrices for computing input-dependent B,, and C., WA. and Wa. represent down and up projections of low-rank weight matrices in the linear layer computing input-dependent step size An. AA represents the bias in this linear layer. D denotes the weight of residual connections.
18

---

Parameter-Eficce Efficieet Fine- Fine-Tuning of State Space Models
Layer
Method
# Parums (%)
All
Easy
Medium
Hard
Extra
All
All
100.00
66.2
09.5
53.4
43-4
LoRA
0.97
56.4
76.2
57.0
47.7
31.3
DoRA
1.02
53.7
77.0
57.0
47.1
29.5
Prompt Tuning
tokens
0.01
43.6
42.4
33.3
Prompt
Prefix Tunin-Tunin
64 tokens
12.81
39.7
65.7
38.6
31.0
15.1
Pa. Convid
51.3
50.9
LoRA
0.51
54.7
75.0
55.6
46.0
31.3
DoRA
0.55
57.2
79.4
68.7
46.0
31.3
Wa,z
DoRA
0.19
58.4
80.2
60.1
49.4
30.7
Lincar Projection Matrices
Wn.
LoRA
0.17
46.3
45.7
36.8
DoRA
0.19
59.8
83.9
60.1
50.6
32.5
Wm. Wis
LoRA
0.34
37.5
77.4
68.7
45.4
37.3
DoRA
0.37
60.7
78.6
62.1
52.9
38.6
Wa.
LoRA
0.17
61.8
81.9
63.2
45.4
30.8
DoRA
61.3
50.0
Full
4.46
56.7
76.6
57.8
46.0
31.9
DoRA
0.48
58.9
77.4
022.1
47.1
34.9
Full
0.23
51.1
71.4
52.5
42.5
25.9
Full
2.29
47.2
72.2
46.9
35.6
22.9
We, We. Wa..
LoRA
0.35
56.0
73.8
06.7
44.3
33.7
Full
1.85
56.8
77.0
59.4
43.7
33.1
Wat
50.4
48.9
DoRA
0.13
50.3
76.2
09.2
42.5
27.1
Convie
Full
0.07
53.2
74.6
52.9
43.7
31.9
Others
D. Laye
50.4
40.2
(a) Full benchmark results on Spider using Mamba-I 1.4B.
Layer
Method
Params (%)
All
Easy
Medium
Hard
Extra
Foll
100.00
71.8
87.6
51.8
LoRA
0.80
70.9
90.7
710
58.6
15.8
Prompt Tuning
64
50.7
19.3
Prompt
Prefix-Tuning
token
10.82
45.1
75.0
45.1
32.2
13.9
Bias
(A Convid
BidFi
0.02
59.9
62.3
60.8
52.9
31.3
58.2
War
LoRA
0.14
06.7
87.9
67.7
66.9
42.8
Láncar Projection Matrices
W
LoRA
0.14
65.1
86.7
68.8
54.6
35.5
W... W...
LoRA
0.28
65.2
89.1
67.3
51.7
38.0
Wo.
LoRA
0.14
67.0
87.1
69.1
52.9
46.4
65.7
LoRA
0.38
63.9
86.3
68.2
49.4
34.3
A
Full
0.19
56.6
77.0
58.1
46.0
33.1
Full
2.27
58.6
79.0
61.0
50.6
31.3
S6
Wg. Wc: Wa.
LoRA
0.29
60.3
82.7
63.0
46.6
33.7
33.7
Wa.t
LoRA
0.10
02.2
80.2
06.6
49.4
36.7
Convid
Full
0.06
62.5
81.9
66.1
51.1
35.5
Others
Full
0.02
51.0
71.0
51.1
29.5
(b) Full benchmark results on Spider using Mamba-I 2.8B.
Table Full benchmark results Spider (Yu et al., 2018) dataset using Mamba-L. We report the accuracy for Spider and its subsets. We consider two models our experiments: Mamba-I 1.4B and Mamba-I 2.8B. In each Mamba block, W and Wi.: are input projections that preprocess the input for SSM modules and the gating branch, respectively. W ou denotes the output projection after the gating mechanism. Wa and We are weight matrices for computing nependeepnndent B,, and C.. WA. and Wa.t represent down up projections of low-rank weight matrices the linear layer computing -ependdennndent step size A... 34 represents the bias this linear layer. D denotes the weight of residual connections.
19

---

Paramiter-Efficient Fine-Tuning of State Space Models
Layer
Method
Params (%)
Accuracy
Pretrained
0.00
0.08
Full
100.00
59.96
All
All
LoRA
1.92
60.35
Sa, Convid
BitFit
0.06
44.40
All
LoRA
1.02
62.79
Win
LoRA
0.34
53.49
Linear Projection Matrices
in.
LoRA
0.34
58.15
W Win,
LoRA
0.68
61.04
W
LoRA
52.04
Full
4.31
55.51
All
LoRA
0.92
43.96
A
Full
0.46
61.21
2.28
49.51
S6
Wa. Wc. WA.1
LoRA
0.69
52.27
Full
34.54
Wa.
LoRA
0.23
56.49
Convid
Full 0.14
55.65
Others
D, LayerNorm
Full
0.04
58.09
Table 10. Full benchmark results the CIFAR-10 (Krizhersky et al., 2009) dataset using Mamba-I 130M. We report accuracy (f), each Mamba block. Wi,,. and = are input projections that preprocess the input SSM modules the gating branch, respectively. W denotes the output projection after the gating mechanism. WB and Wc are weight matrices for computing input-dependent B, C,. Wa.1 Wa.t represent down and up projections low-rank weight matrices the linear layer computing input-dependent step size A.. BA represents the bias this linear layer. D denotes the weight of residual connections.
Hu E H become hy E RR the input sequence X E Px becomes æ E R.,, and the prefix P E Dxx becomes p T We omit the superscript (d) denoting the channel index. To differentiate between the hidden states and output of prefix-tuned S4 (i.e., f(PP, X]; ID.MMMMM and initial state tuned S4 (i.e., f(X; Hō)), we introduce superscripts "PT" and "IST" respectively. The "PT" superscript denotes hidden states and output of S4 after prefix-tuning, while "IST" indicates those after initial state tuning.
We divide the proposition into two statements:
1. For any prefix RM, there exists initial hidden state hj E RR such that the output of S4 after prefix-tuning and that after initial state tuning are identical, i.e., f(x; f(pp x); Mo) for all x E RN.
2. Furthermore, assume that I Iosicign a;) o 0 and I1 by of O. Then the converse (i.c., for any ho E R there exists p³ M such that f((p", ar: ho) (a;; hg) for all IR-N holds if and only if M > H.
We will first prove the first statement and then proceed to prove the second statement.
Statement 1.
The recurrent computation formulation of in (2) implies that for each position i, the output y, depends solely the previous hidden state h4-1 and the current input x,, Thus, demonstrate that f(x; ht) = s(pp ar) M for all æ E N suffices to show that the hidden state for predicting output ffST equals that for predicting output 4M41* where AST and yrr + are outputs corresponding to the input æ for initial state tuning and prefix-tuning, respectively. In model 1 = h is equal to the (M + 1)-th hidden state of prefix-tuned tuned model PT = EM A M Bpm- When this equality holds, the subsequent hidden states and outputs for both versions of S4 will be identical, as the input sequence from that point onward is the same. Therefore,
other words, it is sufficient to show that the initial state of initial-state-tune
20

---

Garameter-Efficeen Fine- Fine-Tuning of State Space Models
Layer
Method
Params (%)
METEOR
BLEU
Full
100.00
66.6
34.9
All
LoRA
1.39
66.9
45.4
W W.. nn
LoRA
1.02
67.1
44.7
Linear Projection Matrices
W
LoRA
0.68
67.1
43.0
W
LoRA
0.34
66.8
42.3
Full
4.17
65.7
39.7
All
LoRA
0.38
64.2
40.1
Full
4.00
66.0
36.2
WB. Wo. WA
LoRA
0.38
64.8
39.5
Table I1. Full benchmark results of LoRA on DART (Nan et al., 2021) dataset using Mamba-II 130M.
Layer
Method
# Params (%
All
Easy
Medium
Hard
Extra
Full
100.00
64.8
85.9
65.7
54.0
42.2
LoRA
0.71
64.5
81.0
66.4
56.9
42.8
W W m
LoRA
0.52
50.4
68.5
52.0
44.8
24.7
Linear Projection Matrices
LoRA
57.5
76.2
59.4
48.9
W
LoRA
0.18
57.9
81.0
56.7
51.7
33.1
Full
2.42
55.1
76.2
56.1
42.5
34.3
LoRA
0.18
54.1
74.2
58.1
46.0
21.7
S6
Ave
Full
0.00
21.5
46.0
18.8
11.5
2.4
Full
2.34
50.3
73.0
52.2
39.7
22.3
WB, We. Wa
LoRA
0.18
55.5
77.4
55.2
46.6
33.1
Table 12. Full benchmark results on the Spider (Yu et al., 2018) dataset using Mamba-1l 1.3B.
we prove the first statement by letting
M Σ m==1
h
Bpm
Statement 2. We aim to investigate the conditions under which there exists he E IR such that for any p E R. f(lp", ho) # f(x; hū). This is equivalent to demonstrating the existence of hi E R such that
M
M m Bpn for all p E RM
h
+
A
mi
This condition can further reformulated as
R
span (A
B,
M-1 B, B,..., +0,
which is equivalent
span A B,A B B...., CR.
(6)
To determine when this condition holds, we analyze three distinct cases: (i) M < H. (i) M H. and (iii) M > H.
(Case 1: When M < Ht). In this scenario, it is obvious that (6) holds. The existence of such a h is guaranteed because the dimension of the span is at most M, which is strictly less than H. This choice of h ensures that it cannot be represented as a linear combination of the vectors in the span, thereby establishing the inequality.
R if and only if (A MAAB.. B.A M B. BB...,B)
(Case 2:
When M H). In this scenario, span A B.A linearly independent. Note that
B.
B
de B. A
B B...., B)
det(A A M-1
AMA-..
by
21

---

Carameter-Efficiee Fine-Tuning of State Space Models
Layer
Methed
# Parums (%)
METEOR
BLEU
Full
100.00
70.8
45.0
Attention
All
LoRA
0,02
63.5
19.7
MLP
All
LoRA
1.37
70.9
46.2
Linear Projection Matrices S6
LoRA
0.31
70.2
Wi
LoRA
0.11
68.9
37.8
Lancar Projection Matrices
War
LoRA
0.06
All
Full
0.54
60.2
35.5
Wr. We. Wa.1
LoRA
0.15
06.6
Table 13. Full benchmark results DART (Nan et al., 2021) dataset using
Task
RTE
MRPC
CoL.A
SST-2
QNLI
QQP
MNLI
Avg. Score
Promut Tuning
$6.0
12.0
76.8
61.5
63.8
Prefix Tuning
69.5
75.7
43.4
91.5
83.4
83.1
35.6
68.6
Initial State Tuning
66.8
75.1
52.4
92.4
86.4
86.1
78.5
76,8
LoRA (Lincar Projection Matrices)
70.4
82.8
60.6
92.4
88.4
87.7
81.5
80.5
Table 14, Comparison of prompt-tuning, prefix-tuning, initial state tuning, and LoRA on seven tasks from the GLUE benchmark. We report Matthews correlation (1) for CoLA, overall (matched and mismatched) accuracy (f) for MNLI, and accuracy for other tasks. Initial state tuning and LoRA are constrained to use less than 0.5% trainable parameters. Bold numbers indicate the best performance across all three methods, while underlined numbers show the highest score among input-injection methods (prefix-tuning and initial state tuning). Initial state tuning outperforms prefix-tuning and prompt-tuning on five out of seven tasks, while LoRA consistently outperforms all input-injection methods.
where
d1
M det(A"
M-1
AA
1) det
02
#
a
a2
(Expand)
4H
0.H
1.
H
(-1)
M(N-1)
II
(aj d:).
(Vandermonde matrix)
(8)
0Si<JSH
Combining (7) and (8) yields
H
H
det
A M M B.... B) = (-1) 1(H-1)
B, A
II
(az
0:)
II
by
0SKjCH
k=
Therefore, if and only if Ihgiejgn(y ui) + 0 and I by + 0. we have
det AA M B, A B.... B...., #0,
which is both necessary and sufficient for the linear independence of (A B, A M-1 condition in (6) be satisfied.
....,B), B), and consequently, for the B,...,
(Case 3: When M > H). The analysis presented in case extends naturally to this scenario.
The combination of the three cases above completes the proof of statement 2.
C.4. Optimal Application of LoRA* in SSM-based Models
Several studies (Hu et al., 2023; He et al., 2021) present findings on Transformers, indicating that applying LoRA* to linear projection matrices yields performance comparable to or marginally superior to that of attention layers. In contrast, our experimental results on SSMs reveal that applying LoRA* * to linear projection matrices is more effective than applying it to S6. To elucidate this phenomenon, we examine the influence of updating linear projection matrices on the model's output.
22

---

Parameter-Efficiet Fine-Tuning of State Space Models
Notations.
To make the analysis tractable, consider simplified SSM-based architecture composed of the following components:
Two input projection matrices: 1in.1, , in. DD
The S6 module parameterized by diagonal state transition matrices {A with A(d) R the weight matrices dd=1
WB. Wc E Hx for computing input-dependent input transition vectors B, E TR. and output mapping vectors Cn E R" the down and up projection matrices WA,, E R WA,t 1 E ERRx RxD (where R is the rank) for low-rank weight matrices for computing the input-dependent step size A, = (A ..... ( E RD. for I....,N. N.
Define Ws = [W W WI, E (2H++B)xD In the Mamba implementation, Wse is implemented as the weight matrix of single linear layer, referred to as x x_proj in the codebase. Let the input sequence be X = (xi NN) E RDNN At each time step n, the S6 module uses two differently projected versions of the input: (i) the input projected via Wm.1 I is used to compute the input-dependent parameters A... B,, and C.. and (ii) the input projected via Wm,2 serves as the actual input to the S6 module. We note that this formulation generalizes the standard case, which uses single input projection matrix before the S6 module. In particular, it reduces to the standard case when W Wm.2. Then, the output at time step N is given by:
Input-dependent CN
Input-dependent Am A(Win, Im))
Input-dependent B., Bn
(d) VN
C(W
Σ
πI m=1
W )n
Input after projection Wi..
Parameters depending en imput after projection Wi.1
To be more specific, the definitions for the relevant terms are:
A n = softplus W WA, Wm, 1T + Ba),
(9)
A, (d) exp ( (4)
B A W WBW W 1Jn
Cn We Wi,, 18
When BA = 0, the output at time step N can be further written as
(d) VN (WoW in. (2.)
Σ
II exp(A ( A (1) A (d) W Wi..in 12 1a,,ienWWWi.,. (
(10)
where A, softplus WA., WA, WT, in, 1.
Theoretical Analysis. Assume none of the parameters are zero and D > 2H + R, where R is the rank of WA. Wa.t. Lemma in the main body shows that applying LoRA" solely to Wm.1 is equivalent to applying it to Ws6- For completeness, we provide the proof below and restate the lemma for the reader's convenience.
Lemma (Expressivity of Fine-Tuning Projection Matrices), Consider two models with the architecture described above. Let:
A larget model I* parameterized by ((A-(d))
Id= W We. W WA, W in Win);:
A frozen model fo parameterized by (d) Id=1 WB, Wc. WA,r. WT, WinD Win)).
The two models share (A'()) W and Win while differing in WB, We. WA,, and Wi..- Then, there exists a
projection matrix Win such that the frozen model matches the output of the target model for any inpul sequence, i.e.,
f(:(A-1) (dd) D Id=1, WB. Wc. WA... W Wm.. Win2) = f (: (= (1) W WG W.,, WT.1 W W 1
(11)
Proof of Lemma 1. To prove (11), we substitute (10) into (11), simplify the expression, and show that the equality holds under the following conditions:
W W
W W mn.1
(12)
m..1
W A
W m.
WA W
W B Wt
E WBWm..
23

---

Parameter-Efficiet Fine-Tuning of State Space Models
WB
Since Ws6-
Wc
the three conditions (12) can be compactly written as
Wa.
S6 W
S6 Wm.11
(13)
S6 in,1
now show that for any Ws6. there exists a matrix Win that satisfies (13). By applying Singular Value Decomposition
(SVD) to Ws6 obtain:
O22H-R)x(0) RDD----xXD22------- V W66 S6 U Σ
(14)
where E (22 E R and V DDD, The diagonal elements Σ are in decreasing
order. We let
Σ U U Wi
W V
(15)
where Q E R 2112211110 D an arbitrary matrix to be determined later. Plugging (14) and (15) back to Ws6 Wm.1 and simplifying results
W6 Win,1
U
W W S6 in.1
= U [ (PH-R)((D-HHRR) V
((14) (15))
= W Wi.1.
(Simplifying)
which demonstrates that (13) satisfied and completes the proof.
Empirical Validation. To experimentally verify Lemma we conduct a small-scale experiment. Specifically, we train Mamba 130M on three GLUE tasks RTE, MRPC, and CoLA for ten epoch under two settings: (1) training only the linear projection layer (Win), and (2) training the S6 modules (WB. B. Wc. WA,t). We experiment with various learning rates and, for each configuration, repeated the best-performing setting five times to ensure robustness. As shown in Fig, (training loss) and Table 15 (validation metrics), our results confirm that optimizing only the linear projection layer is as expressive as training the S6 layers. In fact, in all cases, training only the linear projection not only matches, but even outperforms S6 layer training and converges more quickly.
RTE
MRPC
CoLA
Train loss
0.75
0.75
0.6
Win
0.50
0.50
Wa. We. Wa.
0.4
0.25
0.25
0.2
0.00
0.00
0.0
100
100
200
200
400
Step
Step
Step
Figure Training cross-entropy loss, with shaded area indicating standard deviation, for fine-tuning linear projection layers versus S6 layers. The results show that W., matches the expressivity of WB. Wc, and WA,t. while also achieving faster convergence.
D. Details of Sec. 5: SDT
D.I. Understanding the Roles of State Matrix A, Input Transition Vector B, and Output Mapping Vector C. for Single Channel in S4 Modules
Problem Setting, Inspired by Zeng & Lee (2024)'s theoretical analysis of LoRA's expressive power, we adopt a similar framework to explore the expressive potential of various parameters in the model. Specifically, we assume a target model
24

---

Parameter-Efficient Fine-Tuning of State Space Models
Layers
RTE
MRPC
CoLA
Win
69.9 = 1.2
82.4
61.1 + 2.3
WB. Wc. Wa.t
68.3 # 0.8
79.9 1.2
54.9 + 1.5
Table 15. Mean and standard deviation of validation metries for fine-tuning linear projection layers versus S6 layers. The results demonstrate that Wi effectively captures the expressivity WB. Wc, and Wa, and achieves superior performance on validation metures-accuracy for RTE and MRPC, and Matthews correlation coefficient for CoLA.
that performs well on the intended task and a frozen model, which may be either pretrained or randomly initialized. Our goal is to identify parameter-efficient method to update the frozen model so that becomes functionally equivalent to the target model. aligmment with Zeng & Lee (2024), we assume that the frozen model's capacity equal to or exceeds that of the target model. This assumption is based on two main considerations: (i) analytical tractability, which necessitates that the frozen model must have the potential match the functionality of the target model, and (ii) a practical rationale, given that the models typically used in practice are often overparameterized. Assume that both the target model and the frozen model are S4, with the target model having a hidden state dimension H, and the frozen model having a hidden state dimension H 2 H. Meanwhile, suppose that all the hidden dimensions of both models are valid, meaning that none the parameter elements are zero. The target model, frozen model, and the updated model after tuning the parameters on the frozen model can be formulated using discretized parameters A, B, C' as follows:
n
(Target model)
f*(x), Σ C,A. m- "B,r" B,,m, where diag (A.), R.
n
(Frozen model) fo(x), Σ
CA" n- Bum, where diag B,),
m=
(Updated model)
f(x)) = Σ
CA
--m-n- Bam, where diag( (), B, A), B, C
mm1
Parameter Efficiency Analysis on S4. Let pH denote the set of all H H permutation matrices. Given this formulation, we present our first analysis of parameter efficiency for the S4 model in the following lemma. This analysis based on the parameters after necessary discretization (A, B, C). For the reader's convenience, we restate Lemma 2 below with minor notational changes to facilitate the proof.
Lemma (Minimal Parameter Adjustment for S4 Fine- -uning) Consider the parameters after discretization, i.e. A, B, C. To achieve fanctional equivalence between the updated model and the target model, i.e., == J², the minimum number of tunable parameters is:
eliminating redundant dimensions
aligning used dimensions with target model
milpepr
(diag (A) [
BOC (8.41):
+
[ APA 1.H. A +
[ B o
) I.H.
B. O
aligning the state matrix
aligning input curput interactions
Proof of Lemma 2. The key idea of this proof is straightforward. To facilitate the analysis and update the frozen model to be equivalent the target model, we first equalize the number of hidden state dimensions between the two models. This is achieved by expanding the target model's s A,, B,, and C, to match the H hidden state dimensions of the frozen model, padding the additional H H dimensions with zeros.
Define @ as the element-wiss product. We can express the target model as:
(x)x,, = Σ
C.
A,
m
B
0 o
0
Fm
m=1
Σ
diag (1.
[..
0
0
n-m
[CT
B.
0
0
0
Zm
25

---

Parameter-Effiient Fine-Tuning of State Space Models
Consider any permutation matrix P pH Applying P to permute the frozen model leaves the model functionally unchanged:
fo(Φ) = Σ CA
Bum =
Σ
CP
AP) n-m BBBm
Bam
m=1
m=1
Σ
diag
(P' (P AP) n- (PP P B) Im-
[. 0 0 0
Due to the convolution structure of A, two models are functionally equivalent if and only if P APAA aligns with
and (PT CT (P B) align with c.
B- 0
for some P E pH If they are already matching partially matched
for certain entries, no updates are required for those entries; only the unmatched entries need to be updated. Then, the required trainable parameters for this permutation matrix P are:
[P (diag A) @ B CT " 1m ll. + [P' AP I.H. A.11 + [ I.H. - -B. -B.C o.T
Optimizing the permutation matrix P E plt yields the desired results.
This lemma highlights the significance of identifying essential hidden state dimensions. The term [d"iiig(AA) A) @ B (H. underscores the importance of excluding redundant dimensions. This can be achieved by either directly removing these dimensions from the state matrix A, or by updating B or to ensure that only the selected hidden state dimensions are utilized during the input transition or output mapping phases. Once redundant dimensions are filtered out, tuning only the essential dimensions is sufficient align the updated model with the target model.
Furthermore, based on the lemma, the roles of the input transition vector B and C are nearly identical, as they consistently appear together as the combined term B @ CT, which is also discussed in Gupta et al. (2022). Consequently, one could opt to tune either B or C' exclusively or alternatively, split the indices into two groups, tuning B for the first group and C for the second. Both vectors indicate how information from different hidden state dimensions is integrated, whereas plays a distinct role, determining how the hidden states are stored.
In practice, instead of directly using the discretized parameters A, B, C, S4 is implemented using the continuous parameters A, B, C with step size A. To provide further practical guidance parameter tuning, the following two lemmas analyze the parameter efficiency of continuous parameters under different discretization methods. Two exemplary methods of discretization are bilinear and zero-order hold (ZOH):
= (I- (I-A/A (1 (A2A) (ZOH) B= (I- AB,
A = exp( A A)
(Bilinear)
(16)
B = (AA) ( I) AB.
Lemma 3 (Essential Continuous Parameter Set for $4 with Bilinear Discritization). Consider the parameters before discretization, iee, A, B,C C, which are subsequently discretized using bilinear discretization. To achieve functional equivalence between the updated model and the target model, i.e., f == f", is sufficient to lane the following number of parameters:
elininating reclundaus dimensions
aligning diseensions with torget model
min [APT (diag(T + 4/2A) & B C Dom
[P AP 1:H.. A.
P (B C Din, B. @ C.
aligning slate matric
aliigning input-ourpur interactions
Proof of Lemma 3. Combining Lemma 2 and the Bilinear discretization method in (16) yields the desired results.
Lemma (Essential Continuous Parameter Set for S4 with ZOH Discritization). Consider the parameters before discretiza- tion, i.e., A, B, C, which are subsequently discretized using ZOH discretization. To achieve functional equivalence between
26

---

Parameter-Effiiiel Fine- Fine-Tuning of State Space Models
the updated model and the larget model, i.e., f f*, it is sufficient to lune the following number of parameters:
eliminaainn redundias dimonsions
aligning used dineessoos with target morded
|AP (diag(exp(AA)) 1) a B ( Mun+taa
[P
AP L.H...H.. A.11,
c |P o Mis." B, o C.
aligning matrix
ofigning input-onipue
Proof of Lemma 4. Combining Lemma 2 and the ZOH discretization method in (16) yields the desired results.
The insights provided by Lemma 3 and Lemma 4 are the same as those provided by Lemma 2. The analysis here supports the second step of SDT-P presented in Sec. 5.
D.2. Extension Deep S4 Models
Our previous analysis focused on single-channel S4 models. We now expand our investigation to more complex scenarios involving deep S4 models for both target and frozen architectures, incorporating D channels and varying layer depths. In this section, in addition to SDT-P, we introduce SDT+ The key distinction between SDT+ and SDT-P lies in their treatment of linear projection matrices. SDT-P only operate on SSM modules and additionally requires applying LoRA to modify the linear projection matrices. In contrast, SDT+ applies SDT-P on SSM modules while also updates the columns of weight matrices corresponding to the updatable channels identified through Alg. 2. It is worth noting that the linear projection matrix updates in SDT+ are inherently low-rank, making a specialized case of SDT-P combined with LoRA. Our analysis starts with SDT+, and automatically applies SDT-P combined with LoRA.
In this analysis, we assume that each input token Ie belongs to X, bounded subset of RP, and that the length of the input sequence is finite. Let the frozen model have L. layers, and the target model have L layers, where L 2 L." Similar to the technique used in Zeng & Lee (2024) and Giannou et al. (2023). The basic idea of updating the frozen model to match the functionality of the target model is to utilize every IL/I.*1 layers of the frozen model to approximate every layer of the target model. We start introducing this proof idea from the simplest case where L = 1, L = D. In this scenario, we can simply choose one different channel tune and maintain all other channels at zero every layer. The outputs from the various channels of the deep $4 layers then combined through residual connection. This proof idea inspires to perform channel selection and make use of the residual connections, which is the first and third step of SDT-P presented in Sec. 5. Building on this idea, we present the following results for when the target model has only L* = 1 layer, and = D == 2
Lemma 5. Consider D-dimensional input sequence. Assume that the linear layers in the model have linear activation functions. Using SDT+, any deep S4 model with H hidden states per channel and L, layers can be updated to accurately present any target one-layer deep S4 model without residual connections, having reduced hidden state dimension H < H. Then this can be achieved by selectively fine-tuning at most [D/L] channels, H* hidden states, and residual connections at each layer, while additionally fully fine-tuning the linear projection matrix of the last layer only.
Proof of Lemma 5. In this proof, we start by considering the case where L = D. In this case, update single distinct channel for each layer while setting the other channels to zero. Essentially, we modify the frozen model so that each layer corresponds to and functions as an individual channel in the target model. To be more specific, we fully update the first channel in the first layer to match the first channel of the target model, second channel in the second layer to match the second channel the target model, so on and so forth.
For the (-th layer of the frozen model, we append subscript to all parameters of the deep S4 layer as introduced in (4). For the d-th channel, corresponding notations denoted with superscript (d). We define the t-th intermediate output token of the 1-th deep S4 layer as ž1, e RR. Additionally, the updated S4 module in layer is denoted as $4,, with S4 referring specifically to the sub-function that outputs the 1-th token. Therefore, for the 1-th intermediate output token of the I-th deep
27

---

Parameter-Effiient Fine-Tuning of State Space Models
S4 layer of the updated model can be written
21, Wi S4,,0 1. 21-1,)
Br ū 21-1,
$4 (1-1,111 =111))
W
+ B: + ūz o 211
(D) S4 (4-1,11
(D)
where Wi E DRD E R are the updated weight and biases of the 1-th layer of the frozen model, and ū, E RP is the updated residual connection weight of the frozen model.
For layers/ / <L= D. We follow the steps provided in Sec. 5 to update the 1-th layer of the frozen model such that it functionally equivalent to the 1-th channel of the target model. For the reader's convinence, we restate our strategies here:
(Channel Selection) Select D' & D (D' = here) important channels for making predictions. Any channel d that is not utilized will have their corresponding (i(d) set to zero, eliminating the need to update parameters for (d) and the d-th column of W. To more specific, we let C(d) 0 for all d * this scenario.
(Hidden State Selection) Within the selected channels, select H' < H. important hidden states. For any hidden state that is not used within a selected channel d, the corresponding element in ((4) will set to zero, thus eliminating the need to ()))) tune the corresponding element in (d) To be more specific, we can achieve S4 (+) S4 (-) by Lemma 2.
(Residual and Bias Tuning) Regardless of other selections, SDT consistently tunes the coefficients of residual connections and biases in linear projections, as these components contain a negligible number of parameters. In this scenario, we let 3 0, ü
0
elements
D- elemenis
This construction yields
(1) 1-1t
(1-1)
(1)
(1+1) 2-11
11,
1--11
4-1.1
Consequently, only the /-th channel is active in the (-th layer, while all other layers function as identity mappings, propagating the output of the preceding layer without modification.
For layer L D. Based on the setup of the first L -1 layers, have
(x(())
a(4)]]
2T-1,1
For the last layer, we let
W W. P2 = B., üp 0,
$4 11)
S4 (-) (4). (-), which can be achieved by Lemma 2.
easy to verify that the output of the updated frozen model is identical to the output of the target model, i.e.,
y: = 21, = W. (44
S4!. (x(41) (((--1)
+ B..
Thus far, we have demonstrated that the statement holds when L = D. This analysis can be readily extended to cases where + D by tuning [D/L] channels at each layer. For example, when L = D/2, we can tune two channels per layer using a construction similar to the one described above. This generalization completes the proof.
28

---

Parameter-Efficiet Fine-Tuning of State Space Models
Theorem 2 (Expressive Power of SDT+ on Deep S4 Models). Consider D-dimensional input sequence. Assume that the linear layers in the model have linear activation functions. Using SDT+, ant deep S4 model with H hidden states per channel and L layers be updated to accurately present any target deep S4 model without residual connections, having reduced hidden state dimension H* < H, and fewer layers L* < L. This can be achieved by selectively fine-tuning at most [DL [L] channels, H' hidden states, and residual connections at each layer.
Proof of Theorem 2. We update every [D/L] layers of the frozen model to approximate each layer of the target model. By applying Lemma 5 iteratively to each set of [D/L] layers, we obtain the desired result.
Theorem leads to the following result, which represents the deep S4 model case of Theorem
Theorem 3 (Expressive Power of SDT-P on Deep S4 Models). Consider a D-dimensional input sequence. Assume that the linear layers in the model have linear activation functions. Using SDT-P, any deep S4 model with H hidden states per channel and L layers can be updated to accurately present any target deep S4 model without residual connections, having a reduced hidden state dimension H* < H, and fewer layers It* < L. This can be achieved by selectively fine-tuning at most [DL/LL) channels, H* hidden states on SSM modules, applying rank- +=1 updates on linear projection matrices and updating residual connections and biases at each layer, while additionally fully fine- the linear projection matrix of the last layer only.
Proof of Theorem 1. Since SDT+ is special case of SDT-P, Theorem 2 directly implies the desired statement.
D.3. Extension to S6
In this section, we extend the discussion of SDT-P and SDT+ to S6, following the same logic. We begin by proving results for SDT+ in the scenario where the target model consists of only single layer. In doing so, we extend Theorem 3 to apply to deep S6 models by first generalizing Lemma 5 to Lemma 6.
Lemma 6. Consider D-dimensional input sequence. Assume that the linear layers in the model have linear activation functions. Using SDT+, any deep 56 model with H hidden states per channel and I, layers can be updated to accurately present any target one-layer deep S6 model without residual connections, having reduced hidden state dimension H1 < HH. Then this can be achieved by selectively fin-it at most [D/L] channels, H hidden states, and residual connections at each layer, while additionally fully fine-tuning the linear projection matrix of the last layer only.
Proof of Lemma 6. To demonstrate this, we can follow the same proof strategy as the proof of Lemma 5. particular, the t-th intermediate output token of the I-th deep S6 layer of the updated model can be similarly written as
11, Wi S61, 1,1-11111.. 11)))) + B: + ûg 21-1.
56 (..-1_....,,2...
(1)
W
+ +: + â ④ 21-1,
(D) (D)
(D)
S6 (1--1111--- -111))
where W. E DxD B E D are the updated weight and biases of the I-th layer of the frozen model, and â E R.D is the updated residual connection weight of the frozen model.
For layers/ / < === We follow the steps provided in Sec. 5 to update the I-th layer of the frozen model such that it functionally equivalent to the I-th channel of the target model. For the reader's convinence, we restate our strategies here:
(Channel Selection) Select D' < D (D' here) important channels for making predictions. For any channel d that is not utilized, rather than directly setting the corresponding (d) to zero as in the deep S4 model, we instead set ((ll) to be sufficiently large. According to the computation of SSM parameters described in (17), this ensures that B(d) is set to zero for all d + in this scenario. This approach is equivalent to setting (()) to zero, as both result in the channel producing all zeros.
29

---

Parameter-Efficient Fine-Tuning of State Space Models
(Hidden State Selection) Within the selected channels, select H' v H important hidden states. For any hidden state that is not used within selected channel d, the corresponding entries in A will be set sufficiently small. To be more (1) specific, can achieve S (+) S6 (-) by leveraging the discretized parameters. Lemma provides the conditions for this equality to hold by updating A, B, and C' for the 1-th channel, where these parameters are computed as follows:
n softplus( Wa,t, BB),
(17)
(d) A, 15 = exp( A (d) A (d)),
B (d)
A BB C, WW
Therefore, we can achieve S = S6 by only updating the corresponding values or columns of the weight matrices for each channel and dimension.
(Residual and Bias Tuning) Regardless of other selections, SDT+ consistently tunes the coefficients of residual connec- tions and biases in linear projections, as these components contain negligible number of parameters. In this scenario, we let 3 0, i
1
elements
elements
This construction yields
(1) 1111
(1-1)
1.
(D) 1-1
11,
For the remaining layers, following the same steps leads to the desired results.
Therefore, we similarly obtain the following two results.
Theorem 4 (Expressive Power of SDT+ Deep S6 Models). Consider D-dimensional input sequence. Assume that the linear layers in the model have linear activation functions. Using SDT+, any deep S6 model with H hidden states per channel and L layers can be updated to accurately present any target deep 56 model without residual connections, having reduced hidden state dimension H < H, and fewer layers L* < L. This can be achieved by selectively fine-tuning at most [DL* [ channels. H* hidden states, and residual connections at each layer.
Theorem (Expressive Power of SDT-P on Deep S6 Models). Consider a D-dimensional imput sequence. Assume that the linear layers the model have linear activation functions. Using SDT-P, any deep S6 model with H hidden states per channel and L layers can be updated accurately present any target deep S6 model without residual connections, having reduced hidden state dimension H* < H, and fewer layers L < L. This can be achieved by selectively fine-tuning at most [DL' /L] channels, H* hidden states SSM modules, applying rank- I updates on linear projection matrices and updating residual connections and biases at each layer, while additionally fully fine -tming the linear projection matrix of the last layer only.
Combining Theorem 3 and 5 leads to Theorem 1.
D.4. Sparse Dimension Tuning and Pruning (SDT-P)
Algorithm is our extended algorithm, which includes setting dimensions to zero. However, in practical settings, setting channels to zero is not necessary and omitting it reduces number of hyperparameters, as pruning parameters is effectively equivalent to training them to zero.
30

---

Parameter-Eficient Fine-Tuning of State Space Models
Algorithm 2 Dimension Selection Algorithm of SDT-P
Input: small subset of dataset D, warmup epochs E, number of layers L, total channels D, total states H, state sparsity Sp. channel sparsity ag. state update fraction 3, channel update fraction
/* Warmup Epochs
Perform full update on SSM modules using for epochs * Categorize dimensions
*
for / do
/* Set dimensions as zero
Sort channels based on | (4
Select final (1 D channels as zero channels and denote non-zero channels as set
for d E D do
Sort states based on magnitude of A/ at each state dimension
Select final (1 ag) H states as zero states and denote non-zero states as set
Unfreeze dimensions
Sort non-zero channels D based on changes of | ( Select the top 3|DD] channels as updatable, denoted by IT for d E IV do
Sort non-zero state dimensions based changes of A (4)
Select the top a HI|| states as updatable at the d-th channel
D.5. Extension S5
In this part, we extend Lemma 2 and Theorem to two corresponding results for S5. The extension follows the same procedure as the previous proof, so we omit the details here.
Lemma 7 (Minimal Parameter Adjustment for S5 Fine- Fine-Tuningl Assume all hidden dimensions of the target model f are non-zero, i.e., all elements of diag(A, @ ((.)) ((d) non-zero. To update frozen model fo such that it becomes functionally equivalent to the target model f*, the minimum number of tunable parameters is: afiguing remaining dimensions with torget model
eliminating relundinn dimourionn
m, | I.H. A.11, +E
[ B 2 C(UT
B の C(AT
B.a @ c!
H. +1:8
1.H.
subject to (A, A CoP) P is a permutation matrix).
Theorem 6 (Expressive Power SDT-P with LoRA on Simplified SSM-based Models). Assume all layers use linear activations. Let fo be a frozen deep S4 S5, or 56 model with L layers, each containing H hidden states per channel. Let f* be a smaller target model of the same type (S4, S5 or S6), with no residual connections, L layers, and H* H hidden states per channel. Then, there exists a set of parameter updates to fo satisfying the following conditions such that for any finite-length input sequence X =x11...,,NN) with H,, E X C RD, where X is bounded, the resulting model satisfies f(X) (X):
1. (SDT-P on SSM) In each SSM module, update at most [DL* LL] channels. Within each updated channel, fine fine-tune at most H* hidden states and set the rest to zero. (LoRA* on Linear Projections) Apply rank- L/ updates to each linear projection matrix.
3. (Minimal Additional Updates) Modify only the residual connections, per-layer biases, and the final-layer output projection.
D.6. Memory Usage and Runtime Analysis of SDT
Memory Usage Analysis. To assess the memory usage of SDT and LoRA, we conducted experiments on four different models, including both SSM and hybrid architectures. For each model and method, a dataset was generated with 2,500 batches of data samples, each batch comprising a random sequence 1,500 tokens. The simulation was repeated four times, including dataset generation. All experiments were carried out on a single H100 GPU, and the reported metrics represent
31

---

Parameter-Efficeet Fine-Tuning of State Space Models
130M
370M
790M
1.4B
2.8B
20
50
50
*>--
LoRA
Peak memory (GB)
40
60
40
40
LoRA & SDT
30
40
30
30
10
20
20
5
10
10
10
0
2000
4000
2000
4000
0
2000
4000
1000
2000
500
1000
Sequence Length
Sequence Length
Sequence Length
Sequence Length
Sequence Length
Figure 4. Peak memory usage during training as a function of context length for different Mamba model sizes. Each line represents the mean three configurations; shaded regions indicate min-max ranges. SDT consistently more memory-efficiet than LoRA when applying to SSM module.
averages across the four simulations. Consistent with our previous experiments, we used the original hyperparameter settings, ensuring that SDT includes similar trainable parameters than LoRA. The memory usage of LoRA and SDT is presented in Table 16. Our observations indicate that SDT requires less memory than LoRA. This difference can be attributed to the design of the LoRA adapters, which involve matrix multiplication two low-rank matrices. contrast, tuning SSM with the same number of parameters does not require any matrix multiplication, resulting in lower memory usage.
Memory Usage (GB)
Mamba- 130M
Mamba-1.4B
Tambba-Tiny-199h
Jamba Jamba-Minn-S3 Mini-52B
LoRA
7.753
37.167
7.207
71.986
LoRA SDT
5.738
26.491
6.605
67.193
Table 16. Memory usage comparison between SDT and LoRA various models. Bold numbers indicate the lowest memory usage for each model.
To provide a more fine-grained view, we further analyze how sequence length affects peak memory usage for different Mamba model sizes, as shown in Fig. 4. measure the memory required to process single training batch with varying context lengths using randomly generated data. Each batch contains four examples, with 90% of tokens used as input and 10% output (loss is computed only the output tokens). The experimental settings for both LoRA and SDT follow the setup described in Section 6.2. We evaluate three configurations for each method, matched in parameter budget. In the plot, each line represents the average across the three configurations, and the shaded region for LoRA shows the range (minimum to maximum). SDT shows negligible variance across configurations, so no shading is included. All models are trained for 500 iterations, and results are averaged over these iterations. Experiments were conducted on an NVIDIA H100 80GB GPU.
Runtime Analysis. We similarly analyze the latency of LoRA and SDT, using the same experimental setup as in Table 16. Fine-tuning with SDT consists two stages: (I) dimension selection and (2) standard training. In this study, we first compare the runtime of SDT and LoRA during stage 2 (training) and then evaluate the additional runtime introduced by SDT during stage (dimension selection). Our results show that the dimension selection stage adds only marginal runtime overhead, and SDT is more efficient than LoRA in standard training.
Training: When the channels and states have been selected, the training of SDT is faster than LoRA when the same number of trainable parameters are considered.
The runtimes are reported Table 17. observe that, despite having similar trainable parameters, SDT is faster than LoRA. We attribute this the fact that LoRA introduces additional FLOPs due to the extra matrix multiplication operations required for each update (specifically, the multiplication of two low-rank matrices).
Dimension Selection: For dimension selection, our method first performs Initial Subset Training, and then selects the dimensions based on the magnitude of parameter changes across different dimensions.
1. Initial Subset Training: We update the model by going through only a subset of the dataset (e.g., , 3% of batches in DART
32

---

Parameter-Efficiet Fine-Tuning of State Space Models
Avg. Runtime (Seconds)
Mamba- 130M
Mamba-1.4t
Jamba-Tiny-3190
Jamba-Minii522
LoRA
410.0 80.0
2060.0 135.0
352.5 107.5
3427.5 185.0
LoRA & SDT
330.0 77.5
1697.5 +: 87.5
257.5 +: 72.5
3065.0 + 232.5
Table 17. Runtime comparison of SDT and LoRA during stage 2 (training).
experiments), which sufficient practice.
2. Magnitude-Baser Dimension Selection: After the subset training, select dimensions based on the magnitude of parameter changes observed.
In this experiment, simulate real scenario using dataset with 2,500 batches, considering small subset containing 125 batches (5% of the full dataset). We repeat the experiments 80 times, and the reported numbers are averaged across these simulations. Table 18 demonstrates that the dimension selection stage adds only negligible runtime.
Avg. Runtime (Seconds)
Mamba-1300
Mamba-1.4B
Jamba-Tiny-3199
Jamba-Mini-577
Initial Subset Training
16.250 3.880
85,250 5.130
15.750 + 1.000
163.630 + 10.
Hagnitude-Based Dimension Selection
0.280 + 0.000
0.520 + 0.120
0.090 + 0.000
0.240 + 0.040
Total Time
16.530 + 3.880
85.770 = 5.250
15.840 + 1.000
163.870 + 10.160
Proportion of Training Epoch
0.050
0.051 x
0.062x
0.053x
Proportion of Training 5 Epoch
0.010x
0.010x
0.012x
0.011x
Table 18. Runtime comparison SDT and LoRA during stage 1 (dimension selection).
We further examine how runtime varies with sequence length, using the same experimental setup as in the memory analysis (Fig. 4). Our results in Fig. show that SDT consistently outperforms LoRA in training speed when applied SSM modules.
130M
370M
790M
1.4B
2.8B
1.50
0.35
1.0
LoRA
0.8
Timre per batch (s)
0.30
1.25
0.8
0.25
1.00
0.6
0.6
0.6
0.20
0.75
0.15
0.4
0.50
0.4
0.4
0.10
0.2
0.2
0.05
0.2
2000
4000
2000
2000
4000
1000
2000
500
1000
Sequence Length
Sequence Length
Sequence Length
Sequence Length
Sequence Length
Figure 5. Average training time per batch across different sequence lengths and Mamba model sizes. Each line represents the mean across three configurations; shaded regions indicate min-max ranges. SDT consistently faster than LoRA when applying module.
E. Expanded Sec. 6: Evaluation of SDT
E.1. Experiments on Deep S4 Models
Synthetic. For selecting channels and hidden states, we initiate with warmup learning rate between 1e - 2 and 1e and conduct 20 warmup iterations. Learning rates are adjusted between 5e 2, 1e 2, 5e 3, and le 3. We apply LoRA with ranks of and 4 to the SSM and with ranks of 4, 8, and 16 to the línear projection matrices. Non-zero states are selected from the sets (4, 8), and non-zero channels from (8, 16).
33

---

Parameter-Effiiiet Fine-Tuning of State Space Models
Method
# Params (%)
Accuracy
Frozen
0.00
73.9
LoRA (Proj)
16.00
77.6
LoRA (S4+Proj)
15.52
77.6
LoRA & SDT
11.17
78.0
Full Fine- Fine-Tuning
100.00
77.6
Table 19. Accuracy comparison between SDT and LoRA on deep S4 models for CIEAR-10 (Krizhevsky et al., 2009).
In addition, we compare the convergence speed of LoRA and SDT in terms of training loss for sequence lengths in (100, 500, 1000). We plot the MSE of both methods against wall-clock time. As shown in Fig. 6, SDT consistently converges to lower loss faster than LoRA across all tested sequence lengths.
Sequence Length
100
Sequence Length 500 Sequence Length 1000
10.0
SDT LoRA
7.5
MSE
5.0
2.5
0.0
200
200
200
400
Wall Clock Time (seconds)
Figure 6. Comparison of SDT and LoRA for tuning S4 in deep S4 models, where LoRA is applied to linear projection matrices. Results shown across varying sequence lengths under different time budgets in synthetic experiments.
CIFAR-10 (Krizhevsky et al., 2009). Previous work (Dinh et al., 2022) demonstrates that large language models can be fine-tuned for image classification tasks. In this study, we investigate the adaptation of SSMs for computer vision, focusing on experiments conducted with the CIFAR-10 dataset (Krizhevsky et al., 2009). We employ an eight-layer deep S4 model with a hidden state dimension of 16 and a model dimension of 64. Since pretrained deep S4 models are not available, we simulate pretrained scenario by fully updating the model for 50 epochs first, then subsequently evaluating the PEFT methods over an additional 5 epochs. We adhere to the preprocessing steps for CIFAR-10 as outlined by Gu al. (2022a). The LoRA ranks for linear projection matrices are tuned among (1, 2, 4, 8, 16), and for the S4 component, ranks are set from (1, 2, 4). Non-zero states are chosen from (8, 12, 16), and non-zero channels from {48, 64). The warmup phase includes epoch with learning rate of 1e - 2. For linear projection matrices, LoRA ranks are explored at (2, 4, 8, 16), and for the SSM, ranks at (2, 4, 8). All state dimensions are updated, and channel dimensions considered for updates are (4, 8, 16, 32). The results, as reported in Table 19, indicate that SDT outperforms LoRA with fewer trainable parameters.
E.2. Experiments on M1, Jamba, and LoRA+
Additional Experimental Details. In this paragraph, we provide further experimental details. Unless otherwise stated, our experiment setting identical to Sec. C.I. For LoRA, we consider three different LoRA configurations at each layer, targeting the primary parameters of Mamba. Specifically, we focus on the following matrices: Wo.r (output linear projection), WB B. We (weight matrices for computing imputedependent B,, C,), and W WA.,, Wa.t (down and up projection matrices of LoRA adapters for computing A). The three LoRA application methods are: (i) Wout. WB, Wc. and WA, Wa.ti (ii) W... WB. We and WA.J i and (iii) Won and WA,t- For SDT, scenarios. We select the state freeze ratio a from the set (75%, 90%, 95%} and apply LoRA exclusively to W... to maintain a comparable number of trainable parameters. Residual connections and biases are frozen in this experiment. For the warmup, we employ 500 data batches to fully train the SSM modules prior to dimension selection, except for the RTE task in GLUE, where we use 250 batches due to its limited dataset size. Note that the parameters are reverted back after the warmup stage.
the channel freeze ratio at 99% across all
34

---

Parameter-Efficeet Fine-Tuning of State Space Models
Additional Results Mamba-Il. For Mamba-II, applying SDT is not straightforward because Mamba-II further constrains A such that all (non-zero) entries must have the same value. Therefore, original dimension selection approach cannot be directly applied here. We consider naive extension of SDT by selecting dimensions in the projection matrices for input mapping vector B and the projection matrices for output mapping vector C' using their respective magnitude, and fine-tune the selected dimensions and all elements of state transition matrix A.
Tables 20 and 21 compare the performance on Mamba-II. The results demonstrate that SDT consistently outperforms LoRA on Mamba-ll models.
Model
Mamba-I1-1300
Mamba- -I--b-------.BB
Dataset Metric (t)
DART
SAMSum
Spider
Params (%)
Params (%)
METEOR
BLEU
R1
R2
RL
Acc.
LoRA
0.3354
68.71
48.09
0.1614
49.73
26.14
41.53
72.36
LoRA SDT
0.3393
70.60
48.93
0.1767
50.72
27.21
42.54
84.15
Table 20. Performance comparison between SDT and LoRA on Hamba-II-1300) and Mamba-II-1.33 Bold numbers indicate the best performance for each task.
Model
Mlmma--I-130N
Dataset
GLUE
Params (%)
Accuracy (1)
RTE
MRPC
SST2
QNLI
QQP
MNLI
LoRA
0.3354
63.4
80.9
89.1
85.3
87.1
78.6
LoRA & SDT
0.3393
64.3
82.3
94.1
87.0
88.3
81.1
Table 21. Performance comparison between SDT and LoRA on GLUE (Wang et al., 2019) benchmark using Mamba-II-130M. Bold numbers indicate the best performance for each task.
Additional Results on Jamba. Table shows results for SDT and LoRA on additional datasets. Even though the performance improvement is smaller, our method outperforms pure LoRA in most cases. Mamba layers make up only small part of Jamba, which is possible reason for smaller performance gains.
GLUE
DART
CelebA
SAMSum
Spider Acc.
LinProj
S6
Avg.
BLEU
MET.
Acc.
R1
R2
RL
LoRA
65.5
52.9
73.0
88.5
56.4
33.5
47.9
90.7
LoRA
SDT
67.7
53.1
73.0
88.4
56.5
33.5
48.0
89.8
Table 22, Performance comparison between SDT and LoRA on pretrained Jamba models. Bold numbers indicate the best perfor- mance for each task. We use Jamba-Tiny-319M to compare performance SDT and LoRA on GLUE (Wang et al., 2019), and CelebA (Liu al., 2015) benchmarks. For all other datasets, we employ Jamba-Mini-52B, We report only the best setting out of three for each method.
Additional Results for LoRA+. We extend our investigation to include LoRA+ (Hayou et al., 2024) with SDT and evaluate its performance against LoRA+ across various datasets on both Mamba-I and Mamba-II. The results, presented in Table 23, show that integrating SDT with LoRA+ enhances its effectiveness and achieves superior performance compared using LoRA+ alone.
35

---

Parameter- Efficieet Fine- Fine-Tuning of State Space Models
Model
Mamba-I-130M
Mamba-Il-130M
Mamba-ll-1.38
Dataset Metric (1)
DART
DART
SAMSum
Spider
METEOR
BLEU
METEOR
BLEU
R1
R2
RL
Acc.
LoRA+
70.06
50.91
69.78
49.14
49.83
26.09
41.66
73.75
LoRA+ & SDT
70.58
51.93
70.48
49.99
50.81
27.19
42.4
84.22
Table 23. Performance comparison between LoRA+ and SDT on Mamba-I and Mamba-II. Bold numbers indicate the best perfor- mance for each task. We test all experiments under various parameter settings (<0.4%) for both LoRA+ and SDT, and report the best values.
36


Reviews:
[{"review_id": "7SHUuiHJoy", "reviewer": "Reviewer_Bhh7", "summary": "This paper investigates how PEFT methods perform on State Space Models (SSMs) (e.g. the Mamba architecture) and identifies which model components are best to target. It provides a comprehensive benchmark of existing PEFT techniques on SSM-based language models and hybrid architectures with Jamba. A key finding is that LoRA and its variants consistently outperform other PEFT approaches on SSM models​. However, LoRA’s benefit comes mainly from fine-tuning the linear projection matrices (e.g. input/output projections) – applying LoRA to the SSM-specific components does not yield additional gains​. In fact, prompt-based tuning methods (like prefix or prompt tuning) are found largely ineffective on SSMs, essentially only adjusting the initial hidden state (a severely limited form of fine-tuning, as proven in the paper’s Proposition 1)​. These insights highlighted a gap: none of the standard PEFT methods are well-suited for the internal SSM parameters. To address this, the authors propose a new method called Sparse Dimension Tuning (SDT) – a PEFT strategy tailored for SSM layers. SDT works by selectively fine-tuning only a subset of the state-space channels and state dimensions, while freezing or pruning the rest. When combining SDT (for SSM layers) with LoRA (for the linear projection layers), the paper achieves state-of-the-art fine-tuning performance on multiple benchmarks​. The approach matches or surpasses the best existing methods while training only a small fraction of model parameters, validating the effectiveness of the proposed methodology across extensive experiments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are delighted that the reviewer likes the paper, recognizing it as (i) well-executed, (ii) a systematic study of PEFT for emerging neural architectures (Mamba, Mamba 2, and hybrid variants), and (iii) providing strong and comprehensive empirical results. Thank you for your encouragement!", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "It is related to Parameter Efficient Fine Tuning, State Space Models and Hybrid State Space Models. No previous works on systematic studies of PEFT on SSMs.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Stength:\n\n1. This is a well executed paper that systematically studied PEFT for the emerging neural architectures, including Mamba, Mamba 2 and its hybrid variant. The empirical results are strong and comprehensive.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bhaSMiQSQt", "reviewer": "Reviewer_biYn", "summary": "This paper investigated the PEFT for SSM, like MAMBA.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that (i) our paper is well written, (ii) it tackles the gap where PEFT studies favor Transformer over SSM, and (iii) it contains many experimental results to support the claims.\n\n---\n\n> Q1: The proposed PEFT is incremental, such idea is mainly from the PEFT for transformer-based LLM.\n\nWe respectfully disagree. SDT is not just an incremental tweak of PEFT for Transformer-based LLMs—it’s built on a fresh theoretical analysis tailored for SSMs’ unique parameter structure. Beyond that, our paper delivers substantial contributions that stand firm, even if the method is viewed as incremental by the reviewer: \n* (i) **(One of the First)** A systematic benchmark of PEFT methods on SSM models,\n* (ii) **(First)** A theoretical analysis of PEFT in the SSM setting, and\n* (iii) **(One of the First)** A new PEFT method designed specifically for SSMs, supported by both theory and strong empirical results.\n\nTogether, these contributions offer new insights into PEFT beyond what has been explored in Transformer-based LLMs.\n\n\n> Q2: The theoretical analysis is mainly for S4 or S6, how about S5 and Mamba2?\n\n* **`New Theoretical Results`** **S5**: Although our original analysis primarily focused on S4 and S6, based on your comment, we have extended our theoretical results (Lemma 2 and Theorem 1) to include S5. Specifically, Lemma 2, which characterizes the minimal number of parameters required to update a frozen S4 model for functional equivalence to a target S4, has been extended to S5 with similar conclusions, accounting for multi-channel handling. Additionally, Theorem 1, regarding the expressive power of SDT-P with LoRA on simplified SSM-based models, also holds true for S5. We provide the detailed extension for S5 here: https://anonymous.4open.science/r/ce7/l9.png.\n\n\n* **Mamba-II**: Although extending our theoretical analysis to Mamba-II is non-trivial, we successfully adapt our method to this architecture, as detailed in Sec. C.2 and E.2, and evaluate it on diverse tasks, including GLUE, DART, SAMSum, and Spider. Our method consistently outperforms LoRA alone on Mamba-II, highlighting its generalizability beyond the architectures explicitly covered in our theoretical analysis. We will acknowledge this theoretical limitation explicitly in the conclusion section, emphasizing that extending our analysis to Mamba-II is an important direction for future work.\n\n---\n\n**Final Note:** Thank you for sharing your concerns. While we understand your perspective, we kindly remind you of the substantial contributions our paper makes and the efforts we've undertaken to address your comments. We would greatly appreciate it if you could reconsider your evaluation score and support the acceptance of our paper.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "it mainly focus on PEFT for SSM.", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is well written, and there are many PEFT studies for Transformer-based LLM, not many PEFT for SSM.\n\n2. The paper contains many experiemntal result to support their claim in the supplementary materials.\n\nWeaknesses:\n\n1. The proposed PEFT is incremental, such idea is mainly from the PEFT for transformer-based LLM.\n2. The theoretical analysis is mainly for S4 or S6, how about S5 and Mamba2?", "other_comments_or_suggestions": "No", "questions_for_authors": "1. The proposed PEFT is incremental, such idea is mainly from the PEFT for transformer-based LLM.\n2. The theoretical analysis is mainly for S4 or S6, how about S5 and Mamba2?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SyhG0mvskA", "reviewer": "Reviewer_TAMW", "summary": "This paper investigates the performance of popular parameter-efficient fine-tuning methods (PEFT) (e.g., LoRA and its variants) when applied to SSMs like Mamba and hybrid models such as Jamba. It finds that LoRA-based methods consistently outperform other PEFT approaches, especially when applied to linear projection matrices, but fail to improve performance significantly when directly applied to SSM modules. To address this, the authors propose Sparse Dimension Tuning (SDT), a specialized PEFT method designed explicitly for fine-tuning SSM modules by selectively training specific channel and state dimensions based on theoretical insights. Extensive experiments demonstrate that combining SDT for SSM modules with LoRA for linear projections achieves state-of-the-art performance across diverse tasks, including natural language understanding, generation, and computer vision benchmarks, confirming SDT's effectiveness and efficiency compared to existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging feedback, particularly for recognizing that (i) our method is creative and rational, (ii) our claims are supported by extensive experiments and theoretical analysis, (iii) our theoretical claims are sound, (iv) we advance SOTA results in fine-tuning SSM-based and hybrid models, and (v) our contributions are well-integrated with the broader scientific literature. \n\n\n---\n\n> Q: Limited applicability beyond SSM-based and hybrid models. \n\nOur proposed SDT algorithm is intentionally tailored for SSMs, with our theoretical analysis and experiments specifically focused on this model class.\n\nRegarding your concern, while applying SDT to pure Transformers is beyond our current scope, we demonstrate its broader potential through experiments on Mamba-II, whose SSD module closely resembles attention [1]. When applied to SSD modules with LoRA on linear projection matrices, SDT consistently outperforms LoRA alone on Mamba-II across diverse tasks (Sec E.2).\n\nFurthermore, we believe the theoretical insights and proof techniques from our analysis could prove valuable for developing PEFT methods for other architectures. \n\n\n--- \n\n*References:*\n\n[1] Dao, T. and Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning, 2024.\n\n**Final Note:**  Thank you again for your valuable comments. We are grateful that you appreciate our paper’s contributions in theory, methodology, and experimentation. If you have any remaining questions, please do not hesitate to let us know. Assuming our responses have addressed your concerns satisfactorily, we kindly ask you to consider raising your score and supporting our paper.", "claims_and_evidence": "Yes. The claims are backed by extensive experiments and theoretical analysis.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem. The paper introduces SDT specifically to address the limitations of existing PEFT methods on SSM-based models, and it benchmarks these methods using a diverse set of tasks and datasets—including GLUE, DART, SAMSum, Spider, CIFAR-10, and CelebA—which are well-recognized for assessing performance in both language and vision applications.", "theoretical_claims": "The proofs for the main theoretical claims, e.g., Lemma 1, Lemma 2, and Theorem 1 and their logical structures and derivations appear sound under the stated assumptions.", "experimental_designs_or_analyses": "The authors benchmark PEFT methods on both synthetic datasets (using deep S4 models) and real-world tasks spanning GLUE, DART, SAMSum, Spider, CIFAR-10, and CelebA with SSM-based (Mamba) and hybrid (Jamba) architectures; while these experiments are well-controlled with careful hyperparameter tuning and fair parameter budget comparisons, a minor issue arise: \n- The focus on SSM-based and hybrid models limits the generalizability of the findings, suggesting that broader architectural testing could further strengthen the evidence.", "relation_to_broader_scientific_literature": "The paper’s contributions are well-integrated with the broader scientific literature by extending established PEFT methods—such as prompt tuning, prefix-tuning, BitFit, and especially low-rank adaptations like LoRA—to the realm of SSMs, which have been previously developed for efficient long-sequence modeling.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n- Creative integration of established PEFT techniques like LoRA with a novel Sparse Dimension Tuning (SDT) method tailored for SSM modules.\n- Advances the state-of-the-art in fine-tuning SSM-based and hybrid models, which is valuable for efficient language modeling.\n\nWeakness:\n- Limited evaluation on a broader range of model architectures beyond SSM-based and hybrid models, which may affect the generalizability of the findings.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qn7ybW8fjw", "reviewer": "Reviewer_GPVJ", "summary": "This paper studies the fine-tuning of state-space models, in particular, S4 and S6. Empirical studies on fine-tuning the encoder and the decoder using many different existing tuning mechanisms are shown. Then, an SDT-P fine-tuning strategy of the autoregressive modules is proposed based on pruning and then sparsely fine-tuning a set of latent states. It is also empirically shown that the pruning stage can be pruned. Experiments show that the SDT method works better than the traditional LoRA strategy.\n\n## update after rebuttal\nI have raised my score to weak accept to reflect my current position.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We've addressed all your concerns below.\n\n--- \n\n> Q1: Lemma 2 Clarifications\n> * (i) Use $\\odot$ or $\\circ$ for Hadamard product?\n> * (ii) Clarify \"same permutation\"?\n> * (iii) (5) seems independent of $f_0$—is this correct?\n> * (iv) What does \"all hidden dimensions are active\" mean?\n> * (v) Is it the exact or an upper bound on tunable parameters?\n\n* (i) Fixed.\n* (ii) \"Up to the same permutation\" means that the same permutation matrix $P$ is applied to all three parameters of the initial model $f_0$. This yields a model that is functionally equivalent but with permuted hidden dimensions, giving\n    $$\\Theta_0= \\\\{(P^\\top \\overline{A}_0 P, P^\\top \\overline{B}_0, C_0 P) : P \\text{ is a permutation matrix}\\\\}.$$\n    Here, $\\overline{A}_0$, $\\overline{B}_0$, and $C_0$ are the parameters of $f_0$ before fine-tuning. \n\n* (iii) The quantity in (5) does, in fact, depend on the initial model $f_0$ through the constraint set $\\Theta_0$. To prevent confusion, we rephrased Lemma 2 to highlight this constraint set.\n\n* (iv) It means that (for target model $f_\\star$), all hidden dimensions are non-zero.\n\n* (v) When the assumption (iv) holds, it refers to the exact minimum.\n\n**`Preview`** Revised Lemma 2 with improved clarity (https://anonymous.4open.science/r/ce7/l2.png). Notations are from Sec. 5.1.\n\n> Q2: Theorem 1\n> * (i) Define $L^\\star$ and $H^\\star$—are they from the target model?\n> * (ii) Does \"selectively fine-tuning\" imply a specific algorithm?\n> * (iii) Clarify \"accurately represent.\"\n\n* (i) $L^\\star$ and $H^\\star$ are indeed the number of layers and the hidden dimension of the target model $f_\\star.$ We have updated statement for a better introduction of the target model and the corresponding notations.\n* (ii) The statement focuses solely on model expressiveness, independent of training algorithms.\n* (iii) It means \"functionally equivalent,\" i.e., the fine-tuned model $f$ satisfies $f(x) = f_\\star (x)$ for all input sequences $x$.\n\n**`Preview`** Revised version of Theorem 1 with improved clarity (https://anonymous.4open.science/r/ce7/t1.png). \n\n> Q3: Lemma 1\n> * (i)  Lemma 1 covers only expressiveness, not optimization—please comment.\n> * (ii) Lemma 1 suggests fine-tuning linear projections alone is sufficient; does this diminish your method’s significance?\n\n\n* (i) **`New Exp.`** Lemma 1 analyzes only theoretical expressiveness. Empirical results (https://anonymous.4open.science/r/ce7/f1.png) confirm fine-tuning $W_{\\text{in}}$ alone matches tuning $W_B$, $W_C$, $W_{\\Delta, \\uparrow}$ across three GLUE tasks.\n* (ii) Great point. To clarify, in Lemma 1, $A$ and $W_{\\Delta,\\downarrow}$ are fixed. While $W_{\\text{in}}$'s expressiveness includes that of $W_B$, $W_C$, and $W_{\\Delta,\\uparrow}$, it does not cover the expressiveness of $A$, which is essential for seq2seq operations. Thus, $W_{\\text{in}}$ offers a portion of expressivity for SSM, yet it alone remains insufficient to attain optimal performance.\n\n**`Preview`**  Updated Lemma 1 to emphasize fixed $A$ (https://anonymous.4open.science/r/ce7/l1.png) and will further smooth the Sec. 4–5 transition.\n\n> Q4: Be clear whether the model is S4 or S4D.\n\nWe specifically use S4D (diagonal $A$) and will clarify this in Sec. 1, 3, and 5.\n\n> Q5: Fine-tuning strategies lack mathematical details.\n\nThanks. We'll revise Sec. 3 (\"Preliminaries\") to include mathematical details on fine-tuning methods.\n\n> Q6: What is the fine-tuning time compared to LoRA?\n\n**`New Exp.`** Based on your feedback, we conducted two additional experiments: (i: https://anonymous.4open.science/r/ce7/f2.png) a performance comparison between SDT and LoRA under varying time budgets in a synthetic setting; and (ii: https://anonymous.4open.science/r/ce7/f3.png) a runtime analysis of SDT and LoRA for training on a single batch using a pretrained model, across different sequence lengths. We observe that our method is slightly more efficient than LoRA, particularly as the sequence length increases, because LoRA introduces additional matrix multiplications, while SDT does not.\n\n> Q7: How were $\\alpha$, $\\beta$ in Table 3/4 chosen, and why is it a fair comparison to LoRA?\n\n**`New Exp`** We fix $\\beta = 0.99$ and sweep $\\alpha \\in \\\\{0.75, 0.90, 0.95\\\\}$. To ensure a fair comparison with LoRA, we use similar parameter budgets and hyperparameter sets of the same size for both methods: 45 configurations per method (15 learning rates × 3 method-specific settings). For LoRA, we additionally compare the three chosen configurations against alternative settings with similar parameter budgets (https://anonymous.4open.science/r/ce7/f4.png), showing that our selected configurations perform reasonably well.\n\n---\n\n**Final Note:** In addition, we’ll summarize all notations in the appendix. Since you said raising your score would be “an easy reach” once issues were addressed, we hope our response meet that bar. We’d appreciate an updated score and are happy to clarify anything further.", "claims_and_evidence": "I find the claims about the empirical advantages of SDT well-supported by experiments.", "methods_and_evaluation_criteria": "I find the proposed methods and evaluation criteria appropriate for the problem at hand.", "theoretical_claims": "I did not check the correctness of the proofs. In fact, I am not satisfied with the presentation of some theoretical statements, which I will clarify in the comments/questions sections.", "experimental_designs_or_analyses": "I checked the soundness of the experimental designs and found them appropriate.", "relation_to_broader_scientific_literature": "This paper analyzes the fine-tuning of a promising class of large language models, which is certainly an important problem.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n   * The problem studied in this paper is important and opportune.\n   * The empirical study is comprehensive and convincing.\n\nWeaknesses:\n   * While the central ideas of the paper are clear, I do have some comments on the presentation of the material. See the section below.\n   * The theoretical statements are on the rough side and many statements need rephrasing. Please see the section below.", "other_comments_or_suggestions": "The comments here are mainly about the presentation of the material and the delivery of the theoretical statements. My current evaluation of \"weak reject\" is mainly due to the presentation issues. I can guarantee that once the author(s) put efforts into addressing the following comments, pushing up my evaluation would be an easy reach.\n\n1. Right now, the fine-tuning strategies studied and compared in this paper are only referred to without any other details provided. While some of them are popular, readers who are not experts on fine-tuning may find it hard to parse all of them, and consequently Table 1, all at once. It would be useful to add a brief and mathematical introduction to some of them. An introduction in the supplementary material would be otherwise useful.\n\n2. Lemma 2 and Theorem 1 need rephrasing to be mathematically correct. In particular, consider the following questions:\n   1. In Lemma 2, the quantity in (5) is just a number and does not depend on the initial model $f_0$. This does not seem to correctly reflect the definition of (5).\n   2. In Lemma 2, when saying \"the minimum number of tunable parameters,\" do you mean the precise minimum number (so that we know for sure that if the number goes below it, then we'll not be able to do the job) or an upper bound of the minimum number?\n   3. In Theorem 1, $L^*$ and $H^*$ are not defined. I think they come from the target model. In that case, it needs to be clearly stated what the target model is.\n   4. In Theorem 1, when saying \"selectively fine-tuning,\" what is the training algorithm being used? Or is it just an algorithm-free statement that considers the \"best\" way of modifying the tunable parameters?\n   5. In Theorem 1, what does \"accurately represent\" mean?\n\n3. Some minor comments:\n   1. Technically, an S4 model refers to a model where $\\mathbf{A}$ is the sum of a diagonal matrix plus a rank-one matrix, whereas an S4D model is the one with a diagonal $\\mathbf{A}$. The author(s) should be more precise.\n   2. Throughout the manuscript $\\otimes$ is used for entrywise product. This is a very misleading notation as it is often used for the Kronecker product. Consider using $\\odot$ or $\\circ$ for the Hadamard product instead.\n   3. On line 293-294, please be more precise about what \"up to the same permutation\" means.", "questions_for_authors": "1. Lemma 1 is about the expressiveness of different ways of doing the fine-tuning. It does not consider, for example, the stability of representations, or more naively, how easy it is to reach the target by only fine-tuning the encoder. Can the author(s) comment on this?\n2. Lemma 1 seems to diminish the significance of the proposed method, as it says that fine-tuning can be done without changing the autoregressive units. This may be related to the first point, but can the authors provide more clarification on this in the manuscript so that the transition from section 4 to 5 is smoother?\n3. On line 279, the author(s) wrote \"assume all hidden dimensions are active.\" What does that mean precisely?\n4. What is the efficiency of the proposed method compared to, for example, LoRA? For example, Figure 2 is mainly a comparison based on the same number of tunable parameters. What about a comparison based on the fine-tuning time? Also, how do these change as the sequence lengths grow?\n5. How are $\\alpha$ and $\\beta$ in Algorithm 1 selected in Table 3 and 4 and why is it a fair comparison to LoRA with whatever the hyperparameters you selected there? These are necessary to be discussed in the main manuscript.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work performs a detailed study of parameter-efficient fine-tuning (PEFT) for the recently popular family of state space models (SSM). The contributions include a systematic empirical study of existing PEFT methods, and introducing a new method tailored for SSMs that applies to several variants including S4, Mamba, and more models that the authors included during the reviewer discussion. Reviewers agreed that the problem domain of PEFT for SSMs is important and relatively new, and that the paper's contributions are well-motivated and sound.
Decision: Accept (poster)