Parameter-Efficient Fine-Tuning of State Space Models

## Abstract

Abstract Deep State Space Models (SSMs), such as Mamba (Gu & Dao, 2024 ) , have emerged as powerful tools for language modeling, offering high performance with efficient inference and linear scaling in sequence length.
However, the application of parameter-efficient fine-tuning (PEFT) methods to SSM-based models remains largely unexplored.
This paper aims to systematically study two key questions: (i) How do existing PEFT methods perform on SSM-based models? (ii) Which modules are most effective for fine-tuning?
We conduct an empirical benchmark of four basic PEFT methods on SSM-based models.
Our findings reveal that prompt-based methods (e.g., prefix-tuning) are no longer effective, an empirical result further supported by theoretical analysis.
In contrast, LoRA remains effective for SSM-based models.
We further investigate the optimal application of LoRA within these models, demonstrating both theoretically and experimentally that applying LoRA to linear projection matrices without modifying SSM modules yields the best results, as LoRA is not effective at tuning SSM modules.
To further improve performance, we introduce LoRA with Selective Dimension tuning (SDLoRA), which selectively updates certain channels and states on SSM modules while applying LoRA to linear projection matrices.
Extensive experimental results show that this approach outperforms standard LoRA.

## 1 Introduction

Over the past two years, Large Language Models (LLMs) such as ChatGPT (Achiam et al., ; Brown et al., ) have achieved groundbreaking performance and are now widely used in daily life.
Many models use the Transformer architecture (Vaswani et al., ), with its attention mechanism essential in predicting subsequent tokens based on context. Each token computes attention scores with every preceding one, selectively focusing only on the most relevant during processing. This, however, creates quadratic time complexity, posing challenges when dealing with long sequences.
In response, various alternative architectures like Linear Attention (Katharopoulos et al., ), RWKV (Peng et al., ), RetNet (Sun et al., ), and Mamba (Gu & Dao, ) have been developed to operate with subquadratic time complexity.

As the most popular subquadratic-time architecture currently serving as an alternative to Transformers, SSMs  (Gu et al., , , ; Gu & Dao, ) achieve efficient training and inference.
SSMs are closely related to linear RNNs, which maintain a hidden state to encapsulate the information of previous tokens.
When a new input token is introduced, the prediction of the next token involves only operations on this hidden state and the new token, which enhances inference efficiency. To overcome the limitation of RNNs, which cannot be trained in parallel, S4 (Gu et al., , ) leverages its linearity, enabling it to adopt a convolutional form during training, facilitating parallel computation.
Consequently, SSMs are highly efficient and have demonstrated success in numerous long-sequence tasks (Gu et al., , ).
Recently, a new series of SSM models, Mamba (Mamba-I (Gu & Dao, ) and Mamba-II (Dao & Gu, )), have achieved Transformer-level performance in language modeling.
Although numerous SSM-based models exist, we primarily focus on the deep S4 model and Mamba.
The deep S4 model, serving as the foundational architecture, readily extends its properties to other variants, while Mamba has emerged as one of the most popular SSM-based models.

Consequently, we expect fine-tuning these pretrained SSMs for downstream tasks will become a crucial problem in the near future.
While fine-tuning the entire model is expensive and inefficient, numerous Parameter-Efficient Fine-Tuning (PEFT) methods (Houlsby et al., ; Hu et al., ; He et al., ; Li & Liang, ; Lester et al., ; Zaken et al., ; Liu et al., , ) have been developed for efficient adaptation under resource constraints.
Notably, most popular PEFT methods fall into two categories:
(i) prompt-based tuning, which involves modifying the input sequence (Lester et al., ) or tuning the sequence at each layer (Li & Liang, );
and (ii) parameter-based tuning, which directly updates the model parameters, such as LoRA (Hu et al., ), which modifies the weight matrices, and BitFit (Zaken et al., ), which updates only the bias terms.

Despite the success that existing PEFT methods have achieved in adapting Transformer-based models, their efficacy in adapting SSM-based models remains largely unexplored, leaving many interesting questions open.
For instance, are existing popular PEFT methods still effective for SSM-based models?
If they are applicable, what is the optimal way to apply these methods to SSM-based models, and which parameters should be updated?
If not, can we develop variants specifically tailored for SSMs that perform better?
To answer these questions, to the best of our knowledge, we conduct the first comprehensive study of PEFT on SSM-based models, both theoretically and empirically.

To the best of our knowledge, we are the first to benchmark existing PEFT methods on SSM-based models.
Through extensive experiments, we demonstrate that (Finding 1) prompt-based PEFT methods are no longer effective for SSM-based models, and (Finding 2) LoRA remains effective on SSM-based models.
Meanwhile, the two major components of SSM-based models are the SSM module, which functions analogously to attention in Transformers, and linear projection matrices, which are similar to feed-forward layers.
We next investigate which part of the model is more effective for applying PEFT.
We empirically find that (Finding 3) applying LoRA to linear projection matrices without modifying the SSM module is already effective, while the most effective linear projection matrices differ depending on the dataset.
Notably, Findings 1 and 3 are supported by our theoretical analysis.
While LoRA is not effective for tuning SSM modules, theoretically, tuning additional SSM modules increases expressivity.
Finally, we analyze the architecture of SSM-based models using the theoretical framework of Giannou et al. () and Zeng & Lee ().
We show that, in addition to applying LoRA to linear projection matrices, Selectively updating the channel and state Dimensions of SSM modules further enhances performance.
We dub this method as SDLoRA, the first PEFT method tailored for SSM-based models.
Through extensive experiments, we observe that (Finding 4) SDLoRA outperforms LoRA alone in fine-tuning SSM-based models.

## 2 Related Works

#### State Space Models (SSMs).

Linear State-Space Layers (LSSL) represent one of the earliest SSM layers utilized in deep learning, functioning as continuous-time, recurrent, and convolutional models (Gu et al., ).
LSSL employs HiPPO theory (Gu et al., ) to initialize the state matrix ${\bm{A}}$, enabling the capture of long dependencies.
However, LSSL is computationally expensive, limiting its practical application.
Gu et al. () introduced Structured State Space Models (S4), which optimize computation efficiency by employing a structured state matrix ${\bm{A}}$.
Gupta et al. () proposed DSS, which simplifies the model by using a diagonal matrix for ${\bm{A}}$ and empirically demonstrated that it suffices to achieve performance comparable to S4.
Further, Gu et al. () provided a theoretical explanation for the effectiveness of the diagonal state matrix ${\bm{A}}$ in DSS and introduced S4D, which offers various initialization methods for ${\bm{A}}$.
Subsequently, the diagonal structure of the state matrix ${\bm{A}}$ has been adopted in follow-up methods (Gu & Dao, ).
Despite differences in optimization algorithms, we refer to S4 and its close variants, including DSS and S4D, collectively as S4. This terminology encompasses models that maintain the standard discrete-time SSM form with a diagonal state matrix.

Despite of the remarkable performance of SSMs on certain tasks of sequence modeling, SSMs still showed worse performance than Transformers on language modeling.
Fu et al. () transitioned from synthetic language modeling tasks to real language modeling tasks with SSMs. They proposed H3, which is inspired by Linear Attention (Katharopoulos et al., ), introducing both diagonal SSM and shift SSM.
Recently, Mamba (Gu & Dao, ; Dao & Gu, ) escaped from linear time invariance (LTI) modeling by introducing input-dependent terms and achieved better performance than Transformer on language modeling. Furthermore, several hybrid models (Lieber et al., ; Park et al., ) tried to exploit the advantages of both SSMs and Transformers.

#### Parameter-Efficient Fine-Tuning (PEFT).

Due to the increase in model size, PEFT methods have gained increasing popularity as they achieve good performance while being much more efficient compared to full model updating (Houlsby et al., ; Hu et al., ; He et al., ; Li & Liang, ; Lester et al., ; Zaken et al., ; Liu et al., , ).
Most of the existing popular PEFT methods fall into two categories: (i) prompt-based methods (Li & Liang, ; Lester et al., ; Liu et al., , ), and (ii) parameter tuning methods (Donahue et al., ; Yosinski et al., ; Hu et al., ; Zaken et al., ).
Common prompt-based methods include prompt tuning (Lester et al., ), and prefix-tuning (Li & Liang, ).
Prompt tuning prepends a sequence of learnable virtual tokens, which are continuous vectors.
Prefix-tuning further expands on Prompt tuning by prepending tokens across the model’s depth, making it more powerful.
Therefore, our analysis of prompt-based methods’ limitations will focus on prefix-tuning, with the findings also applicable to the other prompt-based methods.
Conversely, parameter tuning methods, which originated from traditional transfer learning practices, typically involve freezing the initial layers and only tuning the last few layers (Donahue et al., ; Yosinski et al., ).
In recent years, more effective and innovative parameter tuning approaches have emerged (Hu et al., ; Zaken et al., ).
The widely used Low-Rank Adaptation (LoRA) updates a subset of parameters (e.g., attention layers of a Transformer) in a low-rank manner.
BitFit (Zaken et al., ), focuses on tuning only the bias terms of a pretrained model.
In Sec. A, we provide a more detailed description of these baseline methods.

Numerous efforts have been made to theoretically understand existing PEFT methods.
For prompt-based methods, Wang et al. (), Petrov et al. (), and Oymak et al. () have theoretically analyzed the effectiveness and limitations of prompt tuning and prefix-tuning for Transformer-based models.
For LoRA, Zeng & Lee () explored its expressive power by demonstrating that even a randomly initialized model can be adapted to match any smaller target model using LoRA.
Some of our theoretical analysis draws upon the framework established by Zeng & Lee ().
Jang et al. () conducted a theoretical exploration of LoRA within the neural tangent kernel (NTK) regime.

## 3 Preliminaries of State Space Models

#### Scalar-input Scalar-output SSM.

The initial SSM is derived from a specific continuous system that maps a one-dimensional function or signal $x(t)\in{\mathbb{R}}$ to $y(t)\in{\mathbb{R}}$ via an $H$-dimensional latent state $\bm{h}(t)\in{\mathbb{R}}^{H}$, as described in (1).
In (1), input transition vector ${\bm{B}}\in{\mathbb{R}}^{H\times 1}$ indicates the input’s impact on the state of the system,
state matrix ${\bm{A}}\in{\mathbb{R}}^{H\times H}$ characterizes the system’s internal state dynamics,
and the output mapping vector ${\bm{C}}\in{\mathbb{R}}^{1\times H}$ relates the state to the output $y(t)$.(^1^11Note that ${\bm{B}},{\bm{C}}$ are vectors. We use bold capital letters to remain consistent with existing works (Gu et al., ; Gu & Dao, ).)
$\begin{aligned} \bm{h}^{\prime}(t)&={\bm{A}}{\bm{h}}(t)+{\bm{B}}x(t)\\
y(t)&={\bm{C}}\bm{h}(t)\end{aligned}$
(1)
$\begin{aligned} \bm{h}_{t}&=\overline{{\bm{A}}}\bm{h}_{t-1}+\overline{{\bm{B}}
}x_{t},\\
y_{t}&={\bm{C}}\bm{h}_{t}\end{aligned}$
(2)
$\begin{gathered}\overline{{\bm{K}}}=({\bm{C}}\overline{{\bm{B}}},{\bm{C}}
\overline{{\bm{A}}}^{\,}\overline{{\bm{B}}},\ldots,{\bm{C}}{\overline{{\bm{A}}
}}^{t-1}\overline{{\bm{B}}}),\\
(y_{1},\ldots,y_{t})=(x_{1},\ldots,x_{t})*\overline{{\bm{K}}}\end{gathered}$
(3)

To adapt SSMs for deep learning, the continuous parameters $({\bm{A}},{\bm{B}})$ are transformed into discrete counterparts $(\overline{{\bm{A}}},\overline{{\bm{B}}})$ using a learnable step size $\Delta\in{\mathbb{R}}$.
An example of a discretization rule is the zero-order hold, which defines $\overline{{\bm{A}}}=\exp(\Delta{\bm{A}}),\overline{{\bm{B}}}=(\Delta{\bm{A}})^
{-1}(\exp(\Delta{\bm{A}})-{\bm{I}})\cdot\Delta{\bm{B}}$.

The discrete-time SSM is formulated as (2).
For efficient and parallelizable training, the output ${\bm{y}}$ of a length-$N$ input ${\bm{x}}$ in the discrete-time SSM can be computed with a long convolution, as detailed in (3). This convolution operation can be efficiently computed in the frequency domain with FFT.

#### Vector-input Vector-output SSM.

Many deep learning tasks, such as language modeling, often use multi-channel inputs. When the input and output are vectors, denoted as $\bm{x},\bm{y}\in{\mathbb{R}}^{D}$, separate SSMs are used for each of the $D$ input channels. As such, a superscript $(d)$ is introduced to indicate parameters specific to each channel when necessary. This notation may be omitted for simplicity.

#### Structured State Space Sequence Model (S4).

S4 introduced by Gu et al. () represents one of the earliest applications of SSMs in deep learning.
It features a diagonal structure for the state matrix ${\bm{A}}$, a design theoretically validated by Gu et al. () and practically implemented through its subsequent variants, DSS (Gupta et al., ) and S4D (Gu et al., ).

#### Deep S4 Layer.

Since S4 lacks non-linearity and operates with independent channels, a position-wise linear layer and a non-linear activation function are integrated into the deep S4 layer, facilitating information mixing across channels and introducing non-linearity. Furthermore, a residual connection from the input to the output of S4 is introduced.
Let $\otimes$ represent the element-wise product, and $\operatorname{S4}(\cdot)$ denote the S4 mechanism, where the output of each channel is computed according to (3) using its own convolutional kernel $\overline{{\bm{K}}}^{(d)}$.
While the subtle details such as the activation functions may vary slightly from the previous studies (Gu et al., , ), for the theoretical analysis in this paper, we define the deep S4 layer as below.
The output of a deep S4 layer is then formulated as:

$$ $\bm{y}_{t}=\operatorname{ReLU}({\bm{W}}\cdot\operatorname{S4}_{t}(\bm{x}_{1}, \ldots,\bm{x}_{t})+{\bm{\beta}}+{\bm{u}}\otimes\bm{x}_{t}),$ (4) $$

where ${\bm{W}}\in{\mathbb{R}}^{D\times D}$ and ${\bm{\beta}}\in{\mathbb{R}}^{D}$ represent the linear projection matrix and bias, respectively, and ${\bm{u}}\in\mathbb{R}^{D}$ is the coefficient of the residual connection.
Note that in a deep S4 layer, the trainable parameters are SSM parameters $({\bm{A}}^{(d)},{\bm{B}}^{(d)},{\bm{C}}^{(d)},\Delta^{(d)})$ across $D$ channels with ${\bm{A}}^{(d)}$ being diagonal
and the parameters $({\bm{W}},{\bm{\beta}})$ for the linear layer and ${\bm{u}}$ for the residual connection.

#### Selective State Space Models (S6).

A key property of all SSMs mentioned above is linear time invariance (LTI), where model dynamics remain constant over time.
However, LTI models face significant limitations: their constant dynamics fail to selectively extract relevant information from the context or influence the hidden state in an input-dependent manner.
The S6 model, proposed by Gu & Dao (), addresses these limitations by making its parameters input-dependent.

In particular, at each time step $t$, given the input $\bm{x}_{t}\in{\mathbb{R}}^{D}$, they introduce input-dependency to step size $\bm{\Delta}_{t}=(\Delta^{(1)}_{t},\ldots,\Delta^{(D)}_{t})^{\top}\in{\mathbb{R
}}^{D}$, input transition vectors ${\bm{B}}_{t}\in{\mathbb{R}}^{H\times 1}$ and the output mapping vectors ${\bm{C}}_{t}\in{\mathbb{R}}^{1\times H}$ via linear projection:

$$ $\bm{\Delta}_{t}=\operatorname{softplus}(\bm{W}_{\bm{\Delta}}\bm{x}_{t}+{\bm{ \beta}}_{\bm{\Delta}}),\quad{\bm{B}}_{t}={\bm{W}}_{{\bm{B}}}\bm{x}_{t},\quad{ \bm{C}}_{t}={\bm{W}}_{{\bm{C}}}\bm{x}_{t},$ (5) $$

whereas the diagonal state matrices ${\bm{A}}^{(1)},\ldots,{\bm{A}}^{(D)}$ remain input-independent.
Note that $\bm{W}_{\bm{\Delta}}\in{\mathbb{R}}^{D\times D}$ is implemented via a rank-$r$ low-rank parameterization, denoted by $\bm{W}_{\bm{\Delta}}={\bm{W}}_{\bm{\Delta},\uparrow}{\bm{W}}_{\bm{\Delta},\downarrow}$, where ${\bm{W}}_{\bm{\Delta},\uparrow}\in{\mathbb{R}}^{D\times r}$ and ${\bm{W}}_{\bm{\Delta},\downarrow}\in{\mathbb{R}}^{r\times D}$, which is a common method for reducing compute overheads (Wang et al., , ).
To summarize, the trainable parameters in S6 include state matrices ${\bm{A}}^{(d)}$ across $D$ channels, parameters ${\bm{W}}_{\bm{\Delta},\uparrow},{\bm{W}}_{\bm{\Delta},\downarrow}$ and ${\bm{\beta}}_{\bm{\Delta}}$ for computing $\bm{\Delta}_{t}$, and weight matrices ${\bm{W}}_{{\bm{B}}},{\bm{W}}_{{\bm{C}}}\in{\mathbb{R}}^{H\times D}$ for computing ${\bm{B}}_{t},{\bm{C}}_{t}$.
The state matrices and the input transition vectors of S6 are then discretized according to $\overline{{\bm{A}}}_{t}^{(d)}=\exp(\Delta_{t}^{(d)}{\bm{A}}^{(d)}),\overline{{
\bm{B}}}_{t}^{(d)}=\Delta_{t}^{(d)}{\bm{B}}_{t}.$
In contrast to S4, where $\overline{{\bm{B}}}^{(d)}$ varies independently across channels, the differences in $\overline{{\bm{B}}}^{(d)}$ in S6 are solely due to the scalar $\Delta_{t}^{(d)}$.
Additionally, S6 uses the same ${\bm{C}}_{t}$ for all channels at each time step $t$, unlike S4, which has unique ${\bm{C}}^{(d)}$ for each channel.

#### Mamba.

Similar to the Transformer block, which consists of attention and linear layers, the Mamba block proposed by Gu & Dao () features an S6 module, a point-wise 1D causal convolution layer (Conv1d) for token mixing, linear layers — including input ($\bm{W}_{\text{in}}$) and output ($\bm{W}_{\text{out}}$) projection layers and a gated MLP.
Mamba, primarily allocating its parameters in $\bm{W}_{\text{in}}$ and $\bm{W}_{\text{out}}$, is inspired by H3 (Fu et al., ).

## 4 Benchmarking PEFT Methods on SSM-based Models

In this section, we examine the effectiveness of popular PEFT methods when applied naively to SSM-based models, specifically Mamba (130M and 1.4B).

#### Experiment Setup.

We consider two main categories of PEFT methods: parameter-based and prompt-based.
From each category, we evaluate two representative methods.
For parameter-based methods, we select BitFit (Zaken et al., ) and LoRA (Hu et al., ).
For prompt-based methods, we choose prefix-tuning (Li & Liang, ) and prompt tuning (Lester et al., ). For BitFit, fine-tuning is performed on all bias terms present in the Mamba architecture, specifically the biases of the Conv1d and the linear projection layer of step size $\bm{\Delta}$.
For prefix-tuning, we adopted the huggingface implementation (Mangrulkar et al., ) to construct a MLP, employing the overparameterization technique to ensure stable optimization.

We consider five datasets spanning diverse domains: the GLUE natural language understanding benchmark (Wang et al., ), the DART RDF-to-text generation benchmark (Nan et al., ), the Spider text-to-SQL generation benchmark (Yu et al., ), and CIFAR-10 for computer vision tasks (Krizhevsky et al., ).
A more detailed introduction of the datasets considered in this paper is provided in Sec. B.
Notably, prefix-tuning requires substantially more parameters than other PEFT methods, as it employs a multilayer perceptron at each layer to project a fixed sequence into soft tokens for training stability.
For all other PEFT methods, we constrain the trainable parameters to fewer than 0.5% for language tasks and 1% for vision tasks, ensuring a fair comparison.
The higher allowance for vision tasks accommodates the need for extensive fine-tuning for new modalities.
Consequently, LoRA adapters are applied exclusively to linear projection matrices, leaving the SSM modules unchanged to comply with these parameter constraints.

#### Results.

Table 1 presents our results.
Parameter-based PEFT methods generally outperform prompt-based methods significantly, despite using the same number of trainable parameters—except for prefix-tuning, which underperforms despite using more parameters.
LoRA consistently achieves the best performance across all tasks and metrics, occasionally surpassing full fine-tuning while tuning less than 1% of parameters.
We provide the full experimental results, including detailed results of subtasks on GLUE and Spider, in Sec. C.2.

These findings above raise two critical questions: (i) Why do existing prompt-based PEFT methods lose effectiveness when applied to SSM-based models? (ii) Can LoRA achieve better performance when applying on both linear projection matrices and SSM modules?
To address these questions, we conduct both theoretical analysis and further empirical studies on prompt-based PEFT methods and LoRA in the context of SSMs.

**Table 1: Benchmarking popular Parameter-Efficient Fine-Tuning (PEFT) methods on five real-world datasets. R1/R2/RL stand for ROUGE-1/2/L. For all PEFT methods except prefix-tuning, we report the best results for cases where fewer than 0.5% of parameters are tunable for language tasks and fewer than 1% for vision tasks (i.e., CIFAR-10) after comprehensive hyperparameter search. Prefix-tuning is an exception, as it requires training a multilayer perceptron at each layer to project a fixed sequence into soft tokens for training stability, consuming more trainable parameters than our threshold. Bold numbers indicate outperformance over all PEFT methods, while underlined numbers indicate outperformance over full fine-tuning.**
| Dataset | GLUE | DART | SAMSum | Spider | CIFAR-10 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric ($\uparrow$) | Avg. Score | METEOR | BLEU | R1 | R2 | RL | Acc. | Acc. |
| Prompt Tuning | 63.8 | 66.2 | 39.8 | 50.1 | 25.6 | 41.6 | 43.6 | 30.4 |
| Prefix-Tuning | 68.6 | 66.6 | 42.5 | 50.6 | 26.5 | 42.1 | 39.7 | 41.0 |
| BitFit | 76.8 | 67.0 | 43.7 | 50.3 | 25.7 | 41.9 | 48.4 | 44.4 |
| LoRA (Linear Projection Matrices) | 80.5 | 70.4 | 49.1 | 50.9 | 27.0 | 42.3 | 57.5 | 61.1 |
| Full Fine-Tuning | 80.5 | 71.0 | 51.8 | 51.2 | 27.3 | 42.9 | 66.2 | 60.0 |

### 4.1 Limitations of Applying Existing Prompt-based Methods on SSMs

This part addresses our first question arised in this section: Why do existing prompt-based PEFT methods lose effectiveness when applied to SSM-based models?
We approach this by establishing an upper bound on the performance of existing prompt-based PEFT methods.

A key feature of SSMs is their next-token prediction mechanism, which relies solely on the current token and hidden states, without considering previous tokens directly.
The hidden states encapsulate all information from preceding tokens.
Consequently, prepending tokens to an SSM is functionally equivalent to tuning the initial state, as demonstrated by the following proposition.
The formal version and proof of Proposition 1 are presented in Section C.3.

###### Proposition 1 (Informal: Expressivity of Prefix-Tuning on SSMs) .

The maximum expressiveness achievable via prefix-tuning on SSMs is equivalent to the expressiveness of solely tuning the initial hidden state $\bm{h}_{0}$.

To evaluate the performance of initial state tuning, we conducted experiments on the GLUE benchmark, comparing prompt-tuning, prefix-tuning, initial state tuning, and LoRA across seven GLUE tasks.
Table 2 presents our findings.
The results demonstrate that initial state tuning generally outperforms prefix-tuning, corroborating our analysis.
However, LoRA significantly surpasses initial state tuning in performance.
These observations lead us to conclude that the limitations of initial state tuning inherently constrain the potential of existing prompt-based methods, preventing them from outperforming LoRA in the context of SSM-based models.
While the reason for the underperformance of initial state tuning is unclear, we identify explaining it as an interesting direction for future research.
Nevertheless, we propose a plausible explanation.
We hypothesize that SSM’s exclusive reliance on hidden states, without direct access to previous tokens or states, severely restricts the impact of initial state tuning, particularly for long sequences.
This aligns with the findings of Fu et al. (), which demonstrate SSM’s limitations in recalling older tokens.

**Table 2: Comparison of prompt-tuning, prefix-tuning, initial state tuning, and LoRA on seven tasks from the GLUE benchmark. We report the Matthews correlation ($\uparrow$) for CoLA, overall (matched and mismatched) accuracy $(\uparrow)$ for MNLI, and accuracy for other tasks. Initial State Tuning and LoRA are constrained to use less than 0.5% trainable parameters. Bold numbers indicate the best performance across all three methods, while underlined numbers show the highest score among prompt-based methods (prefix-tuning and initial state tuning). Initial state tuning outperforms prefix-tuning and prompt-tuning on five out of seven tasks, while LoRA consistently outperforms all prompt-based methods.**
| Task | RTE | MRPC | CoLA | SST-2 | QNLI | QQP | MNLI | Avg. Score |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Prompt Tuning | .560 | .716 | .120 | .894 | .768 | .796 | .615 | .638 |
| Prefix-Tuning | .695 | .757 | .434 | .915 | .834 | .831 | .356 | .686 |
| Initial State Tuning | .668 | .751 | .524 | .924 | .864 | .861 | .785 | .768 |
| LoRA (Linear Projection Matrices) | .704 | .828 | .606 | .924 | .884 | .877 | .815 | .805 |

###### Proposition 1 (Informal: Expressivity of Prefix-Tuning on SSMs) .

### 4.2 Optimal Application of LoRA in SSM-based Models

In our previous experiments, we applied LoRA exclusively to linear projection matrices.
However, SSM-based models typically comprise various modules, including S4 (convolution layer), S6, and multiple distinct linear projection matrices.
To investigate the impact of applying LoRA to different components, we conduct a comprehensive study across five datasets.

**Table 3: For LoRA, targeting only the linear projection matrices yields better performance than applying it to all modules in Mamba. Consistent rank is maintained across all three methods.**
| Model | Mamba-130M | Mamba-1.4B |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Params. (%) | GLUE | DART | CIFAR-10 | Params. (%) | SAMSum | Spider |  |  |  |
| Metric ($\uparrow$) | Avg. Score | METEOR | BLEU | Acc. | R1 | R2 | RL | Acc. |  |  |
| SSM Modules | .92 | 79.3 | 69.9 | 50.8 | 44.0 | .46 | 50.5 | 26.4 | 42.2 | 56.3 |
| Linear Projection Matrices | 1.02 | 80.5 | 71.2 | 49.2 | 62.8 | .51 | 50.8 | 26.9 | 42.8 | 54.7 |
| Both | 1.92 | 80.2 | 71.0 | 49.5 | 60.4 | .97 | 50.8 | 26.6 | 42.7 | 56.4 |

We examine LoRA’s performance when applied to SSM modules and linear projection matrices separately, as well as in combination.
For linear projections, we test LoRA on all possible matrices.
For SSM modules, we apply LoRA to all weight matrices (e.g., weight matrices of input-dependent step size $\bm{\Delta}$) in SSM modules.
For the state transition matrices ${\bm{A}}$, given their diagonal structure for each channel, we treat them as vectors, concatenate the channels into a matrix, and apply LoRA.
The results are presented in Table 3.
We observe that applying LoRA to linear projection matrices achieves superior performance on six out of eight metrics.
Interestingly, additional tuning of SSM modules lead to decreased performance in some cases.
This suggests that LoRA might not be well-suited for tuning SSM modules, while being highly effective for linear projection matrices.

To further elucidate this concept, we present the following lemma, which examines a simplified model architecture consisting of S6 with a linear input projection matrix at each layer.
We demonstrate that fine-tuning the projection matrix $\bm{W}_{\text{in}}$ encompasses the expressivity of fine-tuning the parameters $\bm{W}_{{\bm{B}}}$, $\bm{W}_{{\bm{C}}}$, and ${\bm{W}}_{\bm{\Delta},\uparrow}$.

###### Lemma 2 (Expressivity of Fine-Tuning Projection Matrices) .

Consider an S6 with an additional linear input projection matrix $\bm{W}_{\text{in}}$.
Denote the input-dependent SSM parameters $\{\{\overline{{\bm{A}}}^{(d)}_{n}\}_{d=1}^{D},\overline{{\bm{B}}}_{n},{\bm{C}}
_{n}\}_{n=1}^{N}$ as
${\bm{\theta}}(\cdot;\{{\bm{A}}^{(d)}\}_{d=1}^{D},\bm{W}_{{\bm{B}}},\bm{W}_{{
\bm{C}}},{\bm{W}}_{\bm{\Delta},\uparrow},{\bm{W}}_{\bm{\Delta},\downarrow},\bm
{W}_{\text{in}}).$
For any given $\overline{\bm{W}}_{{\bm{B}}}$, $\overline{\bm{W}}_{{\bm{C}}}$, and $\overline{\bm{W}}_{\Delta,\uparrow}$, there exists a $\widehat{\bm{W}_{\text{in}}}$ such that for any input sequences ${\bm{X}}\in{\mathbb{R}}^{D\times N}$,

$$ $\displaystyle{\bm{\theta}}({\bm{X}};\{{\bm{A}}^{(d)}\}_{d=1}^{D},\overline{\bm {W}}_{{\bm{B}}},\overline{\bm{W}}_{{\bm{C}}},\overline{\bm{W}}_{\Delta, \uparrow},{\bm{W}}_{\bm{\Delta},\downarrow})={\bm{\theta}}({\bm{X}};\{{\bm{A}} ^{(d)}\}_{d=1}^{D},\bm{W}_{{\bm{B}}},\bm{W}_{{\bm{C}}},{\bm{W}}_{\bm{\Delta}, \uparrow},{\bm{W}}_{\bm{\Delta},\downarrow},\widehat{{\bm{W}}}_{\text{in}}).$ (6) $$

We expand upon this section in Sec. C.4, where we provide more detailed statements of the above assertion and its corresponding proofs.
Additionally, we empirically examine applying LoRA to different weight matrices of Mamba, which incorporates multiple linear projection matrices in each layer, including output projection matrices $\bm{W}_{\text{out}}$ after the S6 module and input projection matrices $\bm{W}_{\text{in}}$ before the gating and convolutional layer.
Our experiment results, however, reveal that applying LoRA to different matrices achieves similar performance, as detailed in Sec. C.4.

###### Lemma 2 (Expressivity of Fine-Tuning Projection Matrices) .

## 5 Dimension Selection for Tuning State-Space Models

In Sec. 4.2, we demonstrate the efficacy of LoRA in fine-tuning linear projection matrices.
Theoretically, fine-tuning all components should offer greater expressive power. However, Table 3 indicates that applying LoRA to SSM modules might paradoxically decrease performance. Therefore, we aim to develop an algorithm specifically tailored for tuning SSM modules. To achieve this, we first seek to understand the relative importance of different parameters within SSM modules.

### 5.1 Understanding the Roles of State Matrix A , Input Transition Vector B , and Output Mapping Vector C for a Single Channel in S4 Modules

#### Problem Setting.

Inspired by Zeng & Lee ()’s theoretical analysis of LoRA’s expressive power, we adopt a similar framework to explore the expressive potential of various parameters in the S4 model.
Specifically, we assume a target model that performs well on the intended task and a frozen model, which may be either pretrained or randomly initialized.
Our goal is to identify a parameter-efficient method to update the frozen model so that it becomes functionally equivalent to the target model.
In alignment with Zeng & Lee (), we assume that the frozen model’s capacity is equal to or exceeds that of the target model.
This assumption is based on two main considerations:
(i) analytical tractability, which necessitates that the frozen model must have the potential to match the functionality of the target model,
and (ii) a practical rationale, given that the models typically used in practice are often overparameterized.
Assume that both the target model and the frozen model are S4, with the target model having a hidden state dimension $H_{\star}$ and the frozen model having a hidden state dimension $H\geq H_{\star}$.
Meanwhile, suppose that all the hidden dimensions of both models are valid, meaning that none of the parameter elements are zero.
The target model, frozen model, and the updated model after tuning the parameters on the frozen model can be formulated using discretized parameters $\overline{{\bm{A}}},\overline{{\bm{B}}},{\bm{C}}$ as follows:

$$ (Target model) $\displaystyle f^{\star}({\bm{x}})_{n}=\sum\nolimits_{m=1}^{n}{\bm{C}}_{\star} \overline{{\bm{A}}}_{\star}^{m-n}\overline{{\bm{B}}}_{\star}x_{m},\text{ where }\operatorname{diag}(\overline{{\bm{A}}}_{\star}),\overline{{\bm{B}}}_{\star} ,{\bm{C}}_{\star}\in{\mathbb{R}}^{H_{\star}},$ (7) (Frozen model) $\displaystyle f_{0}({\bm{x}})_{n}=\sum\nolimits_{m=1}^{n}{\bm{C}}\overline{{ \bm{A}}}^{m-n}\overline{{\bm{B}}}x_{m},\text{ where }\operatorname{diag}( \overline{{\bm{A}}}),\overline{{\bm{B}}},{\bm{C}}\in{\mathbb{R}}^{H},$ (8) (Updated model) $\displaystyle\hat{f}({\bm{x}})_{n}=\sum\nolimits_{m=1}^{n}\widehat{{\bm{C}}} \widehat{\overline{{\bm{A}}}}^{m-n}\widehat{\overline{{\bm{B}}}}x_{m},\text{ where }\operatorname{diag}(\widehat{\overline{{\bm{A}}}}),\widehat{\overline{{ \bm{B}}}},\widehat{{\bm{C}}}\in{\mathbb{R}}^{H}.\vspace{-.1in}$ (9) $$

#### Parameter Efficiency Analysis on S4.

Let ${\mathcal{P}}^{H}$ denote the set of all $H\times H$ permutation matrices.
Given this formulation, we present our first analysis of parameter efficiency for the S4 model in the following lemma.
This analysis is based on the parameters after necessary discretization $(\overline{{\bm{A}}},\overline{{\bm{B}}},{\bm{C}})$.

###### Lemma 3 (Essential Discretized Parameter Set for S4) .

Consider the parameters after discretization, i.e., $\overline{{\bm{A}}},\overline{{\bm{B}}},{\bm{C}}$.
To achieve functional equivalence between the updated model and the target model, i.e., $\hat{f}\equiv f^{\star}$, it is sufficient to tune the following number of parameters:

$$ $\min_{{\bm{P}}\in{\mathcal{P}}^{H}}\overbrace{\left\lVert\left[{\bm{P}}^{\top} (\operatorname{diag}(\overline{{\bm{A}}})\otimes\overline{{\bm{B}}}\otimes{\bm {C}}^{\top})\right]_{(H_{\star}+1):H}\right\rVert_{0}}^{\text{eliminating redundant dimensions}}+\overbrace{\underbrace{\left\lVert\left[{\bm{P}}^{\top} \overline{{\bm{A}}}{\bm{P}}\right]_{1:H_{\star},1:H_{\star}}-\overline{{\bm{A} }}_{\star}\right\rVert_{0}}_{\text{aligning the state matrix}}+\underbrace{ \left\lVert\left[{\bm{P}}^{\top}(\overline{{\bm{B}}}\otimes{\bm{C}}^{\top}) \right]_{1:H_{\star}}-\overline{{\bm{B}}}_{\star}\otimes{\bm{C}}_{\star}^{\top }\right\rVert_{0}}_{\text{aligning input-output interactions}}}^{\text{ aligning used dimensions with target model}}.$ (10) $$

This lemma highlights the significance of identifying essential hidden state dimensions.
The term $\left\lVert\left[{\bm{P}}^{\top}(\operatorname{diag}(\overline{{\bm{A}}})
\otimes\overline{{\bm{B}}}\otimes{\bm{C}}^{\top})\right]_{(H_{\star}+1):H}
\right\rVert_{0}$ underscores the importance of excluding redundant dimensions.
This can be achieved by either directly removing these dimensions from the state matrix $\overline{{\bm{A}}}$, or by updating $\overline{{\bm{B}}}$ or ${\bm{C}}$ to ensure that only the selected hidden state dimensions are utilized during the input transition or output mapping phases.
Once redundant dimensions are filtered out, tuning only the essential dimensions is sufficient to align the updated model with the target model.

###### Lemma 3 (Essential Discretized Parameter Set for S4) .

### 5.2 SSM Dimension Selection Algorithm

Inspired by Lemma 3, we introduce the Dimension Selection algorithm to construct adapters on SSMs for fine-tuning.
This algorithm first selects unimportant dimensions and sets them to zero, filtering out irrelevant information based on Lemma 3.
For enhanced parameter efficiency, we then update only the most important channels and state dimensions within these selected subsets.
Regardless of other selections, we consistently tune the coefficients of residual connections and biases in linear projections, as these components contain a negligible number of parameters.
However, we will later demonstrate that in practice, tuning residual connections and biases is unnecessary.
The detailed pseudo-code is presented in Alg. 1.
Given that tuning ${\bm{C}}$ alone is as effective as tuning both ${\bm{B}}$ and ${\bm{C}}$ for S4 (Gupta et al., ), subsequent discussions on S4 will focus solely on ${\bm{C}}$, excluding $\{{\bm{B}}^{(d)}\}_{d=1}^{D}$ for simplicity, without loss of generality.

Figure: Algorithm 1 Dimension Selection Algorithm for S4

We refer to our method as SDLoRA.
This approach extends beyond applying LoRA to linear projection matrices by Selectively updating certain subset of channels and states Dimensions, which are chosen by Alg. 1.
Our analysis considers cases where each input token $x_{t}\in{\mathcal{X}}$, with ${\mathcal{X}}\in{\mathbb{R}}^{D}$ bounded, and the input sequence length is finite.
The following theorem elucidates the expressive capacity of SDLoRA on deep S4 models.
For proof and additional details, refer to Sec. D.2.

###### Theorem 4 (Expressive Power of SDLoRA on Deep S4 Models) .

Consider a $D$-dimensional input sequence.
Assume that the linear layers in the model have linear activation functions.
Using SDLoRA, any deep S4 model with $H$ hidden states per channel and $L$ layers can be updated to accurately present any target deep S4 model without residual connections, having a reduced hidden state dimension $H^{\star}<H$, and fewer layers $L^{\star}<L$.
This can be achieved by selectively fine-tuning at most $\lceil DL^{\star}/L\rceil$ channels, $H^{\star}$ hidden states on SSM modules, applying rank-$\lceil\frac{L}{L^{\star}}\rceil$ updates on linear projection matrices and updating residual connections and biases at each layer, while additionally fully fine-tuning the linear projection matrix of the last layer only.

This theorem demonstrates that a larger pretrained model requires selecting fewer channels and hidden states at each layer.
Furthermore, if the target task is less complex — evidenced by a smaller target model with fewer layers $L^{\star}$ and hidden states $H^{\star}$ — the number of channels and hidden states needed is also reduced.
This finding aligns with the theoretical analysis of LoRA presented in Zeng & Lee (), which shows that larger pretrained models require fewer learnable parameters (referred to as “lower rank” in their context) during fine-tuning, especially for simpler tasks.
Although this theorem is constrained by the assumptions of linear activations and the absence of residual connections in the target model, while also requiring fully fine-tuning the linear project matrix of last layer, our findings have broader implications.
Our following experimental results suggest that these findings generalize beyond these restrictions.

###### Theorem 4 (Expressive Power of SDLoRA on Deep S4 Models) .

### 5.3 Empirical Evaluation on Deep S4 Models

In this experiment, we seek to validate the theoretical guarantees for SDLoRA under more general conditions, including residual connections in the target model and ReLU activations in both frozen model and target model, without full fine-tuning the linear projection matrix of the last layer.
Additionally, we assess SDLoRA’s empirical performance on both synthetic and real datasets.
More experiment setup details are provided in Sec. D.3.

Figure: Figure 1: Approximation error of PEFT methods on deep S4 models for synthetic experiments.

| Method | # Params (%) | Accuracy |
| --- | --- | --- |
| Frozen | 0.00 | 73.9 |
| LoRA (Proj) | 16.00 | 77.6 |
| LoRA (S4+Proj) | 15.52 | 77.6 |
| SDLoRA | 11.17 | 78.0 |
| Full Fine-Tuning | 100.00 | 77.6 |

Table: Table 4: Accuracy comparison between SDLoRA and LoRA on deep S4 models for CIFAR-10 (Krizhevsky et al., ).

#### Synthetic Dataset.

For the synthetic dataset, we employ a regression setting to validate our theoretical results.
(Experiment Setting) We randomly initialize two models: a one-layer deep S4 model as the target and a four-layer deep S4 model as the frozen model.
The task is to update the frozen model to match the functionality of the target model.
We generate an input sequence ${\bm{X}}$ of length $200$ and dimension $64$, with values uniformly drawn from integers between 0 and 9.
This input is then processed through the target model to obtain the corresponding outputs.
These input-output pairs are used to train the frozen model over 500 iterations using the Mean Squared Error (MSE) loss.

(Results)
Figure 1 displays the MSE, averaged across all tokens, plotted against the trainable parameters of different methods.
We observe that by using only $\approx 28\%$ of the total parameters of the frozen S4 model, SDLoRA closely approximates the performance of the target S4 model, achieving results comparable to full fine-tuning, thereby substantiating our theorem.
Meanwhile, we observe that SDLoRA outperforms both the approach of applying LoRA solely to linear projection matrices and the approach of applying LoRA to both the S4 module and linear projection matrices.
In this latter approach, the diagonal vectors of state matrices ${\bm{A}}^{(d)}$, input transition vectors ${\bm{B}}^{(d)}$ and output mapping vectors ${\bm{C}}^{(d)}$ are naively concatenated across $D$ channels into three $D\times H$ matrices before low-rank updates are applied.
In Sec. D.3, we also evaluate an extension of SDLoRA that performs sparse tuning on the linear projection matrices by updating only the columns corresponding to the channels selected by Alg.1, instead of applying LoRA.
This extension shows promising results.

#### CIFAR-10.

We also conduct experiments on the CIFAR-10 dataset (Krizhevsky et al., ).
We employ an eight-layer deep S4 model with a hidden state dimension of 16 and a model dimension of 64.
Since pretrained deep S4 models are not available, we simulate a pretrained scenario by fully updating the model for 50 epochs first, then subsequently evaluating the PEFT methods over an additional 5 epochs.
The results, as reported in Table 4, indicate that SDLoRA outperforms LoRA with fewer trainable parameters.

### 5.4 Empirical Evaluation on Mamba

Lastly, we conduct experiments on pretrained Mamba models. We consider four datasets, using Mamba-130M for GLUE and DART, and Mamba-1.4B for SAMSum and Spider.
We evaluate three configurations each for LoRA and SDLoRA, applying LoRA to distinct parameter subsets and varying SDLoRA’s state freeze ratios while maintaining a 99% channel freeze ratio.
In this experiment, we allow channels and states to learn directly from the datasets without manually setting any to zero.
We then select a LoRA-rank such that all configurations have a similar number of trainable parameters for a fair comparison.
Residual connections and biases are frozen in this experiment.
All reported values represent averages across three simulations, with learning rates independently selected for each simulation.
For more details, please see Sec. D.4.
The experimental results are reported in Table 5.
The results demonstrate that SDLoRA outperforms LoRA alone for fine-tuning the SSM even when 99% of the channels are frozen.
This result underscores the efficacy of SDLoRA.

**Table 5: Performance comparison between SDLoRA and LoRA on pretrained Mamba models. Bold numbers indicate the best performance for each task. Underlined numbers indicate that the model outperforms all models fine-tuned via the alternative method for the same task (e.g., SDLoRA outperforms all LoRA methods, or vice versa). On Mamba-130M, we compare the performance of SDLoRA and LoRA on GLUE (Wang et al., ) and DART (Nan et al., ) benchmarks. On Mamba-1.4B, we compare performance of SDLoRA and LoRA on SAMSum (Gliwa et al., ) and Spider (Yu et al., ) benchmarks. R1, R2, and RL represent ROUGE-1, ROUGE-2, and ROUGE-L, respectively.**
| Model | Mamba-130M | Mamba-1.4B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Params<br>(%) | GLUE | DART | Params<br>(%) | SAMSum | Spider |  |  |  |  |  |  |  |  |  |  |
| Avg. Score ($\uparrow$) | BLEU ($\uparrow$) | METEOR ($\uparrow$) | R1 ($\uparrow$) | R2 ($\uparrow$) | RL ($\uparrow$) | Acc. ($\uparrow$) |  |  |  |  |  |  |  |  |  |  |
| Val | Test | Val | Test | Val | Test | Val | Test | Val | Test | Val | Test | Val | Test |  |  |  |
| LoRA | .3178 | 80.71 | 78.74 | 50.44 | 41.27 | 70.00 | 65.84 | .1594 | 51.59 | 50.56 | 27.66 | 26.49 | 42.87 | 42.22 | 82.08 | 61.19 |
| .3600 | 80.79 | 79.39 | 51.03 | 42.02 | 70.16 | 66.18 | .1810 | 51.61 | 51.03 | 28.15 | 26.81 | 43.18 | 42.36 | 83.52 | 62.64 |  |
| .3883 | 80.39 | 79.49 | 50.70 | 41.55 | 69.83 | 65.98 | .1947 | 51.48 | 50.90 | 27.90 | 26.63 | 43.26 | 42.41 | 82.98 | 59.25 |  |
| SDLoRA | .3492 | 80.93 | 79.75 | 51.45 | 42.37 | 70.45 | 66.60 | .1760 | 51.63 | 50.90 | 27.97 | 26.86 | 43.32 | 42.52 | 84.36 | 62.57 |
| .3498 | 81.05 | 79.16 | 51.47 | 43.85 | 70.46 | 66.38 | .1761 | 51.61 | 50.76 | 28.02 | 26.65 | 43.38 | 42.29 | 84.48 | 59.96 |  |
| .3509 | 80.67 | 78.73 | 51.54 | 42.56 | 70.45 | 66.45 | .1764 | 51.74 | 50.86 | 28.08 | 26.54 | 43.39 | 42.19 | 84.19 | 61.25 |  |

## 6 Conclusion & Discussion

In this paper, we present the first study on the performance of PEFT methods applied to SSM-based models.
Our evaluation of existing PEFT methods provides valuable insights and guidelines for future researchers to parameter-efficiently fine-tune SSM-based models to other domains.
Moreover, we take a first step in establishing a theoretical framework for studying PEFT methods on SSM-based models.
Furthermore, we introduce SDLoRA, the first PEFT method specifically tailored for SSM-based models, which outperforms existing methods.

While our work offers numerous valuable insights, some limitations exist.
Theoretically, our guarantees for SDLoRA are limited to linear activations and require full fine-tuning of the last layer.
However, our experiments demonstrate that SDLoRA does not suffer from these limitations in practice.
Removing such restrictions for SDLoRA in theory or developing new PEFT methods under more general theoretical cases is an interesting future direction.
Additionally, our theory only demonstrates that updating a subset of channels and states is sufficient, without providing guidance on optimal selection.
Our channel and state selection, based on a warmup stage and parameter magnitude, may not be optimal.
Further investigation into the impact of channel/state selection and development of improved dimension selection algorithms presents an interesting avenue for future work.
Lastly, our work primarily focuses on SSM-based models. Studying PEFT methods on SSM-Transformer hybrid models (Lieber et al., ; Park et al., ), is an interesting future direction.

## Acknowledgement

The work of Kangwook Lee is supported in part by NSF CAREER Award CCF-2339978, Amazon Research Award, and a grant from FuriosaAI.

## Reproducibility Statement

The code for all experiments reported in this paper is publicly accessible. For the purpose of reproducibility, the code can be found at the following GitHub repository: [https://github.com/furiosa-ai/ssm-peft](https://github.com/furiosa-ai/ssm-peft).

## Appendix A In-depth Introduction of Baselines

In this section, we provide a more detailed description of the baseline methods.

#### LoRA (Hu et al., 2021 ) .

LoRA (Low-Rank Adaptation) focuses on fine-tuning large models by freezing most of the pretrained parameters and injecting trainable low-rank matrices into each layer of the Transformer’s architecture. The intuition behind using low-rank matrices comes from linear algebra, where a large matrix can be closely approximated by the product of two smaller matrices. The number of trainable parameters can be controlled with the rank of the low-rank matrices. LoRA also uses a scaling parameter (LoRA alpha) for the weight matrices to control the balance of the original model weights and LoRA weights during training. After fine-tuning, LoRA weights can be merged with the original model weights, introducing no additional inference overhead.

#### Prompt Tuning (Lester et al., 2021 ) .

Prompt tuning freezes all model weights and prepends a trainable soft prompt to the input prompt. The soft prompt consists of trainable virtual tokens, which are continuous. At inference time, prompt tuning introduces an inference overhead based on the number of virtual tokens used.

#### Prefix-Tuning (Li & Liang, 2021 ) .

Prefix-tuning also prepends trainable tokens to the input like prompt tuning but injects separate prefixes in every layer. For each Transformer layer, prefix-tuning prepends trainable embeddings to the attention’s ${\bm{K}}$ and ${\bm{V}}$ matrix. The authors have found that directly training these prefixes can lead to unstable training, so they propose to over-parameterize them with a large MLP to increase training stability. After training, the MLP can be dropped. Like prompt tuning, prefix-tuning introduces an inference overhead, scaling linearly with the number of trainable embeddings.

#### BitFit (Zaken et al., 2022 ) .

BitFit is a simple but effective PEFT method that freezes all model weights except the bias terms, consequently greatly reducing the number of trainable parameters. As no additional parameters are added, no inference overhead occurs.

## Appendix B Details of Datasets

In this paper, we consider five datasets across three domains: (i) Natural Language Understanding (NLU), represented by GLUE (Wang et al., ); (ii) Natural Language Generation (NLG), including SAMSum (Gliwa et al., ), Spider (Yu et al., ) and DART (Nan et al., ); and (iii) Computer Vision (CV), represented by CIFAR-10 (Krizhevsky et al., ).

#### GLUE (Wang et al., 2019 ) .

The GLUE (General Language Understanding Evaluation) benchmark is a collection of datasets used for training, evaluating, and analyzing natural language understanding models across a range of diverse tasks. The benchmark includes nine sentence- or sentence-pair language understanding tasks that require various features of understanding, such as sentiment analysis, linguistic acceptability, semantic textual similarity, and question answering. We use seven datasets from the GLUE benchmark (RTE, MRPC, CoLA, SST-2, QNLI, QQP, MNLI) where the model has to choose between two or three (for MNLI) different choices for the respective task. Except for CoLA, we evaluate all used datasets with the accuracy metric. For CoLA, Matthews correlation is employed.

#### SAMSum (Gliwa et al., 2019 ) .

SAMSum is a dataset for dialogue summarization research, comprising approximately 16,000 synthetic text conversations with accompanying summaries. Created by English-fluent linguists, these exchanges simulate real-world digital communications across various topics and styles. The conversations range from informal to formal, incorporating elements like slang and emoticons to reflect authentic messaging patterns. Each dialogue is paired with a concise, third-person summary, capturing its essential content. This structure makes SAMSum particularly useful for developing and evaluating automated summarization systems capable of processing conversational text.

#### Spider (Yu et al., 2018 ) .

Spider is a large-scale, complex, and cross-domain semantic parsing and text-to-SQL dataset. It contains about 10,000 annotated SQL queries, distributed across 200+ databases, each with multiple tables. We follow Scholak et al. () and use about 7,000 examples for training and about 1,000 examples for validation, where we ignore sequences longer than 1536 tokens. The dataset consists of English question and SQL query pairs, which cover a wide range of SQL operations including SELECT, WHERE, COUNT, GROUP BY, ORDER BY, JOIN, and more. Given an English question and an SQL database scheme, the task for the model is to translate the English question into an appropriate SQL statement. Evaluation is performed via accuracy where the output is considered as correct if the model’s predicted SQL query and the included GT SQL query give the same result when executed on the database. The dataset additionally categorizes each query into easy (25%), medium (40%), hard (20%), and extra hard (15%) based on the complexity of the required SQL statement. For evaluation, we report the execution accuracy of all categories.

#### DART (Nan et al., 2021 ) .

The DART (DAta Record to Text) benchmark is a large-scale, structured dataset designed for RDF-to-text (Resource Description Framework-to-text) generation with 80,000+ instances. The DART benchmark is composed of a collection of structured data triples and corresponding text summaries which are organized into different categories. The task of the DART benchmark is to generate natural language summaries that correctly represent the given structured data inputs. DART is typically evaluated with METEOR and BLEU.

#### CIFAR-10 (Krizhevsky et al., 2009 ) .

The CIFAR-10 (Canadian Institute For Advanced Research) dataset is a collection of images that are commonly used to train machine learning and computer vision algorithms. It is one of the most widely used datasets for image classification. The CIFAR-10 dataset contains 60,000 (50,000 for training, 10,000 for validation) 32$\times$32 color images in 10 different classes. The 10 different classes are: airplane, car, bird, cat, deer, dog, frog, horse, ship, and truck. There are 6,000 images of each class. For training, we center crop each image to 24$\times$24 pixels and flatten each image to a string, with a total of 24$\times$24$\times$3 words, where each word is a number between 0-255 representing the respective pixel value. Although CIFAR-10 is a dataset for computer vision, previous work (Dinh et al., ) showed that Transformers can be adapted to the vision domain from the language domain, and we tested this ability on the state-space model.

**Table 6: Datasets and models for our experiments. For each dataset, we report the number of training, validation, and test samples, maximum sequence length, training epochs, model size, and evaluation metric used.**
| Data | Size (Train) | Size (Val) | Size (Test) | Max. seq. len. | #Epochs | Mamba Size | Metrics |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GLUE | RTE | 1992 | 498 | 277 | 291 | 10 | 130M | Accuracy |
| MRPC | 2934 | 734 | 408 | 105 | 10 | 130M | Accuracy |  |
| CoLA | 6840 | 1711 | 1043 | 47 | 10 | 130M | Matthews corr. |  |
| SST-2 | 53879 | 13470 | 872 | 68 | 10 | 130M | Accuracy |  |
| QNLI | 83794 | 20949 | 5463 | 602 | 10 | 130M | Accuracy |  |
| QQP | 291076 | 72770 | 40430 | 316 | 3 | 130M | Accuracy |  |
| MNLI | 314161 | 78541 | 19647 | 425 | 3 | 130M | Accuracy |  |
| Spider | 5543 | 1375 | 1034 | 1412 | 10 | 1.4B, 2.8B | Accuracy |  |
| SAMSum | 14732 | 818 | 819 | 1174 | 10 | 1.4B | ROUGE |  |
| DART | 62659 | 2768 | 5097 | 491 | 10 | 130M | METEOR, BLEU |  |
| CIFAR-10 | 40000 | 10000 | 10000 | 1730 | 5 | 130M | Accuracy |  |

The dataset characteristics, including our train, validation and test set sizes, sequence lengths, and number of epochs, are summarized in Table 6.

## Appendix C Details of Sec. 4 : Benchmarking PEFT Methods on SSM-based Models

In this section, we provide a comprehensive experimental setup, proofs and further discussion of theoretical results, and more detailed experimental outcomes.

### C.1 Experiment Setup

For each dataset, we choose the model size of Mamba depending on how challenging the dataset is and perform a small grid search for one epoch on a subset of the data (1k-2k instances) with learning rates $\{$4\text{\times}{10}^{-1}$,$2\text{\times}{10}^{-1}$,$1\text{\times}{10}^{-1}
$,...,$1\text{\times}{10}^{-5}$\}$ to find the optimal learning rate of each PEFT method. Afterward, we train the best setting for each PEFT method on the full data for several epochs (Table 6) using an NVIDIA RTX 3090 GPU for the 130M model and an NVIDIA A100 for the larger 1.4B and 2.8B models in mixed precision (BF16).
We only report the validation metric of the best epoch during training (early stopping) in our results.
We fine-tune the Mamba models (Gu & Dao, ) pretrained from Pile (Gao et al., ) with AdamW with a linear learning rate decay schedule.
For LoRA we set rank to 8, alpha to 8, and dropout to 0.1 for all experiments.
For evaluating NLG tasks, we employ beam search with five beams and a maximum beam length of 1024.

### C.2 Extended Results on Benchmarking Existing PEFT Methods

We present comprehensive fine-tuning results for the GLUE benchmark (Wang et al., ), DART dataset (Nan et al., ), SAMSum dataset (Gliwa et al., ) and Spider dataset (Yu et al., ) in Table 7, Table 8, Table 9 and Table 10, respectively.
These experimental results encompass various LoRA implementations (on different weight matrices and modules) and provide more fine-grained results across all subtasks.

**Table 7: Full experimental results on the GLUE (Wang et al., ) benchmark. We report accuracy ($\uparrow$) for RTE, MRPC, SST-2, QNLI, QQP, and MNLI tasks. CoLA performance is measured using Matthews Correlation Coefficient ($\uparrow$). Mamba-130M is employed in this experiment. In each Mamba block, $\bm{W}_{\text{in},x}$ and $\bm{W}_{\text{in},z}$ are input projections that preprocess the input for SSM modules and the gating branch, respectively. $\bm{W}_{\text{out}}$ denotes the output projection after the gating mechanism. $\bm{W}_{{\bm{B}}}$ and $\bm{W}_{{\bm{C}}}$ are weight matrices for computing input-dependent ${\bm{B}}_{n}$ and ${\bm{C}}_{n}$. ${\bm{W}}_{\bm{\Delta},\downarrow}$ and ${\bm{W}}_{\bm{\Delta},\uparrow}$ represent down and up projections of low-rank weight matrices in the linear layer computing input-dependent step size $\bm{\Delta}_{n}$. ${\bm{\beta}}_{\bm{\Delta}}$ represents the bias in this linear layer. ${\bm{D}}$ denotes the weight of residual connections.**
| Layer | Method | # Params (%) | RTE | MRPC | CoLA | SST-2 | QNLI | QQP | MNLI | Avg. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pretrained |  |  | $0.0$ | $0.469\,314$ | $0.678\,922$ | $0.000\,000$ | $0.524\,083$ | $0.505\,217$ | $0.368\,118$ | $0.322\,594$ | $0.409\,750$ |
| All | All | Full | $100.0$ | $0.711\,191\,356\,182\,098\,4$ | $0.806\,372\,523\,307\,800\,3$ | $0.631\,936\,013\,698\,577\,9$ | $0.922\,018\,349\,170\,684\,8$ | $0.874\,061\,882\,495\,880\,1$ | $0.878\,703\,951\,835\,632\,3$ | $0.807\,655\,096\,054\,077\,1$ | $0.804\,562\,738\,963\,535\,8$ |
| LoRA | $1.922\,527\,283\,438\,617\,3$ | $0.699\,275\,374\,412\,536\,6$ | $0.808\,823\,525\,905\,609\,1$ | $0.614\,111\,959\,934\,234\,6$ | $0.918\,577\,969\,074\,249\,3$ | $0.884\,312\,629\,699\,707$ | $0.876\,131\,594\,181\,060\,8$ | $0.811\,370\,670\,795\,440\,7$ | $0.801\,800\,532\,000\,405\,5$ |  |  |
| Prompt | Prompt Tuning | 16 tokens | $0.009\,514\,691\,278\,001\,438$ | $0.559\,566\,795\,825\,958\,3$ | $0.715\,686\,261\,653\,900\,1$ | $0.119\,891\,807\,436\,943\,05$ | $0.894\,495\,427\,608\,49$ | $0.767\,893\,075\,942\,993\,2$ | $0.795\,819\,938\,182\,830\,8$ | $0.614\,863\,812\,923\,431\,4$ | $0.638\,316\,731\,367\,792\,4$ |
| Prefix-Tuning | 1 token (no MLP) | $0.028\,538\,643\,106\,431\,304$ | $0.675\,090\,253\,353\,118\,9$ | $0.757\,352\,948\,188\,781\,7$ | $0.434\,039\,652\,347\,564\,7$ | $0.915\,137\,588\,977\,813\,7$ | $0.834\,157\,049\,655\,914\,3$ | $0.831\,066\,012\,382\,507\,3$ | $0.356\,390\,297\,412\,872\,3$ | $0.686\,176\,257\,474\,081\,8$ |  |
| Bias | ${\bm{\beta}}_{\bm{\Delta}},\texttt{Conv1d}$ | BitFit | $0.057$ | $0.694\,545\,447\,8$ | $0.803\,921\,580\,3$ | $0.546\,525\,955\,2$ | $0.919\,724\,762\,4$ | $0.861\,614\,525\,3$ | $0.852\,782\,607\,1$ | $0.771\,669\,983\,9$ | $0.778\,683\,551\,71$ |
| linear projection matrices | All | LoRA | $1.017\,230\,519\,270\,278\,4$ | $0.700\,361\,013\,412\,475\,6$ | $0.823\,529\,422\,283\,172\,6$ | $0.577\,130\,556\,106\,567\,4$ | $0.933\,486\,223\,220\,825\,2$ | $0.887\,241\,423\,130\,035\,4$ | $0.887\,261\,927\,127\,838\,1$ | $0.825\,316\,846\,370\,697$ | $0.804\,903\,915\,950\,230\,1$ |
| $\bm{W}_{\text{in},x}$ | LoRA | $0.341\,392\,002\,181\,115\,6$ | $0.703\,971\,147\,537\,231\,4$ | $0.821\,078\,419\,685\,363\,8$ | $0.573\,514\,342\,308\,044\,4$ | $0.917\,431\,175\,708\,770\,8$ | $0.883\,031\,308\,650\,970\,5$ | $0.877\,145\,707\,607\,269\,3$ | $0.812\,185\,049\,057\,006\,8$ | $0.798\,336\,735\,793\,522\,4$ |  |
| $\bm{W}_{\text{in},z}$ | LoRA | $0.341\,392\,002\,181\,115\,6$ | $0.700\,361\,013\,412\,475\,6$ | $0.823\,529\,422\,283\,172\,6$ | $0.581\,120\,252\,609\,252\,9$ | $0.924\,311\,935\,901\,641\,8$ | $0.873\,146\,593\,570\,709\,2$ | $0.873\,212\,933\,540\,344\,2$ | $0.803\,990\,423\,679\,351\,8$ | $0.797\,096\,082\,142\,421\,2$ |  |
| $\bm{W}_{\text{in},x},\bm{W}_{\text{in},z}$ | LoRA | $0.680\,460\,965\,049\,587\,4$ | $0.703\,971\,147\,537\,231\,4$ | $0.843\,137\,264\,251\,709$ | $0.624\,137\,103\,557\,586\,7$ | $0.925\,458\,729\,267\,120\,4$ | $0.886\,326\,193\,809\,509\,3$ | $0.883\,353\,948\,593\,139\,6$ | $0.817\,427\,575\,588\,226\,3$ | $0.811\,973\,137\,514\,931\,8$ |  |
| $\bm{W}_{\text{out}}$ | LoRA | $0.341\,392\,002\,181\,115\,6$ | $0.703\,971\,147\,537\,231\,4$ | $0.828\,431\,367\,874\,145\,5$ | $0.605\,746\,030\,807\,495\,1$ | $0.924\,311\,935\,901\,641\,8$ | $0.884\,312\,629\,699\,707$ | $0.876\,675\,724\,983\,215\,3$ | $0.814\,831\,793\,308\,258\,1$ | $0.805\,468\,661\,444\,527\,8$ |  |
| S6 | All | Full | $4.310\,565\,285\,913\,944$ | $0.696\,750\,879\,287\,719\,7$ | $0.789\,215\,683\,937\,072\,8$ | $0.590\,752\,780\,437\,469\,5$ | $0.915\,137\,588\,977\,813\,7$ | $0.880\,651\,652\,812\,957\,8$ | $0.875\,290\,632\,247\,924\,8$ | $0.804\,906\,606\,674\,194\,3$ | $0.793\,243\,689\,196\,450\,4$ |
| LoRA | $0.923\,912\,723\,244\,536\,3$ | $0.660\,649\,836\,063\,385$ | $0.786\,764\,681\,339\,263\,9$ | $0.578\,182\,518\,482\,208\,3$ | $0.908\,256\,888\,389\,587\,4$ | $0.877\,722\,859\,382\,629\,4$ | $0.868\,859\,767\,913\,818\,4$ | $0.798\,137\,128\,353\,118\,9$ | $0.782\,653\,382\,846\,287\,3$ |  |  |
| ${\bm{A}}$ | Full | $0.456\,748\,639\,567\,040\,37$ | $0.682\,310\,461\,997\,985\,8$ | $0.821\,078\,419\,685\,363\,8$ | $0.542\,106\,330\,394\,744\,9$ | $0.909\,403\,681\,755\,065\,9$ | $0.863\,811\,075\,687\,408\,4$ | $0.878\,898\,799\,419\,403\,1$ | $0.793\,505\,370\,616\,912\,8$ | $0.784\,444\,877\,079\,555$ |  |
| $\bm{W}_{{\bm{B}}},\bm{W}_{{\bm{C}}},{\bm{W}}_{\bm{\Delta},\downarrow}$ | Full | $2.283\,743\,197\,835\,202$ | $0.696\,750\,879\,287\,719\,7$ | $0.769\,607\,841\,968\,536\,4$ | $0.558\,246\,970\,176\,696\,8$ | $0.913\,990\,855\,216\,98$ | $0.854\,475\,557\,804\,107\,7$ | $0.850\,157\,082\,080\,841\,1$ | $0.767\,954\,409\,122\,467$ | $0.773\,026\,227\,951\,049\,8$ |  |
| LoRA | $0.692\,192\,750\,875\,576\,5$ | $0.678\,700\,387\,477\,874\,8$ | $0.789\,215\,683\,937\,072\,8$ | $0.488\,315\,135\,240\,554\,8$ | $0.913\,990\,855\,216\,98$ | $0.868\,753\,433\,227\,539\,1$ | $0.858\,443\,200\,588\,226\,3$ | $0.786\,114\,215\,850\,830\,1$ | $0.769\,076\,130\,219\,868\,2$ |  |  |
| ${\bm{W}}_{\bm{\Delta},\uparrow}$ | Full | $1.398\,792\,708\,674\,061$ | $0.660\,649\,836\,063\,385$ | $0.752\,451\,002\,597\,808\,8$ | $0.566\,903\,471\,946\,716\,3$ | $0.910\,550\,475\,120\,544\,4$ | $0.862\,479\,388\,713\,836\,7$ | $0.871\,308\,445\,930\,481$ | $0.784\,801\,721\,572\,876$ | $0.772\,734\,905\,992\,235\,5$ |  |
| LoRA | $0.234\,957\,666\,087\,183\,55$ | $0.671\,480\,119\,228\,363$ | $0.799\,019\,634\,723\,663\,3$ | $0.550\,651\,848\,316\,192\,6$ | $0.909\,403\,681\,755\,065\,9$ | $0.527\,375\,936\,508\,178\,7$ | $0.865\,866\,959\,095\,001\,2$ | $0.786\,634\,087\,562\,561$ | $0.730\,061\,752\,455\,575\,1$ |  |  |
| Conv1d | Full | $0.142\,733\,949\,864\,700\,12$ | $0.682\,310\,461\,997\,985\,8$ | $0.784\,313\,738\,346\,099\,9$ | $0.578\,959\,524\,631\,500\,2$ | $0.910\,550\,475\,120\,544\,4$ | $0.859\,600\,961\,208\,343\,5$ | $0.859\,980\,225\,563\,049\,3$ | $0.779\,508\,292\,675\,018\,3$ | $0.779\,317\,668\,506\,077\,3$ |  |
| Others | ${\bm{D}},\texttt{LayerNorm}$ | Full | $0.043\,414\,909\,750\,512\,95$ | $0.653\,429\,627\,418\,518\,1$ | $0.791\,666\,686\,534\,881\,6$ | $0.403\,467\,684\,984\,207\,15$ | $0.910\,550\,475\,120\,544\,4$ | $0.839\,069\,962\,501\,525\,9$ | $0.859\,824\,895\,858\,764\,6$ | $0.670\,331\,358\,909\,606\,9$ | $0.732\,620\,098\,761\,149\,9$ |

**Table 8: Full experimental results on the DART (Nan et al., ) benchmark. We report METEOR $(\uparrow)$ and BLEU $(\uparrow)$ scores. Mamba-130M is utilized in this experiment. In each Mamba block, $\bm{W}_{\text{in},x}$ and $\bm{W}_{\text{in},z}$ are input projections that preprocess the input for SSM modules and the gating branch, respectively. $\bm{W}_{\text{out}}$ denotes the output projection after the gating mechanism. $\bm{W}_{{\bm{B}}}$ and $\bm{W}_{{\bm{C}}}$ are weight matrices for computing input-dependent ${\bm{B}}_{n}$ and ${\bm{C}}_{n}$. ${\bm{W}}_{\bm{\Delta},\downarrow}$ and ${\bm{W}}_{\bm{\Delta},\uparrow}$ represent down and up projections of low-rank weight matrices in the linear layer computing input-dependent step size $\bm{\Delta}_{n}$. ${\bm{\beta}}_{\bm{\Delta}}$ represents the bias in this linear layer. ${\bm{D}}$ denotes the weight of residual connections.**
| Layer | Method | # Params (%) | METEOR | BLEU |  |
| --- | --- | --- | --- | --- | --- |
| All | All | Full | $100.0$ | $71.008\,300\,781\,25$ | $51.802\,164\,316\,177\,37$ |
| LoRA | $1.922\,527\,283\,438\,617\,3$ | $70.968\,002\,080\,917\,36$ | $49.521\,303\,176\,879\,88$ |  |  |
| Prompt | Prompt Tuning | 64 tokens | $0.038\,047\,904\,689\,998\,75$ | $66.186\,875\,104\,904\,17$ | $39.826\,393\,127\,441\,406$ |
| Prefix-Tuning | 64 tokens | $22.688\,043\,993\,029\,535$ | $66.589\,879\,989\,624\,02$ | $42.462\,074\,756\,622\,314$ |  |
| Bias | ${\bm{\beta}}_{\bm{\Delta}},\texttt{Conv1d}$ | BitFit | $0.057$ | $67.0$ | $43.7$ |
| linear projection matrices | All | LoRA | $1.017\,230\,519\,270\,278\,4$ | $71.176\,528\,930\,664\,06$ | $49.160\,829\,186\,439\,514$ |
| $\bm{W}_{\text{in},x}$ | LoRA | $0.341\,392\,002\,181\,115\,6$ | $70.250\,225\,067\,138\,67$ | $48.860\,117\,793\,083\,19$ |  |
| $\bm{W}_{\text{in},z}$ | LoRA | $0.341\,392\,002\,181\,115\,6$ | $70.425\,111\,055\,374\,15$ | $49.059\,996\,008\,872\,986$ |  |
| $\bm{W}_{\text{in},x},\bm{W}_{\text{in},z}$ | LoRA | $0.680\,460\,965\,049\,587\,4$ | $70.937\,132\,835\,388\,18$ | $49.452\,367\,424\,964\,905$ |  |
| $\bm{W}_{\text{out}}$ | LoRA | $0.341\,392\,002\,181\,115\,6$ | $70.727\,145\,671\,844\,48$ | $46.977\,826\,952\,934\,265$ |  |
| S6 | All | Full | $4.310\,565\,285\,913\,944$ | $70.348\,870\,754\,241\,94$ | $48.673\,865\,199\,089\,05$ |
| LoRA | $0.923\,912\,723\,244\,536\,3$ | $69.898\,468\,255\,996\,7$ | $50.779\,956\,579\,208\,374$ |  |  |
| ${\bm{A}}$ | Full | $0.456\,748\,639\,567\,040\,37$ | $69.333\,219\,528\,198\,24$ | $48.095\,324\,635\,505\,676$ |  |
| $\bm{W}_{{\bm{B}}},\bm{W}_{{\bm{C}}},{\bm{W}}_{\bm{\Delta},\downarrow}$ | Full | $2.283\,743\,197\,835\,202$ | $70.056\,617\,259\,979\,25$ | $49.983\,921\,647\,071\,84$ |  |
| LoRA | $0.692\,192\,750\,875\,576\,5$ | $68.773\,013\,353\,347\,78$ | $47.991\,544\,008\,255\,005$ |  |  |
| ${\bm{W}}_{\bm{\Delta},\uparrow}$ | Full | $1.398\,792\,708\,674\,061$ | $69.576\,507\,806\,777\,95$ | $47.242\,343\,425\,750\,73$ |  |
| LoRA | $0.234\,957\,666\,087\,183\,55$ | $68.856\,698\,274\,612\,43$ | $47.047\,722\,339\,630\,13$ |  |  |
| Conv1d | Full | $0.142\,733\,949\,864\,700\,12$ | $68.621\,623\,516\,082\,76$ | $47.934\,231\,162\,071\,23$ |  |
| Others | ${\bm{D}},\texttt{LayerNorm}$ | Full | $0.043\,414\,909\,750\,512\,95$ | $67.027\,038\,335\,800\,17$ | $44.228\,667\,020\,797\,73$ |

**Table 9: Full experimental results on the SAMSum (Gliwa et al., ) benchmark. R1, R2, and RL represent ROUGE-1 ($\uparrow$), ROUGE-2 ($\uparrow$), and ROUGE-L ($\uparrow$), respectively. Mamba-1.4B is utilized in this experiment. In each Mamba block, $\bm{W}_{\text{in},x}$ and $\bm{W}_{\text{in},z}$ are input projections that preprocess the input for SSM modules and the gating branch, respectively. $\bm{W}_{\text{out}}$ denotes the output projection after the gating mechanism. $\bm{W}_{{\bm{B}}}$ and $\bm{W}_{{\bm{C}}}$ are weight matrices for computing input-dependent ${\bm{B}}_{n}$ and ${\bm{C}}_{n}$. ${\bm{W}}_{\bm{\Delta},\downarrow}$ and ${\bm{W}}_{\bm{\Delta},\uparrow}$ represent down and up projections of low-rank weight matrices in the linear layer computing input-dependent step size $\bm{\Delta}_{n}$. ${\bm{\beta}}_{\bm{\Delta}}$ represents the bias in this linear layer. ${\bm{D}}$ denotes the weight of residual connections.**
| Layer | Method | # Params (%) | R1 | R2 | RL |  |
| --- | --- | --- | --- | --- | --- | --- |
| All | All | Full | $100.0$ | $51.2$ | $27.3$ | $42.9$ |
| LoRA | $0.972\,818\,142\,718\,422\,3$ | $50.844\,812\,39$ | $26.647\,731\,66$ | $42.692\,086\,1$ |  |  |
| Prompt | Prompt Tuning | 64 tokens | $0.009\,551\,198\,153$ | $50.1$ | $25.6$ | $41.6$ |
| Prefix-Tuning | 64 tokens | $12.807\,268\,91$ | $50.6$ | $26.5$ | $42.1$ |  |
| Bias | ${\bm{\beta}}_{\bm{\Delta}},\texttt{Conv1d}$ | BitFit | $0.028\,656\,331\,48$ | $50.3$ | $25.7$ | $41.9$ |
| linear projection matrices | All | LoRA | $0.513\,166\,979\,724\,261\,8$ | $50.820\,308\,92$ | $26.873\,826\,98$ | $42.773\,035\,17$ |
| $\bm{W}_{\text{in},x}$ | LoRA | $0.171\,642\,869\,594\,627\,82$ | $49.831\,703$ | $25.436\,693$ | $41.155\,553$ |  |
| $\bm{W}_{\text{in},z}$ | LoRA | $0.171\,642\,869\,594\,627\,82$ | $50.021\,547\,08$ | $26.050\,487\,16$ | $41.673\,398\,02$ |  |
| $\bm{W}_{\text{in},x},\bm{W}_{\text{in},z}$ | LoRA | $0.342\,697\,523\,326\,188\,86$ | $50.872\,570\,28$ | $26.967\,412\,23$ | $42.283\,281\,68$ |  |
| $\bm{W}_{\text{out}}$ | LoRA | $0.171\,642\,869\,594\,627\,82$ | $49.860\,501$ | $25.443\,378$ | $41.459\,155$ |  |
| S6 | All | Full | $4.456\,059\,545\,468\,792$ | $51.134\,622\,1$ | $26.891\,985\,54$ | $42.242\,348\,19$ |
| LoRA | $0.464\,394\,215\,127\,723\,3$ | $50.520\,539\,28$ | $26.363\,494\,99$ | $42.184\,033\,99$ |  |  |
| ${\bm{A}}$ | Full | $0.229\,250\,651\,856\,915\,34$ | $50.090\,897\,08$ | $25.944\,900\,51$ | $41.724\,771\,26$ |  |
| $\bm{W}_{{\bm{B}}},\bm{W}_{{\bm{C}}},{\bm{W}}_{\bm{\Delta},\downarrow}$ | Full | $2.292\,506\,518\,569\,153\,8$ | $50.457\,620\,62$ | $25.973\,382\,59$ | $41.786\,766\,05$ |  |
| LoRA | $0.347\,144\,240\,365\,408\,7$ | $50.416\,523\,22$ | $25.998\,947\,02$ | $41.848\,915\,82$ |  |  |
| ${\bm{W}}_{\bm{\Delta},\uparrow}$ | Full | $1.848\,333\,380\,596\,38$ | $50.262\,951\,85$ | $25.663\,086\,77$ | $41.611\,900\,93$ |  |
| LoRA | $0.118\,067\,802\,522\,655\,75$ | $50.183\,439\,25$ | $25.424\,981\,12$ | $41.260\,328\,89$ |  |  |
| Conv1d | Full | $0.071\,640\,828\,705\,286\,06$ | $50.085\,217$ | $25.710\,830$ | $41.920\,885$ |  |
| Others | ${\bm{D}},\texttt{LayerNorm}$ | Full | $0.021\,641\,500\,338\,055\,16$ | $49.580\,908$ | $24.796\,230$ | $41.105\,807$ |

**(a) Comprehensive experimental results on Spider using Mamba-1.4B.**
| Layer | Method | # Params (%) | All | Easy | Medium | Hard | Extra |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| All | All | Full | $100.0$ | $66.150\,867\,938\,995\,36$ | $84.274\,190\,664\,291\,38$ | $69.506\,728\,649\,139\,4$ | $53.448\,277\,711\,868\,286$ | $43.373\,495\,340\,347\,29$ |
| LoRA | $0.972\,818\,142\,718\,422\,3$ | $56.382\,977\,962\,493\,9$ | $76.209\,676\,265\,716\,55$ | $56.950\,670\,480\,728\,15$ | $47.701\,150\,178\,909\,3$ | $34.337\,350\,726\,127\,625$ |  |  |
| Prompt | Prompt Tuning | 64 tokens | $0.009\,551\,198\,153$ | $43.617\,022\,04$ | $65.322\,577\,95$ | $42.376\,682\,16$ | $33.333\,334\,33$ | $25.301\,206\,11$ |
| Prefix-Tuning | 64 tokens | $12.807\,268\,91$ | $39.651\,837\,94$ | $65.725\,809\,34$ | $38.565\,021\,75$ | $31.034\,481\,53$ | $15.060\,241\,52$ |  |
| Bias | ${\bm{\beta}}_{\bm{\Delta}},\texttt{Conv1d}$ | BitFit | $0.028\,656\,331\,48$ | $51.257\,252\,69$ | $74.193\,549\,16$ | $50.896\,859\,17$ | $43.103\,447\,56$ | $26.506\,024\,6$ |
| linear projection matrices | All | LoRA | $0.513\,166\,979\,724\,261\,8$ | $54.738\,879\,203\,796\,39$ | $75.0$ | $55.605\,381\,727\,218\,63$ | $45.977\,011\,322\,975\,16$ | $31.325\,301\,527\,976\,99$ |
| $\bm{W}_{\text{in},x}$ | LoRA | $0.171\,642\,869\,594\,627\,82$ | $60.831\,719\,636\,917\,114$ | $76.612\,901\,687\,622\,07$ | $63.452\,917\,337\,417\,6$ | $52.873\,563\,766\,479\,49$ | $38.554\,215\,431\,213\,38$ |  |
| $\bm{W}_{\text{in},z}$ | LoRA | $0.171\,642\,869\,594\,627\,82$ | $46.324\,950\,456\,619\,26$ | $68.548\,387\,289\,047\,24$ | $45.739\,910\,006\,523\,13$ | $36.781\,609\,058\,380\,13$ | $24.698\,795\,378\,208\,16$ |  |
| $\bm{W}_{\text{in},x},\bm{W}_{\text{in},z}$ | LoRA | $0.342\,697\,523\,326\,188\,86$ | $57.543\,522\,119\,522\,095$ | $77.419\,352\,531\,433\,1$ | $58.744\,394\,779\,205\,32$ | $45.402\,297\,377\,586\,365$ | $37.349\,396\,944\,046\,02$ |  |
| $\bm{W}_{\text{out}}$ | LoRA | $0.171\,642\,869\,594\,627\,82$ | $61.798\,840\,761\,184\,69$ | $81.854\,838\,132\,858\,28$ | $65.246\,635\,675\,430\,3$ | $45.402\,297\,377\,586\,365$ | $39.759\,036\,898\,612\,976$ |  |
| S6 | All | Full | $4.456\,059\,545\,468\,792$ | $56.673\,115\,491\,867\,065$ | $76.612\,901\,687\,622\,07$ | $57.847\,535\,610\,198\,975$ | $45.977\,011\,322\,975\,16$ | $34.939\,759\,969\,711\,304$ |
| LoRA | $0.464\,394\,215\,127\,723\,3$ | $56.286\,269\,426\,345\,825$ | $75.0$ | $56.502\,240\,896\,224\,976$ | $50.574\,713\,945\,388\,794$ | $33.734\,938\,502\,311\,71$ |  |  |
| ${\bm{A}}$ | Full | $0.229\,250\,651\,856\,915\,34$ | $51.063\,829\,660\,415\,65$ | $71.370\,965\,242\,385\,86$ | $52.466\,368\,675\,231\,934$ | $42.528\,736\,591\,339\,11$ | $25.903\,615\,355\,491\,638$ |  |
| $\bm{W}_{{\bm{B}}},\bm{W}_{{\bm{C}}},{\bm{W}}_{\bm{\Delta},\downarrow}$ | Full | $2.292\,506\,518\,569\,153\,8$ | $47.195\,357\,084\,274\,29$ | $72.177\,422\,046\,661\,38$ | $46.860\,986\,948\,013\,306$ | $35.632\,184\,147\,834\,78$ | $22.891\,566\,157\,341\,003$ |  |
| LoRA | $0.347\,144\,240\,365\,408\,7$ | $55.029\,010\,772\,705\,08$ | $73.790\,323\,734\,283\,45$ | $56.726\,455\,688\,476\,56$ | $44.252\,872\,467\,041\,016$ | $33.734\,938\,502\,311\,71$ |  |  |
| ${\bm{W}}_{\bm{\Delta},\uparrow}$ | Full | $1.848\,333\,380\,596\,38$ | $56.769\,824\,028\,015\,14$ | $77.016\,127\,109\,527\,59$ | $59.417\,039\,155\,960\,08$ | $43.678\,161\,501\,884\,46$ | $33.132\,529\,258\,728\,03$ |  |
| LoRA | $0.118\,067\,802\,522\,655\,75$ | $58.027\,076\,721\,191\,406$ | $78.629\,034\,757\,614\,14$ | $59.417\,039\,155\,960\,08$ | $48.850\,575\,089\,454\,65$ | $33.132\,529\,258\,728\,03$ |  |  |
| Conv1d | Full | $0.071\,640\,828\,705\,286\,06$ | $53.191\,488\,981\,246\,95$ | $74.596\,774\,578\,094\,48$ | $52.914\,798\,259\,735\,11$ | $43.678\,161\,501\,884\,46$ | $31.927\,710\,771\,560\,67$ |  |
| Others | ${\bm{D}},\texttt{LayerNorm}$ | Full | $0.021\,641\,500\,338\,055\,16$ | $49.613\,153\,934\,478\,76$ | $70.564\,514\,398\,574\,83$ | $50.448\,429\,584\,503\,174$ | $40.229\,883\,790\,016\,174$ | $25.903\,615\,355\,491\,638$ |

**Table 11: Full experimenal results on the CIFAR-10 (Krizhevsky et al., ) dataset. We report accuracy ($\uparrow$). Mama-130M is utilized in this experiment. In each Mamba block, $\bm{W}_{\text{in},x}$ and $\bm{W}_{\text{in},z}$ are input projections that preprocess the input for SSM modules and the gating branch, respectively. $\bm{W}_{\text{out}}$ denotes the output projection after the gating mechanism. $\bm{W}_{{\bm{B}}}$ and $\bm{W}_{{\bm{C}}}$ are weight matrices for computing input-dependent ${\bm{B}}_{n}$ and ${\bm{C}}_{n}$. ${\bm{W}}_{\bm{\Delta},\downarrow}$ and ${\bm{W}}_{\bm{\Delta},\uparrow}$ represent down and up projections of low-rank weight matrices in the linear layer computing input-dependent step size $\bm{\Delta}_{n}$. ${\bm{\beta}}_{\bm{\Delta}}$ represents the bias in this linear layer. ${\bm{D}}$ denotes the weight of residual connections.**
| Layer | Method | # Params (%) | Accuracy |  |
| --- | --- | --- | --- | --- |
| Pretrained |  |  | $0.0$ | $0.081\,500$ |
| All | All | Full | $100.0$ | $59.960\,001\,707\,077\,026$ |
| LoRA | $1.922\,527\,283\,438\,617\,3$ | $60.350\,000\,858\,306\,885$ |  |  |
| Bias | ${\bm{\beta}}_{\bm{\Delta}},\texttt{Conv1d}$ | BitFit | $0.06$ | $44.4$ |
| linear projection matrices | All | LoRA | $1.017\,230\,519\,270\,278\,4$ | $62.790\,000\,438\,690\,186$ |
| $\bm{W}_{\text{in},x}$ | LoRA | $0.341\,392\,002\,181\,115\,6$ | $53.490\,000\,963\,211\,06$ |  |
| $\bm{W}_{\text{in},z}$ | LoRA | $0.341\,392\,002\,181\,115\,6$ | $58.149\,999\,380\,111\,694$ |  |
| $\bm{W}_{\text{in},x},\bm{W}_{\text{in},z}$ | LoRA | $0.680\,460\,965\,049\,587\,4$ | $61.040\,002\,107\,620\,24$ |  |
| $\bm{W}_{\text{out}}$ | LoRA | $0.341\,392\,002\,181\,115\,6$ | $52.039\,998\,769\,760\,13$ |  |
| S6 | All | Full | $4.310\,565\,285\,913\,944$ | $55.510\,002\,374\,649\,05$ |
| LoRA | $0.923\,912\,723\,244\,536\,3$ | $43.959\,999\,084\,472\,656$ |  |  |
| ${\bm{A}}$ | Full | $0.456\,748\,639\,567\,040\,37$ | $61.210\,000\,514\,984\,13$ |  |
| $\bm{W}_{{\bm{B}}},\bm{W}_{{\bm{C}}},{\bm{W}}_{\bm{\Delta},\downarrow}$ | Full | $2.283\,743\,197\,835\,202$ | $49.509\,999\,155\,998\,23$ |  |
| LoRA | $0.692\,192\,750\,875\,576\,5$ | $52.270\,001\,173\,019\,41$ |  |  |
| ${\bm{W}}_{\bm{\Delta},\uparrow}$ | Full | $1.398\,792\,708\,674\,061$ | $34.540\,000\,557\,899\,475$ |  |
| LoRA | $0.234\,957\,666\,087\,183\,55$ | $56.489\,998\,102\,188\,11$ |  |  |
| Conv1d | Full | $0.142\,733\,949\,864\,700\,12$ | $55.650\,001\,764\,297\,485$ |  |
| Others | ${\bm{D}},\texttt{LayerNorm}$ | Full | $0.043\,414\,909\,750\,512\,95$ | $58.090\,001\,344\,680\,786$ |

### C.3 Limitations of Applying Prompt-based Methods on SSMs

We provide the formal version of Proposition 1 and its corresponding proof here.
We start by introducing the necessary notations.
Denote the space of S4 mechanisms with $D$ channels as ${\mathcal{F}}_{\text{S4},D}$. Let ${\bm{H}}_{0}=({\bm{h}}_{0}^{(1)},{\bm{h}}_{0}^{(2)},\dots,{\bm{h}}_{0}^{(D)})
\in{\mathbb{R}}^{H\times D}$ represent the initial hidden state, and ${\bm{X}}=({\bm{x}}_{1},{\bm{x}}_{2},\ldots,{\bm{x}}_{N})\in{\mathbb{R}}^{D
\times N}$ denote the input sequence.
The output of the S4 mechanism is represented as $f({\bm{X}};{\bm{H}}_{0})$.
Furthermore, for $d$-th channel, let state transition matrix $\overline{{\bm{A}}}^{(d)}=\operatorname{diag}{(a_{1}^{(d)},\cdots,a_{H}^{(d)})}$ and input transition vector $\overline{{\bm{B}}}^{(d)}=(b_{1},\cdots,b_{H})^{\top}$, where $d=1,\ldots,D$.
For any vector ${\bm{v}}\in{\mathbb{R}}^{n}$, we use ${\bm{v}}_{i:j}\in{\mathbb{R}}^{j-i}$ to denote the subvector of ${\bm{v}}$ containing elements from $i\in{\mathbb{N}}^{+}$ to $j\in{\mathbb{N}}^{+}$, where $i<j$.
Similarly, for any matrix ${\bm{M}}\in{\mathbb{R}}^{m\times n}$, we use ${\bm{M}}_{i_{1}:j_{1},i_{2}:j_{2}}$ to denote the submatrix containing rows $i_{1}\in{\mathbb{N}}^{+}$ to $j_{1}\in{\mathbb{N}}^{+}$ and columns $i_{2}\in{\mathbb{N}}^{+}$ to $j_{2}\in{\mathbb{N}}^{+}$, where $i_{1}<j_{1}$, $i_{2}<j_{2}$.

###### Proposition 5 (Formal Version of Proposition 1 ) .

Let $f\in{\mathcal{F}}_{\text{S4},D}$ be an S4 mechanism.
Consider prefix-tuning that prepends a sequence ${\bm{P}}=({\bm{p}}_{1},\ldots,{\bm{p}}_{M})\in{\mathbb{R}}^{D\times M}$ to the input sequence ${\bm{X}}=({\bm{x}}_{1},{\bm{x}}_{2},\ldots,{\bm{x}}_{N})\in{\mathbb{R}}^{D
\times N}$.
For any prefix ${\bm{P}}\in{\mathbb{R}}^{D\times M}$, there exists an initial hidden state ${\bm{H}}_{0}^{\star}\in{\mathbb{R}}^{H\times D}$ such that the output of S4 after prefix-tuning and that after initial state tuning are identical, i.e., $f({\bm{X}};{\bm{H}}_{0}^{\star})\equiv f([{\bm{P}},{\bm{X}}];{\bm{H}}_{0})_{1:
D,M+1:M+N}$ for all ${\bm{X}}\in{\mathbb{R}}^{D\times N}$.

Furthermore, assume that $\prod_{0\leq i<j\leq H}(a_{j}^{(d)}-a_{i}^{(d)})\neq 0$ and $\prod_{k=1}^{H}b_{k}^{(d)}\neq 0$ for all channels $d=1,\ldots,D$.
Then the converse (i.e., for any ${\bm{H}}_{0}\in{\mathbb{R}}^{H\times D}$, there exists a ${\bm{P}}^{\star}\in{\mathbb{R}}^{D\times M}$ such that $f([{\bm{P}}^{\star},{\bm{X}}];{\bm{H}}_{0})_{1:D,M+1:M+N}\equiv f({\bm{X}};{
\bm{H}}_{0}^{\star})$ for all ${\bm{X}}\in{\mathbb{R}}^{D\times N}$) holds if and only if $M\geq H$.

###### Proof of Proposition 5 .

Given that operations in S4 are independent across all channels, we can, without loss of generality, consider the case where the number of channels $D=1$.
Consequently, we can simplify our notation: the initial hidden states ${\bm{H}}_{0}\in{\mathbb{R}}^{H\times D}$ become ${\bm{h}}_{0}\in{\mathbb{R}}^{H}$, the input sequence ${\bm{X}}\in{\mathbb{R}}^{D\times N}$ becomes ${\bm{x}}\in{\mathbb{R}}^{N}$, and the prefix ${\bm{P}}\in{\mathbb{R}}^{D\times M}$ becomes ${\bm{p}}\in{\mathbb{R}}^{M}$.
We omit the superscript $(d)$ denoting the channel index.
To differentiate between the hidden states and output of prefix-tuned S4 (i.e., $f([{\bm{P}},{\bm{X}}];{\bm{H}}_{0})_{1:D,M+1:M+N}$) and initial state tuned S4 (i.e., $f({\bm{X}};{\bm{H}}_{0}^{\star})$), we introduce superscripts “PT” and “IST” respectively.
The “PT” superscript denotes hidden states and output of S4 after prefix-tuning, while “IST” indicates those after initial state tuning.

We divide the proposition into two statements:

- 1.
For any prefix ${\bm{p}}\in{\mathbb{R}}^{M}$, there exists an initial hidden state ${\bm{h}}_{0}^{\star}\in{\mathbb{R}}^{H}$ such that the output of S4 after prefix-tuning and that after initial state tuning are identical, i.e., $f({\bm{x}};{\bm{h}}_{0}^{\star})\equiv f([{\bm{p}},{\bm{x}}];{\bm{h}}_{0})_{M+
1:N+M}$ for all ${\bm{x}}\in{\mathbb{R}}^{N}$.
- 2.
Furthermore, assume that $\prod_{0\leq i<j\leq H}(a_{j}-a_{i})\neq 0$ and $\prod_{k=1}^{H}b_{k}\neq 0$.
Then the converse (i.e., for any ${\bm{h}}_{0}\in{\mathbb{R}}^{H}$, there exists a ${\bm{p}}^{\star}\in{\mathbb{R}}^{M}$ such that $f([{\bm{p}}^{\star},{\bm{x}}];{\bm{h}}_{0})_{M+1:N+M}\equiv f({\bm{x}};{\bm{h}
}_{0}^{\star})$ for all ${\bm{x}}\in{\mathbb{R}}^{N}$) holds if and only if $M\geq H$.

We will first prove the first statement and then proceed to prove the second statement.

Statement 1.
The recurrent computation formulation of S4 in (2) implies that for each position $i$, the output $y_{i}$ depends solely on the previous hidden state $h_{i-1}$ and the current input $x_{i}$.
Thus, to demonstrate that $f({\bm{x}};{\bm{h}}_{0}^{\star})\equiv f([{\bm{p}},{\bm{x}}];{\bm{h}}_{0})_{M+
1:N+M}$ for all ${\bm{x}}\in{\mathbb{R}}^{N}$, it suffices to show that the hidden state for predicting output $y^{\text{IST}}_{1}$ equals that for predicting output $y^{\text{PT}}_{M+1}$, where $y^{\text{IST}}_{1}$ and $y^{\text{PT}}_{M+1}$ are outputs corresponding to the input $x_{1}$ for initial state tuning and prefix-tuning, respectively.
In other words, it is sufficient to show that the initial state of initial-state-tuned model ${\bm{h}}^{\text{IST}}_{0}={\bm{h}}_{0}^{\star}$ is equal to the $(M+1)$-th hidden state of prefix-tuned model ${\bm{h}}^{\text{PT}}_{M+1}=\sum_{m=1}^{M}\overline{{\bm{A}}}^{M-m}\overline{{
\bm{B}}}p_{m}$.
When this equality holds, the subsequent hidden states and outputs for both versions of S4 will be identical, as the input sequence from that point onward is the same.
Therefore, We prove the first statement by letting

$$ $\displaystyle{\bm{h}}_{0}^{\star}=\sum_{m=1}^{M}\overline{{\bm{A}}}^{M-m} \overline{{\bm{B}}}p_{m}.$ (11) $$

Statement 2.
We aim to investigate the conditions under which there exists a ${\bm{h}}_{0}^{\star}\in{\mathbb{R}}^{H}$ such that for any ${\bm{p}}\in{\mathbb{R}}^{M}$,
$f([{\bm{p}}^{\star},{\bm{x}}];{\bm{h}}_{0})_{M+1:N+M}\neq f({\bm{x}};{\bm{h}}_
{0}^{\star})$. This is equivalent to demonstrating the existence of ${\bm{h}}_{0}^{\star}\in{\mathbb{R}}^{H}$ such that

$$ ${\bm{h}}_{0}^{\star}\neq\sum_{m=1}^{M}\overline{{\bm{A}}}^{M-m}\overline{{\bm{ B}}}p_{m},\quad\text{for all }{\bm{p}}\in{\mathbb{R}}^{M}.$ (12) $$

This condition can be further reformulated as

$$ ${\mathbb{R}}^{H}\setminus\text{span}(\overline{{\bm{A}}}^{M}\overline{{\bm{B}} },\overline{{\bm{A}}}^{M-1}\overline{{\bm{B}}},\ldots,\overline{{\bm{B}}})\neq\emptyset,$ (13) $$

which is equivalent to

$$ $\text{span}(\overline{{\bm{A}}}^{M}\overline{{\bm{B}}},\overline{{\bm{A}}}^{M- 1}\overline{{\bm{B}}},\ldots,\overline{{\bm{B}}})\subsetneq{\mathbb{R}}^{H}.$ (14) $$

To determine when this condition holds, we analyze three distinct cases: (i) $M<H$, (ii) $M=H$, and (iii) $M>H$.

*(Case 1: When $M<H$).*
In this scenario, it is obvious that (14) holds.
The existence of such a ${\bm{h}}_{0}^{\star}$ is guaranteed because the dimension of the span is at most $M$, which is strictly less than $H$. This choice of ${\bm{h}}_{0}^{\star}$ ensures that it cannot be represented as a linear combination of the vectors in the span, thereby establishing the inequality.

*(Case 2: When $M=H$).* In this scenario, $\text{span}(\overline{{\bm{A}}}^{M}\overline{{\bm{B}}},\overline{{\bm{A}}}^{M-
1}\overline{{\bm{B}}},\ldots,\overline{{\bm{B}}})={\mathbb{R}}^{H}$ if and only if $(\overline{{\bm{A}}}^{M}\overline{{\bm{B}}},\overline{{\bm{A}}}^{M-1}\overline
{{\bm{B}}},\ldots,\overline{{\bm{B}}})$ are linearly independent.
Note that

$$ $\displaystyle\det(\overline{{\bm{A}}}^{M}\overline{{\bm{B}}},\overline{{\bm{A} }}^{M-1}\overline{{\bm{B}}},\ldots,\overline{{\bm{B}}})=\det(\overline{{\bm{A} }}^{M},\overline{{\bm{A}}}^{M-1},\ldots,{\bm{1}})\prod_{k=1}^{H}b_{k},$ (15) $$

where

$$ $\displaystyle\det(\overline{{\bm{A}}}^{M},\overline{{\bm{A}}}^{M-1},\ldots,{ \bm{1}})$ $\displaystyle=\det\begin{bmatrix}a_{1}^{H-1}&\cdots&a_{1}^{2}&a_{1}&1\\ a_{2}^{H-1}&\cdots&a_{2}^{2}&a_{2}&1\\ \vdots&\ddots&\vdots&\vdots&\vdots\\ a_{H}^{H-1}&\cdots&a_{H}^{2}&a_{H}&1\\ \end{bmatrix}$ $\displaystyle\quad(\text{Expand})$ (16) $\displaystyle=(-1)^{\frac{H(H-1)}{2}}\prod_{0\leq i<j\leq H}^{H}(a_{j}-a_{i}).$ $\displaystyle\quad(\text{Vandermonde matrix})$ (17) $$

Combining (15) and (17) yields

$$ $\displaystyle\det(\overline{{\bm{A}}}^{M}\overline{{\bm{B}}},\overline{{\bm{A} }}^{M-1}\overline{{\bm{B}}},\ldots,\overline{{\bm{B}}})=(-1)^{\frac{H(H-1)}{2} }\prod_{0\leq i<j\leq H}^{H}(a_{j}-a_{i})\prod_{k=1}^{H}b_{k}.$ (18) $$

Therefore, if and only if $\prod_{1\leq i<j\leq H}(a_{j}-a_{i})\neq 0$ and $\prod_{k=1}^{H}b_{k}\neq 0$, we have

$$ $\displaystyle\det(\overline{{\bm{A}}}^{M}\overline{{\bm{B}}},\overline{{\bm{A} }}^{M-1}\overline{{\bm{B}}},\ldots,\overline{{\bm{B}}})\neq 0,$ (19) $$

which is both necessary and sufficient for the linear independence of $(\overline{{\bm{A}}}^{M}\overline{{\bm{B}}},\overline{{\bm{A}}}^{M-1}\overline
{{\bm{B}}},\ldots,\overline{{\bm{B}}})$, and consequently, for the condition in (14) to be satisfied.

*(Case 3: When $M>H$).* The analysis presented in case 2 extends naturally to this scenario.

The combination of the three cases above completes the proof of statement 2.
∎

###### Proposition 5 (Formal Version of Proposition 1 ) .

###### Proof of Proposition 5 .

### C.4 Optimal Application of LoRA in SSM-based Models

Several studies (Hu et al., ; He et al., ) present findings on Transformers, indicating that applying LoRA to linear projection matrices yields performance comparable to or marginally superior to that of attention layers.
In contrast, our experimental results on SSMs reveal that applying LoRA to linear projection matrices is more effective than applying it to S6 (see Table 3).
To elucidate this phenomenon, we examine the influence of updating linear projection matrices on the model’s output.

#### Notations.

For the feasibility of the analysis, we consider a simplified SSM-based architecture which only consists of the input projection matrix $\bm{W}_{\text{in}}\in{\mathbb{R}}^{D\times D}$ and the S6 module parameterized by diagonal state transition matrices $\{{\bm{A}}^{(d)}\}_{d=1}^{D}$ with ${\bm{A}}^{(d)}\in{\mathbb{R}}^{H\times H}$, the weight matrices $\bm{W}_{{\bm{B}}},\bm{W}_{{\bm{C}}}\in{\mathbb{R}}^{H\times D}$ for computing input-dependent input transition vectors ${\bm{B}}_{n}\in{\mathbb{R}}^{H}$ and output mapping vectors ${\bm{C}}_{n}\in{\mathbb{R}}^{H}$, the down and up projection matrices ${\bm{W}}_{\bm{\Delta},\downarrow}\in{\mathbb{R}}^{D\times R},{\bm{W}}_{\bm{
\Delta},\uparrow}\in{\mathbb{R}}^{R\times D}$ (where $R$ is the rank) for low-rank weight matrices for computing the input-depdenent step size $\bm{\Delta}_{n}=(\Delta^{(1)}_{n},\ldots,\Delta^{(D)}_{n})\in{\mathbb{R}}_{D}$, for $n=1,\ldots,N$.
Define $\bm{W}_{\text{S6}}=[\bm{W}_{{\bm{B}}}^{\top},\bm{W}_{{\bm{C}}}^{\top},{\bm{W}}
_{\bm{\Delta},\uparrow}^{\top}]^{\top}\in{\mathbb{R}}^{(2H+R)\times D}$.
In the Mamba implementation, $\bm{W}_{\text{S6}}$ is implemented as the weight matrix of a single linear layer, referred to as x_proj in the codebase.
Therefore, the parameters of the S6 can be formulated as

$$ $\displaystyle{\bm{\theta}}(\cdot;\{{\bm{A}}\}_{d=1}^{D},\bm{W}_{\text{S6}},{ \bm{W}}_{\bm{\Delta},\downarrow},\bm{W}_{\text{in}})=\{\overline{{\bm{A}}}_{n} ,\overline{{\bm{B}}}_{n},{\bm{C}}_{n}\}_{n=1}^{N}.$ (20) $$

Consider input sequence ${\bm{X}}=({\bm{x}}_{1},\ldots,{\bm{x}}_{N})\in{\mathbb{R}}^{D\times N}$.
Let ${\bm{Z}}=({\bm{z}}_{1},\ldots,{\bm{z}}_{N})\in{\mathbb{R}}^{D\times N}$ denote the intermediate output after the input projection.
The intermediate output at position $n\in\{1,\ldots,N\}$ is

$$ $\displaystyle{\bm{z}}_{n}=\bm{W}_{\text{in}}\bm{x}_{n}.$ (21) $$

Note that

$$ $\displaystyle{\bm{B}}_{n}=\bm{W}_{{\bm{B}}}{\bm{z}}_{n},\quad{\bm{C}}_{n}=\bm{ W}_{{\bm{C}}}{\bm{z}}_{n},\quad\bm{\Delta}_{n}=\operatorname{softplus}({\bm{W} }_{\bm{\Delta},\uparrow}{\bm{W}}_{\bm{\Delta},\downarrow}{\bm{z}}_{n}+{\bm{ \beta}}_{\bm{\Delta}}),$ (22) $$

and after discretization, we have

$$ $\displaystyle\overline{{\bm{A}}}_{n}^{(d)}=\exp(\Delta_{n}^{(d)}{\bm{A}}^{(d)} ),\quad\overline{{\bm{B}}}_{n}=\Delta_{n}^{(d)}{\bm{B}}_{n}=\Delta_{n}^{(d)} \bm{W}_{{\bm{B}}}{\bm{z}}_{n}.$ (23) $$

Combining (21), (22) and (23) yields

$$ $\displaystyle{\bm{\theta}}({\bm{X}};\{{\bm{A}}\}_{d=1}^{D},\bm{W}_{\text{S6}}, {\bm{W}}_{\bm{\Delta},\downarrow},\bm{W}_{\text{in}})=\{\overline{{\bm{A}}}_{n },\overline{{\bm{B}}}_{n},{\bm{C}}_{n}\}_{n=1}^{N},\text{ where}$ (24) $\displaystyle\overline{{\bm{A}}}_{n}^{(d)}=\exp(\Delta_{n}^{(d)}{\bm{A}}^{(d)} ),\quad\overline{{\bm{B}}}_{n}=\Delta_{n}^{(d)}\bm{W}_{{\bm{B}}}\bm{W}_{\text{ in}}\bm{x}_{n},\quad{\bm{C}}_{n}=\bm{W}_{{\bm{C}}}\bm{W}_{\text{in}}\bm{x}_{n},$ (25) $\displaystyle\bm{\Delta}_{n}=\operatorname{softplus}({\bm{W}}_{\bm{\Delta}, \downarrow}{\bm{W}}_{\bm{\Delta},\uparrow}\bm{W}_{\text{in}}\bm{x}_{n}+{\bm{ \beta}}_{\bm{\Delta}}).$ (26) $$

#### Theoretical Analysis.

In the following theorem, we demonstrate that applying LoRA exclusively to $\bm{W}_{\text{in}}$ is equivalent to applying it to $\bm{W}_{\text{S6}}$.

###### Lemma 6 (Detailed Version of Lemma 2 ) .

Consider a model consists of an S6 module augmented with a linear input projection $\bm{W}_{\text{in}}\in{\mathbb{R}}^{D\times D}$.
For any fine-tuned model where only $\bm{W}_{\text{S6}}$ is updated to $\overline{\bm{W}}_{\text{S6}}$,
there exists $\widehat{{\bm{W}}}_{\text{in}}$ such that updating only $\bm{W}_{\text{in}}$ to $\widehat{{\bm{W}}}_{\text{in}}$ yields:

$$ $\displaystyle{\bm{\theta}}({\bm{X}};\{{\bm{A}}^{(d)}\}_{d=1}^{D},\overline{\bm {W}}_{\text{S6}},{\bm{W}}_{\bm{\Delta},\downarrow},\bm{W}_{\text{in}})={\bm{ \theta}}({\bm{X}};\{{\bm{A}}^{(d)}\}_{d=1}^{D},\bm{W}_{\text{S6}},{\bm{W}}_{ \bm{\Delta},\downarrow},\widehat{{\bm{W}}}_{\text{in}})$ (27) $$

###### Proof of Lemma 6 .

In this proof,
we use $\overline{\cdot}$ to denote the corresponding notations for the model with only $\bm{W}_{\text{S6}}$ updated, and use $\widehat{\cdot}$ to denote the corresponding notations for the model with only $\bm{W}_{\text{in}}$ updated.

To demonstrate (27), it is sufficient, according to (24), to find $\widehat{{\bm{W}}}_{\text{in}}$ that satisfies the following equations:

$$ $\displaystyle\overline{{\bm{W}}}_{{\bm{C}}}\bm{W}_{\text{in}}=\bm{W}_{{\bm{C}} }\widehat{{\bm{W}}}_{\text{in}}$ (28) $\displaystyle\overline{{\bm{W}}}_{\bm{\Delta},\uparrow}\bm{W}_{\text{in}}={\bm {W}}_{\bm{\Delta},\uparrow}\widehat{{\bm{W}}}_{\text{in}}$ (29) $\displaystyle\overline{{\bm{W}}}_{{\bm{B}}}\bm{W}_{\text{in}}=\bm{W}_{{\bm{B}} }\widehat{{\bm{W}}}_{\text{in}}.$ (30) $$

Since $\bm{W}_{\text{S6}}=\begin{bmatrix}\bm{W}_{{\bm{B}}}\\
\bm{W}_{{\bm{C}}}\\
{\bm{W}}_{\bm{\Delta},\uparrow}\end{bmatrix}$, the three conditions (28) can be written as

$$ $\displaystyle\overline{\bm{W}}_{\text{S6}}\bm{W}_{\text{in}}=\bm{W}_{\text{S6} }\widehat{{\bm{W}}}_{\text{in}}.$ (31) $$

By applying Singular Value Decomposition (SVD) to $\bm{W}_{\text{S6}}$ and $\left(\bm{W}_{\text{S6}}-\overline{\bm{W}}_{\text{S6}}\right)\bm{W}_{\text{in}}$, we obtain:

$$ $\displaystyle\bm{W}_{\text{S6}}={\bm{U}}\begin{bmatrix}{\bm{\Sigma}}&{\bm{O}}_ {(2H+R)\times(D-2H-R)}\end{bmatrix}{\bm{V}}^{\top},$ (32) $\displaystyle\left(\bm{W}_{\text{S6}}-\overline{\bm{W}}_{\text{S6}}\right)\bm{ W}_{\text{in}}={\bm{U}}^{\prime}\begin{bmatrix}{\bm{\Sigma}}^{\prime}&{\bm{O}} _{(2H+R)\times(D-2H-R)}\end{bmatrix}{\bm{V}}^{\prime\top},$ (33) $$

where ${\bm{U}},{\bm{U}}^{\prime}\in{\mathbb{R}}^{(2H+R)\times(2H+R)}$, ${\bm{\Sigma}},{\bm{\Sigma}}^{\prime}\in{\mathbb{R}}^{(2H+R)\times(2H+R)}$, and ${\bm{V}},{\bm{V}}^{\prime}\in{\mathbb{R}}^{D\times D}$. The diagonal elements of ${\bm{\Sigma}}$ and ${\bm{\Sigma}}^{\prime}$ are in decreasing order.

We let

$$ $\displaystyle\bm{\widehat{W}}_{\textrm{in}}={\bm{V}}\begin{bmatrix}{\bm{\Sigma }}^{-1}{\bm{U}}^{\top}\overline{\bm{W}}_{\text{S6}}\bm{W}_{\text{in}}\\ {\bm{Q}}\end{bmatrix},$ (34) $$

where ${\bm{Q}}\in{\mathbb{R}}^{(D-2H-R)\times D}$ is an arbitrary matrix to be determined later.
Plugging (32) and(34) back to $\bm{W}_{\text{S6}}\widehat{{\bm{W}}}_{\text{in}}$ and simplifying results in

$$ $\displaystyle\bm{W}_{\text{S6}}\widehat{{\bm{W}}}_{\text{in}}$ (35) $\displaystyle={\bm{U}}\begin{bmatrix}{\bm{\Sigma}}&{\bm{O}}_{(2H+R)\times(D-2H -R)}\end{bmatrix}{\bm{V}}^{\top}{\bm{V}}\begin{bmatrix}{\bm{\Sigma}}^{-1}{\bm{ U}}^{\top}\overline{\bm{W}}_{\text{S6}}\bm{W}_{\text{in}}\\ {\bm{Q}}\end{bmatrix}$ (( 32 ) & ( 34 )) (36) $\displaystyle=\overline{\bm{W}}_{\text{S6}}\bm{W}_{\text{in}},$ (Simplifying) (37) $$

which demonstrates that (31) is satisfied and completes the proof.
∎

###### Lemma 6 (Detailed Version of Lemma 2 ) .

###### Proof of Lemma 6 .

## Appendix D Details of Sec. 5 : Selective Dimension Tuning

### D.1 Understanding the Roles of State Matrix A , Input Transition Vector B , and Output Mapping Vector C for a Single Channel in S4 Modules

#### Problem Setting.

Inspired by Zeng & Lee ()’s theoretical analysis of LoRA’s expressive power, we adopt a similar framework to explore the expressive potential of various parameters in the S4 model.
Specifically, we assume a target model that performs well on the intended task and a frozen model, which may be either pretrained or randomly initialized.
Our goal is to identify a parameter-efficient method to update the frozen model so that it becomes functionally equivalent to the target model.
In alignment with Zeng & Lee (), we assume that the frozen model’s capacity is equal to or exceeds that of the target model.
This assumption is based on two main considerations:
(i) analytical tractability, which necessitates that the frozen model must have the potential to match the functionality of the target model,
and (ii) a practical rationale, given that the models typically used in practice are often overparameterized.
Assume that both the target model and the frozen model are S4, with the target model having a hidden state dimension $H_{\star}$ and the frozen model having a hidden state dimension $H\geq H_{\star}$.
Meanwhile, suppose that all the hidden dimensions of both models are valid, meaning that none of the parameter elements are zero.
The target model, frozen model, and the updated model after tuning the parameters on the frozen model can be formulated using discretized parameters $\overline{{\bm{A}}},\overline{{\bm{B}}},{\bm{C}}$ as follows:

$$ (Target model) $\displaystyle f^{\star}({\bm{x}})_{n}=\sum_{m=1}^{n}{\bm{C}}_{\star}\overline{ {\bm{A}}}_{\star}^{m-n}\overline{{\bm{B}}}_{\star}x_{m},\text{ where } \operatorname{diag}(\overline{{\bm{A}}}_{\star}),\overline{{\bm{B}}}_{\star},{ \bm{C}}_{\star}\in{\mathbb{R}}^{H_{\star}},$ (38) (Frozen model) $\displaystyle f_{0}({\bm{x}})_{n}=\sum_{m=1}^{n}{\bm{C}}\overline{{\bm{A}}}^{m -n}\overline{{\bm{B}}}x_{m},\text{ where }\operatorname{diag}(\overline{{\bm{A }}}),\overline{{\bm{B}}},{\bm{C}}\in{\mathbb{R}}^{H},$ (39) (Updated model) $\displaystyle\hat{f}({\bm{x}})_{n}=\sum_{m=1}^{n}\widehat{{\bm{C}}}\widehat{ \overline{{\bm{A}}}}^{m-n}\widehat{\overline{{\bm{B}}}}x_{m},\text{ where } \operatorname{diag}(\widehat{\overline{{\bm{A}}}}),\widehat{\overline{{\bm{B}} }},\widehat{{\bm{C}}}\in{\mathbb{R}}^{H}.$ (40) $$

#### Parameter Efficiency Analysis on S4.

Let ${\mathcal{P}}^{H}$ denote the set of all $H\times H$ permutation matrices.
Given this formulation, we present our first analysis of parameter efficiency for the S4 model in the following lemma.
This analysis is based on the parameters after necessary discretization $(\overline{{\bm{A}}},\overline{{\bm{B}}},{\bm{C}})$.

###### Lemma 0 (Essential Discretized Parameter Set for S4) .

Consider the parameters after discretization, i.e., $\overline{{\bm{A}}},\overline{{\bm{B}}},{\bm{C}}$.
To achieve functional equivalence between the updated model and the target model, i.e., $\hat{f}\equiv f^{\star}$, it is sufficient to tune the following number of parameters:

$$ $\min_{{\bm{P}}\in{\mathcal{P}}^{H}}\overbrace{\left\lVert\left[{\bm{P}}^{\top} (\operatorname{diag}(\overline{{\bm{A}}})\otimes\overline{{\bm{B}}}\otimes{\bm {C}}^{\top})\right]_{(H_{\star}+1):H}\right\rVert_{0}}^{\text{eliminating redundant dimensions}}+\overbrace{\underbrace{\left\lVert\left[{\bm{P}}^{\top} \overline{{\bm{A}}}{\bm{P}}\right]_{1:H_{\star},1:H_{\star}}-\overline{{\bm{A} }}_{\star}\right\rVert_{0}}_{\text{aligning the state matrix}}+\underbrace{ \left\lVert\left[{\bm{P}}^{\top}(\overline{{\bm{B}}}\otimes{\bm{C}}^{\top}) \right]_{1:H_{\star}}-\overline{{\bm{B}}}_{\star}\otimes{\bm{C}}_{\star}^{\top }\right\rVert_{0}}_{\text{aligning input-output interactions}}}^{\text{ aligning used dimensions with target model}}.$ (41) $$

###### Proof of Lemma 3 .

The key idea of this proof is straightforward.
To facilitate the analysis and update the frozen model to be equivalent to the target model, we first equalize the number of hidden state dimensions between the two models. This is achieved by expanding the target model’s ${\bm{A}}_{\star}$, ${\bm{B}}_{\star}$, and ${\bm{C}}_{\star}$ to match the $H$ hidden state dimensions of the frozen model, padding the additional $H-H^{\star}$ dimensions with zeros.

Define $\otimes$ as the element-wise product.
We can express the target model as:

$$ $\displaystyle f^{\star}(\bm{x})_{n}$ $\displaystyle=\sum_{m=1}^{n}\begin{bmatrix}{\bm{C}}_{\star}&{\bm{0}}^{\top} \end{bmatrix}\begin{bmatrix}\overline{{\bm{A}}}_{\star}&{\bm{O}}\\ {\bm{O}}&{\bm{O}}\\ \end{bmatrix}^{n-m}\begin{bmatrix}\overline{{\bm{B}}}_{\star}\\ {\bm{0}}\end{bmatrix}x_{m}$ (42) $\displaystyle=\sum_{m=1}^{n}\operatorname{diag}\left(\begin{bmatrix}\overline{ {\bm{A}}}_{\star}&{\bm{O}}\\ {\bm{O}}&{\bm{O}}\\ \end{bmatrix}\right)^{n-m}\left(\begin{bmatrix}{\bm{C}}_{\star}^{\top}\\ {\bm{0}}\end{bmatrix}\otimes\begin{bmatrix}\overline{{\bm{B}}}_{\star}\\ {\bm{0}}\end{bmatrix}\right)x_{m}$ (43) $$

Consider any permutation matrix ${\bm{P}}\in{\mathcal{P}}^{H}$.
Applying ${\bm{P}}$ to permute the frozen model leaves the model functionally unchanged:

$$ $\displaystyle f_{0}(\bm{x})_{n}$ $\displaystyle=\sum_{m=1}^{n}{\bm{C}}\overline{{\bm{A}}}^{n-m}\overline{{\bm{B} }}x_{m}=\sum_{m=1}^{n}{\bm{C}}{\bm{P}}\left({\bm{P}}^{\top}\overline{{\bm{A}}} {\bm{P}}\right)^{n-m}{\bm{P}}^{\top}\overline{{\bm{B}}}x_{m}$ (44) $\displaystyle=\sum_{m=1}^{n}\operatorname{diag}\left({\bm{P}}^{\top}\overline{ {\bm{A}}}{\bm{P}}\right)^{n-m}\left(({\bm{P}}^{\top}{\bm{C}}^{\top})\otimes({ \bm{P}}^{\top}\overline{{\bm{B}}})\right)x_{m}$ (45) $$

Therefore, to make the updated model equivalent to the target model, we need to update ${\bm{P}}^{\top}\overline{{\bm{A}}}{\bm{P}}$ to align with $\begin{bmatrix}\overline{{\bm{A}}}_{\star}&{\bm{O}}\\
{\bm{O}}&{\bm{O}}\\
\end{bmatrix}$, and $({\bm{P}}^{\top}{\bm{C}}^{\top})\otimes({\bm{P}}^{\top}\overline{{\bm{B}}})$ to align with $\begin{bmatrix}{\bm{C}}_{\star}^{\top}\\
{\bm{0}}\end{bmatrix}\otimes\begin{bmatrix}\overline{{\bm{B}}}_{\star}\\
{\bm{0}}\end{bmatrix}$.
If they are already matching or partially matched for certain entries, no updates are required for those entries; only the unmatched entries need to be updated.
Then, the required trainable parameters for this permutation matrix ${\bm{P}}$ are:

$$ $\left\lVert\left[{\bm{P}}^{\top}(\operatorname{diag}(\overline{{\bm{A}}}) \otimes\overline{{\bm{B}}}\otimes{\bm{C}}^{\top})\right]_{(H_{\star}+1):H} \right\rVert_{0}+\left\lVert\left[{\bm{P}}^{\top}\overline{{\bm{A}}}{\bm{P}} \right]_{1:H_{\star},1:H_{\star}}-\overline{{\bm{A}}}_{\star}\right\rVert_{0}+ \left\lVert\left[{\bm{P}}^{\top}(\overline{{\bm{B}}}\otimes{\bm{C}}^{\top}) \right]_{1:H_{\star}}-\overline{{\bm{B}}}_{\star}\otimes{\bm{C}}_{\star}^{\top }\right\rVert_{0}.$ (46) $$

Optimizing the permutation matrix ${\bm{P}}\in{\mathcal{P}}^{H}$ yields the desired results.
∎

This lemma highlights the significance of identifying essential hidden state dimensions.
The term $\left\lVert\left[{\bm{P}}^{\top}(\operatorname{diag}(\overline{{\bm{A}}})
\otimes\overline{{\bm{B}}}\otimes{\bm{C}}^{\top})\right]_{(H_{\star}+1):H}
\right\rVert_{0}$ underscores the importance of excluding redundant dimensions.
This can be achieved by either directly removing these dimensions from the state matrix $\overline{{\bm{A}}}$, or by updating $\overline{{\bm{B}}}$ or ${\bm{C}}$ to ensure that only the selected hidden state dimensions are utilized during the input transition or output mapping phases.
Once redundant dimensions are filtered out, tuning only the essential dimensions is sufficient to align the updated model with the target model.

Furthermore, based on the lemma, the roles of the input transition vector $\overline{{\bm{B}}}$ and ${\bm{C}}^{\top}$ are nearly identical, as they consistently appear together as the combined term $\overline{{\bm{B}}}\otimes{\bm{C}}^{\top}$, which is also discussed in Gupta et al. ().
Consequently, one could opt to tune either $\overline{{\bm{B}}}$ or ${\bm{C}}$ exclusively or alternatively, split the indices into two groups, tuning $\overline{{\bm{B}}}$ for the first group and ${\bm{C}}$ for the second.
Both vectors indicate how information from different hidden state dimensions is integrated, whereas $\overline{{\bm{A}}}$ plays a distinct role, determining how the hidden states are stored.

In practice, instead of directly using the discretized parameters $\overline{{\bm{A}}},\overline{{\bm{B}}},{\bm{C}}$, S4 is implemented using the continuous parameters ${\bm{A}},{\bm{B}},{\bm{C}}$ with step size $\Delta$.
To provide further practical guidance on parameter tuning, the following two lemmas analyze the parameter efficiency of continuous parameters under different discretization methods:
Two exemplary methods of discretization are bilinear and zero-order hold (ZOH):

$$ $\displaystyle\text{(Bilinear)}\leavevmode\nobreak\ \begin{cases}\overline{{\bm {A}}}=({\bm{I}}-\Delta/2{\bm{A}})^{-1}({\bm{I}}+\Delta/2{\bm{A}})\\ \overline{{\bm{B}}}=({\bm{I}}-\Delta/2{\bm{A}})^{-1}\cdot\Delta{\bm{B}},\end{ cases}\hfill(\text{ZOH})\leavevmode\nobreak\ \begin{cases}\overline{{\bm{A}}}= \exp(\Delta{\bm{A}})\\ \overline{{\bm{B}}}=(\Delta{\bm{A}})^{-1}(\exp(\Delta{\bm{A}})-{\bm{I}})\cdot \Delta{\bm{B}}.\end{cases}$ (47) $$

###### Lemma 7 (Essential Continuous Parameter Set for S4 with Bilinear Discritization) .

Consider the parameters before discretization, i.e., ${\bm{A}},{\bm{B}},{\bm{C}}$, and they are discretized via bilinear discretization.
To achieve functional equivalence between the updated model and the target model, i.e., $\hat{f}\equiv f^{\star}$, it is sufficient to tune the following number of parameters:

$$ $\min_{{\bm{P}}\in{\mathcal{P}}^{H}}\overbrace{\left\lVert\left[\Delta{\bm{P}}^ {\top}(\operatorname{diag}({\bm{I}}+\Delta/2{\bm{A}})\otimes{\bm{B}}\otimes{ \bm{C}}^{\top})\right]_{(H_{\star}+1):H}\right\rVert_{0}}^{\text{eliminating redundant dimensions}}+\overbrace{\underbrace{\left\lVert\left[{\bm{P}}^{\top} {\bm{A}}{\bm{P}}\right]_{1:H_{\star},1:H_{\star}}-{\bm{A}}_{\star}\right\rVert _{0}}_{\text{aligning the state matrix}}+\underbrace{\left\lVert\left[{\bm{P}} ^{\top}({\bm{B}}\otimes{\bm{C}}^{\top})\right]_{1:H_{\star}}-{\bm{B}}_{\star} \otimes{\bm{C}}_{\star}^{\top}\right\rVert_{0}}_{\text{aligning input-output interactions}}}^{\text{aligning used dimensions with target model}}.$ (48) $$

###### Proof of Lemma 7 .

Combining Lemma 3 and the Bilinear discretization method in (47) yields the desired results.
∎

###### Lemma 8 (Essential Continuous Parameter Set for S4 with ZOH Discritization) .

Consider the parameters before discretization, i.e., ${\bm{A}},{\bm{B}},{\bm{C}}$, and they are discretized via ZOH discretization.
To achieve functional equivalence between the updated model and the target model, i.e., $\hat{f}\equiv f^{\star}$, it is sufficient to tune the following number of parameters:

$$ $\min_{{\bm{P}}\in{\mathcal{P}}^{H}}\overbrace{\left\lVert\left[\Delta{\bm{P}}^ {\top}(\operatorname{diag}(\exp(\Delta{\bm{A}})-{\bm{I}})\otimes{\bm{B}} \otimes{\bm{C}}^{\top})\right]_{(H_{\star}+1):H}\right\rVert_{0}}^{\text{ eliminating redundant dimensions}}+\overbrace{\underbrace{\left\lVert\left[{ \bm{P}}^{\top}{\bm{A}}{\bm{P}}\right]_{1:H_{\star},1:H_{\star}}-{\bm{A}}_{ \star}\right\rVert_{0}}_{\text{aligning the state matrix}}+\underbrace{\left \lVert\left[{\bm{P}}^{\top}({\bm{B}}\otimes{\bm{C}}^{\top})\right]_{1:H_{\star }}-{\bm{B}}_{\star}\otimes{\bm{C}}_{\star}^{\top}\right\rVert_{0}}_{\text{ aligning input-output interactions}}}^{\text{aligning used dimensions with target model}}.$ (49) $$

###### Proof of Lemma 8 .

Combining Lemma 3 and the ZOH discretization method in (47) yields the desired results.
∎

The insights provided by Lemma 7 and Lemma 8 are the same as those provided by Lemma 3.
The analysis here supports the second step of SDLoRA presented in Sec. 5.

###### Lemma 0 (Essential Discretized Parameter Set for S4) .

###### Proof of Lemma 3 .

###### Lemma 7 (Essential Continuous Parameter Set for S4 with Bilinear Discritization) .

###### Proof of Lemma 7 .

###### Lemma 8 (Essential Continuous Parameter Set for S4 with ZOH Discritization) .

###### Proof of Lemma 8 .

### D.2 Extension to Deep S4 Models

Our previous analysis focused on single-channel S4 models.
We now expand our investigation to more complex scenarios involving deep S4 models for both target and frozen architectures, incorporating $D$ channels and varying layer depths.
In this section, we consider two PEFT methods: (i) Selective Dimension Tuning (SDT) and (ii) SDLoRA.
The key distinction between SDT and SDLoRA lies in their treatment of linear projection matrices.
SDT exclusively updates the columns of weight matrices corresponding to the updatable channels identified through Alg. 1.
In contrast, SDLoRA employs LoRA to modify these matrices.
It is worth noting that the linear projection matrix updates in SDT are inherently low-rank, making it a specialized case of SDLoRA.
Our analysis starts with SDT, and it automatically applies to SDLoRA.

In this analysis, we assume that each input token $x_{t}$ belongs to ${\mathcal{X}}$, a bounded subset of ${\mathbb{R}}^{D}$, and that the length of the input sequence is finite.
Let the frozen model have $L$ layers, and the target model have $L^{\star}$ layers, where $L\geq L^{\star}$.
Similar to the technique used in Zeng & Lee () and Giannou et al. ().
The basic idea of updating the frozen model to match the functionality of the target model is to utilize every $\lceil L/L^{\star}\rceil$ layers of the frozen model to approximate every layer of the target model.
We start introducing this proof idea from the simplest case where $L^{\star}=1,L=D$.
In this scenario, we can simply choose one different channel to tune and maintain all other channels at zero at every layer.
The outputs from the various channels of the deep S4 layers are then combined through a residual connection.
This proof idea inspires us to perform channel selection and make use of the residual connections, which is the first and third step of SDLoRA presented in Sec. 5.
Building on this idea, we present the following results for when the target model has only $L^{\star}=1$ layer, and $L=D=2$.

###### Lemma 9 .

Consider a $D$-dimensional input sequence.
Assume that the linear layers in the model have linear activation functions.
Using SDT, any deep S4 model with $H$ hidden states per channel and $L$ layers can be updated to accurately present any target one-layer deep S4 model without residual connections, having a reduced hidden state dimension $H^{\star}<H$.
Then this can be achieved by selectively fine-tuning at most $\lceil D/L\rceil$ channels, $H^{\star}$ hidden states, and residual connections at each layer, while additionally fully fine-tuning the linear projection matrix of the last layer only.

###### Proof of Lemma 9 .

In this proof, we start by considering the case where $L=D$.
In this case,
we update a single distinct channel for each layer while setting the other channels to zero.
Essentially, we modify the frozen model so that each layer corresponds to and functions as an individual channel in the target model.
To be more specific, we fully update the first channel in the first layer to match the first channel of the target model, second channel in the second layer to match the second channel of the target model, so on and so forth.

For the $l$-th layer of the frozen model , we append subscript $l$ to all parameters of the deep S4 layer as introduced in (4).
For the $d$-th channel, corresponding notations are denoted with a superscript $(d)$. We define the $t$-th intermediate output token of the $l$-th deep S4 layer as ${\bm{z}}_{l,t}\in{\mathbb{R}}^{D}$. Additionally, the updated S4 module in layer $l$ is denoted as $\widehat{\operatorname{S4}}_{l}$, with $\widehat{\operatorname{S4}}_{l,t}$ referring specifically to the sub-function that outputs the $t$-th token.
Therefore, for the $t$-th intermediate output token of the $l$-th deep S4 layer of the updated model can be written as

$$ $\displaystyle\begin{split}{\bm{z}}_{l,t}&=\widehat{{\bm{W}}}_{l}\cdot\widehat{ \operatorname{S4}}_{l,t}({\bm{z}}_{l-1,1},\ldots,{\bm{z}}_{l-1,t})+\widehat{{ \bm{\beta}}}_{l}+\widehat{{\bm{u}}}_{l}\otimes{\bm{z}}_{l-1,t}\\ &=\widehat{{\bm{W}}}_{l}\cdot\begin{bmatrix}\widehat{\operatorname{S4}}_{l,t}^ {(1)}(z_{l-1,1}^{(1)},\ldots,z_{l-1,t}^{(1)})\\ \vdots\\ \widehat{\operatorname{S4}}_{l,t}^{(D)}(z_{l-1,1}^{(D)},\ldots,z_{l-1,t}^{(D)} )\\ \end{bmatrix}+\widehat{{\bm{\beta}}}_{l}+\widehat{{\bm{u}}}_{l}\otimes{\bm{z}} _{l-1,t},\end{split}$ (50) $$

where $\widehat{{\bm{W}}}_{l}\in{\mathbb{R}}^{D\times D},\widehat{{\bm{\beta}}}_{l}
\in{\mathbb{R}}^{D}$ are the updated weight and biases of the $l$-th layer of the frozen model, and $\widehat{{\bm{u}}}_{l}\in{\mathbb{R}}^{D}$ is the updated residual connection weight of the frozen model.

###### Lemma 9 .

###### Proof of Lemma 9 .

#### For layers l < L = D 𝑙 𝐿 𝐷 l<L=D italic_l < italic_L = italic_D .

We follow the steps provided in Sec. 5 to update the $l$-th layer of the frozen model such that it functionally equivalent to the $l$-th channel of the target model.
For the reader’s convinence, we restate our strategies here:

- •
(Channel Selection) Select $D^{\prime}\leq D$ ($D^{\prime}=1$ here) important channels for making predictions.
Any channel $d$ that is not utilized will have their corresponding ${\bm{C}}^{(d)}$ set to zero, eliminating the need to update parameters for ${\bm{A}}^{(d)}$ and the $d$-th column of ${\bm{W}}$.
To be more specific, we let ${\bm{C}}^{(d)}={\bm{0}}$ for all $d\neq l$ in this scenario.
- •
(Hidden State Selection) Within the selected channels, select $H^{\prime}\leq H$ important hidden states.
For any hidden state that is not used within a selected channel $d$, the corresponding element in ${\bm{C}}^{(d)}$ will be set to zero, thus eliminating the need to tune the corresponding element in ${\bm{A}}^{(d)}$.
To be more specific, we can achieve $\widehat{\operatorname{S4}}_{l,t}^{(l)}(\cdot)=\operatorname{S4}_{\star,t}^{(l
)}(\cdot)$ by Lemma 3.
- •
(Residual and Bias Tuning) Regardless of other selections, SDLoRA consistently tunes the coefficients of residual connections and biases in linear projections, as these components contain a negligible number of parameters.
In this scenario, we let $\widehat{{\bm{\beta}}}_{l}={\bm{0}},\leavevmode\nobreak\ \widehat{{\bm{u}}}_{l
}=\begin{bmatrix}\smash{\underbrace{\begin{matrix}1&\cdots&1\end{matrix}}_{l-1
\text{ elements}}}&0&\smash{\underbrace{\begin{matrix}1&\cdots&1\end{matrix}}_
{D-l\text{ elements}}}\end{bmatrix}^{\top}.$

This construction yields

$$ $\displaystyle{\bm{z}}_{l,t}=\begin{bmatrix}z_{l-1,t}^{(1)}&\ldots&z_{l-1,t}^{( l-1)}&\operatorname{S4}_{\star,t}^{(l)}(z_{l,1}^{(l)},\ldots,z_{l,t}^{(l)})&z_ {l-1,t}^{(l+1)}&\ldots&z_{l-1,t}^{(D)}\end{bmatrix}^{\top}.$ (51) $$

Consequently, only the $l$-th channel is active in the $l$-th layer, while all other layers function as identity mappings, propagating the output of the preceding layer without modification.

#### For layer l = L = D 𝑙 𝐿 𝐷 l=L=D italic_l = italic_L = italic_D .

Based on the setup of the first $L-1$ layers, we have

$$ $\displaystyle{\bm{z}}_{L-1,t}=\begin{bmatrix}\operatorname{S4}_{\star,t}^{(1)} (x^{(1)})&\cdots&\operatorname{S4}_{\star,t}^{(L-1)}(x^{(L-1)})&x^{(L)}\end{ bmatrix}^{\top}.$ (52) $$

For the last layer, we let

$$ $\displaystyle\widehat{{\bm{W}}}_{L}={\bm{W}}_{\star},\quad\widehat{{\bm{\beta} }}_{L}={\bm{\beta}}_{\star},\quad\widehat{{\bm{u}}}_{L}={\bm{0}},$ (53) $\displaystyle\widehat{\operatorname{S4}}_{L,t}^{(L)}(\cdot)=\operatorname{S4}_ {\star,t}^{(L)}(\cdot),\leavevmode\nobreak\ \text{which can be achieved by Lemma\leavevmode\nobreak\ \ref{lemma:discretization}}.$ (54) $$

It is easy to verify that the output of the updated frozen model is identical to the output of the target model, i.e.,

$$ $\displaystyle\bm{y}_{t}={\bm{z}}_{L,t}={\bm{W}}_{\star}\begin{bmatrix} \operatorname{S4}_{\star,t}^{(1)}(x^{(1)})&\cdots&\operatorname{S4}_{\star,t}^ {(L-1)}(x^{(L-1)})&\operatorname{S4}_{\star,t}^{(L)}(x^{(L)})\end{bmatrix}^{ \top}+{\bm{\beta}}_{\star}.$ (55) $$

Thus far, we have demonstrated that the statement holds when $L=D$.
This analysis can be readily extended to cases where $L\neq D$ by tuning $\lceil D/L\rceil$ channels at each layer.
For example, when $L=D/2$, we can tune two channels per layer using a construction similar to the one described above. This generalization completes the proof.
∎

###### Theorem 10 (Expressive Power of SDLoRA on Deep S4 Models) .

Consider a $D$-dimensional input sequence.
Assume that the linear layers in the model have linear activation functions.
Using SDT, any deep S4 model with $H$ hidden states per channel and $L$ layers can be updated to accurately present any target deep S4 model without residual connections, having a reduced hidden state dimension $H^{\star}<H$, and fewer layers $L^{\star}<L$.
This can be achieved by selectively fine-tuning at most $\lceil DL^{\star}/L\rceil$ channels, $H^{\star}$ hidden states, and residual connections at each layer.

###### Proof of Theorem 10 .

We update every $\lceil D/L\rceil$ layers of the frozen model to approximate each layer of the target model.
By applying Lemma 9 iteratively to each set of $\lceil D/L\rceil$ layers, we obtain the desired result.
∎

For reader’s convience, we restate the following statement presented in the main body again here.

###### Theorem 0 (Expressive Power of SDLoRA on Deep S4 Models) .

Consider a $D$-dimensional input sequence.
Assume that the linear layers in the model have linear activation functions.
Using SDLoRA, any deep S4 model with $H$ hidden states per channel and $L$ layers can be updated to accurately present any target deep S4 model without residual connections, having a reduced hidden state dimension $H^{\star}<H$, and fewer layers $L^{\star}<L$.
This can be achieved by selectively fine-tuning at most $\lceil DL^{\star}/L\rceil$ channels, $H^{\star}$ hidden states on SSM modules, applying rank-$\lceil\frac{L}{L^{\star}}\rceil$ updates on linear projection matrices and updating residual connections and biases at each layer, while additionally fully fine-tuning the linear projection matrix of the last layer only.

###### Proof of Theorem 4 .

Since SDT is a special case of SDLoRA, Theorem 10 directly implies the desired statement.
∎

###### Theorem 10 (Expressive Power of SDLoRA on Deep S4 Models) .

###### Proof of Theorem 10 .

###### Theorem 0 (Expressive Power of SDLoRA on Deep S4 Models) .

###### Proof of Theorem 4 .

#### SDLoRA for Mamba.

In the Mamba model, the output mapping vector ${\bm{C}}$ is input-dependent, making it unsuitable for direction modification.
Therefore, we focus our channel and hidden state selection solely on ${\bm{A}}$.
For any channels or hidden states that are not selected, we set the corresponding elements of ${\bm{A}}$ to minimal values, effectively setting the associated entries in $\overline{{\bm{A}}}$ to zero.
For channels and states that are updatable, we update the corresponding entries for ${\bm{A}}$.
However, since ${\bm{B}}^{(d)}$ and ${\bm{C}}^{(d)}$ cannot be directly updated, we modify the corresponding weight matrices that compute these vectors. Specifically, for updatable channels, we update the corresponding columns in ${\bm{W}}_{{\bm{B}}}$ and ${\bm{W}}_{{\bm{C}}}$; for updatable states, we adjust the corresponding rows in these weight matrices.

### D.3 Experiments on Deep S4 Models

#### Synthetic.

For selecting channels and hidden states, we initiate with a warmup learning rate between $1e-2$ and $1e-3$ and conduct 20 warmup iterations. Learning rates are adjusted between $5e-2$, $1e-2$, $5e-3$, and $1e-3$. We apply LoRA with ranks of 2 and 4 to the SSM and with ranks of 4, 8, and 16 to the linear projection matrices. Non-zero states are selected from the sets {4, 8}, and non-zero channels from {8, 16}.

We additionally consider SDT (Selective Dimension Tuning), which is introduced in Sec. D.2, and the results are visualized in Fig. 2.
We observe that SDT even outperforms SDLoRA in this synthetic experiments, demonstrating highly promising performance.
Unfortunetly, we fail to make it work on pretraned Mamba, and identify it as one of the promising future directions.

Figure: Figure 2: Performance comparison between various methods. SDT (Selective Dimension Tuning) is compared to SDLoRA. Unlike SDLoRA, which applies LoRA to linear projection matrices, SDT performs sparse tuning on linear projection matrices by updating only the columns corresponding to channels selected via Alg. 1. Notably, SDT achieves superior performance, matching full fine-tuning results while using only 25% of the parameters, and even surpassing full fine-tuning with more parameters. Extending SDT to real datasets is considered a promising future direction for SDLoRA.
Refer to caption: x2.png

#### CIFAR-10 (Krizhevsky et al., 2009 ) .

We adhere to the preprocessing steps for CIFAR-10 as outlined by Gu et al. (). The LoRA ranks for linear projection matrices are tuned among {1, 2, 4, 8, 16}, and for the S4 component, ranks are set from {1, 2, 4}. Non-zero states are chosen from {8, 12, 16}, and non-zero channels from {48, 64}.
A warmup phase includes 1 epoch with a learning rate of $1e-2$.
For linear projection matrices, LoRA ranks are explored at {2, 4, 8, 16}, and for the SSM, ranks at {2, 4, 8}.
All state dimensions are updated, and channel dimensions considered for updates are {4, 8, 16, 32}.

### D.4 Experiments on Pretrained Mamba

Here, we provide more experiment details.
Unless otherwise stated, our experiment setting is identical to Sec. C.1.
For LoRA, we consider three different LoRA configurations at each layer, involving the following matrices which comprise most of the parameters: $\bm{W}_{\text{out}}$ (output linear projection), $\bm{W}_{{\bm{B}}},\bm{W}_{{\bm{C}}}$ (weight matrices for computing input-dependent ${\bm{B}}_{n},{\bm{C}}_{n}$), and ${\bm{W}}_{\bm{\Delta},\downarrow},{\bm{W}}_{\bm{\Delta},\uparrow}$ (down and up projection matrices of LoRA adapters for computing $\bm{\Delta}$).
The three LoRA application methods are: (i) $\bm{W}_{\text{out}}$, $\bm{W}_{{\bm{B}}},\bm{W}_{{\bm{C}}}$, and ${\bm{W}}_{\bm{\Delta},\downarrow},{\bm{W}}_{\bm{\Delta},\uparrow}$; (ii) $\bm{W}_{\text{out}},\bm{W}_{{\bm{B}}},\bm{W}_{{\bm{C}}}$ and ${\bm{W}}_{\bm{\Delta},\downarrow}$; and (iii) $\bm{W}_{\text{out}}$ and ${\bm{W}}_{\bm{\Delta},\uparrow}$.
For SDLoRA, we set the channel freeze ratio at 99% across all scenarios. We select the state freeze ratio $\alpha$ from the set ${75\%,90\%,95\%}$ and apply LoRA exclusively to $\bm{W}_{\text{out}}$ to maintain a comparable number of trainable parameters.
Residual connections and bias are frozen in this experiment.
For the warmup, we employ 500 data batches to fully train the SSM modules prior to dimension selection, except for the RTE task in GLUE, where we use 250 batches due to its limited dataset size.
Note that the parameters are reverted back after the warmup stage.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	During the process of selective dimension tuning, the authors select the target channels and states based on magnitude, any other metrics have been tried?
2.	Will SDLoRA's training speed be slower compared to vanilla LoRA? How much slower will it be?
3.	What is the accuracy of SDLoRA on a larger data set, such as ImageNet?
4.	Some other advanced parameter-efficient tuning method like DoRA [1] can be adapted to Mamba? or the proposed SDLoRA can be adapted to Jamba?

[1] DoRA: Weight-Decomposed Low-Rank Adaptation
soundness: 3
strengths: 1.	Reasonable theoretical analysis and comprehensive experiments.
2.	The introduced SDLoRA is novel and effective.
3.	Through extensive experiments, the findings in this paper is useful and inspired.
summary: This paper presents the first study on the performance of PEFT methods applied to SSM-based models. Specifically, prompt-based and parameter-based methods are involved. With theoretical analysis and extensive experiments, LoRA tends to achieve better performance. To further improve the performance, this paper introduces SDLoRA, which selectively updates certain channels and states on SSM modules while applying LoRA to linear projection matrices.
weaknesses: 1.	The speed of SDLoRA is nor reported.
2.	Experimental results on larger datasets are needed.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See the weaknesses
soundness: 3
strengths: (1) Efficiency and Scalability: By focusing on selective parameter updates, the SDLoRA method enhances computational efficiency, which is crucial for large-scale models. Experimental results show that SDLoRA consistently outperforms traditional LoRA across several benchmarks, proving its efficacy in SSM architectures.
(2) Adaptability: The proposed SDLoRA method demonstrates adaptability across multiple tasks, including NLP tasks and vision tasks
summary: The paper explores parameter-efficient fine-tuning (PEFT) methods for deep State Space Models (SSMs), especially in the context of language modeling tasks. It investigates the effectiveness of various PEFT methods, such as prompt-based prefix-tuning and Low-Rank Adaptation (LoRA), applied to SSM architectures like Mamba. A new variant called SDLoRA (Selective Dimension LoRA) is proposed in this paper to selectively update channels and states in the SSM modules, aiming to enhance the fine-tuning performance while reducing parameters. The results indicate that SDLoRA outperforms conventional LoRA when fine-tuning SSM-based models.
weaknesses: (1) Limited Applicability Beyond SSMs: The focus on SSMs means SDLoRA may not generalize well to non-SSM architectures or hybrid models such as Transformer models or Transformer-SSM combinations. Its broader applicability to other architectures remains untested.
(2) Parameter Selection: The dimension selection process in SDLoRA relies on parameter magnitudes, which may not be optimal and could benefit from a more sophisticated selection algorithm. And what if the magnitude of each channel changes during the fine-tuning stage?

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Have you looked at all into Hybrid Architectures?
soundness: 2
strengths: The paper is looking at something which definitely needs to be studied, as it is not a foregone conclusion that adapters, prompt tuning, etc... will have the same benefits for SSMs as they do for transformers. A detailed study understanding the different tradeoffs brought on by the SSM architecture should be explored.

The paper has a good mix of theoretical and empirical analysis.
summary: The paper aims to explore the application of LoRA to SSMs. It explores different adapter types as well as different ways of how to apply them to the SSM block. The paper additionally includes theoretical results to justify their choices.
weaknesses: The paper, to me, does not have enough substance. It needs either more detailed theoretical results, which result in some innovation or needs more empirical results which help the community understand how different adapters perform with transformers vs SSMs.

Here are some specifics:
1. Mamba2 is not included in any of the results, the paper should be updated to look at this architecture as well
2. The theoretical analysis is largely only for the S4 block, its not clear to me the conclusions would extend to S6, and aren't convincing to me for that reason.
3. The empirical results are lacking. The SDLoRA module is slightly different application of the standard LoRA block, essentially targeting different layers within the block instead of new a design. Also only two adapters are compared within this work. To have a true empirical study of this, many more experiments need to be conducted, even in the presence of theoretical results.

I think the paper might be more interesting if it were to do something like the following:
- Compare many adapters in a standardized for Transformers, Mamba1, and Mamba2
- Isolate differences in which adapters perform well for each class of model
- See if these insights give rise to a new adapter design specifically for this model class
- Draw theoretical results to try to explain why certain adapters perform differently than in Transformers

The current structure doesn't feel as though it is contributing much

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. Could the authors comment on how SDLoRA benefit to hybrid models that combine SSMs and Transformers? 
2. On Mamba-130M, the authors use GLUE and DART benchmarks, while on Mamba-1.4B, they use SAMSum and Spider benchmarks. Could the authors elaborate on the considerations behind this benchmark selection strategy?
3. In Table 4, SDLoRA outperforms Full Fine-Tuning. Was this outcome expected, and if so, could the authors provide insights into why this might be the case? Additionally, have the authors considered conducting experiments on more challenging visual tasks, such as Imagenet-1k, to further validate the effectiveness of SDLoRA?
soundness: 3
strengths: The paper systematically analyzes how existing PEFT methods perform on SSM-based models and which modules are most effective for fine-tuning, revealing that prompt-based methods like prefix-tuning are no longer effective. Additionally, it shows that applying LoRA to linear projection matrices without modifying SSM modules yields the best results. The paper further introduces SDLoRA, a novel approach to parameter-efficient fine-tuning (PEFT) for SSMs. This method's innovativeness lies in its selective dimension updating strategy within SSM modules. The document also references various datasets used for evaluating the proposed methods, and the clarity of the paper is generally good.
summary: This paper presents a systematic study on the application of parameter-efficient fine-tuning (PEFT) methods to Deep State Space Models (SSMs). The paper reveals that prompt-based methods are less effective for SSMs, while LoRA remains effective. The authors further propose SDLoRA, which selectively updates certain channels and states on SSM modules while applying LoRA to linear projection matrices, demonstrating improved performance over standard LoRA.
weaknesses: 1. While the paper presents a novel approach for parameter-efficient fine-tuning SSMs, the innovation seems to be more incremental than groundbreaking. 
2. The paper lacks a detailed analysis of the computational overhead from the selective dimension tuning process. This is crucial to understanding the trade-offs between parameter reduction and computational efficiency in SDLoRA.
3. The paper would benefit from a  detailed analysis of SDLORA with hybrid models that combine SSMs and Transformers, as these models are becoming increasingly popular and have shown promise in various domains.

Meta-review: This paper presents a systematic investigation into parameter-efficient fine-tuning (PEFT) methods for Deep State Space Models (SSMs), with a particular focus on language modeling tasks. They introduce SDLoRA, a novel approach that selectively updates certain channels and states on SSM modules while applying LoRA to linear projection matrices, showing improved performance over standard LoRA.

One of the main concerns raised by the reviewers was the limited applicability of the proposed SDLoRA method beyond SSMs. While the authors have demonstrated the effectiveness of SDLoRA on hybrid models like Jamba, more comprehensive testing on a broader range of architectures would be beneficial. The comparison with existing PEFT methods was noted as insufficient by some reviewers. While the authors defended their approach, a more comprehensive comparison could have been made to better position the proposed method against existing benchmarks. The complexity of the proposed model, particularly with components like the selective dimension tuning, raised concerns about its practicality, which was not fully addressed in the rebuttal.

The reviewers expressed mixed opinions, with some leaning towards acceptance due to the novelty and potential of the work, especially in the context of SSMs, while others remained skeptical due to the perceived incremental innovation and the lack of broader architectural testing. The paper shows promise in advancing the field of parameter-efficient fine-tuning for SSMs but would benefit from further refinement, particularly in addressing the concerns about broader applicability and providing more robust comparisons with existing methods.

This paper received an average score that, while close to the acceptance threshold, is not competitive among this year's submissions. Given the balance of strengths and weaknesses, the final recommendation is to reject this submission in its current form.

Decision: Reject