MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba

## Abstract

Abstract An ecosystem of Transformer-based models has been established by building large models with extensive data.
Parameter-efficient fine-tuning (PEFT) is a crucial technology for deploying these models to downstream tasks with minimal cost while achieving effective performance. Recently, Mamba, a State Space Model (SSM)-based model, has attracted attention as a potential alternative to Transformers. While many large-scale Mamba-based models have been proposed, efficiently adapting pre-trained Mamba-based models to downstream tasks remains unexplored.
In this paper, we conduct an exploratory analysis of PEFT methods for Mamba. We investigate the effectiveness of existing PEFT methods for Transformers when applied to Mamba. We also modify these methods to better align with the Mamba architecture. Additionally, we propose new Mamba-specific PEFT methods that leverage the distinctive structure of Mamba. Our experiments indicate that PEFT performs more effectively for Mamba than Transformers. Lastly, we demonstrate how to effectively combine multiple PEFT methods and provide a framework that outperforms previous works. To ensure reproducibility, we will release the code after publication.

## 1 Introduction

Modern large-scale models, also known as Foundation Models, are heavily based on Transformers (Vaswani, ). Transformer-based pre-trained models span diverse domains such as language, vision, and multi-modal applications.
Despite their widespread use, Transformers have a notable drawback: their computational inefficiency with long sequences. The computational complexity of the attention module scales with the square of the sequence length.

To address this fundamental drawback, Gu & Dao () proposed Mamba, a linear-time sequence model that leverages the strengths of State Space Models (SSMs). While Transformers are constructed from attention modules, Mamba is based on the SSM architecture, allowing it to handle long sequences more efficiently. Additionally, Mamba has been shown to perform better than Transformers with the same number of parameters on major tasks such as natural language processing (NLP) (Gu & Dao, ) and computer vision (CV) (Zhu et al., ). This fact makes Mamba stand out from other post-Transformer architectures with sub-square time complexity (Peng et al., ; Sun et al., ). Mamba-based models have been proposed across a wide range of domains (Behrouz & Hashemi, ; Liang et al., ; Zhang et al., ; Li et al., ). We believe that Mamba has the potential to go beyond the Transformer ecosystem.

Parameter-efficient fine-tuning (PEFT) is essential for adapting such large-scale models to downstream tasks. Fine-tuning all parameters of these models results in high computational costs. PEFT enables additional training for large-scale Transformers with limited computing resources and data.
Early examples of PEFT include the fine-tuning of pre-trained language models for NLP tasks (Houlsby et al., ; Hu et al., ). Subsequently, it has been extensively adopted across a wide range of applications (Yeh et al., ; Wang et al., ; ).
While many PEFT methods have been extensively studied for Transformers (Lialin et al., ; Han et al., ), research on PEFT methods for Mamba remains limited.

In this paper, we provide an exploratory and comprehensive investigation of PEFT for Mamba. First, we adapt representative PEFT methods used in Transformers to the Mamba architecture and conduct extensive experiments. We also propose new PEFT methods specific to the Mamba architecture.

Figure: Figure 1: An overview of our proposed MambaPEFT. We investigate, improve, and propose 20 variations of seven PEFT methods for Mamba and search for the best combination.
Refer to caption: x1.png

In the experiments, we benchmarked Mamba using PEFT methods, including seven main methods and a total of 20 derived variations (see Figure 1). Our benchmarks indicate that PEFT is more crucial for Mamba than for Transformers, with several methods outperforming the PEFT methods used for Transformers.
Additionally, we demonstrate that these PEFT methods can be combined to surpass the performance of individual methods. We propose an efficient search technique to identify optimal PEFT combinations and hyperparameters. Unlike existing works that focus on specific high-performance methods, we explore a wide variety of PEFT methods. This exploration reveals suitable PEFT combinations and shows that merely combining high-performing methods is not sufficient.

The main contributions are as follows. First, to the best of our knowledge, we perform the first extensive and comprehensive benchmarking of PEFT methods for Mamba, including proposed Mamba-specific methods that are distinct from all PEFT methods for Transformers. Second, we propose a framework for achieving higher performance by combining multiple PEFT methods, which are obtained through our efficient search technique. Third, our results indicate that PEFT is more effective for Mamba than for Transformers, and several Mamba-specific phenomena are discovered through the experiments.

## 2 Related work and preliminary

In this section, we review related work and present the motivation behind our research. We begin with an introduction of Mamba. Subsequently, we discuss PEFT methods for Transformers. Finally, we highlight that research on PEFT for Mamba is limited.

### 2.1 Mamba

SSMs are powerful methods for modeling time-dependent dynamic systems. It was developed and analyzed in detail to address the signal processing technique classically known as the Kalman filter (Kalman, ) and has been used to date in a wide range of fields,
such as control engineering (Maciejowski & Huzmezan, ).
An SSM (Gu et al., ; ) takes an input $X_{t}$, converts it to a hidden state $H_{t}$, and then outputs $Y_{t}$.
This procedure at each time step $t$ is

$$ $H_{t}=A_{t}H_{t-1}+B_{t}X_{t},\quad Y_{t}=C_{t}H_{t}+D_{t}X_{t}.$ (1) $$

As this model describes the relationship between continuous quantities, it must be described to deal with discrete quantities such as NLP. During step size $d$, the input is assumed to be invariant and the well-known theory of zero-order hold is applied. Putting

$$ $\overline{A}:=(I-d/2\cdot A)^{-1}(I+d/2\cdot A),\quad\overline{B}:=(I-d/2\cdot A )^{-1}dB,\quad\overline{C}:=C,\quad\overline{D}:=D$ (2) $$

allows us to obtain the final discrete representation:

$$ $H_{t}=\overline{A}H_{t-1}+\overline{B}X_{t},\quad Y_{t}=\overline{C}H_{t}+ \overline{D}X_{t}.$ (3) $$

There have been numerous efforts to apply this mechanism to deep learning (e.g. convolutional representations (Gu et al., ) and the initialization by Hippo matrices (Gu et al., )).
As a result, a time series model, Mamba (Gu & Dao, ), with a dynamic inference system called Selective State Space Model (S6), was developed as,

$$ $A_{t}=\exp(-\Delta_{t}A),\ B_{t}=\Delta_{t}W_{B}X_{t},\ C_{t}=W_{C}X_{t},\ D_{ t}=W_{D}X_{t}$ (4) $$

where $\Delta_{t}=\mathrm{softplus}(W_{\Delta}(W_{X}X_{t})+b_{\Delta})$ for any $t$, $W_{\Delta}$. The $W_{X}$, $W_{B}$, $W_{C}$, $W_{D}$, and $A$ are trained matrices, and $b_{\Delta}$ is a trained bias.

Impressive results of Mamba in NLP have inspired researchers to adapt it to visual tasks.
Vim (Zhu et al., ) is a pioneering work in this area.
Similar to ViT (Dosovitskiy et al., ), it divides images into patches and then inputs the patch sequences into SSMs.
There are other Mamba-based methods for visual tasks. For example, VMamba (Liu et al., ) is a hybrid approach combining Mamba with 2D Convolutions. MambaVision (Hatamizadeh & Kautz, ) further integrates Attention.
In this paper, we conduct experiments with Vim, which is a variant of the vanilla Mamba.

### 2.2 Parameter-Efficient Fine-Tuning for Transformer

PEFT methods have been actively studied due to the large model size of Transformers extensively in both the NLP and CV communities. These studies can be categorized into four methods; partial, additive, reparametrization, and hybrid according to Lialin et al. (); Han et al. ().

Partial-tuning methods.
While usual fine-tuning updates all parameters in the network, Partial-tuning updates a part of them.
BitFit (Zaken et al., ) is a successful method in this category which tunes only the bias terms of the linear layers in Transformers. However, most of the linear layers in Mamba do not have the bias terms, which implies that it cannot be applied a priori.

Additive methods.
Additive methods add a small number of additional parameters or small networks to be fine-tuned. They can be further divided into additive adapter and additive token methods.
The former adds adapter modules whose rank $r$ is reduced from the input dimension $d$ $(r<<d)$ by a bottleneck structure. Adapter (Houlsby et al., ) attaches the adapter in series with the feed-forward network (FFN) module in Transformers. In the following methods, Adapter+ (Steitz & Roth, ) improves the position of the input feature acquisition to the adapter. ParallelAdapter (He et al., ) and AdaptFormer (Chen et al., ) improve performance by inserting the adapter in parallel to the FFN. We adopt ParallelAdapter for Mamba because it is successful in both language (He et al., ) and vision (Chen et al., ) tasks.
It attaches the adapter

$$ $x_{\ell}^{\prime}=\mathrm{FFN}(x_{\ell})+s\cdot\mathrm{ReLU}(x_{\ell}\cdot W_{ \mathrm{down}})\cdot W_{\mathrm{up}},$ (5) $$

where the second term is the adapter with additional parameters $W_{\mathrm{down}}\in\mathbb{R}^{d\times r}$ and $W_{\mathrm{up}}\in\mathbb{R}^{r\times d}$, and a scaling hyperparameter $s$.
In the latter category, Prompt-tuning (Liu et al., ) (resp. Prefix-tuning (Li & Liang, )) adds learnable soft tokens to the input of the network (resp. soft tokens inside each Attention layer).
Successive methods improve how to embed soft tokens (Liu et al., ; Razdaibiedina et al., ; Zhang et al., ). Though prompt-tuning for ViT adds learnable tokens at the beginning of the input (Jia et al., ; Wang et al., ), it is unclear where the token should be added for Mamba because the order of tokens makes sense in SSM. In fact, Vim (Zhu et al., ) adds a class token in the middle of the input sequence. Hence, we investigate multiple choices of soft token insertion.

Reparameterization methods.
Although reparameterization methods add low-rank adapters to Linear layers similar to the additive adapter methods, they adopt adapters that can be merged into the weights of the original Linear layer at the inference time. Due to this restriction, activation cannot be used in the adapter, while it eliminates the inference time overhead.
The most notable one is LoRA (Hu et al., ), a method that reparameterizes the pre-trained Linear weights $W$ as $W^{\prime}=W+s\cdot W_{\mathrm{down}}\cdot W_{\mathrm{up}}$.
While subsequent studies have improved the efficiency by finding ways to reparameterize the weights (Hayou et al., ; Liu et al., ; Jiang et al., ), we first investigate how many ranks of the adapter should be attached to which linear weights in Mamba. In this regard, we will use a simple LoRA with few hyperparameters.

Hybrid methods.
Hybrid methods use multiple PEFT methods. While many approaches manually combine these methods (Mao et al., ; He et al., ; Karimi Mahabadi et al., ), several methods automatically tune the combination of several PEFT methods. $S_{4}$ (Chen et al., ) utilizes random search while changing the dimension of the search space. AutoPEFT (Zhou et al., ) and NOAH (Zhang et al., ) train a supernet containing all possible PEFT methods and discover the optimal combination using an evolutionary algorithm.

Supernet-based methods can only be used to search for each task individually. However, we aim to find the optimal combination for Mamba that can be generalized across multiple tasks, making it useful for various future applications. Hence, we propose an efficient search technique to find a compact combination of PEFT methods. This is necessary because we have many more PEFT methods and hyperparameters than previous works, resulting in a large search space.

### 2.3 Parameter-Efficient Fine-Tuning for Mamba

In contrast to the case of Transformers, there are limited studies on PEFT for Mamba. To the best of our knowledge, the only study in this context is Halloran et al. (), which applied the same rank of LoRA to all the linear layers in Mamba.
With this motivation, we conduct an exploratory investigation and provide benchmarks of PEFT methods for Mamba.

## 3 Investigation of PEFT methods for Mamba and beyond

In this section, we explore PEFT methods for the Mamba architecture. First, we discuss how existing PEFT methods for Transformers can be adapted to Mamba. Next, we present methods that have been improved and modified based on the characteristics of Mamba. Finally, we propose new PEFT methods specifically for Mamba and hybrid PEFT methods to search optimal combinations efficiently.

### 3.1 Simple adaptation of existing PEFT methods to Mamba

Some PEFT methods can be directly applied to Mamba because they are network architecture-independent. For example, ParallelAdapter attaches a parallel adapter to each FFN in Transformers. The counterpart of the FFN in Mamba is the out_proj layer, and hence we attach it to the $out_proj$.

LoRA is also a network architecture-independent method and can be applied directly to Mamba. However, it is still unclear which Linear layer should be targeted and what hyperparameter values should be used. We individually investigate LoRA on each module as LoRA(embedding), LoRA($in_proj$), LoRA($x_proj$), LoRA($dt_proj$), and LoRA($out_proj$) to clarify the appropriate way to apply LoRA to Mamba (see Figure 1).

### 3.2 Re-designing and improving existing PEFT methods for Mamba

Figure: Figure 2: (a) Prefix-tuning designed for Transformer. It can’t be applied to Mamba. (b) The proposed Affix-tuning, which we re-design for Mamba. It discards prefixes after SSM. This design allows us to insert affix tokens at arbitrary locations. (c) Additional-scan that we design for Mamba. In this method, we add a learnable dimension to the hidden state in SSM.
Refer to caption: x2.png

Partial LoRA. Mamba is a network characterized by numerous intermediate features with diverse properties, such as $X$, $Z$, $dt$, $B$, $C$, and so on (see Figure 1). In the previously mentioned normal LoRA, the inputs for multiple intermediate features are compressed together in the narrow rank of LoRA, even though they have different properties.
Therefore, we now develop Partial LoRAs
LoRAp($X$), LoRAp($Z$), LoRAp($dt$), LoRAp($B$), and LoRAp($C$), where we apply LoRA to only a part of the weights in the linear layer according to the output features.
Since the dimensions of linear layers vary widely in Mamba (e.g. from 16 to 2048 in Mamba-1.3B), we also investigate whether there is an optimal dimension per layer or not to attach LoRA.

Prompt-tuning. Prompt-tuning can be applied directly to Mamba. As with Transformer, we simply add soft prompts to the input sequence. In normal Prompt-tuning, a prompt is inserted at the beginning of the input sequence. For Vim, however, we adjust the insertion position. This is because, unlike ViT, Mamba is a time-series model and works differently depending on where the prompt is inserted. In fact, Vim improved accuracy by inserting the class token in the middle of the sequence rather than the beginning. In addition, Mamba Register (Darcet et al., ) found it is preferable to insert the register tokens at equal intervals. Hence, in this paper, we construct three prompt types; prefix (resp. infix, suffix) type, inserting prompt tokens at the beginning (resp. with equal intervals, at the end).

Affix-tuning. We cannot apply Prefix-tuning directly to time series models such as Mamba, since it is designed for the Attention mechanism based on query, key, and value. As shown in Figure 2, by adding soft tokens to only $K$ and $V$, Prefix-tuning successfully adds new information to all tokens in the input sequence without breaking the positional relationship of the original input sequence.
Motivated by these observations, we propose a PEFT method that adds soft tokens before input to the SSMs and discards the output tokens of the inserted position (see Figure 2). This preserves the positional relationship between input and output tokens even when tokens are added. Furthermore, for Vim, we adjust the insertion position in the same way as for Prompt-tuning. Since the name Prefix-tuning can be misleading, we generalize by calling it Affix-tuning.

### 3.3 New PEFT methods designed for Mamba

Partial-tuning.
BitFit (Zaken et al., ) is effective among Partial-tuning methods because it fine-tunes bias terms with low degrees of freedom and they do not break the pre-trained model.
Different situations for the Partial-tuning in Mamba is that, unlike Transformers, it does not utilize bias terms in linear layers without $dt_proj$, while it has various other parameters with low degrees of freedom such as the parameters $A$, $D$, and the weights of the causal convolution whose dimension is one.
Based on this situation, we construct Partial-tuning for Mamba, named Bias-tuning, A-tuning, D-tuning, and Conv1d-tuning, which fine-tune the previously mentioned parameters.
For Vim, we also make positional embedding and class tokens learnable. In addition, we enhance our method from conventional Partial-tuning by modifying the weight decay. Specifically, instead of the usual weight decay of $|W|^{2}$, we propose applying a weight decay of $|W-W_{\mathrm{pretrain}}|^{2}$ to preserve the pre-trained model.

Additional-scan. We propose a new efficient PEFT with fewer parameters by leveraging the SSM architecture. By increasing the state dimension of the SSM, we aim to store information in it to adapt to new tasks, called Additional-scan. It corresponds to additive methods. Specifically, we increase the state dimension of the input parameters of SSMs, $A$, $B$, and $C$, by $N^{\prime}$. Eq. 3
then becomes as follows:

$$ $\overline{A}^{[T,L,N+\mathbf{\underline{N^{\prime}}}]}=\exp(-\Delta^{[T,L,1]} \circ A^{[1,L,N+\mathbf{\underline{N^{\prime}}}]}),\quad\overline{B}^{[T,L,N+ \mathbf{\underline{N^{\prime}}}]}=\Delta^{[T,L,1]}\circ B^{[T,1,N+\mathbf{ \underline{N^{\prime}}}]},$ (6) $$

where $T$, $L$, and $N$ are the token length, hidden size, and state dimension of the SSM, and $P^{[d_{1},d_{2},d_{3}]}$ represents $P\in\mathbb{R}^{d_{1}\times d_{2}\times d_{3}}$. Then, the core recurrent computation in SSMs becomes

$$ $\begin{split}H_{t}^{[L,N+\mathbf{\underline{N^{\prime}}}]}&=\overline{A}_{t}^{ [L,N+\mathbf{\underline{N^{\prime}}}]}\circ H_{t-1}^{[L,N+\mathbf{\underline{N ^{\prime}}}]}+\overline{B}_{t}^{[L,N+\mathbf{\underline{N^{\prime}}}]}\circ X_ {t}^{[D,1]}\\ Y_{t}^{[L,1]}&=H_{t}^{[L,N+\mathbf{\underline{N^{\prime}}}]}\cdot(C_{t}^{\top} )^{[N+\mathbf{\underline{N^{\prime}}},1]}+D^{[L,1]}.\end{split}$ (7) $$

In Mamba, a diagonal SSM is employed and each hidden dimension forms an independent recurrence. Hence, we can write down more about one hidden state dimension as follows;

$$ $\begin{split}[h_{t,1},\>...,\>h_{t,N},\>h_{t,N+\mathbf{\underline{1}}},\>..., \>h_{t,N+\mathbf{\underline{N^{\prime}}}}]=[&\exp(-\delta_{t,0}a_{0,1})h_{t-1, 1}+\delta_{t,0}b_{t,1}x_{t,0},\>...,\\ &\exp(-\delta_{t,0}a_{0,{N}})h_{t-1,N}+\delta_{t,0}b_{t,N}x_{t,0},\\ &\exp(-\delta_{t,0}\mathbf{\underline{a_{0,N+\mathbf{1}}}})h_{t-1,N+\mathbf{ \underline{1}}}+\delta_{t,0}\mathbf{\underline{b_{t,N+\mathbf{1}}}}x_{t,0},\>. ..,\\ &\exp(-\delta_{t,0}\mathbf{\underline{a_{0,N+\mathbf{N^{\prime}}}}})h_{t-1,N+ \mathbf{\underline{N^{\prime}}}}+\delta_{t,0}\mathbf{\underline{b_{t,N+\mathbf {N^{\prime}}}}}x_{t,0}].\end{split}$ (8) $$

It can be observed that the added parameters do not affect the original hidden state, $h_{t,1},\>...,\>h_{t,N}$. This enables us to store new information with additional selective scan without affecting the pre-trained hidden memory and selective scan mechanism.
To add new dimension to $A$, $B$, and $C$, we add additional trainable parameters to $A$ and to the Linear $x_proj$ that generates $B$ and $C$ (see Figure 2).
The parameter $A$ in SSM needs careful initialization (Gu et al., ), known as the Hippo theory. Mamba uses S4D initialization (Gu et al., ), $A_{*,*,n}=n$,
while in our case, the other $A$ values have already been changed by training, which implies that S4D initialization is not theoretically better. Thus, we propose an initialization of $A$ for Additional-scan. Specifically, the additional parameters of $A$ are initialized with the values of neighborhood pre-trained $A$. It is simple, while stable training is achieved.
Accounting for eq. 8 and the pre-trained model use S4D initialization, the top of the state dimension is intended for long-term memory or selective scan from distant tokens, while the bottom is intended for short-term memory or selective scan from neighbor tokens.
Thus, when fine-tuning, the suitable position of the additional state dimension is checked whether at the bottom as in eq. 8 or at the top.

### 3.4 Hybrid PEFT Search

We have introduced the primary seven methods as follows: ParallelAdapter, LoRA, Partial LoRA, Prompt-tuning, Affix-tuning, Partial-tuning, and Additional-scan. Specifically, there are five variations of LoRA, five variations of Partial LoRA, and six variations of Partial-tuning, making a total of 20 methods. This variety motivated us to propose a hybrid PEFT for Mamba, which combines multiple PEFT methods to enhance performance. There are three key elements to consider in our hybrid PEFT: the combination of PEFT methods, the hyperparameters of each PEFT method (e.g., a rank of LoRA), and the training hyperparameters for each method (e.g., learning rate). Exploring them simultaneously results in an enormous search space compared to the search space of previous works which target only several PEFT methods. Furthermore, the hyperparameters have a conditional search space, meaning they are only considered if the corresponding PEFT method is active. Thus, we propose a two-step approach that explicitly divides the problem into two parts to improve efficiency in the search process.

In the first step, we restrict the search space to combinations of PEFT methods with minimal trainable parameters within each method, i.e., we explore whether to activate each PEFT method with fixed hyperparameters with minimal trainable parameters. This phase is based on our observation that performance degrades when too many PEFT methods or trainable parameters are added.
In the second step, based on the PEFT method combination suggested by the first step, we greedily search the hyperparameters of both PEFT methods and their training. Since this phase inevitably increases the number of trainable parameters, we also include an option to remove a specific PEFT method. This approach helps to prevent excessive parameter growth and allows us to allocate more parameters to the more critical methods. We utilize the TPE algorithm (Bergstra et al., ) implemented in Optuna (Akiba et al., ) for the search in each step. Detailed algorithm is provided in Appendix B.

## 4 Experiments and discussion

In this section, we present our experiments on 7 PEFT methods with 20 variations. We conduct experiments on both image and language tasks. Our methods are compared with state-of-the-art methods for Transformers. We provide detailed results and analysis based on these comprehensive experiments.

**Table 1: Test accuracy on the VTAB-1k benchmark pre-trained on ImageNet-1K using the training technique of DeiT (Touvron et al., ). The best hyperparameter settings maximizing the averaged accuracy of six datasets written in the ablation studies are used for each PEFT method. The number of trainable parameters are on the Cifar100 task. The best performance is highlighted in bold, and the second-best performance is underlined.**
| Model | Method | #Params (K) | Natural | Specialized | Structured | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| ViT-S | Scratch | 21,704 | 10.66 | 56.12 | 24.83 | 26.20 |
| Full | 21,704 | 51.79 | 72.79 | 45.27 | 53.47 |  |
| Linear Proving | 9 | 60.87 | 78.13 | 30.57 | 51.74 |  |
| \cdashline2-7 | FacT-TK | 16 | 72.87 | 82.34 | 54.10 | 66.96 |
|  | LoRA | 628 | 73.60 | 82.22 | 57.61 | 68.68 |
|  | Adaptformer | 333 | 73.63 | 83.15 | 57.80 | 68.97 |
|  | Adapter+ | 122 | 74.68 | 83.57 | 58.82 | 69.87 |
| Vim-S | Scratch | 25,450 | 8.33 | 49.87 | 28.16 | 25.42 |
| Full | 25,450 | 59.35 | 68.74 | 34.39 | 47.08 |  |
| Linear Probing | 9 | 62.50 | 77.25 | 31.97 | 52.75 |  |
| \cdashline2-7 | CLS-token-tuning | 9 | 62.50 | 77.20 | 32.20 | 52.84 |
|  | Pos-embed-tuning | 84 | 64.25 | 74.60 | 39.77 | 56.12 |
|  | Bias-tuning | 37 | 68.94 | 79.16 | 45.05 | 61.03 |
|  | D-tuning | 45 | 67.14 | 78.56 | 40.10 | 58.16 |
|  | A-tuning | 598 | 72.01 | 80.72 | 49.59 | 64.40 |
|  | Conv1d-tuning | 156 | 74.33 | 82.45 | 57.83 | 69.09 |
| \cdashline2-7 | Prompt-tuning (w/o proj) | 12 | 69.92 | 79.20 | 47.75 | 62.54 |
|  | Prompt-tuning | 307 | 63.19 | 77.88 | 35.82 | 54.76 |
|  | Affix-tuning (w/o proj) | 230 | 63.90 | 77.66 | 34.21 | 54.30 |
|  | Affix-tuning | 117,000 | 75.84 | 83.29 | 58.94 | 70.29 |
|  | Additional-scan | 672 | 74.63 | 82.68 | 56.40 | 68.65 |
|  | ParallelAdapter | 663 | 76.10 | 83.97 | 59.97 | 70.96 |
| \cdashline2-7 | LoRA(embed) | 45 | 64.66 | 77.53 | 43.83 | 58.60 |
|  | LoRA(x_proj) | 2,540 | 74.41 | 81.92 | 54.88 | 67.77 |
|  | LoRA(dt_proj) | 2,442 | 75.35 | 83.05 | 57.12 | 69.30 |
|  | LoRA(out_proj) | 2,663 | 76.42 | 83.96 | 60.08 | 71.12 |
|  | LoRA(in_proj) | 1,483 | 76.58 | 84.08 | 60.16 | 71.25 |
|  | LoRAp(d) | 2,442 | 73.25 | 80.91 | 50.93 | 65.46 |
|  | LoRAp(C) | 2,417 | 72.78 | 81.57 | 51.35 | 65.61 |
|  | LoRAp(B) | 2,417 | 72.95 | 81.66 | 52.26 | 66.07 |
|  | LoRAp(Z) | 1,778 | 76.15 | 84.26 | 59.72 | 70.94 |
|  | LoRAp(X) | 1,778 | 76.64 | 83.89 | 60.84 | 71.52 |
| \cdashline2-7 | All (w/ proj) | 119,765 | 74.67 | 82.96 | 53.92 | 67.68 |
|  | Hybrid (w/ proj) | 117,236 | 77.00 | 84.41 | 61.55 | 72.05 |
|  | Hybrid (w/o proj) | 1,044 | 76.85 | 84.42 | 61.06 | 71.80 |

### 4.1 Experimental settings

Models. We use Vim-S (Zhu et al., ) as a base model in our experiments. We also experiment with ViT (Dosovitskiy et al., ) for comparison. We adopt pre-trained weights trained with ImageNet-1k (Deng et al., ) dataset using DeIT (Touvron et al., ) training framework in all models.

Dataset. We conduct our evaluation on the VTAB-1k image classification dataset (Zhai et al., ). This dataset contains tasks from 19 domains. These tasks are grouped into three categories: Natural, Specialized, and Structured. For each task, 1000 images are used for training.

Baselines. We adopt LoRA (Hu et al., ), Adaptformer (Chen et al., ), FacT-TK (Jie & Deng, ), and Adapter+ (Steitz & Roth, ) as existing PEFT methods for Transformers. For Mamba, we evaluate the 20 PEFT methods explained in Section 3. As a baseline for Hybrid PEFT, we use a combination of all PEFT methods in the experiments.

Implementation Details. We follow the setup of Jie & Deng () in our experiments, using AdamW optimizer (Loshchilov & Hutter, ) and training the model for 100 epochs. The learning rate is set to 1e-3, with a cosine scheduler and a warmup period of 10 epochs. A weight decay with 1e-4 magnitude is applied. We do not perform data augmentation. For ablation studies, we use the average accuracy on six tasks: CIFAR-100, Sun397, Camelyon, Retinopathy, Clevr-Count, and sNORB-Elev.

### 4.2 Results for individual PEFT methods

Overall Results.
Table 1 shows the performance with hyperparameters that maximize accuracy.
Many PEFT methods for Vim outperform the state-of-the-art ones for ViT.
Interestingly, the PEFT methods for ViT start over-fitting early when the number of trainable parameters is increased and are unable to improve accuracy. On the other hand, Vim continues to improve its accuracy even with more trainable parameters.
This is probably due to the highly modular structure of Mamba, which prevents its pre-trained memory from being corrupted by additional parameters.
In addition, Vim has a greater performance improvement margin between full fine-tuning and PEFT than ViT has.
These results suggest that PEFT is more effective for Mamba than Transformer.

Figure: (a)

**Table 2: Ablation studies. (a) The position and number of tokens in Affix-tuning with projection. If the position is “Before”, tokens are inserted before the in_proj layer. (b) The initialization method in Additional-scan. (c) The performance improvement in Partial-tuning with our proposed Weight Decay.**
| Position | Type | #Tokens | Acc. (%) |
| --- | --- | --- | --- |
| Before | Prefix | 1 | 23.85 |
| After | Prefix | 1 | 62.91 |
| After | Infix | 1 | 61.21 |
| After | Prefix | 2 | 63.36 |
| After | Prefix | 3 | 63.51 |
| After | Prefix | 4 | 56.57 |

LoRA. In our experiments, LoRA turns out to be one of the most effective PEFT methods with Vim. As shown in Table 1, PEFT methods perform well regardless of where LoRA is applied. The best performance is achieved with the partial LoRA, LoRAp(X). This result stems from our detailed investigation, indicating that the effectiveness of LoRA depends on the nature of the output features.

Another notable finding regarding LoRA is that when it is attached to a linear weight with a small dimension, the accuracy continues to increase even if the rank of LoRA exceeds the full-rank, as shown in Figure 3. Full-rank refers to the smaller value of the input dimension or output dimension. This phenomenon is likely because the effective degrees of freedom are bounded by the full-rank.
It follows that adding more parameters does not cause overfitting. Instead, it positively contributes to over-parameterization, similar to the findings in Ding et al. (). More details regarding the LoRA on large lineaer weights are described in Appendix Section A.1.

Affix-tuning and Prompt-tuning. The ablation studies on affix-tuning are shown in Table 2. First, our Affix-tuning has multiple choices about where to add soft tokens because it is architecture-independent compared to Prefix-tuning for transformers, as shown in Figure 2. We test two positions, before or after the in_proj layer. If before, the output tokens are discarded just before the out_proj layer. If after, they are discarded after SSM. We find that it is important to affect additional tokens only on SSM by adding after the in_proj layer. Moreover, the best performance is achieved by adding an affix at the beginning of the input tokens, just as with Prefix-tuning for Transformer, and different from how to handle additional tokens for Vim in Zhu et al. (); Darcet et al. (). This should be because it is impossible to learn which positions are additional tokens rather than patch tokens unless they are trained from scratch. A detailed analysis of the positions and numbers of inserted tokens is provided in the Section A.1.

Note that these methods have the option of applying embedding projection to the prompts, which significantly increases the number of parameters. However, it can be merged during inference, resulting in zero computational cost increase. Our results show that embedding projection is effective in improving accuracy, and hence it is recommended to use it if memory allows.

PEFT methods designed for Mamba. We perform experiments on two newly proposed PEFT methods for Mamba. The results of Additional-scan are presented in Table 1. Additional-scan shows competitive performance with the best settings of LoRA and Affix-tuning, with fewer trainable parameters. Figure 3 and Table 2 list the experiments with different initialization for Additional-scan. It shows that our proposed initialization consistently outperforms the original S4D used in Mamba.
The results of Partial-tuning are presented in Table 1. Performance highly depends on which components are made learnable. The highest performance is achieved by making conv1d learnable. Our proposed modified weight decay works effectively around 1e-3 strength (Table 2).
We analyze the number of trainable parameters and hyperparameters in Appendix A.2 in detail.

### 4.3 Results of Hybrid PEFT

The bottom of Table 1 shows the effectiveness of our two-step optimization approach. Combining all PEFT methods results in lower performance compared to using a single method. In contrast, our two-stage search method achieves higher performance with fewer parameters. These results indicate that selecting a preferred combination of PEFT methods is crucial.

We find that the high-performing single methods are not necessarily selected among the optimal combinations. An example of this combination is provided in Table 6 in Appendix B.2. We hypothesize that this phenomenon is related to the importance of ensuring model diversity in ensemble methods (Dietterich, ). Combining PEFT methods with different mechanisms can offset individual errors, mitigate overfitting, and enhance generalization performance. Detailed verification of this aspect remains a direction for future research.

**Table 3: Experimental results of the commonsense reasoning tasks. Language Model Evaluation Harness (Gao et al., ) is used to benchmark. The best performance is highlighted in bold, and the second-best performance is underlined, excluding Full fine-tuning.**
| Model | Method | #Params(%) | BoolQ | PIQA | SIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | QBQA | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pythia 160M | Full | 100 | 58.7 | 62.2 | 39.0 | 29.1 | 52.1 | 42.3 | 23.9 | 28.4 | 42.0 |
| LoRA | 0.72 | 61.0 | 62.0 | 36.3 | 30.3 | 52.0 | 38.2 | 24.6 | 28.0 | 41.6 |  |
| \cdashline1-12<br>Mamba 130M | Full | 100 | 56.1 | 65.3 | 38.7 | 35.3 | 52.0 | 46.4 | 25.7 | 32.8 | 43.8 |
| SLL LoRA | 1.45 | 56.3 | 63.3 | 38.2 | 34.6 | 51.6 | 43.5 | 23.6 | 30.6 | 42.7 |  |
| Additional-scan | 0.51 | 57.8 | 64.1 | 37.5 | 34.5 | 53.0 | 41.3 | 23.5 | 30.0 | 42.7 |  |
| Affix-tuning (w/o proj) | 0.17 | 55.1 | 61.4 | 36.5 | 32.9 | 51.5 | 36.8 | 23.5 | 27.2 | 40.6 |  |
|  | Affix-tuning | 64.64 | 59.7 | 64.3 | 38.2 | 35.2 | 51.9 | 42.9 | 24.0 | 29.0 | 43.2 |
|  | LoRAp(X) | 2.67 | 61.7 | 64.0 | 39.5 | 34.3 | 52.2 | 43.5 | 25.3 | 29.4 | 43.7 |
| Pythia 410M | Full | 100 | 55.4 | 66.8 | 40.0 | 34.2 | 50.5 | 45.2 | 26.5 | 29.4 | 43.5 |
| LoRA | 0.77 | 61.3 | 67.7 | 40.8 | 39.2 | 54.9 | 48.1 | 24.7 | 28.6 | 45.7 |  |
| \cdashline1-12<br>Mamba 370M | Full | 100 | 58.1 | 69.9 | 41.9 | 45.7 | 53.8 | 52.7 | 29.7 | 33.4 | 48.2 |
| SLL LoRA | 2.30 | 59.5 | 69.6 | 42.2 | 44.1 | 54.9 | 50.6 | 26.3 | 30.8 | 47.3 |  |
| Additional-scan | 0.47 | 61.9 | 69.3 | 41.2 | 45.3 | 54.9 | 28.4 | 49.5 | 31.4 | 47.7 |  |
| Affix-tuning (w/o proj) | 0.16 | 62.0 | 67.7 | 39.3 | 46.3 | 54.1 | 47.8 | 28.2 | 31.0 | 47.0 |  |
|  | Affix-tuning | 68.88 | 61.2 | 68.4 | 39.6 | 46.2 | 55.4 | 48.2 | 28.2 | 30.6 | 47.2 |
|  | LoRAp(X) | 2.67 | 60.8 | 68.8 | 42.1 | 44.7 | 56.2 | 50.4 | 27.4 | 32.2 | 47.8 |
| Pythia 1B | Full | 100 | 57.3 | 64.4 | 40.3 | 47.1 | 51.8 | 41.5 | 25.0 | 26.8 | 44.0 |
| LoRA | 0.41 | 60.0 | 69.3 | 40.9 | 45.3 | 53.6 | 49.8 | 27.2 | 31.0 | 47.1 |  |
| \cdashline1-12<br>Mamba 790M | Full | 100 | 62.0 | 72.1 | 44.8 | 54.0 | 55.9 | 57.7 | 31.2 | 35.2 | 51.6 |
| SLL LoRA | 3.1 | 60.7 | 72.0 | 42.4 | 54.7 | 56.9 | 55.3 | 29.4 | 34.2 | 50.7 |  |
| Additional-scan | 0.33 | 63.0 | 71.9 | 41.9 | 54.2 | 57.1 | 54.9 | 30.0 | 32.6 | 50.7 |  |
| Affix-tuning (w/o proj) | 0.22 | 57.2 | 71.7 | 41.4 | 55.0 | 55.8 | 52.6 | 29.8 | 33.0 | 49.6 |  |
| Affix-tuning | 69.99 | 61.0 | 72.5 | 41.0 | 54.9 | 55.6 | 54.6 | 29.6 | 33.8 | 50.4 |  |
| LoRAp(X) | 1.75 | 59.9 | 72.2 | 44.2 | 52.8 | 58.0 | 53.7 | 30.8 | 34.8 | 50.8 |  |
| Pythia 1.4B | Full | 100 | 53.1 | 68.9 | 40.6 | 39.8 | 50.6 | 49.0 | 26.9 | 29.4 | 44.8 |
| LoRA | 0.44 | 60.1 | 71.3 | 42.5 | 50.1 | 58.9 | 57.6 | 29.6 | 33.6 | 50.5 |  |
| \cdashline1-12<br>Mamba 1.4B | Full | 100 | 61.4 | 73.3 | 43.9 | 56.9 | 59.0 | 59.7 | 34.0 | 35.4 | 53.0 |
| SLL LoRA | 4.64 | 59.7 | 73.5 | 43.1 | 56.9 | 60.7 | 59.7 | 31.7 | 36.0 | 52.7 |  |
| Additional-scan | 0.26 | 63.0 | 73.5 | 42.8 | 57.5 | 60.5 | 60.9 | 32.4 | 37.4 | 53.5 |  |
| Affix-tuning (w/o proj) | 0.09 | 63.6 | 74.0 | 41.9 | 58.9 | 60.6 | 61.6 | 33.4 | 36.8 | 53.9 |  |
| LoRAp(X) | 1.36 | 63.1 | 73.5 | 42.7 | 57.7 | 61.6 | 60.4 | 32.9 | 37.4 | 53.7 |  |

### 4.4 Language tasks

In addition to the image tasks, we evaluate our method on language tasks using the vanilla Mamba Gu & Dao (). We experiment with a commonsense reasoning task, following the setup and dataset of Hu et al. (). This task consists of eight sub-tasks, which are evaluated on each task after training on a dataset that integrates training data from all tasks.
We use Pythia (Biderman et al., ), which is pre-trained with the same dataset as Mamba, as a Transformer baseline. For a PEFT method for Mamba, we compare with SLL LoRA (Halloran et al., ). To the best of our knowledge, SLL LoRA is the only PEFT method for Mamba.

In the language tasks, the proposed Additional-scan is found to work as an efficient PEFT. We hypothesize that this is related to the amount of data. This experiment uses 170k datasets, in contrast to the 1k used for VTAB-1k. We hypothesize that learning a selective mechanism requires a relatively large amount of data, and Additional-scan works powerfully in such cases. In Affix-tuning, we find that, as the size of the base model increases, it achieves sufficient accuracy without the embedding projection. This is a valuable discovery because reducing memory costs is crucial for larger models. As discussed in Section A.4, we find that Additional-scan, Affix-tuning (w/o proj), and LoRAp(X) are all useful depending on the application and should be used accordingly.

### 4.5 Limitations

This paper focuses on an exploratory investigation. Evaluating the applicability of our findings to larger models, such as vision models trained with ImageNet-21K (Ridnik et al., ) or large language models with billions of parameters (Touvron et al., ), is an area for future work. We will release the code after publication to stimulate future research from communities.

## 5 Conclusion

We conducted a comprehensive investigation of PEFT for Mamba. Our investigation started by applying existing PEFT methods for Transformers to Mamba. We proposed Affix-tuning and Partial LoRA as modifications of existing PEFT methods and introduced Partial-tuning and Additional-scan as new Mamba-specific PEFT methods. We also provided an effective framework for combining them. These comprehensive experiments on image and language tasks led to several findings.

The experiments demonstrate that Mamba benefits from PEFT more than Transformers. It improves accuracy without overfitting, even with a larger number of parameters, especially when fine-tuning data is limited. LoRA-based methods, particularly LoRAp(X), are effective with limited data. In contrast, the proposed Additional-scan method excels with larger datasets. We found that initialization significantly impacts performance when adding an additional dimension to A, despite Mamba’s robustness to initialization compared to other SSMs. Moreover, the proposed Affix-tuning was effective especially for large Mamba models. As to adapters on small linear weights, interestingly, over-parameterizing the original weight rather than decomposing it to a lower rank can be beneficial, which should holds for other architectures than Mamba. In hybrid PEFT, simply combining individually high-performance methods is inefficient. It is crucial to find the appropriate combination of PEFT methods. We believe that these results make a valuable contribution to the Mamba research community.

## Appendix A More detailed ablation studies

### A.1 Individual PEFT methods in detail

In the experiments, we need to determine the appropriate parameters for each method. To this end, we investigate the optimal settings by varying the degrees of freedom for each PEFT method. Specifically, the rank of LoRA ($r$), the number of additional tokens for Affix-tuning and Prefix-tuning ($n$), the number of dimensions for Additional-scan, and the strength of the proposed weight decay for Partial-tuning. Same as the ablation study in Table 2 and Figure 3 in the main text, we evaluate the averaged accuracy across six tasks: CIFAR-100, Sun397, Camelyon, Retinopathy, Clevr-Count, and sNORB-Elev.

First, each LoRA applied to large weights is evaluated. As shown in Figure 4, the maximum accuracy with ViT can be obtained with approximately $r=32$, whereas that with Vim can be obtained with a higher rank, as shown in Figure 4. One reason is that the pre-trained Vim is less prone to collapse even when a large LoRA is added. Interestingly, the in_proj layer itself is prone to collapse and begins to degrade in accuracy at $r>32$, while LoRAp(X) and LoRAp(Z), which apply LoRA to partial weights of the in_proj layer, are able to continue improving accuracy up to $r=64$. By dividing LoRA for each output feature, the collapse is suppressed, and further accuracy improvement is possible with large parameters in LoRA. We have already discussed LoRA on small linear weights in the main text (Figure 3)

Figure: (a)

For Prompt-tuning, adding soft tokens to both the beginning and end is the most effective, as shown in Figure 4.
Figure 4 also shows that the number of affixes improves performance up to 3 tokens. However, 4 or more affix tokens degrade performance, due to over-fitting caused by increasing the number of parameters.

### A.2 Efficiency of PEFT methods for Mamba

In Table 1, we present a benchmark with the hyperparameter settings that maximize accuracy. As discussed in Section A.1, Vim is less prone to collapse and benefits more from increasing the number of training parameters in PEFT modules. Consequently, the number of trainable parameters for Vim in Table 1 is larger than that for ViT. Hence, we analyze the trade-off between accuracy and the size of PEFT modules. Specifically, all methods evaluated in Figure 3 and Figure 4 are plotted in Figure 5. The results indicate that the superior PEFT methods for Mamba achieve high accuracy even with a lower number of parameters and exhibit a better trade-off compared to those for ViT.

Figure: Figure 5: Scatter plot of the evaluations for the methods in Section A.1, showing the relationship between the proportion of trainable parameters and performance

### A.3 Per-task evaluation on VTAB-1k

We present the results for each task in VTAB-1k, including different base model sizes and training data. Specifically, we also report the accuracy on other base model architectures such as Vim-tiny (Zhu et al., ), ViT-tiny (Touvron et al., ), and ViT-B (Dosovitskiy et al., ), as well as different pre-training datasets such as ImageNet-21k (Deng et al., ) or using the training framework of (Dosovitskiy et al., ).
The compared PEFT methods for Transformers are LoRA (Hu et al., ), Adaptformer (Chen et al., ), Adapter+ (Steitz & Roth, ), FacT-TK (Jie & Deng, ), Head2Toe (Evci et al., ), VQT (Tu et al., ), BitFit (Zaken et al., ), and LoSA (Mercea et al., ).
The results are shown in Section A.3.

Many PEFT methods for Vision Transformers have benchmarked with ViT-B pre-trained on ImageNet-21K. First, we clarify the accuracy change due to differences in our experimental settings and their settings. By using ImageNet-1K instead of ImageNet-21K, accuracy decreases by around 5 points. Changing the training framework to DeiT does not significantly affect accuracy. Reducing the model size from ViT-B to ViT-S results in an accuracy drop of about 2 points.

Next, we compare PEFT for Transformers and PEFT for Mamba using base models of a similar size. For small-sized base models, as mentioned in the main paper, we achieve a performance improvement of an additional 1.5 points over the state-of-the-art PEFT for Transformers. With even smaller models, such as ViT-tiny and Vim-tiny, it shows similar accuracy improvements. This confirms that our proposed method is effective for model sizes other than Vim-S used in the main paper.

Finally, we evaluate each task individually. PEFT for Vim outperforms PEFT for ViT on most tasks when compared within a similar base model size. Notably, PEFT for Vim is inferior to PEFT for ViT only in the dSpr-Loc task, which classifies the $x$-coordinates of objects of various colors, shapes, and sizes. We consider that one reason for this is that the attention mechanism can locate the objects more straightforwardly than Mamba. On the other hand, in the Clevr/count task, the accuracy with Vim is much higher than that with ViT. This task involves counting the number of objects in the image, and the recurrent nature of Mamba may be more suitable for counting the number of objects in the hidden state.

**Table 4: Detailed test results on the VTAB-1k benchmark. “Mean” denotes the average accuracy for each category, and “Overall Mean” shows the average accuracy over 19 tasks. The (A), (T), and (D) stand for the network architecture, training framework, and dataset used for pre-training.**
|  | Natural | Specialized | Structured |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | CIFAR-100 | Caltech101 | DTD | Flowers102 | Pets | SVHN | Sun397 | Mean | Camelyon | EuroSAT | Resisc45 | Retinopathy | Mean | Clevr-Count | Clevr-Dist | DMLab | KITTI-Dist | dSpr-Loc | dSpr-Ori | sNORB-Azim | sNORB-Elev | Mean | Overall Mean |
| (A) ViT-B (T) ViT (D) IN21K |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Full | 68.9 | 87.7 | 64.3 | 97.2 | 86.9 | 87.4 | 38.8 | 75.9 | 79.7 | 95.7 | 84.2 | 73.9 | 83.4 | 56.3 | 58.6 | 41.7 | 65.5 | 57.5 | 46.7 | 25.7 | 29.1 | 47.6 | 68.9 |
| Linear | 64.4 | 85.0 | 63.2 | 97.0 | 86.3 | 36.6 | 51.0 | 69.1 | 78.5 | 87.5 | 68.5 | 74.0 | 77.1 | 34.3 | 30.6 | 33.2 | 55.4 | 12.5 | 20.0 | 9.6 | 19.2 | 26.9 | 57.6 |
| BitFit | 72.8 | 87.0 | 59.2 | 97.5 | 85.3 | 59.9 | 51.4 | 73.3 | 78.7 | 91.6 | 72.9 | 69.8 | 78.3 | 61.5 | 55.6 | 32.4 | 55.9 | 66.6 | 40.0 | 15.7 | 25.1 | 44.1 | 65.2 |
| LoRA | 67.1 | 91.4 | 69.4 | 98.8 | 90.4 | 85.3 | 54.0 | 79.5 | 84.9 | 95.3 | 84.4 | 73.6 | 84.6 | 82.9 | 69.2 | 49.8 | 78.5 | 75.7 | 47.1 | 31.0 | 44.0 | 59.8 | 74.5 |
| FacT-TK | 70.6 | 90.6 | 70.8 | 99.1 | 90.7 | 88.6 | 54.1 | 80.6 | 84.8 | 96.2 | 84.5 | 75.7 | 85.3 | 82.6 | 68.2 | 49.8 | 80.7 | 80.8 | 47.4 | 33.2 | 43.0 | 60.7 | 75.6 |
| Adapter+ | 83.7 | 94.2 | 71.5 | 99.3 | 90.6 | 88.2 | 55.8 | 83.3 | 87.5 | 97.0 | 87.4 | 72.9 | 86.2 | 82.9 | 60.9 | 53.7 | 80.8 | 88.4 | 55.2 | 37.3 | 46.9 | 63.3 | 77.6 |
| LoSA | 82.5 | 92.8 | 76.1 | 99.7 | 90.5 | 82.0 | 55.8 | 82.8 | 86.6 | 97.1 | 87.0 | 76.7 | 86.9 | 81.5 | 62.3 | 48.6 | 82.1 | 94.2 | 61.7 | 47.9 | 45.6 | 65.5 | 78.4 |
| (A) ViT-B (T) ViT (D) IN1K |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Scratch | 7.6 | 19.1 | 13.1 | 29.6 | 6.7 | 19.4 | 2.3 | 14.0 | 71.0 | 71.0 | 29.3 | 72.0 | 60.8 | 31.6 | 52.5 | 27.2 | 39.1 | 66.1 | 29.7 | 11.7 | 24.1 | 35.3 | 32.8 |
| Full | 44.3 | 84.5 | 54.1 | 84.7 | 74.7 | 87.2 | 26.9 | 65.2 | 85.3 | 95.0 | 76.0 | 70.4 | 81.7 | 71.5 | 60.5 | 46.9 | 72.9 | 74.5 | 38.7 | 28.5 | 23.8 | 52.2 | 63.2 |
| Linear | 50.6 | 85.6 | 61.4 | 79.5 | 86.5 | 40.8 | 38.0 | 63.2 | 79.7 | 91.5 | 71.7 | 65.5 | 77.1 | 41.4 | 34.4 | 34.1 | 55.4 | 18.1 | 26.4 | 16.5 | 24.8 | 31.4 | 52.7 |
| Head2Toe | 54.4 | 86.8 | 64.1 | 83.4 | 82.6 | 78.9 | 32.1 | 68.9 | 81.3 | 95.4 | 81.2 | 73.7 | 82.9 | 49.0 | 57.7 | 41.5 | 64.4 | 52.3 | 32.8 | 32.7 | 39.7 | 46.3 | 62.3 |
| VQT | 58.4 | 89.4 | 66.7 | 90.4 | 89.1 | 81.1 | 33.7 | 72.7 | 82.2 | 96.2 | 84.7 | 74.9 | 84.5 | 50.8 | 57.6 | 43.5 | 77.2 | 65.9 | 43.1 | 24.8 | 31.6 | 49.3 | 65.3 |
| FacT-TK | 56.0 | 89.0 | 66.8 | 90.0 | 91.3 | 84.8 | 40.2 | 74.0 | 83.9 | 94.7 | 82.8 | 72.3 | 83.4 | 73.8 | 59.6 | 45.1 | 75.5 | 68.0 | 49.4 | 23.8 | 34.1 | 53.7 | 70.4 |
| LoRA | 52.7 | 89.1 | 66.4 | 90.8 | 90.7 | 85.4 | 40.0 | 73.6 | 84.6 | 94.7 | 82.6 | 73.2 | 83.8 | 76.6 | 62.5 | 50.0 | 80.0 | 75.8 | 48.2 | 27.8 | 36.7 | 57.2 | 71.5 |
| \cdashline1-24                (A) ViT-B (T) DeiT (D) IN1K |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Linear | 35.4 | 86.5 | 61.8 | 83.9 | 88.3 | 43.4 | 37.7 | 62.4 | 80.0 | 90.8 | 74.3 | 73.4 | 79.6 | 40.6 | 37.3 | 36.7 | 59.6 | 20.3 | 25.0 | 16.6 | 24.2 | 32.5 | 58.2 |
| FacT-TK | 61.0 | 89.1 | 67.0 | 91.8 | 91.6 | 87.6 | 42.3 | 75.8 | 84.0 | 94.8 | 84.4 | 72.4 | 83.9 | 76.1 | 58.1 | 47.2 | 79.7 | 76.6 | 55.9 | 28.6 | 33.0 | 56.9 | 72.2 |
| LoRA | 61.3 | 89.5 | 68.2 | 92.9 | 91.3 | 88.4 | 42.8 | 76.3 | 84.0 | 95.0 | 85.0 | 73.2 | 84.3 | 79.4 | 58.8 | 49.5 | 81.7 | 81.0 | 54.2 | 31.2 | 38.1 | 59.2 | 70.8 |
| (A) ViT-S (T) DeiT (D) IN1K |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Scratch | 4.9 | 11.1 | 9.7 | 24.4 | 3.4 | 19.4 | 1.6 | 10.7 | 65.0 | 57.3 | 28.6 | 73.6 | 56.1 | 20.9 | 48.1 | 26.2 | 45.0 | 7.3 | 27.1 | 6.2 | 17.8 | 24.8 | 26.2 |
| Full | 30.4 | 69.3 | 37.6 | 65.8 | 50.2 | 87.8 | 21.5 | 51.8 | 76.2 | 85.2 | 66.8 | 63.0 | 72.8 | 33.5 | 56.8 | 40.3 | 66.8 | 73.5 | 40.4 | 28.0 | 22.8 | 45.3 | 53.5 |
| Linear | 41.4 | 83.9 | 62.0 | 76.5 | 88.8 | 37.6 | 35.9 | 60.9 | 80.5 | 88.6 | 69.4 | 74.0 | 78.1 | 35.3 | 34.0 | 34.0 | 65.7 | 18.3 | 20.5 | 15.3 | 21.5 | 30.6 | 51.7 |
| Fact-TK | 54.4 | 86.6 | 65.4 | 89.3 | 89.8 | 85.1 | 39.6 | 72.9 | 84.4 | 93.4 | 79.9 | 71.7 | 82.3 | 71.9 | 58.0 | 44.0 | 74.5 | 78.8 | 50.6 | 25.2 | 29.7 | 54.1 | 67.0 |
| LoRA | 55.3 | 87.6 | 67.3 | 90.2 | 90.2 | 85.1 | 39.6 | 73.6 | 81.7 | 94.1 | 80.5 | 72.7 | 82.2 | 77.6 | 59.1 | 47.0 | 81.6 | 81.5 | 49.6 | 29.4 | 35.1 | 57.6 | 68.7 |
| Adaptformer | 56.0 | 88.0 | 65.6 | 90.5 | 90.5 | 85.3 | 39.5 | 73.6 | 84.0 | 93.8 | 82.2 | 72.7 | 83.2 | 77.5 | 59.0 | 46.7 | 79.3 | 83.7 | 50.4 | 30.5 | 35.4 | 57.8 | 69.0 |
| Adapter+ | 56.6 | 89.2 | 66.8 | 91.1 | 90.2 | 88.2 | 40.7 | 74.7 | 84.8 | 94.2 | 82.9 | 72.4 | 83.6 | 76.8 | 59.7 | 48.4 | 80.5 | 87.8 | 51.9 | 32.4 | 33.1 | 58.8 | 69.9 |
| \cdashline1-24                (A) Vim-S (T) DeiT (D) IN1K |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Scratch | 5.6 | 11.9 | 5.9 | 12.1 | 5.0 | 16.3 | 1.6 | 8.3 | 61.6 | 62.3 | 13.8 | 61.6 | 49.8 | 28.9 | 53.2 | 22.5 | 40.9 | 38.6 | 11.8 | 11.3 | 18.3 | 28.2 | 25.4 |
| Full | 51.6 | 83.0 | 61.5 | 71.1 | 45.0 | 88.4 | 14.9 | 59.4 | 66.5 | 88.1 | 63.3 | 57.1 | 68.7 | 48.8 | 60.4 | 38.8 | 55.0 | 6.3 | 11.8 | 30.0 | 24.1 | 34.4 | 50.8 |
| Linear | 48.3 | 85.2 | 59.5 | 75.5 | 88.7 | 40.7 | 39.6 | 62.5 | 78.8 | 89.0 | 68.2 | 73.0 | 77.3 | 37.0 | 36.7 | 33.3 | 66.0 | 18.0 | 25.2 | 16.5 | 23.2 | 32.0 | 52.8 |
| CLS-token-tuning | 48.0 | 85.1 | 59.2 | 75.7 | 88.8 | 41.0 | 39.4 | 62.4 | 78.7 | 89.2 | 68.5 | 73.1 | 77.4 | 37.3 | 36.7 | 33.6 | 66.4 | 18.3 | 25.4 | 17.1 | 23.5 | 32.3 | 52.9 |
| Pos-embed-tuning | 50.6 | 85.1 | 59.4 | 74.6 | 88.3 | 50.5 | 39.9 | 64.1 | 77.2 | 87.6 | 68.2 | 66.4 | 74.8 | 32.9 | 46.3 | 32.4 | 65.0 | 58.6 | 29.8 | 19.0 | 29.5 | 39.2 | 55.9 |
| D-tuning | 54.2 | 86.0 | 63.1 | 81.3 | 88.9 | 55.5 | 41.1 | 67.1 | 78.8 | 91.3 | 73.0 | 71.2 | 78.6 | 45.3 | 42.7 | 36.9 | 70.0 | 41.5 | 38.5 | 19.9 | 26.0 | 40.1 | 58.2 |
| A-tuning | 58.7 | 87.4 | 65.0 | 86.5 | 90.2 | 74.0 | 42.4 | 72.0 | 81.1 | 93.2 | 78.4 | 70.2 | 80.7 | 62.7 | 53.9 | 40.0 | 77.5 | 60.6 | 46.0 | 23.9 | 32.1 | 49.6 | 64.4 |
| Conv1d-tuning | 59.7 | 89.1 | 65.2 | 86.5 | 89.4 | 88.4 | 42.1 | 74.3 | 82.9 | 94.2 | 79.2 | 73.5 | 82.5 | 76.5 | 63.3 | 45.7 | 79.5 | 80.2 | 51.3 | 30.8 | 35.4 | 57.8 | 69.1 |
| Prompt-tuning | 55.6 | 87.2 | 62.7 | 82.8 | 89.5 | 70.3 | 41.5 | 69.9 | 78.1 | 91.4 | 74.9 | 72.4 | 79.2 | 59.1 | 54.8 | 38.8 | 73.7 | 58.9 | 45.7 | 22.1 | 28.9 | 47.8 | 62.5 |
| Affix-tuning | 61.9 | 89.3 | 67.8 | 89.7 | 91.0 | 88.0 | 43.2 | 75.8 | 82.6 | 94.8 | 82.2 | 73.6 | 83.3 | 80.4 | 59.5 | 46.1 | 81.3 | 83.6 | 50.9 | 30.5 | 39.4 | 58.9 | 70.3 |
| Additional-scan | 59.6 | 90.0 | 66.6 | 88.9 | 90.7 | 83.6 | 43.1 | 74.6 | 83.2 | 94.4 | 81.3 | 71.9 | 82.7 | 75.1 | 63.2 | 43.7 | 80.0 | 75.0 | 48.1 | 29.1 | 36.9 | 56.4 | 68.7 |
| ParallelAdapter | 64.0 | 89.7 | 68.9 | 91.4 | 91.3 | 83.4 | 44.1 | 76.1 | 85.3 | 95.2 | 83.4 | 72.0 | 84.0 | 83.9 | 63.4 | 49.8 | 80.2 | 75.5 | 53.0 | 32.8 | 41.3 | 60.0 | 71.0 |
| LoRA(embed) | 51.7 | 84.9 | 59.2 | 75.5 | 88.0 | 53.4 | 39.8 | 64.7 | 77.5 | 91.2 | 71.5 | 70.0 | 77.5 | 42.9 | 52.0 | 34.9 | 73.0 | 54.9 | 47.0 | 19.7 | 26.2 | 43.8 | 58.6 |
| LoRA(x_proj) | 59.2 | 88.0 | 67.5 | 88.9 | 90.4 | 83.9 | 43.0 | 74.4 | 82.7 | 93.9 | 80.3 | 70.7 | 81.9 | 72.5 | 60.7 | 43.6 | 80.6 | 75.0 | 45.8 | 27.8 | 33.1 | 54.9 | 67.8 |
| LoRA(dt_proj) | 61.6 | 90.1 | 66.7 | 89.0 | 90.7 | 86.1 | 43.4 | 75.4 | 83.2 | 94.9 | 81.2 | 72.9 | 83.1 | 76.9 | 59.8 | 47.0 | 79.8 | 75.6 | 50.7 | 30.3 | 37.0 | 57.1 | 69.3 |
| LoRA(out_proj) | 64.2 | 89.8 | 68.6 | 91.4 | 91.2 | 86.0 | 43.8 | 76.4 | 84.9 | 94.9 | 83.4 | 72.7 | 84.0 | 84.5 | 62.7 | 49.5 | 81.3 | 77.1 | 52.0 | 32.8 | 40.9 | 60.1 | 71.1 |
| LoRA(in_proj) | 63.4 | 90.3 | 68.2 | 91.0 | 91.0 | 88.6 | 43.5 | 76.6 | 84.7 | 95.0 | 83.8 | 72.9 | 84.1 | 84.0 | 61.3 | 48.6 | 80.0 | 83.5 | 52.6 | 31.8 | 39.5 | 60.2 | 71.3 |
| LoRAp(d) | 57.1 | 87.4 | 65.9 | 87.6 | 90.4 | 81.8 | 42.7 | 73.3 | 80.9 | 93.6 | 78.9 | 70.2 | 80.9 | 64.2 | 55.4 | 42.8 | 76.9 | 64.5 | 47.4 | 25.5 | 30.8 | 50.9 | 65.5 |
| LoRAp(C) | 55.8 | 87.8 | 66.9 | 87.6 | 90.5 | 78.6 | 42.3 | 72.8 | 82.1 | 94.1 | 79.7 | 70.5 | 81.6 | 60.5 | 59.2 | 41.3 | 80.9 | 70.4 | 42.8 | 25.9 | 29.7 | 51.4 | 65.6 |
| LoRAp(B) | 56.5 | 87.5 | 66.3 | 88.1 | 90.5 | 79.0 | 42.8 | 73.0 | 82.9 | 93.9 | 79.6 | 70.3 | 81.7 | 67.2 | 58.8 | 41.5 | 78.3 | 69.4 | 44.4 | 26.8 | 31.7 | 52.3 | 66.1 |
| LoRAp(Z) | 61.7 | 90.6 | 68.1 | 90.3 | 90.7 | 88.4 | 43.2 | 76.2 | 85.6 | 95.2 | 83.5 | 72.8 | 84.3 | 82.4 | 60.6 | 48.2 | 81.9 | 81.6 | 52.4 | 31.4 | 39.5 | 59.7 | 70.9 |
| LoRAp(X) | 63.3 | 89.3 | 69.4 | 90.6 | 91.5 | 88.9 | 43.6 | 76.6 | 84.5 | 95.3 | 83.4 | 72.4 | 83.9 | 84.9 | 62.9 | 48.6 | 81.4 | 82.8 | 52.7 | 33.1 | 40.4 | 60.8 | 71.5 |
| Hybrid | 63.7 | 90.2 | 69.4 | 90.9 | 90.1 | 90.1 | 43.9 | 77.0 | 86.3 | 95.2 | 83.9 | 72.2 | 84.4 | 84.6 | 61.7 | 50.0 | 81.0 | 86.4 | 54.0 | 34.6 | 40.2 | 61.6 | 72.1 |
| (A) ViT-tiny (T) DeiT (D) IN1K |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Linear | 35.7 | 81.3 | 57.1 | 72.4 | 84.5 | 35.2 | 29.9 | 56.6 | 81.0 | 86.8 | 67.0 | 74.2 | 77.2 | 32.5 | 32.7 | 33.1 | 59.9 | 14.2 | 17.7 | 13.6 | 19.2 | 27.9 | 48.8 |
| FacT-TK | 44.7 | 83.6 | 59.5 | 85.9 | 91.5 | 81.6 | 32.9 | 68.5 | 82.0 | 91.4 | 75.0 | 70.0 | 79.6 | 66.3 | 58.3 | 43.1 | 72.9 | 77.4 | 45.0 | 24.3 | 31.1 | 52.3 | 66.8 |
| LoRA | 45.5 | 85.9 | 59.5 | 85.3 | 86.2 | 84.7 | 32.3 | 68.5 | 80.5 | 93.0 | 76.1 | 70.9 | 80.1 | 72.2 | 60.0 | 45.4 | 76.1 | 79.7 | 45.9 | 27.6 | 32.8 | 55.0 | 65.2 |
| \cdashline1-24                (A) Vim-tiny (T) DeiT (D) IN1K |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Linear | 43.0 | 83.7 | 56.9 | 73.5 | 87.4 | 40.6 | 36.9 | 60.3 | 79.0 | 87.1 | 68.3 | 73.1 | 76.9 | 32.6 | 36.1 | 34.2 | 66.5 | 15.2 | 22.4 | 15.0 | 21.3 | 30.4 | 51.2 |
| Additional-scan | 52.6 | 87.1 | 62.3 | 85.7 | 88.5 | 81.7 | 37.0 | 70.7 | 80.2 | 93.0 | 78.8 | 72.0 | 81.0 | 69.6 | 62.2 | 42.6 | 78.6 | 77.0 | 43.5 | 27.0 | 32.4 | 54.1 | 65.9 |
| Conv1d-tuning | 50.0 | 86.4 | 60.0 | 82.8 | 85.6 | 89.2 | 35.2 | 69.8 | 82.5 | 93.3 | 77.7 | 69.7 | 80.8 | 71.5 | 63.9 | 44.1 | 77.1 | 77.9 | 48.1 | 29.9 | 34.4 | 55.9 | 66.3 |
| Affix-tuning | 53.3 | 87.4 | 60.4 | 85.9 | 88.6 | 84.4 | 37.3 | 66.8 | 79.4 | 93.8 | 77.7 | 70.0 | 80.2 | 74.3 | 64.2 | 44.2 | 77.9 | 79.6 | 48.0 | 27.0 | 38.3 | 56.7 | 66.9 |
| LoRAp(X) | 56.0 | 87.6 | 64.5 | 87.7 | 89.1 | 87.4 | 38.2 | 72.9 | 83.2 | 94.5 | 82.2 | 71.8 | 80.8 | 78.7 | 59.3 | 47.2 | 76.1 | 81.0 | 50.6 | 29.7 | 37.4 | 57.5 | 68.5 |

### A.4 Ablation studies on language tasks

We conduct ablation studies for language tasks. Figure 6 shows that the optimal learning rate varies for each method. In contrast, as shown in Figure 6 and Figure 6, the optimal learning rate and the optimal number of trainable parameters remain relatively stable despite changes in model size. Hence, the number of trainable parameters is fixed regardless of the model size, and the learning rate is coarsely tuned for evaluation in Table 3. The detailed settings are described in Section B.1.

Figure: (a)

The trade-off between the number of trainable parameters and accuracy for Additional-scan, Affix-tuning (w/o proj), and LoRAp(X) is described in Figure 6. Each method excels in certain aspects and should be chosen based on the specific application. LoRAp(X) is effective when sufficient memory is available, Affix-tuning (w/o proj) maintains accuracy with the fewest trainable parameters, and Additional-scan falls in between. Since even PEFT may not fit into GPU memory with large-scale models, it is crucial to select the appropriate method based on the application and environment.

## Appendix B Implementation details

### B.1 Language Tasks

We follow the fine-tuning setup of Liu et al. (); Hu et al. () for commonsense reasoning tasks. Each model is fine-tuned with about 140,000 data for three epochs with a batch size of 16. A linear learning rate scheduler is used with a warmup period of 100 iterations.

As to the learning rate, we use suitable values for each method. It is set to 5e-5 for the full fine-tuning of Mamba and Pythia (Biderman et al., ). For SLL LoRA on Mamba, we follow the original learning rate per model size (Halloran et al., ). We also use a roughly tuned learning rate for our PEFT methods obtained through the ablation studies in Section A.4. The configurations are shown in Table 5.

**Table 5: The learning rate configurations for language tasks.**
| Method | Mamba 160M | Mamba 370M | Mamba 790M | Mamba 1.4B |
| --- | --- | --- | --- | --- |
| Additional-scan | 5e-3 | 1e-3 | 1e-3 | 1e-4 |
| Affix-tuning | 1e-4 | 1e-5 | 1e-5 | 1e-5 |
| LoRAp(X) | 1e-3 | 5e-4 | 5e-4 | 1e-4 |
| LoRA on Pythia | 1e-5 | 1e-4 | 1e-5 | 1e-5 |

For the hyperparameters of each PEFT method, we use the optimal values obtained through the ablation studies on the VTAB-1K dataset. For Pythia, We apply LoRA with $r=8$ to dense, query_key_value, dense_4h_to_h, and dense_h_to_4h layers.

### B.2 Hybrid PEFT Search

Since optimizing over all 19 tasks is computationally expensive, we optimize over the average accuracy of 5 tasks in the representative subset of VTAB-1k (Zhai et al., ): Caltech101, EuroSAT, Pets, Camelyon, and Resisc45. By processing five tasks in parallel on one A100 GPU, one trial can be completed in 20 to 30 minutes, with minimal dependency on the type and size of the applied PEFT methods.

As mentioned in the Section 3.4, the first step of our search verifies the effectiveness of each PEFT method and seeks the preferable combination with minimum trainable parameters.
Specifically, we search for 100 trials using TPE algorithm (Bergstra et al., ) about whether to use each PEFT method with the fixed hyperparameters shown in step 1 of Table 6. As a result, it turns out that the combination of eight PEFT methods, CLS-token-tuning, A-tuning, Affix-tuning, LoRA(out_proj), LoRA(in_proj), LoRA(dt_proj), LoRAp(X), and LoRAp(B), is the best.

In the second step, the hyperparameters for the eight methods are searched. In addition, a search dimension is added which selects a PEFT method that should be removed or decides not to remove any at all. This is intended to eliminate less important PEFT methods and allocate more parameters to the more significant ones. The specific search space is shown in step 2 of Table 6.

The detailed algorithm is provided in Algorithm 1. In our experiments, we use TPE as the search algorithm, top-1 accuracy as the objective function, and the number of trials is set to $N=100$.

Figure: Algorithm 1 HybridPEFT Search Algorithm

**Table 6: The search space and the fixed hyperparameters in our Hybrid PEFT search of the first step and the second step. In the first step, the combination of CLS-token-tuning, A-tuning, Affix-tuning, LoRA(out_proj), LoRA(in_proj), LoRA(dt_proj), LoRAp(X), and LoRAp(B) is the best.**
| Step 1 |  |  | Step 2 |  |  |
| --- | --- | --- | --- | --- | --- |
| boolean search<br>to use or not | fixed<br>hyperparameters |  |  | fixed methods to use | hyperparameter<br>search |
| CLS-token-tuning | wd=1e-3 |  |  | CLS-token-tuning | lr [1e-4, 5e-3] |
| Bias-tuning | wd=1e-3 |  |  |  | wd [1e-5, 1e-2] |
| \cdashline5-6<br>Pos-embed-tuning | wd=1e-3 |  |  | A-tuning | lr [1e-4, 5e-3] |
| D-tuning | wd=1e-3 |  |  |  | wd [1e-5, 1e-2] |
| \cdashline5-6<br>A-tuning | wd=1e-3 |  |  | Affix-tuning | n [1, 3] |
| Conv1d-tuning | wd=1e-3 |  |  |  | lr [1e-4, 5e-3] |
| Prompt-tuning | n=1 |  |  |  | wd [1e-6, 1e-3] |
| \cdashline5-6<br>Affix-tuning | n=1 |  |  | LoRA(out_proj) | r [4, 16] |
| Additional-scan | n=1 |  |  |  | s [1e-2, 1] |
| ParallelAdapter | r=8, s=0.1 |  |  |  | lr [1e-4, 5e-3] |
| LoRA(embed) | r=8, s=0.1 |  |  |  | wd [1e-6, 1e-3] |
| \cdashline5-6<br>LoRA(x_proj) | r=8, s=0.1 |  |  | LoRA(in_proj) | r [4, 16] |
| LoRA(dt_proj) | r=4, s=0.1 |  |  |  | s [1e-2, 1] |
| LoRA(out_proj) | r=8, s=0.1 |  |  |  | lr [1e-4, 5e-3] |
| LoRA(in_proj) | r=8, s=0.1 |  |  |  | wd [1e-6, 1e-3] |
| \cdashline5-6<br>LoRAp(d) | r=4, s=0.1 |  |  | LoRA(dt_proj) | r [4, 16] |
| LoRAp(B) | r=4, s=0.1 |  |  |  | s [1e-2, 1] |
| LoRAp(C) | r=4, s=0.1 |  |  |  | lr [1e-4, 5e-3] |
| LoRAp(X) | r=8, s=0.1 |  |  |  | wd [1e-6, 1e-3] |
| \cdashline5-6<br>LoRAp(Z) | r=8, s=0.1 |  |  | LoRAp(X) | r [4, 16] |
|  |  |  |  |  | s [1e-2, 1] |
|  |  |  |  |  | lr [1e-4, 5e-3] |
|  |  |  |  |  | wd [1e-6, 1e-3] |
| \cdashline5-6 |  |  |  | LoRAp(B) | r [4, 12] |
|  |  |  |  |  | s [1e-2, 1] |
|  |  |  |  |  | lr [1e-4, 5e-3] |
|  |  |  |  |  | wd [1e-6, 1e-3] |
| \cdashline5-6 |  |  |  |  | select $\begin{cases}\rm remove\;CLS\mathchar 45\relax token\mathchar 45\relax tuning<br>\\<br>\rm remove\;A\mathchar 45\relax tuning\\<br>\rm remove\;Affix\;tuning\\<br>\rm remove\;LoRA(out_proj)\\<br>\rm remove\;LoRA(in_proj)\\<br>\rm remove\;LoRA(dt_proj)\\<br>\rm remove\;LoRA_{p}(X)\\<br>\rm remove\;LoRA_{p}(B)\\<br>\rm not\;remove\\<br>\end{cases}$ |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please further explain the advantage and effectiveness of the proposed method compared with the baselines.
Please further explain the novelty of the proposed method.
soundness: 2
strengths: 1. The paper not only improves existing PEFT algorithms to suit Mamba but also proposes targeted, innovative algorithms, providing the community with a broader selection of PEFT methods.
2. The experiments comprehensively showcase the performance of each PEFT algorithm, offering a framework for optimal algorithmic combinations.
3. The extensive experimental workload demonstrates thorough exploration of the algorithms.
summary: This paper presents a comprehensive and detailed exploration of parameter-efficient fine-tuning (PEFT) for Mamba. By applying and refining existing PEFT methods, introducing novel PEFT approaches（Partial-tuning and Additional-scan）, and exploring optimal algorithmic combinations, it achieves outstanding performance in both image and language tasks. This work provides valuable insights for efficient fine-tuning in Mamba.
weaknesses: 1. The improved and original PEFT algorithms did not achieve the best results. In the VTAB-1k benchmark, the performance of the improved Partial LoRA is nearly identical to that of LoRA, while Additional-scan underperforms ParallelAdapter with comparable training parameters. Compared to traditional PEFT algorithms, these methods lack strong competitive advantage.
2. Although LoRA demonstrates outstanding performance on the VTAB-1k benchmark, its comparison with the modified Partial LoRA on language tasks is missing, making it unclear whether Partial LoRA is indeed the optimal PEFT method.
3. The algorithmic novelty is limited, as the concept of Partial-tuning appears relatively straightforward.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Table 1 shows that Adapter-based and LoRA-based methods perform best. Did the authors try SequentialAdapter, or, as in the original LoRA paper, apply LoRA across multiple locations simultaneously in the model?
2. In Table 1, the Hybrid approach shows no significant advantage over individual PEFT methods (e.g., ParallelAdapter or LoRA) despite having far more parameters (117,236). The performance of Hybrid is not markedly better than LoRA(in_proj), which has only 1,483 learnable parameters.
3. Beyond parameter efficiency, computational efficiency is also crucial for PEFT methods. Did the authors consider comparing the computational efficiency of different methods (e.g., in terms of FLOPs or running time)?
soundness: 3
strengths: 1. The paper focuses on a novel and competitive model architecture, SSMs, systematically analyzing the applicability of existing PEFT methods for the Mamba model, offering insights for designing efficient fine-tuning schemes for SSMs.
2. The paper adapts existing PEFT methods specifically for SSMs, proposing new approaches tailored for Mamba, achieving good performances.
3. The extensive experiments, covering both image and text modalities, provide comprehensive benchmarks for all methods and reveal differences in PEFT performance between the Mamba and Transformer models.
summary: This paper explores the application of parameter-efficient fine-tuning (PEFT) in the Mamba model. It begins by systematically analyzing the suitability of existing PEFT methods for the Mamba model, then adapts certain PEFT methods to better align with the characteristics of State Space Models (SSMs), and finally investigates hybrid PEFT search strategies. Extensive experiments were conducted across image and text modalities, providing thorough benchmarks for various PEFT methods and demonstrating that the Mamba model exhibits distinct fine-tuning behavior compared to Transformer models.
weaknesses: 1. The paper covers too many aspects, including various PEFT methods, individual analyses, and hybrid architecture search. The authors should concentrate on one aspect and provide valuable conclusions for each part instead of describing each in detail, as this could reduce readability and divert focus.
2. The paper should consider including the most advanced works within each PEFT category, such as GPS[1] and SPT[2] for partial-tuning methods, and SSF[3] in addition to LoRA-based methods for reparameterization approaches.
3. Experiments should involve larger-scale models, where PEFT methods are particularly valuable. Evaluation across different model scales is crucial for PEFT methods. Notably, "scale" here refers to model parameter size rather than training data size (e.g., a model trained on ImageNet-21K in Section 4.5 is not necessarily larger than one trained on ImageNet-1K; model parameter size is more relevant in this context).
3. Presentation quality requires improvement, including spelling errors (e.g., "Linear Probing" instead of "Linear Proving" in Table 1’s ViT-S section) and inconsistent heights in sub-tables within Table 2.

[1] Zhang Z, Zhang Q, Gao Z, et al. Gradient-based Parameter Selection for Efficient Fine-Tuning[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 28566-28577.

[2] He H, Cai J, Zhang J, et al. Sensitivity-aware visual parameter-efficient fine-tuning[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11825-11835.

[3] Lian D, Zhou D, Feng J, et al. Scaling & shifting your features: A new baseline for efficient model tuning[J]. Advances in Neural Information Processing Systems, 2022, 35: 109-123.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Could the authors provide additional results using Vim-Base or larger-scale architectures?
- How was the data utilized in the hybrid search method? Was a held-out set created for the searches?
soundness: 2
strengths: - The paper is well-written and easy to follow.
- Searching for diverse PEFT options for Mamba is informative and would be a valuable contribution for those who employ Mamba architectures. 
- Studying extensive PEFT options based on a baseline yields convincing results.
summary: This paper explores parameter-efficient tuning methods such as the prominent methods like Adapter, LoRA, Prompt-tuning, and Partial tuning for the Mamba architecture. The authors highlight that these methods have not been studied in Mambas compared to the extensive research conducted on Transformers. The primary focus of this paper is on the Vision Mamba (Vim) architecture, with a minor portion addressing language models with Mamba compared with the Transformer counterparts Pythia. The study compares Vim and Vision Transformer models using PEFT methods, which address some key architectural and methodological choices like layer selection for LoRA and prompt placement in Prompt-tuning, for example. The authors present a two-step PEFT search method that hybridizes various PEFT options to offer a near-optimal solution (in terms of performance). Finally, the authors present a new PEFT method called Additional-scan, which increases the state dimension of the SSM, and an additional initialization method for the state matrix A is also presented. The experimental results employing Vim-S pre-trained on ImageNet-1K to demonstrate the effectiveness of the diverse PEFT options; Mamba-language models compared with Pythias are also presented.
weaknesses: #### #  Main concerns
- The authors' claim in line 22 that PEFT is more effective for Mamba than for Transformers lacks adequate rationale and supporting evidence. There is no clear intuition as to why Mamba would benefit more from PEFT; the experiments appear biased due to an unfair comparison - specifically, Transformers were tested with only a limited range of PEFT options, while Mamba underwent more comprehensive testing (as illustrated in Table 1).
- The authors primarily tested Vision Mamba rather than the broader Mamba architecture, yet they frequently used "Mamba" as an abbreviation for Vision Mamba. This reviewer is concerned this may be misleading, especially as the result table for the language models offers only limited experiments. The authors should clarify their terminology to ensure readers understand that Vision Mamba is the specific variant.
- The authors presented a hybrid PEFT approach as an optimal-like solution, which is identified through their proposed search method. However, its conclusion does not specify a practical, generalizable option. This reviewer feels It's important to note that the effectiveness of the identified hybrid approach may not be universal, and the search process itself would be resource-intensive for a new architecture.

#### # Missing/wrong details in preliminary
- Section 2.1, titled *Mamba*, should provide foundational knowledge about Mamba as a preliminary, but it mainly presents on S4 or early SSMs. Since Mamba has slightly distinct notations and a signature architecture, the authors should revise this section to include more comprehensive details about Mamba.
- As the experiments involve the Vision Mamba architecture, the reviewer suggests including the original architecture, which has subtle design differences (e.g., the definition of B).
- The discretization presented in Eq.(2) is not the zero-order hold discretization as implied but rather the bilinear one. The authors should correct this detail.

####  # Some concerns about experiments
- The experiments used a limited set of baseline models. Although the PEFT options explored were diverse and met the experimental standards of a scientific study, only the Vim-S baseline is used (since the PEFT methods are more likely to be applied to larger architectures, Vim-S may not be a proper choice as a baseline), where this reviewer feels the main comparison focuses on Vision Mambas versus Vision Transformers.
- Vision Transformers are not thoroughly explored compared to the fully examined Mamba models. For instance, ViT-S in Table 1 has only a few options compared to Mamba's more comprehensive use of PEFTs.
- Some experimental comparisons lacked control:
  - In language models, for instance, Pythia 1.4B and the counterpart Mamba 1.4B had significantly different baseline performances (44.8 vs. 53.0), skewing the reported improvements, as higher starting accuracy inflates gains.

#### # Minors
- There is a related work that covers a similar topic, including PEFTs in Mamba, which suggests that this paper is not the first to study PEFT for the Mamba architecture as mentioned in lines 71-74. This reviewer understands that the authors could not include the related work due to its publication date being after this submission. However, it is suggested to compare this study with that work in future revisions to provide context and emphasize differences.
  - Parameter-Efficient Fine-Tuning of State Space Models, NeurIPS 2024 workshop (https://arxiv.org/abs/2410.09016)

- Typo in Table 1: Linear 'Proving'

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: see Weaknesses.
soundness: 2
strengths: 1 The authors extend PEFT methods from Transformer to the Mamba architecture and propose several Mamba-specific PEFT methods.

2 The authors conduct experiments not only in the vision domain but also in the language domain.
summary: This paper explores the PEFT methods for the Mamba. By modifying and adapting commonly used PEFT methods on Transformer, the paper proposes novel PEFT techniques tailored to the Mamba architecture. Experiments demonstrate that Mamba outperforms Transformer when applying PEFT methods. The paper also showcases how to effectively combine multiple PEFT methods to improve performance and introduces a new framework.
weaknesses: 1 Why wasn’t Vim-B selected for comparison experiments with ViT-B? The authors seem to have not provided an explanation in the paper.

2 In Table 1, the PEFT methods selected by the authors are relatively outdated, such as FACT, which originates from AAAI23.

3 Conducting PEFT experiments on Vmamba, another Mamba-based vision model, is necessary to demonstrate the generality of the proposed methods.

4 In section 3.4, the mixed PEFT search strategy proposed by the authors lacks sufficient theoretical support, especially regarding the principles behind why different combinations of PEFT methods can improve performance, which the authors have not sufficiently explained.

5 For the vision experiments, the authors only consider smaller models without discussing the applicability of these methods to larger-scale models.

6 For the new architecture—Mamba—the training duration of these PEFT methods is worth considering and comparing.

Meta-review: a) This paper explores the application of parameter-efficient fine-tuning (PEFT) for Mamba. It analyzes the suitability of existing PEFT methods for Mamba, adapts certain PEFT methods to better align with the characteristics of State Space Models (SSMs), and investigates hybrid PEFT search strategies. Extensive experiments on image and text modalities for various PEFT methods show that Mamba exhibits distinct fine-tuning behaviour compared to Transformers.

b) The paper improves existing PEFT algorithms to suit Mamba but also proposes targeted algorithms, providing the community with a broader selection of PEFT methods. The extensive experiments, covering both image and text modalities, provide comprehensive benchmarks for all methods and reveal differences in PEFT performance between the Mamba and Transformer models.

c) The algorithmic novelty of the paper is limited, as the proposed approaches are not very different form standard PEFT. For some results (e.g. PEFT is more effective for Mamba than for Transformers), authors provide empirical evidence but not theory, a rationale or intuition about the results. 

d) Overall, I consider that the paper deserves publication because: 
PEFT methods for Mamba architectures are still under-explored. The authors provided additional experimental results in their rebuttal, which addressed most of the concerns raised by reviewers. 
The paper includes extensive experiments covering various PEFT methods, different Mamba architectures, models of different scales, and different tasks, which can be have a strong impact for further research in the field.
However, I still think authors should work on further improving the presentation of the paper, for instance clarifying the contribution and providing a better understanding of the empirical results obtained, as suggested by reviewers.

Decision: Accept (Poster)