MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba
Authors: Masakazu Yoshimura
Teruaki Hayashi
Yota Maeda
Venue: ICLR 2025 Poster (2025)
Keywords: parameter-efficient fine-tuning
PEFT
Mamba
State Space Model
SSM
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We explore and propose parameter-efficient fine-tuning for Mamba.

Abstract: An ecosystem of Transformer-based models has been established by building large models with extensive data. 
Parameter-efficient fine-tuning (PEFT) is a crucial technology for deploying these models to downstream tasks with minimal cost while achieving effective performance. Recently, Mamba, a State Space Model (SSM)-based model, has attracted attention as a potential alternative to Transformers. While many large-scale Mamba-based models have been proposed, efficiently adapting pre-trained Mamba-based models to downstream tasks remains unexplored.
In this paper, we conduct an exploratory analysis of PEFT methods for Mamba. We investigate the effectiveness of existing PEFT methods for Transformers when applied to Mamba. We also modify these methods to better align with the Mamba architecture. Additionally, we propose new Mamba-specific PEFT methods that leverage the distinctive structure of Mamba. Our experiments indicate that PEFT performs more effectively for Mamba than Transformers. Lastly, we demonstrate how to effectively combine multiple PEFT methods and provide a framework that outperforms previous works. To ensure reproducibility, we will release the code after publication.

Published as a conference paper at ICLR 2025
MAMBAPEFT: EXPLORING PARAMETER-EFFICHEE FINE- TUNING FOR MAMBA
Masakazu Yoshimura", Teruaki Hayashi" & Yota Maeda
Sony Group Corporation Japan
yoshimura, yoaaskkauuyysshuuraa, (masakazu.
teruaki. .hanakiihhayshi,, yota.maede @snny com
ABSTRACT
An ecosystem of Transformer-based models has been established by building large models with extensive data. Parameter-efficien fine-tuning (PEFT) is crucial technology for deploying these models to downstream tasks with minimal cost while achieving effective performance. Recently, Mamba, State Space Model (SsM)-based model, has attracted attention as a potential alternative to Trans- formers. While many large-scale Mamba-based models have been proposed, ef- ficiently adapting pre-trained Mamba-based models to downstream tasks remains unexplored. In this paper, we conduct an exploratory analysis of PEFT meth- ods for Mamba. We investigate the effectiveness of existing PEFT methods for Transformers when applied Mamba. We also modify these methods bet- ter align with the Mamba architecture. Additionally, we propose new Mamba- specific PEFT methods that leverage the distinctive structure of Mamba. Our experiments indicate that PEFT performs more effectively for Mamba than Trans- formers. Lastly, we demonstrate how effectively combine multiple PEFT meth- ods and provide framework that outperforms previous works. The source code is available at: stttps://githii com/ sony /mambapefbbtttt
INTRODUCTION
Modern large-scale models, also known as Foundation Models, are heavily based on Transformers Vaswani 2017). Transformer-based pre-trained models span diverse domains such language, vision, and multi-modal applications. Despite their widespread use, Transformers have a notable drawback: their computational inefficiency with long sequences. The computational complexity of the attention module scales with the square of the sequence length.
To address this fundamental drawback, Gu & Dao (023) proposed Mamba, linear-time sequence model that leverages the strengths of State Space Models (SSMs). While Transformers are con- structed from attention modules, Mamba is based on the SSM architecture, allowing it handle long sequences more efficiently. Additionally, Mamba has been shown to perform better than Trans- formers with the same number of parameters on major tasks such natural language processing (NLP) (Gu & Dao 2023) and computer vision (CV) (Zhu et al.) 2024). This fact makes Mamba stand out from other post-Tran Transfrrnfrrm architectures with sub-square time complexity (Peng et al.) 2023 Sun et al. (203). Mamba-based models have been proposed across wide range of domains [eehrouz Hashemi] 2024 Liang et al., 2024| Zhang et al. 2024b et al. 2024). Mamba has the potential to go beyond the Transformer ecosystem.
Parameter-efffcient fine-tuning (PEFT) is essential for adapting such large-scale models to down- stream tasks. Fine-tuning all parameters of these models results in high computational costs. PEFT enables additional training for large-scale Transformers with limited computing resources and data. Early examples of PEFT include the fine-tuning of pre-trained language models for NLP tasks (Houlsby et al.) 2019: Hu et al.) 2021). Subsequently. it has been extensively adopted across a wide range of applications (Ych al.) [2024] Wang al., [023 2024a). While many PEFT methods have been extensively studied for Transformers (Lialin et al. 2023 Han et al. 2024), research on
PEFT methods for Mamba remains limited.
"Equally contributed

---

Published as a conference paper at ICLR 2025
In this paper, provide an exploratory and comprehensive investigation of PEFT for Mamba. First, we adapt representative PEFT methods used in Transformers to the Mamba architecture and conduct PEFT methods specifie to the Mamba architecture.
extensive experiments. We also propose In the experiments, we benchmarked Mamba using PEFT methods, including seven main methods and total of 20 derived variations (see Figure D. Our benchmarks indicate that PEFT is more crucial for Mamba than for Transformers, with several methods outper- forming the PEFT methods used for Transform- ers. Additionally, we demonstrate that these PEFT methods can be combined to surpass the performance of individual methods. We pro- pose efficient search technique to identify optimal PEFT combinations and hyperparam- eters. Unlike existing works that focus spe- cific perf--eerormance methods, we explore wide variety of PEFT methods. This explo- ration reveals suitable PEFT combinations and shows that merely combining ger-operforiin methods is not sufficient.
Paratiolidapper
Lineat
SBM
Selective Scan
Lineaf
Linear,
Convid
tuning
The main contributions are as follows. First, to the best of our knowledge, we perform the first extensive and comprehensive benchmark- ing of PEFT methods for Mamba, including proposed Mamba-specifie methods that are dis- tinet from all PEFT methods for Transformers. Second, we propose framework for achiev- ing higher performance by combining multiple PEFT methods, which are obtained through our efficient search technique. Third, results in- dicate that PEFT is more effective for Mamba than for Transformers, and several Mamba- specific phenomena are discovered through the experiments.
Linear,
RMS Norm
Partal-tuning
Prompe 8 :Di
funing
Addtive method
LaRAmadule
Embeddiing Layer
CLS taken
Figure 1: baPEFT. We investigate, improve, and propose 20 variations of seven PEFT methods for Mamba and search for the best combination.
An overview of our proposed Mam-
RELATED WORK AND PRELIMINARY
In this section, we review related work and present the motivation behind our research. We begin with an introduction Mamba. Subsequently, we discuss PEFT methods for Transformers. Finally, highlight that research on PEFT for Mamba is limited.
2. MAMBA
SSMs are powerful methods for modeling dime-dependent dynamic systems. It was developed and analyzed in detail address the signal processing technique classically known the Kalman filter [Kalman] 1960) and has been used to date in a wide range of fields, such as control engineering MMciiejossi & Huzmezan 2007). An SSM (Gu et al.) 2021a takes an input Xt, converts it to hidden state H,, and then outputs Y, This procedure at each time step t is
H, A,H,-1 + B,X,, Y C,H, + D,X,-
As this model describes the relationship between continuous quantities, it must be described to deal with discrete quantities such as NLP. During step size d, the input is assumed to be invariant and the well-known theory of zero-order hold is applied. Putting
A = exp(dA),
B (dA) (dA))-((xpp(de (exp(dA) - T)dB, C = C, D:=D
allows us obtain the final discrete representation:
H, AH BX,
Y, CH, DX.
(3)
2

---

Published as a conference paper at ICLR 2025
While many SSM-based DNN have tuned the A, B. C. and D (Gu et al. 2020 State Space Model (S6) was introduced in Mamba (Gu Dao 2023), in which the A, B, C, and D dynamic and data dettndeennddnt follows:
202 Selective
(4) where A. - softplus (WapllussWWWWa (W + ba) for any t, Wa. The Wx, WB, We, WD. and A are trained matrices, and مa is trained bias.
A exp(- B, = A.W C WoX, Dj = WDX,
Impressive results of Mamba in NLP have inspired researchers to adapt it to visual tasks. Vim (Zhu et al., 2024) is a pioneering work in this area. Similar to VIT (Dosovitskiy et al., 2020), it divides images into patches and then inputs the patch sequences into SSMs. Vim improved performance in visual tasks by processing patches bidirectionally with S6 and placing a class token in the middle of the patch tokens instead of at the beginning. There are other namba-based methods for visual tasks. For example, VMamba (Liu et al., 2024b) is hybrid approach combining Mamba with 2D Convolutions. Mamba Vision (Hatamizadeh & Kautz 2024) further integrates Attention. In this paper, we conduct experiments with Vim, which a variant of the vanilla Mamba.
2.2 PARAMETER- EFFICIENT FINE- TUNING FOR TRANSFORMER
PEFT methods have been actively studied due to the large model size of Transformers extensively in both the NLP and CV communities. These studies can be categorized into four methods; partial, additive, reparametrization, and hybrid according to Lialin et al. (023)) Han et al. (2024).
Partial-tuning methods. While usual fine-tuning updates all parameters in the network, Partial- tuning updates a part of them. BitFit (Zaken et al., 202T) is successful method in this category which tunes only the bias terms of the linear layers in Transformers. However, most of the linear layers in Mamba do not have the bias terms, which implies that it cannot be applied priori. Other methods use pruning to decide where to tune Zhang et al. (024c ; He et al. (2023).
Additive methods. Additive methods add a small number of additional parameters or small net- works to fine-tuned. They can be further divided into additive adapter and additive token methods. The former adds adapter modules whose rank is reduced from the input dimension d (r << d) by a bottleneck structure. Adapter (Houlsby et al.) (019)) attaches the adapter in series with the feed-forward network (FFN) module in Transformers. In the following methods, Adapter+ (Steitz) & Roth 2024) improves the position of the input feature acquisition to the adapter. ParallelAdapter (He et al. 2022 and AdaptFormer (Chen et al. 2022) improve performance by inserting the adapter in parallel to the FFN. We adopt Adaptr for Mamba because it is successful in both language
(He et al. 2022 and vision (Chen et al. 2022) tasks. attaches the adapter
(5) where the second term is the adapter with additional parameters Wdown E Rdxr and W up RR and a scaling hyperparameter s. In the latter category, Prompt-tuning (Liu et al. 202T) (resp. Prefix- tuning (Li & Liang 2021)) adds learnable soft tokens to the input of the network (resp. soft tokens inside each Attention layer). Successive methods improve how to embed soft tokens (Liu et al.) 2022 Razdaibiedina et al. 2023 Zhang al. 2023). Though prompt-tuning for ViT adds learnable tokens at the beginning of the input (Jia et al. 2022; Wang et al., 2024b). it is unclear where the token should be added for Mamba because the order of tokens makes sense in SSM. In fact, Vim (Zhu et al. 2024) adds class token in the middle of the input sequence. Hence, we investigate multiple choices of soft token insertion.
(F,( + ReLU(xz Wdown) W
Reparameterization methods. Although reparameterization methods add low-rank adapters to Linear layers similar to the additive adapter methods, they adopt adapters that can be merged into the weights of the original Linear layer at the inference time. Due to this restriction, activation cannot be used the adapter, while it eliminates the inference time overhead. The most notable one is LoRA (Hu et al.) 2021), a method that reparameterizes the pre-trained Linear weights W as W' W + s- W down W Wup- While subsequent studies have improved the efficiency by finding ways to reparameterize the weights (Lian et al. 2022 HHaouu et al., 2024 Liu et al.) 2024a Jiang et al. 2024). we first investigate how many ranks of the adapter should be attached to which linear weights in Mamba. In this regard, we will use simple LoRA with few hyperparameters.
Hybrid methods. Hybrid methods use multiple PEFT methods. While many approaches manually combine these methods (Mao et al. 2021 He al. 2022 Karimi Mahabadi et al.) 2021), several
3

---

Published as a conference paper at ICLR 2025
methods automatically tune the combination of several PEFT methods. S2 (Chen al. 2023) uti- lizes random search while changing the dimension of the search space. AutoPEFT (Zhou et al. 2024) and NOAH (Zhang et al., 2024a) train supernet containing all possible PEFT methods and discover the optimal combination using an evolutionary algorithm.
Supemee-bbaeed methods only be used to search for each task individually. However, aim to find the optimal combination for Mamba that can be generalized across multiple tasks, making useful for various future applications. Hence, we propose efficient search technique to find a compact combination of PEFT methods. This is necessary because we have many more PEFT methods and hyperparameters than previous works, resulting in large search space.
2.3 PARAMETER- EFFICIENT TINE-TUNING FOR MAMBA
In contrast the case of Transformers, there are limited studies PEFT for Mamba. To the best of our knowledge, the only study in this context Halloran et al. (2024). which applied the same rank of LoRA to all the linear layers in Mamba. With this motivation, conduct an exploratory investigation and provide benchmarks of PEFT methods for Mamba.
INVESTIGATION OF PEFT METHODS FOR MAMBA AND BEYOND
In this section, explore PEFT methods for the Mamba architecture. First, we discuss how ex- isting PEFT methods for Transformers can be adapted to Mamba. Next, we present methods that have been improved and modified based on the characteristics of Mamba. Finally, propose new PEFT methods specifically for Mamba and hybrid PEFT methods to search optimal combinations efficiently.
3.1 SIMPLE ADAPTATION EXISTING PEFT METHODS TO MAMBA
Some PEFT methods can be directly applied to Mamba because they are network architecture- independent. For example, ParallelAdapter attaches a parallel adapter to each FFN in Transform- The counterpart of the FFN in Mamba is the out proj layer, and hence attach it to the out proj.
LoRA is also network architecture-indeeeddee method and be applied directly Mamba. However, it is still unclear which Linear layer should be targeted and what hyperparameter val- ues should be used. We individually investigate LoRA on each module as LoRA(embedding). LoRA(in-projj) LoRA(x-proy) LoRA(dt-proo) and LoRA(out-proo) to clarify the appro- priate way to apply LoRA to Mamba (see Figure
3.2 RE-DESIGNING AND IMPROVING EXISTING PEFT METHODS FOR MAMBA
Partial LoRA. Mamba is network characterized by numerous intermediate features with diverse properties, such as X, Z, dt, B, C, and so on (see Figure 1. In the previously mentioned normal LoRA. the inputs for multiple intermediate features are compressed together in the narrow rank of LoRA, even though they have different properties. Therefore, we now develop Partial LoRAs LoRA,(X), LoRA, (Z), (dt), LoRA, and (o)A,(C), where we apply LoRA only part of the weights in the linear layer according to the output features. Since the dimensions of linear layers vary widely Mamba (e.g. from 16 2048 in Mamba-1.32 1.3B), we also investigate whether there is an optimal dimension per layer or not to attach LoRA.
Prompt-tuning. Prompt-tuning can be applied directly to Mamba. As with Transformer, we simply add soft prompts to the input sequence. In normal Prompt-tuning, prompt is inserted at the be- ginning the input sequence. For Vim, however, we adjust the insertion position. This because, unlike ViT, Mamba is a time-series model and works differently depending on where the prompt is inserted. In fact, Vim improved accuracy by inserting the class token in the middle of the sequence. In addition, Mamba Register DDrcett et al. 2024) found it is preferable insert the register tokens at equal intervals. Hence, in this paper, we construct three prompt types; prefix (resp. infix, suffix) type, inserting prompt tokens at the beginning (resp. with equal intervals, at the end).
4

---

Published as a conference paper at ICLR 2025
(8) Prefix-tuning
Affix-tuning
(c) Additional-scan for Mamba
Transformer
Dacaad
MatMul
Softmax
SSM
Linear
Pv
B
Wa
Wy
Linear
Figure 2: (a) Prefix-tuning designed for Transformer. It can't be applied to Mamba. (b) The pro- posed Affix-tuning, which we re-design for Mamba. It discards prefixes after SSM. This design allows us insert affix tokens at arbitrary locations. (c) Additional-scan that design for Mamba. In this method, we add learnable dimension to the hidden state in SSM.
Aftix-tuning, We cannot apply Prelix-tuning directly to time series models such as Mamba, since it designed for the Attention mechanism based on query, key, and value. As shown in Figure 2 by adding soft tokens to only K and V, Prefix-tune tiniig successfully adds new information to all tokens in the input sequence without breaking the positional relationship of the original input sequence. Motivated by these observations, we propose PEFT method that adds soft tokens before input to SSMs and discards the output tokens the inserted position (see Figure 2 This preserves the positional relationship between input and output tokens even when tokens are added. Furthermore, for Vim, we adjust the insertion position the same way as for Prompt-tuning Since the name Prefix-tuning can be misleading, we generalize by calling it Affix-tuning.
3.3 NEW PEFT METHODS DESIGNED FOR MAMBA
Cartial-tuning, BitFit Zaken et al. 2021 is effective among Partial-tunin methods because it fine- tunes bias terms with low degrees freedom and they do not break the pre-trained model. Different situations for the Partial-tuning in Mamba is that, unlike Transformers, it does not utilize bias terms in linear layers without dt uproj, while it has various other parameters with low degrees freedom such as the parameters A, D. and the weights of the causal convolution whose dimension is one. Based on this situation, we construct Partial-tuming for Mamba, named Bias-tuning, A-tuning, D- tuning, and Convld-tunins also make positional embedding and class tokens learnable. In addition, we enhance our method from conventional Partial-tre Partial-tuning by modifying the weight decay. Specifically, instead of the usual weight decay of [ we propose applying weight decay of IW- W Wpretrain| 2 to preserve the pre-trained model.
which fine-tune the previously mentioned parameters. For Vim,
Additional-scan. We propose new efficient PEFT with fewer parameters by leveraging the SSM architecture. By increasing the state dimension of the SSM, we aim to store information in adapt to new tasks, called Additional-scan. It corresponds to additive methods. Specifically, we increase the state dimension of the input parameters of SSMs, A, B. and C, by N'. Eq. then becomes as follows:
TLN++N = exxp( exp(-A[T.LJ) AU.L,+N')), T II..LA+N' = AT.LI BIT.LNNN1, (6)
where T, L., and N are the token length, hidden size, and state dimension of the SSM, and pld, represents P E IRd xds, Then, the core recurrent computation in SSMs becomes
M INN+N [L.N+N), HILN+NI + BB [L,N+N I
y|I..]] -.HLL (L,NN' -CT, (NNT). DI4.1)
5

---

Published as a conference paper at ICLR 2025
In Mamba, diagonal SSM is employed and each hidden dimension forms independent recur- rence. Hence, we can write down more about one hidden state dimension as follows;
[he,1 ... Ne, N .. NN = eepp( - 00,00 1 + 8t,0b1,1
exp( 00,000,N 14-1,N + be,0be,
exp( -
exp(
(8)
can be observed that the added parameters do not affect the original hidden state, h NNNN This enables us to store new information with additional selective scan without affecting the pre- trained hidden memory and selective scan mechanism. To add new dimension to A, B, and C. we add additional trainable parameters to A and to the Linear a proj that generates B and C' (see Figure [) The parameter A SSM needs careful initialization (Gu et al., (020). known as the Hippo theory. Mamba uses S4D initialization (Gu et al., 2022), A,,,, n, while in our case, the other A values have already been changed by training. which implies that S4D initialization is not theoretically better. Thus, propose an initialization of A for Additional-scam Specifically, the additional parameters of A are initialized with the values of neighborhood pre-trained A. is simple, while stable training is achieved. Accounting for eq. g and the pre-trained model use S4D) initialization, the top of the state dimension intended for long-term memory or selective scan from distant tokens, while the bottom is intended for short-term memory selective sean from neighbor tokens. Thus, when fine-tuning, the suitable position of the additional state dimension is checked whether at the bottom in eq. or at the top.
3.4 HYBRID PEFT SEARCH
We have introduced the primary seven methods follows: ParalleLAdapter, LoRA, Partial LoRA, Trompt-tuning Affix-tuning. Partial-tuning and Additional-scan Specifically, there are five varia- tions of LoRA, five variations of Partial LoRA, and six variations of Partial-tuning, making a total of 20 methods. This variety motivated us to propose hybrid PEFT for Mamba, which combines multiple PEFT methods to enhance performance. There are three key elements to consider in our hybrid PEFT: the combination of PEFT methods, the hyperparameters of each PEFT method (e.g., a rank of LoRA), and the training hyperparameters for each method (e.g., learning rate). Exploring them simultaneously results in enormous search space compared to the search space of previous works which target only several PEFT methods. Furthermore, the hyperparameters have a condi- tional search space, meaning they are only considered if the corresponding PEFT method is active. Thus, we propose a two-step approach that explicitly divides the problem into two parts to improve efficiency in the search process.
In the first step, we restrict the search space to combinations of PEFT methods with minimal train- able parameters within each method, i.e., we explore whether activate each PEFT method with fixed hyperparameters with minimal trainable parameters. This phase is based on our observation that performance degrades when too many PEFT methods trainable parameters are added. In the second step, based on the PEFT method combination suggested by the first step, we greedily search the hyperparameters of both PEFT methods and their training. Since this phase inevitably increases the number of trainable parameters, we also include an option remove a specific PEFT method.
This approach helps to prevent excessive parameter growth and allows us to allocate more parame- ters to the more critical methods. We utilize the TPE algorithm (Bergstra al. (011)) implemented in Optuna (Akiba et al. 2019) for the search in each step. The detailed algorithm is provided in
Appendix B
4 EXPERIMENTS AND DISCUSSION
In this section, we present our experiments on 7 PEFT methods with 20 variations. We conduct experiments both image and language tasks. Our methods compared with oate-of-the-an art methods for Transformers. We provide detailed results and analysis based on these comprehensive experiments.

---

Published as a conference paper at ICLR 2025
Table 1: Test accuracy on the VTAB-IK benchmark pre-trained on ImageNet-IK using the training technique of DeiT' (Touvron et al. (02T). The best hyperparameter settings maximizing the averaged aceuracy of six datasets written the ablation studies are used for each PEFT method. The Time Ratio represents the ratio of the training time to the time taken for full fine-tuning. The number of trainable parameters the average of all tasks.
Model
Method
#Params (K)
TTme Ratio
Natural
Specialized
Structured
Avg.
Scratch
21,704
10,66
56.12
24.83
26.20
Full
21,704
51.79
72.79
45.27
53.47
Lineur Probing
60.87
78.13
30.57
51.74
VIT-S
PaCT-TK
16
72.87
82.34
54.10
66.96
LoRA
628
73.60
82.22
57.61
68.68
Adapttormer
333
73.63
83.15
57.80
68.97
SPT-LoRA
74.75
$6.99
69.38
Adapter+
122
74.68
83.57
58.82
69.87
Scratch
25,450
1.00
8.33
49.87
28.16
25.42
Full
25,450
1.00
59.35
68.74
34.39
47.08
Lineur Probing
0.19
62.50
77.25
31.97
52.75
CLS-token-Tuning
62.30
77.20
32.20
32,84
Pos-embedd-uningg
84
0.52
64.25
74,60
39.77
56.12
Bius-tuning
37
0.54
68.94
79.16
45.05
61.03
D-tuning
45
0.56
67.14
78.56
40.10
58.16
A-tuning
72.01
Conv lenvlddtuninn
156
0.54
74.33
82.45
57.83
69.09
Prompt- tuning (wlo proj)
12
0.53
65.40
78.51
38.35
56.77
Prompt-tuning
0.54
69.92
79.20
47.75
62.54
Affix tuning (wlo proj)
230
72.26
50.73
65.04
Aflix-tuning
117.000
0.66
75.84
83.29
58.94
70.29
Vim-$
Additional-scan
672
0.65
74.63
82.68
56.40
68.65
ParallelAdapter
0.55
76.10
83.97
59,97
70.96
LoRACembed)
45
0.52
64.66
77.55
43.33
36.60
LoR LoRA(x_pro)
2,540
0.57
74.41
81.92
$4.88
67.77
LoRA prop
2,442
0.57
75.35
83.05
57.12
69.30
LoR A(cutlpmm)
2,663
76.42
71.12
LoRA(in.proj
1,483
0.67
76.58
84.08
60.16
71.25
(d)
2,442
0.63
73.25
80.91
50.93
65.46
LoR (C.)
2,417
0.57
72.78
81.57
$1.35
65.61
(B)
2.417
72.95
52.26
LoR (,))
1,778
0.67
76.15
84.26
59.72
70.94
LoRA,
1,778
0.66
76.64
83.89
60.84
71.52
(w/ proj)
119,765
0.86
74.67
82.96
53,92
67.68
LoRA (==8)
1,228
LoRA(in Aiin_proj-ourppoo)
709
0.72
75.69
84.42
59.43
70.68
Hybrid (w/ proj.
117,236
0.86
77.00
84.41
61.55
72.05
Hybrid (w/o proj)
1.044
0.83
76.85
84.42
61.06
71.80
4.1 EXPERIMENTAL SETTINGS
Models. We Vim-S Zhu al. 2024 base model in our experiments. We also experiment with ViT DDssoviskkiy et al. 2020) for comparison. We adopt pre-trained weights trained with
ImageNet- Ik (Deng et al. al. 2009) using DeIT (Touvron 2021) training framework in all models.
Dataset. We conduct our evaluation on the VTAB-IK Ik image classification dataset (Zhai et al. 2019) This dataset contains tasks from 19 domains. These tasks are grouped into three categories: Natural, Specialized, and Structured. For each task, 1000 images are used for training.
Baselines. We adopt LoRA (Hu et 2021). Adaptformer (Chen al. 2022), FacT-TK (Jie &
Deng 2023). SPT-LORA A (He et al. 2023). and Adapter+ (Steitz & Roth 2024 existing PEFT methods for Transformers. For Vim, we evaluate the 20 PEFT methods explained Section 3 baseline for Hybrid PEFT, we use combination of all PEFT methods in the experiments.
Implementation Details. We follow the setup Jie & Deng (0033 our experiments, using AdamW optimizer (Loshchilov & Hutter 2017) and training the model for 100 epochs. The learn- ing rate is set to 1e-3, with a cosine scheduler and a warmup period of 10 epochs. A weight decay with 1e-4 magnitude is applied. We do not perform data augmentation. For ablation studies, the average accuracy on six tasks: CIFAR- 100, Sun397. Camelyon, Retinopathy, Clevr-Couna and SNORB-Elev. In several previous studies, hyperparameters tuned for each task, and the maxi-

---

Published as a conference paper at ICLR 2025
62
60
60
Acc. (%)
Áoe. (9)
58
56
54
10
20
30
40
50
60
70
Rank LoRA
Additional dimensiio
x proj 56))
Top (84I) init)
Top (our init)
(full-rank=16) + mfull-mnk=11)
Bottom (S4D init) + Bottom (our init))
proj (full-rank-21)
(b)
(a)
Figure 3:
Ablation studies for (a) the relationship between LoRA rank and accuracy on small linear weights, and (b) the impact of additional dimensions in Additional-scan performance.
Table 2: Ablation studies. (a) The position and number of tokens in Affix-tuning with projection. If position "Before", tokens are inserted before the prop layer. (b) The initialization method in Additional-scan. (c) The performance improvement in Partial-tuning with proposed Weight Decay.
Type
#Tokees
(%)
Location
Init.
Acc. (%)
Method
wd=le-4
wd=le:3
wd=1e-2
Bottom
S4D
Bias-tuning
+0.05
+0.02
+0.17
Aller
After
After
Profis
Profix
Infix
2
63.36
61:21
62.91
Const(O.1 I)
Const(1)
61.25
61.33
Cls token-tuning
Conv ddtunddgg
-0.40
-0.01
+0.08
+0.18
-0.01
After
Prefix
3
6.3.11
Const(5)
60.05
Pos-ombed-tuning
+0.09
Afler
Prefix
56.57
Ours
61.46
A-tuning
+0.07
+0.15
+0.11
(a)
Top
S4D
57.56
D-tuiing
-0.08
+0.02
-0.04
Ours
61.61
(c)
(b)
mum validation accuracy during training was reported. In contrast, we tuned the hyperparameters to maximize the average of six tasks used in ablation, ensuring that the results obtained are universal and beneficial for many future studies. Additionally, the evaluation conducted using the weights from the final epoch.
4.2 RESULTS FOR INDIVIDUAL PEFT METHOOSS
Overall Results. Table shows the performance with hyperparameters that maximize accuracy. Many PEFT methods for Vim outperform the state-of-the-ar art ones for VIT. Interestingly, the PEFT methods for ViT start over-fitting early when the number of trainable parameters increased and unable improve accuracy. On the other hand, Vim continues to improve its accuracy even with more trainable parameters. This is probably due to the highly modular structure of Mamba, which prevents its pre-trained memory from being corrupted by additional parameters. In addition, Vim has greater performance improvement margin between full fine-tuning and PEFT than ViT has. These results suggest that PEFT more effective for Mamba than Transformer.
LoRA. In our experiments, LoRA turns out to be one of the most effective PEFT methods with Vim. As shown in Table [ PEFT methods perform well regardless of where LoRA is applied. The best performance is achieved with the partial LoRA, LoRA, This result stems from our detailed investigation, indicating that the effectiveness of LoRA depends on the nature of the output features.
Another notable finding regarding LoRA is that when it is attached to linear weight with small dimension, the accuracy continues to increase even if the rank of LoRA exceeds the full-rank, as shown in Figure 3al Full-rank refers to the smaller value of the input dimension output dimension. This phenomenon is likely because the effective degrees of freedom are bounded by the full-rank. parameters does not cause overfitting. Instead, it positively contributes It follows that adding similar to the findings in Ding et al, (202I). More details regarding the to over peeeraaamettrizationn
LoRA on large linear weights described in Appendix Appendix A.1
8

---

Published as a conference paper at ICLR 2025
Affix-tuning and Prompt-tuning. The ablation studies affix-tuning shown in Table Za First, our Affix-tuning has multiple choices about where to add soft tokens because it is architecture- independent compared to Prefix-tuning for transformers, shown in Figure We test two positions, before or after the in proj layer. If before, the output tokens are discarded just before the out proj layer. If after, they are discarded after SSM. We find that it is important to affect additional tokens only on SSM by adding after the in-proj layer. Moreover, the best performance is achieved by adding an aflix at the beginning of the input tokens, just as with Prefix-tunin -unnin for Transformer, and different from how to handle additional tokens for Vim in Zhu et al. (2024): Darcet et al. (202)). This should be because unless they are trained from scratch. A detailed analysis of the positions and numbers of inserted tokens is provided in the Appendix A..
impossible to learn which positions are additional tokens rather than patch tokens
Note that these methods have the option of applying embedding projection to the prompts, which significantly increases the number of parameters. However, it can be merged during inference, re- sulting in zero computational cost increase. Our results show that embedding projection is effective in improving accuracy, and hence is recommended to use if memory allows.
PEFT methods designed for Mamba. We perform experiments on two newly proposed PEFT methods for Mamba. The results of Additional-scan are presented in Table Additional scan shows competitive performance with the best settings of LoRA and Affix-tuning, with fewer trainable parameters. Figure 35]and and Table 26 list the experiments with different initialization for Additional- scan. It shows that our proposed Initialization consistently outperforms the original S4D used in Mamba. The results of Partial-tuning are presented in Table [] Performance highly depends on which components are made learnable. The highest performance is achieved by making conv Id learnable. Our proposed modified weight decay works effectively around le-3 strength (Table 2c We analyze the number of trainable parameters and hyperparameters in Appendix A.2 detail.
4.3 RESULTS OF HYBRID PEFT
The bottom of Table T shows the effectiveness of our two-step optimization approach. Combining all PEFT methods results in lower performance compared to using single method. In contrast, our two-stage search method achieves higher performance with fewer parameters. These results indicate that selecting a preferred combination of PEFT methods is crucial.
We find that the high-performing single methods combinations. An example of this combination is provided in Table I0 in Appendix [B.2] We hy- pothesize that this phenomenon is related the importance of ensuring model diversity in ensemble methods (Dettteric)] (000). Combining PEFT methods with different mechanisms can offset indi- vidual errors, mitigate overfitting, and enhance generalization performance. Detailed verification of this aspect remains direction for future research.
necessarily selected among the optimal
4.4 LANGUAGE TASKS
In addition to the image tasks, we evaluate our method on language tasks using the vanilla Mamba Gu & Dao (2023). We experiment with commonsense reasoning task, following the setup and dataset of Hu et al. (023)). This task consists of eight sub-tasks, which are evaluated on each task after training on dataset that integrates training data from all tasks. We use Pythia (Biderman al. 2023] as the Transformer baseline, which is pre-trained with the same dataset as Mamba. For PEFT method for Mamba, we compare with SLL LoRA (Halloran et al., 2024). To the best of our knowledge, SI.. LoRA is the only PEFT method for Mamba.
In the language tasks, the proposed Additional-scan is found to work as an efficient PEFT. We hy- pothesize that this is related to the amount of data. This experiment uses 170k datasets, in contrast to the Ik used for VTAB-IK. We hypothesize that learning a selective mechanism requires a relatively large amount of data, and Additional-scan works powerfully in such cases. In Affix-tuning, we find that, as the size of the base model increases, it achieves sufficient accuracy without the embedding projection. This is valuable discovery because reducing memory costs is crucial for larger models. As discussed in Appendix(A.4) we find that Additional-seaan Affix-tuning (w/o proj), and LXRA,((X) are all useful depending on the application and should be used accordingly.

---

Published as a conference paper at ICLR 2025
Table 3: Experimental results of the commonsense reasoning tasks. Language Model Evaluation Hamess (Gao et al. 2024) is used to benchmark. The best performance is highlighted in bold, and the second-best performance is underlined, excluding Full fine-tuning.
Model
Method
#Params(3)
BoolQ
PIQA
HellaSwag
WaneCrande
OBQA
Avg.
Pythia 100M
Fall
100
61.3
62.9
37.1
30.7
50.6
41.5
24.3
27.8
42.0
LoRA
0.72
61.0
62.0
36.3
30.3
52.0
38.2
24.6
28.0
41.6
Pull
56.1
63.3
MET
35.3
3200
46.2
25.7
52.3
Mamba 130M
SLL LoKA
1.45
56.3
63.3
38.2
34.6
51.6
43.5
23.6
30.6
42.7
Addtiona--san
0.51
57.8
64.1
37.5
34,5
$3.0
41.3
23.5
30.0
42.7
tuning (wlo proj)
51.5
Affix-runing
64.64
59.7
64.3
38.2
35,2
51.9
42.9
24.0
29.0
43.2
LoRAfin proj)
2.23
53.5
62.9
38.2
33.8
53.1
46.4
23.7
30.8
42.8
LXRA,
Pythia 410M
Full
100
55.0
68-4
42.
40.8
$3.9
50.8
26.7
30.0
46.0
LoRA
0.77
61.3
67.7
40.8
39.2
$4.9
48.1
24.7
28.6
45.7
38.
Mamba 370M
SLL LoKA
2.30
59.5
69.6
42.2
44.1
54.9
50.6
26.
30.8
41.3
Additional-scan
0.47
61.9
69.3
45.3
54.9
28.4
49.5
31,4
47.7
Aflix-tuning (wlo proj)
67.7
39.3
46.3
54.1
47.8
28.2
31,0
Allis tuning
68.88
61.2
68.4
39.6
46.2
55.4
48.2
28.2
30.6
47.2
LoRAfin proj)
2.07
55.4
68.6
41.0
44.7
54.1
$2.4
28.3
33.4
47.2
LoRA,(X)
60.8
68.8
42.1
44.7
32.2
Pythia
LoRA
Full
0:41
100
60.0
$5.0
69.3
70.2
42.5
47.5
45.3
$4.4
53.6
54.1
49.8
27.2
29.7
33.2
31.0
48.3
47.1
72N
35.
57.77
SLL LoRA
3.1
60.7
72.0
42.4
$4.7
$6.9
55.3
29.4
34.2
50.7
Mamba 790M
Affix-runing (wlo proj)
Additionalsscan
0.33
63.0
57.2
71.9
71.7
41.9
55.0
54.2
$7.1
54.9
52.6
29.8
30.0
32.6
33.0
50.7
Allis-tuning
69.99
61.0
72.5
41.0
$4.9
35
54.6
29.6
33.8
50.4
LoRAfin projj
61.7
71.9
44.0
50.8
56.7
56.3
30.5
33.8
50.7
LoRA,(X)
59,9
22.2
Pythia 1.4B
Full
100
$8.6
21.1
42.7
$3.6
55.1
58.5
29.9
34.88
50.5
6..
733
56.9
SLL LoRA
4.64
59.7
73.5
43.1
56.9
60.7
59.7
31.7
36.0
52.7
Additiona-scan
73.5
37.4
Aflis- tuning (wlo proj)
0.09
63.6
74.0
41.9
58.9
60.6
61.6
33.4
36.8
53,9
LoRAfin proj)
1.13
62.6
73.6
43.7
55.6
59.7
58.3
31.7
35.6
$2.6
73.
37.4
4.5 LIMITATIONS
This paper focuses on an exploratory investigation. Evaluating the applicability of our findings to larger models, such as vision models trained with ImageNet-2IKK (Ridnik al. (021 or large language models with billions of parameters (Tourron et al. 2023), is an area for future work. We release the source code stimulate future research from communities.
CONCLUSION
We conducted comprehensive investigation of PEFT for Mamba. Our investigation started by applying existing PEFT methods for Transformers to Mamba. We proposed Affix- affix-tuning and Partial LoRA modifications of existing PEFT methods and introduced Partial-tuning and Additional-scan as new Mamba-specific PEFT methods. We also provided an effective framework for combining them. These comprehensive experiments image and language tasks led several findings.
Our findings are as follows. (I) Mamba benefits from PEFT more than Transformers. It improves accuracy without overfitting, even with larger number of parameters in PEFT (Compare Figure and Figure 4d). (2) LoRA-based methods, particularly LoRA, are effective with limited data
(See Table (3) In contrast, the proposed Additional-scan excels with larger datasets (See Table
and Figure (4) We found that initialization significantly impacts performance when adding an additional dimension to A, despite Mamba's robustness to initialization compared other SSMs
(See Figure 3b (4) The proposed Affix-tuning was effective, especially for large Mamba models
(See Table 3p. (5) When adding LoRA on small linear weights, interestingly, over-parameterrzinn
the original weight rather than decomposing it to a lower rank can be beneficial, which should hold for other architectures than Mamba (Figure Bay. (6) In hybrid PEFT, simply combining individually high- pergoperffrmaece methods, many previous works did, is inefficient. It is crucial to find the appropriate combination of PEFT methods. (See Table [ make valuable contribution to the Mamba research community.
and Table 9 We believe that these results
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We would like to thank Takeshi Ohashi, Wei- Yao Wang, Helen Suzuki, and Julian Tanke for their helpful comments to this manuscript.
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase. Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 262226222222222 2631, 2019.
Ali Behrouz and Farnoosh Hashemi. Graph mamba: Towards learning on graphs with state space models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, 119-130, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704901. doi: 02.1445/36375283377244 URL https://doi.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 24, 2011.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. Parameter- fine- tuning design spaces. In The Eleventh International Conference on Learning Representations, 2023.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances Neural Information Processing Systems, 35:16664-1663 166788 2022.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference Learning Representattions, ICLR 2024, Vi- enna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL URL/ittps://oeeereriee net//
Sorum?id=2cd03mm
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Puttern Recognition, pp. 248-255, 2009. doi: 10. 109/C/P..200000 200.5508884.. 5206848.
Thomas G Dietterich. Ensemble methods in machine learning. International workshop on multi- ple classifier systems, pp. 1-15. Springer, 2000.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma. Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vtg-style convnets great again. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 13733-13742, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 1rXi::2010.11929 2020.
Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2toe: Utilizing in- termediate representations for better transfer learning. In International Conference on Machine Learning, pp. 6009-6033. PMLR, 2022.
Kevin Galim, Wonjun Kang, Yuchen Zeng, Hyung Koo, and Kangwook Lee. Pafemeter-efffcient fine-tuning of state space models, 2024. URL htt.h:ttttt/aattee www.:iiivorraggaee wtt.:///////a/////oo

---

Published as a conference paper at ICLR 2025
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa. Jason Phang. Laria Reynolds. Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. Utt_htttss://eenooo wwtp://or/rrrrrroo///o////oooose 12608602
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 2023.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.0003 2021a.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Com- bining recurrent, convolutional, and continuous-time models with linear state space layers. Ad- vances in neural information processing systems, 34:572-585, 2021b.
Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971- 35983, 2022.
John T. Halloran, Manbir Gulati, and Paul F. Roysdon. Mamba state-space models can be strong downstream learners. arXiv preprint arXiv: 2406.00209, 2024.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-eeffcient fine- tune-tuning for large models: A comprehensive survey, 2024. URL htttp:://arxxvv.orr//bbb/////// 14608
Ali Hatamizadeh and Jan Kautz. Mambavision: A hybrid mamba-transformer vision backbone, 2024.
2200b./4407..880 08083
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models, 2024.
Haoyu He. Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-aware visual parameter-efficient fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11825- 11835, 2023.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrics and Graham Neubig. Towards unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, 2022. URL httpaas://ppeeeeeiieei wttp://nwweeeeeeeeeeeouuuuuueneo /tt////roomn--00DddAAA00-
net.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficiee transfer learning for nlp. In International conference machine learning, pp. 2790- 2799-2799. PMLR, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09635, 2021.
Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, R. Lee, Lidong Bing, and Soujanya Poria. Lim-adapters: An adapter family for parameter-efficient fine-tuning of large language models. Conference Empirical Methods in Natural Language Processing, 2023. doi: 10. 48550/anXXix.2044.1992
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie. Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
12

---

Published as a conference paper at ICLR 2025
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang. et al. Mora: High-rank updating for parameter-efficimt fine- tuning. arXiv preprint arXiv:2405.12130 2024.
Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 1060-1068, 2023.
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances Neural Information Processing Systems, 34:1022- 1035, 2021.
Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-nd: Selective state space modeling for multi-dimensional data. arXiv preprint arXiv:2402.05882, 2024.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.001 2021.
Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: guide to parameter-efficient fine-tuning. arXiv preprint arXiv: 2303. 15647, 2023.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35: 109-123, 123, 2022.
Dingkang Liang, Xin Zhou, Wei Xu, Xingkui Zhu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. arXiv preprint 200709940..0079, 2024.
Shih-yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P- tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning across scales and tasks. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, 2022.
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint vrXiv:2401.10166 2024b.
I. Loshchilov and Hutter. Decoupled weight decay regularization. International Conference on Learning Representations, 2017.
Jan M Maciejowski and Mihai Huzmezan. Predictive control. In Robust Flight Control: A Design Challenge, pp. 125-134. Springer, 2007.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, and Madian Khabsa. Unipelt: A unified framework for parameter-efficient language model tuning. arXiv preprint arXiv:21 2021.
Otniel- Boniel-Bggdan Mercea, Alexey Gritsenko, Cordelia Schmid, and Anurag Arna.. Time-memone memory-an and parameter-efficien visual adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5536-5545, 2024.
13

---

Published as a conference paper at ICLR 2025
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derezynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanistaw Wožniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Lin- guistics: EMNLP 2023, pp. 14048-14077 Singapore, dec 2023. Association for Computational Linguistics. doi: 10.18653/v1/20222 1n2d...nnnn/......nn. findingseemnll.9ee weeee..mn......... URL https: org/ emw. emmlp 936
Anastasiaa Razdaibiedina, Yuning Mao, Madian Khabsa, Mike Lewis, Rui Hou, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: improving prompt tuning with residual reparameteri- zation. In The 6l Annual Meeting Of The Association For Computational Linguistics, 2023.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-214 pretraining for the masses, 2021.
Jan-Martin O Steitz and Stefan Roth. Adapters strike back. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 21449, 2024.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue. Jianyong Wang, and Furu Wei. Retentive network: A successor transformer for large language models. arXiv
preprint arXiv: 2307.08621, 2023.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and
Herve Jegou. Training data-efficient image transformers &amp; distillation through attention. In International Conference on Machine Learning, volume 139, pp. 10347-10357, July 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuming Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv: 2307.09288, 2023.
Cheng- Hao Tu, Zheda Mai, and Wei-Lun Chao. Visual query tuning: Towards effective usage of intermediate representations for parameter and memory efficient transfer learning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7725-7735, 2023.
Ashish Vaswani. Attention is all you need, arXiv preprint arXiv:1706.037600 2017.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024a.
Yuzhu Wang, Lechao Cheng, Chaowei Fang, Dingwen Zhang, Manni Duan, and Meng Wang. Re- visiting the power of prompt for visual tuning. In Forty-first International Conference on Machine Learning. 2024b.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko. Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10-
14

---

Published as a conference paper at ICLR 2025
http://aaaaapp..eeeee/////////// / 1a879 La238b88Seeaab225555222222222222 orgpraaa--aa2--aattaatt-----raaa
16, 2023, 2023. URL
Shih- Shih-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard B. W. Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lycoris fine-tuning model evaluation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: https::/openzevieew net formm?id wf wf2XaBe783
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efffcient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106. 10199, 2021.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv 2019.
preprint
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Zeyu Zhang, Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, and Hao Tang, Motion mamba: Efficient and long sequence motion generation. arXiv preprint arXiv: 2403.07487, 2024b.
Zuen-Ru Zhang, Chuanqi Tan, Haiyang Xu, Chengyu Wang, Jun Huang, and Songfang Huang. To- wards adaptive prefix tuning for parameter-efficient language model fine-tuning. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1239-1248, 2023.
Zhi Zhang, Qizhe Zhang. Zijun Gao, Renrui Zhang, Ekaterina Shutova, Shiji Zhou, and Shang- hang Zhang. Gradient-based parameter selection for efficient fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28566-288888 28577, 2024c.
Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen. Autopeft: Automatic configuration search for parameter-efffcient fine-tuning. Transactions of the Association for Computational Linguistics, 12:525-542, 2024.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. Forty-first International Conference on Machine Learning, 2024.
A MORE DETAILED ABLATION STUDIES
A.1 INDIVIDUAL PEFT METHODS IN DETAIL
In the experiments, need to determine the appropriate parameters for each method. To this end, we investigate the optimal settings by varying the degrees of freedom for each PEFT method. Specifically, the rank of LoRA (r), the number of additional tokens for Affix-tuning and Prefix- tuning (n)), the number of dimensions for Additional-scan, and the strength of the proposed weight decay for Partial-tuning. Same as the ablation study in Table and Figure 3 in the main text, we evaluate the averaged accuracy across six tasks: CIFAR- 100, Sun397, Camelyon, Retinopathy, Clevr-Count, and sNORB-Elev.
First, each LoRA applied to large weights is evaluated. As shown in Figure 4d the maximum accuracy with ViT can be obtained with approximately r 32, whereas that with Vim can be obtained with a higher rank, as shown in Figure 4c. One reason is that the pre-trained Vim is less prone to collapse even when a large LoRA added. Interestingly, the ir_proj layer itself is prone to collapse and begins to degrade in accuracy at 32, while (X) and LoRA, (Z), which apply LoRA to partial weights of the in in-proj layer, are able to continue improving accuracy up to = 64. By dividing LoRA for each output feature, the collapse is suppressed, and further accuracy improvement is possible with large parameters in LoRA. We have already discussed LoRA on small linear weights in the main text (Figure 3a
15

---

Published as a conference paper at ICLR 2025
65
(%)
00
45
68
Tokeus
# #Tokens
Prefix (single) - Prefix (dual)
prefix
infix
Infix (single)
-sefix--suffi
(a)
(b)
65
60.5
(96)
60
39.5
64
96
Rank of
Rank LoRA
in proj out proj ParallelAdapter
LoRA(ViI) - Adaptformerr (VI)
(d)
(c)
Figure 4: Ablation studies for vision tasks. (a) shows the relationship between the number of tokens and performance in Affix-tuning with projection. (b) shows the relationship between the number of tokens and performance in Prompt-tuning. (c) shows the relationship between the rank of LoRA and performance in Mamba. (d) shows the relationship between the rank of LoRA and performance in VIT. Performance consistently decreases as the rank increases up to 64, contrary to the case in Mamba.
For Prompt-tuning, adding soft tokens to both the beginning and end is the most effective, as shown in Figure 4b Figure 4a also shows that the number of affixes improves performance up tokens. However, or more affi tokens degrade performance due to over-fitting caused by increasing the number parameters.
A.2 EFFICIENCY OF PEFT METHODS FOR VISION MAMBA
In Table we present benchmark with the hyperparameter settings that maximize accuracy. As discussed in Appendix A.I Vim is less prone to collapse and benefits more from increasing the number of training parameters in PEFT modules. Consequently, the number of trainable parameters for Vim Table
larger than that for VIT. Hence, we analyze the trade-off between accuracy and the size of PEFT modules. Specifically, all methods evaluated in Figure and Figure 4 are plotted in Figure 5 The results indicate that the superior PEFT methods for Mamba achieve high accuracy with lower number of parameters and exhibit better trade-off compared those for VIT.
16

---

Published as a conference paper at ICLR 2025
ViT
65
Vim(LoRA small width)
Vim(L.RRA large width)
Vim(Prompt-tunin
60
R
Vim( Com(Additioal-scaan
Vim(Partial-uning)
8
o Vim(Full)
55
Acc. (5)
50
45
10-3
10
10
109
Trainable parameter ratio
Figure 5: Scatter plot of the evaluations for the methods in Appendix A.1 showing the relationship between the proportion trainable parameters and performance
A.3 PER-TASK EVALUATION ON VTAB-1K
We present the results for each task VTAB-IK, including different base model sizes and training data. Specifically, we also report the accuracy on other base model architectures such as Vim-B. Vim-tiny (Zhu al. 2024). ViT-tiny (Tuuvron al. 2021 and VIT-B (Dosovitskiy et al. 2020). as well as different pre- pre-training datasets such as ImageNet-21k 21k (Deng et al., 2009) or using the training framework (Dosovitskiy et al.) 2020). The information Vim-B recently released, so we additionally experimented the performance here in the Appendix. The compared PEFT methods for Transformers are LoRA (Hu et al.) 202T). Adaptformer (Chen et al. 2022), Adapter+ (Steitz) & Roth, 2024),, FacT-TK (Jie & Deng 2023). Head2Toe (Evci et al. 2022) VQT (Tu et al.) (023),
BitFit (Zaken ct al. (021), SPT-LoRA He et al. 2023), and LoSA (Mercea et al, 2024). The results shown in Table
Many PEFT methods for Vision Transformers have benchmarked with VIT-B pre-trained on ImageNet-21K. First, clarify the accuracy change due to differences in experimental set- tings and their settings. By using ImageNet-1K instead of ImageNet-21K,, accuracy decreases by around 5 points. Changing the training framework to DeiT does not significantly affect accuracy. Reducing the model size from ViT-B to ViT-S results in an accuracy drop of about 2 points.
Next, we compare PEFT for Transformers and PEFT for Mamba using base models of similar size. For small-sized base models, as mentioned in the main paper, we achieve a performance improvement of an additional 1.5 points over the state-of-the-art PEFT for Transformers. With even smaller models, such as ViT-tiny and Vim-tiny, it shows similar accuracy improvements. This confirms that our proposed method is effective for model sizes other than Vim-S used in the main paper.
Finally, we evaluate each task individually, PEFT for Vim outperforms PEFT ViT on most tasks when compared within a similar base model size. Notably, PEFT for Vim is inferior to PEFT for ViT only the dSpr-Loc task, which classifies the z-coordinates and sizes. We consider that one reason for this is that the attention mechanism can locate the objects more straightforwarddy than Mamba. On the other hand, in the Clevr/count task, the accuracy with Vim is much higher than that with VIT. This task involves counting the number of objects in the image, and the recurrent nature of Mamba may be more suitable for counting the number of objects in the hidden state.
objects various colors, shapes,
17

---

Published as a conference paper at ICLR 2025
Table 4: Detailed test results on the VTAB-IK benchmark. "Mean" denotes the average accuracy for each category, and "Overall Mean" shows the average accuracy over 19 tasks. The (A), (T), and (D) stand for the network architecture, training framework, and dataset used for pre- pre-tngin
Nauual
Specialioed
Suocwued
SNORD Adin
Overail Mean
CIPAK
Caltchh0
DTD
Flovess
SWIN
550297
Moa
Mon
Clevr Dkr
DMLA
KOTTLOW
48g- Loc
SNORB Bler
Mesu
Medbod
(T)
VIT INT1K
57.7
64.3
75.9
57.5
29.1
Linex
51.0
06.5
74.0
55.
26.9
52.3
Bidfi
12.8
82.0
99.2
97.5
85.3
59.9
51.4
73.3
73.3
91.5
12.9
89.8
78.3
51.5
55.6
53.4
55.9
65.6
48.0
15.7
25.1
68.1
62.1
LaKA
67.1
91.4
09.4
99.8
934
15.3
56.0
79.5
94.9
95.3
64.6
73.6
84.6
$2.9
69.2
09.8
78.5
75.2
47.1
31.0
46.0
59.8
71.3
Fac1-TK
90.6
70.8
88.6
56.1
50.6
34.8
64.5
13.2
85.3
52.6
58.3
90.7
90.3
41.4
00.7
13.2
10SA
02.8
76.1
97.5
55.6
86.3
K2.1
91.3
05.5
76.1
(A)
VIT INIK
Serrchh
7.6
19.1
13.1
29.6
6.7
19.4
2.3
14.0
71.0
11.0
26.5
72.0
6038
31.6
52.5
273
36.1
66.1
19.7
11.7
24.1
3533
32.8
Full
46.3
$4.5
54.1
84.2
14.3
26.9
65-2
95.3
95.0
76.0
70.4
R1.7
71.5
60.5
66.9
72.9
74.5
35-3
28.5
23.6
52.2
63.2
Lanzr
85.6
61.4
40.5
11.7
65.5
73.1
36.1
55.
18.1
26.5
31.4
51.7
32.1
40.0
VigT
16.4
95.4
86.1
11.1
55.7
55.3
K4.7
74.9
84.5
1018
57.6
77.
65.
43.1
76.8
31.6
09.3
05.1
Foc T-TK
56.0
50.0
06.8
M000
91.1
56.%
10.3
73.0
81.9
94.7
K2.6
17.5
81.
-94
88.1
75.
65.0
49-4
13.6
36.1
53.7
67.
LoRA
19.1
06-
90.8
K5.4
4000
716
41A
94.7
K7.6
T12
818
76.5
60.
400
8010
25.5
48.3
77.6
16.7
DET
(D)
INIK
Linza
35.4
56.5
61.8
43.4
57.7
796
40.6
596
20.3
25.0
24.2
32.5
53.5
Paz I-T
61.0
67.0
91.3
91.6
87.6
42.5
15.8
54.0
64.4
12.4
81.
58.
47.2
79.7
76A
35.9
26.5
35.0
56.9
66.5
LaKtA
51.3
50.5
68.2
92.9
91.3
58-4
42.8
$4.0
95.9
15.0
132
915
194
58.
095
81.2
81.0
54.3
11.1
18.1
56.2
70.
62.1
89.8
09.7
02.6
91.6
63.4
78.9
35.6
88.
738
85-2
1h-1
49.3
821
76.3
55.4
31.8
32.1
58.3
16.8
08.6
$9.5
43.0
735
511
11.2
836
51.5
71.3
(1)
Full
28.5
65.0
5.5
63.2
42.4
16.4
16.5
43.9
65.1
55.5
15.
73A
095
13.0
14.1
20.7
33.3
35.1
41.3
7.1
14.9
27.8
43.5
Linse
46.6
56.8
63.9
900
90.2
45.1
413
9222
59.5
71.4
738
792
37-3
38.0
3633
63.6
153
25.5
18.2
22.6
52.2
$4.0
Allis- (wol proj)
59.0
58.1
66.8
84.5
934
525
45.0
62.
738
822
620
435
768
59.9
33.8
50.8
64.2
720
00.3
08.9
15.
45.
$1.0
507
46.
601
55%
62.6
50.85
68.4
M1.5
91.3
16.3
65.0
70.6
81.5
05.0
K5D
819
15.1
64.
412
76.
$1.0
35.6
00.4
71.3
LoRAioot prog)
68.8
50.5
68.0
91.7
91.1
$5.3
44.
76-7
$3.9
94.5
55.0
724
840
53.6
63.4
50.
81.3
75.3
52.5
35.0
JU6
(6.1
71.3
pra))
63.7
67.7
76-7
35.0
725
54.
518
79.9
812
20.4
11.0
Senach
4.9
11:1
9.3
24.4
1944
1.6
10.7
A5.0
17.3
78.6
73.6
96.1
46.
762
4800
7.5
37.1
1288
24.8
16.3
Tull
30.6
89.5
57.6
61.8
50.3
21.5
51.8
16.3
K5.3
16.8
738
15.
56%
40.5
66.8
73.5
40.4
76.0
328
45.5
57.5
41.4
K3.9
62.0
116
15.9
60.9
30.5
1400
75
15-1
540
65.
18.1
20.9
31.5
10.6
51.8
85.1
72.9
71.7
81.
71.9
440
74.5
768
50.5
56.1
67.0
LokA
$7.6
59.6
35.1
Adleyioomzr
16.0
88.0
65.6
90.5
85.3
19.5
73.
$4.0
93.6
K2.2
12.7
81.2
17.5
59.0
46.7
793
83.0
594
10.5
15.4
57.8
60:0
55.8
50.5
88.1
90.6
$6.4
406
90.2
74.8
$4.5
95.7
64.8
76.1
848
75.6
58.5
47.4
50.5
80-4
50.0
29.6
35.7
52.0
60.4
56.5
09.2
86.8
DUL
18.1
40.
24.7
04.2
72.4
63.6
1%.6
68.4
80.5
87.8
51.
59.
60/9
15a/
Senutch
11.9
5.5
12.1
16.3
1.6
5.3
61.6
053
28.9
22.5
409
36.
11.5
18.5
29.
25.4
Pull
$1.6
83.0
61.5
71.1
45.0
B5.4
14.9
51.6
06.5
15.1
63.3
57.1
68.
15.6
10
388
250
6.3
11.5
10.0
26.1
54.
50.8
Liazz
46.3
65.2
59.5
75.5
35.3
4037
30.6
62.5
75.8
59.0
68.2
73.0
72.3
37.0
36.3
33.3
65.0
150
25.2
16.5
23.2
52.0
51.8
CLS-sorr-ttniiu
48.0
55.1
99.2
41.0
39.4
62.4
68.5
73.1
73.4
37-3
33.6
66.4
19.3
33.5
32.3
519
Donning
56.2
86.0
18.5
42.7
16
510
196
76.
40.1
Anninng
96.7
67.6
65.
96-4
00.7
76.0
42.4
22.0
K1.1
91.2
18.6
70.
W07
55.9
40
77.
60.
46.0
15.9
12.1
08.6
64.4
Carild tuning
59.7
39.1
65.2
85.5
89.4
85.4
42.1
74.3
$2.9
94.2
70.
13.5
82.5
76.5
63.3
457
29.5
902
51.3
30.5
35.4
57.
69.1
rusing
57.
02.7
41.5
724
79.2
730
58.9
Allls-tuing
30.4
70.3
Adhinunal-azz
59.6
00.0
66.6
85.0
15.6
45.1
24.6
51.2
119
82.7
457
800
750
29.1
36.9
56.4
66.7
66.0
60.
68.9
91.4
91.3
53.8
66.1
76.1
55.3
05.1
K3.6
72.0
540
$5.9
K3.4
10.2
75.3
53.0
32.6
41.3
60.0
71.0
LoRAiembed)
51.7
$4.9
512
75.5
15.0
53.4
39
64.7
77.5
91.
11.5
700
77.5
43.9
569
730
519
41.
19.
26.2
43%
58.6
67.5
70.7
750
56.1
M61
45.4
35.0
57.1
69.
LoRAioe prg)
64.2
59.8
68.6
91A
91.2
860
458
7644
94.9
94.9
63.4
12.7
$4.0
$4.5
62.
49.5
81.
71.1
520
12.5
40.9
60.1
11.
LoRAis prs)
53.4
90.3
692
91.
91.0
186
43.
76.6
95.0
13.
72.9
84.1
$4.0
51.3
686
50.0
83.5
52.6
31.5
39.
682
11.3
LaRk,,
57.1
65.9
51.
427
7333
303
70.
909
66.1
760
64.5
25.5
30.
50.9
65.5
790
43%
150
15.0
505
57.
16
41.
76.5
69.4
46.4
11.7
635
66.1
Lokk,1Z)
61.7
90.6
68.1
90.3
90.
85.4
452
762
35.5
25.2
55.5
72.
843
12.4
50.6
482
81.9
81.6
52.4
31.4
36.5
59.7
709
63.3
50.5
09.4
90.6
91.5
15.9
45.6
70.6
34.5
7244
83.9
$49
62.9
45.6
81.4
83.8
52.5
35.1
404
21.5
Rylald
50.2
09.4
90.1
45.9
77.0
84.
94.6
$1.0
95.4
54.9
61.6
73.1
Dal (D) INIX
K1.5
40.9
14.3
37.9
45.5
FocT-TK
46.7
83.6
85.9
$1.6
12.9
$1.0
15.0
70.0
14.6
45.1
73.9
77.4
45.0
36.3
11.1
53.5
64.0
LakA
45.5
$5.9
59.5
85-3
$5.3
$6.7
52.5
68.
90.5
03.0
16.1
10.1
12.1
100
05.6
76.1
760
45.9
37.6
12.6
55.0
65.3
SPTLLRA
47.5
35.0
02.3
85.3
$6.3
85.6
34.6
09.7
33.3
94.5
58.6
72.8
80.8
64.6
58.6
47.4
79.5
79.7
45.6
30.9
35.6
55.4
66.4
33.9
017
71.6
79.6
72.7
78.0
830
36.2
66.1
Linae
45.0
63.7
56.9
23.5
31.4
40.6
36.9
60.5
96.0
68.5
25.1
76.9
32.6
36.1
562
66.5
152
22.4
15.0
21.5
30.4
51.2
52.6
82.1
62.5
85.2
$6.5
$1.7
32.0
20.2
S0.2
93.0
18.
7200
$1.0
59.6
62.
42.
25.6
7100
43.5
27.0
32.
56.1
65.9
sowlldniing
50.0
56.
6010
35.6
19.3
353
09
43.5
93.3
77.7
1078
71.5
63.9
771
71.
48.1
19.9
559
66.
750
(NRA,
50L8
57.6
72.9
76.1
87.
A.4 ABLATION STUDIES ON LANGUAGE TASKS
We conduct ablation studies for language tasks. Figure shows that the optimal learning rate varies for each method. In contrast, as shown in Figure 6c and Figure [d], the optimal learning rate and the optimal number of trainable parameters remain relatively stable despite changes in model size. Hence, the number of trainable parameters is fixed regardless of the model size, and the learning rate is coarsely tuned for evaluation in Table The detailed settings described in Appendix [..
The trade-off off between the number of trainable parameters and accuracy for Additional-scan, Affix- tuning (w/o proj). and LX)A,(X) is described in Figure 6b) Each method excels in certain aspects
18

---

Published as a conference paper at ICLR 2025
44
47.8
47.6
Aoc (%)
Aec. (%)
47.4
47.2
36
46.8
Ge-6
1e-5
Gre-5
1o-d
hi-44
16-1
Be-l
16-2
0.1
Learuing rate
Trainable parameter ratio (%)
Aflix-tuning (w/o proj)
Additional-saa
Affx-tuning [w/o proj)
Additional-scan
(X)
LXRA,(X)
(b)
55
54
50
50
Apc. (%))
Aec. (%6
48
44
40
40
10
5 x 10 $10
5 x 10
Additional dimension
Learning rate
Mamba 130M Mamba 370M
Mamba 130M Mamba 370M
Mamba 790M
Mamba 790M + Meaba 1.4B
(d)
Figure 6: This ablation study investigates how optimal hyperparameters differ per PEFT method and the size of base models language tasks. (a) Optimal learning rate per PEFT method. (b) The relationship between the ratio of trainable parameters and performance on Mamba 370M base model. (c) Suitable learning rate per base model size for Additional-scan, (d) Optimal additional dimension per base model size for Additional-scan.
and should be chosen based on the specific application. LXRA,(X) is effective when sufficient mem- ory available, Affix-tuning (w/o proj) maintains accuracy with the fewest trainable parameters, and Additional-scan falls in between. Since even PEFT may not fit into GPU memory with large-scale models, it is crucial to select the appropriate method based on the application and environment.
Table 6 shows the experimental results of applying LoRA individual modules in both Mamba and Pythia. In Mamba, applying LoRA to individual modules yielded better performance than SLL LoRA. which applies LoRA to all modules. Conversely, in Pythia, applying LoRA to all modules tended to perform better than applying LoRA to individual modules. These experimental results demonstrate the differences in behavior between Mamba and Transformer when LoRA is applied.
19

---

Published as a conference paper at ICLR 2025
75
77
00
65
Arc. (20
85
Asc. (8)
24
Arc. (90)
88
70
40
10"
3
10a
Xx
Slow teaining dats.
Stoe texining datts
Slon teaining dats.
proj] 1oRAp(X)
Aftis-tuiss (wyj pou)) Additionlaon LoRdAgXX
pro)]
LoRAy/X)
(a)
(c)
Figure 7: Task-specific relationship between size of training data and performance of PEFT in Vim. (a) is on Sum397, (b) is on Retinopathy, and (c) is on DMLab task.
Base Model
Method
#Params (K)
Natural
Specialized
Structured
Average
VMamba-S
Full
49,379
66.4
81.8
62.6
Linear
18
67.9
73.6
34.2
54.9
Conv2d-tuning
160
75.8
83.3
58.9
70.3
Affix-tuming proj)
198
77.3
82.7
59.7
71.0
Additional-scam
395
83.3
60.2
71.0
Adapter
774
77.8
84.5
63.7
73.2
LoRA(out.proy) (X)
1,529
77.9
85.9
65.6
74.4
1,529
78.2
84.5
65.2
74.0
VMamba-B B
Full
87,533
66.8
81.4
60.1
67.1
Lincar
24
68.4
79.1
30.2
54.6
Sonv2d-tuning Affix-tuning' proj)
189
78.0
84.1
63.4
73.1
276
77.5
83.3
59.2
71.0
Additional-scan
527
77.0
83.8
60.1
71.3
ParallelAdapter
1,032
78.6
85.1
63.3
73.5
pro)(out.proj)
2,039
78.6
85.7
64.9
74.3
LoRAp(XX
2,039
78.3
85.3
65.2
74.3
Table 5: Comparison of methods for VMamba.
In VMamba, LXRA,(X) is the same as
LoRA(in.projj
A.5 EFFECT OF DATA SIZE
Table I demonstrates that Partial LoRA performs better in image tasks. On the other hand, Table shows that Affix-tuning and Additional-scan achieve competitive accuracy with Partial LoRA de- spite fewer parameters in language tasks. The major differences between the experiments the modality and the amount of training data. In the image experiment, fine-tuning was performed with IK images, while in the language experiment, fine-tuning was conducted with 170K data. To investigate whether the differences in trends are due to the amount of data, we conducted experi- ments scale the amount of training data in the image tasks. detail, we used official test data of VTAB-1K as training data and vice versa. We chose Sun397, Retinopathy, and DMLab from each category of VTAB-IK, which contains a large number images for this experiment. fig. 7 shows that, as the data size increases, the best methods become Additional-scan for (a), Affix-tuning for (b), and LoRAp (X) for (c). These experiments suggest that LoRA,(() excels with smaller data sizes, but with larger data sizes, the optimal method depends on the specific task. We suggest select- ing a method that balances the trade-off between performance and parameters based on the task or searching HybridPEFT.
20

---

Published as conference paper at ICLR 2025
Table 6: Experimental results applying LoRA to individual modules in Pythia and Mamba.
Model
Method
BoolQ
PIQA
SIQA
HellaSwag
WineGrande
ARC-e
ARC-c
OBQA
Avg.
Pull
61.3
62.9
37.1
30.7
50.6
41.5
24.3
27.8
42.0
Pythia 160M
LoRA
61.0
62.0
36.3
30.3
52.0
38.2
24.6
28.0
41.6
LoRAiquery key, value)
62.0
36.8
30.2
50.7
40.5
27.0
40.2
LoR ARRAidense_hi 4h
50.7
61.7
37.9
29.9
49.6
39.7
22.4
27.0
39.8
LoRAídense 4h)
61.3
29,9
36.2
26.8
39,9
LoRAidense)
49.9
61.6
36,9
30.1
51.1
41.1
23.3
26.0
40.0
Full
36.
768.7
310
-23.7
"22.""
IT8
Mamba 130M
SLL
56.3
63.3
38.2
34.6
51.6
43.5
23.6
30.6
42.7
Affix-tuning tw/ proj)
55.1
61.4
36,5
32.9
51.5
36.8
23.5
27.2
40.6
LoRAGin proj) proj)
62.9
46.4
30,8
LoRA,(X)
61.7
64.0
39.5
34.1
52.2
43.5
25.3
29.4
43.7
55.0
68.4
42.1
40.8
53.9
50.8
26.7
30.0
46.0
Pythia 410M
LoRA
61.3
67.7
39.2
48.1
28.6
45.7
Lo LoRAíquery value)
60.5
67.5
41.3
39.2
54.3
45.4
24.0
29.0
45.1
ARArdense_Ah_ton 4h_to_))
47.1
LoRA(dense h.t0.4h)
61.5
67.0
39.9
39.0
53.7
46.2
23.8
28.6
45.0
LoRAídense)
55.7
67.4
40.5
39.8
54.3
45.2
24.1
30.6
44.7
Full
38.1
69.9
43.7
35.8
52.7
"29.7
33.3
36.2
Mamba 370M
SLL.L.R4
59.5
69.6
42.2
44.1
54,9
50.6
26.3
30.8
47.3
Affix-tnnng (w/o proj)
62.0
67.7
39.3
46.3
54.1
47.8
28.2
31.0
47.0
LoR proj)
55.4
68.6
41.0
44.7
54.1
52.4
28.3
33.4
47.2
LxRA,(X)
68.8
42.1
44.7
56.2
50.4
27.4
32.2
Pythia
Full
55.0
70.2
42.5
47.5
54.4
54.1
29.7
33.2
48.3
LoRA
60.0
69.3
40.9
45.3
53.6
49.8
27.2
31.0
47.1
LoRAíquery key value)
69.4
44.7
50.3
30.8
47.0
LoRAidens 4h
49.2
69.0
42.0
45.2
53.8
52.3
27.5
31.0
46.3
LaRAidense Joh
30.2
LoRAdemse)
53.3
69.4
41.0
46.5
52.4
49.4
27.8
32.6
46.6
$470
359
"7.77
"37.2
".5.2
3/66
SLL LoRA
60.7
72.0
42.4
54.7
56.9
55.3
29.4
34.2
50.7
Mamba 790M
Affix-tuning (w/o proj)
$7.2
71.7
41.4
$5.0
55.8
52.6
29.8
33.0
49.6
LoRAfin proj)
71.9
50.8
56.3
33.8
50.7
LoRA,(X)
$9.9
72.2
44.2
$2.8
58.0
53.7
30.8
34.8
50.8
Pythia 1.4B
58.6
71.1
42.7
$3.6
55.1
58.5
29.9
34.8
50.5
LoRA
71.3
42.5
50.1
$7.6
33.6
50.5
Lo LoRAíquery value)
60.7
70.9
42.2
50.0
58.3
35.4
28.2
33.2
49.9
LoRAidense
71.9
50.5
LoRA(dense hoto4h))
61.4
70,7
42.9
50.3
57.6
51.7
26.7
32.8
49.3
LoRArdeuse)
62.1
71.1
41.6
51.6
58.2
54.7
29.4
31.6
50.0
Full
61.4
73.3
33.9
$6.9
39.0
59.7
"A..
35.3
53.0
SLL LoRA
59.7
73.5
43.1
$6.9
60.7
59.7
31.7
36.0
$2.7
Mamba
Aflix-uning (w/o proj)
63.6
74,0
41.9
58.9
60.6
61.6
33.4
36.8
53.9
LoR LoRACin
62.6
73.6
43.7
55.6
59.7
58.3
31.7
35.6
$2.6
(X)A,(X)
73.5
57.7
61.6
A.6 EFFECT OF MODEL
cobtainso experimental results for Vim and ViT with size variations. From Vim-Tiny to Vim- the overall accuracy is improved. However, from Vim-S Vim-B, there little improvement, whereas ViT-B improved from VIT-S. Because this phenomenon is not observed in vanilla Mamba for language tasks (see table(). it seems it is not due to PEFT methods for Mamba. This suggests that large Vin/Mamba models might face potential challenges when fine-tuning with limited data. Note that even with Vim-B, PEFT still significantly outperforms full fine-tuning and achieves equivalent or higher accuracy than state-of-the-ar PEFT with ViT-B, indicating that PEFT for Vim remains effective.
A.7 EXPERIMENTS WITH VMAMBA
To investigate the generality of PEFT Mamba for visual tasks, we conduct experiments using VMamba (Liu et al.) 2024b), variant other than Vim. VMamba made several improvements over the original Mamba. eliminated the state dimension (i.e., reduced the state dimension to 1), in- troduced hierarchical structure similar to CNNs, replaced causal ID convolutions with 2D con- volutions, removed the Gated MLP structure, and substituted half of the Mamba blocks with MLP blocks. Due these changes, there some differences in trends, but PEFT also works well with VMamba, as shown Table [5) The accuracy is higher than that of Vim, as the original VMamba performs better on ImageNetTK evaluation. Similar to the results with Vim, partial LoRA shows
21

---

Published as a conference paper at ICLR 2025
Table 7: Replacing simple structure PEFT methods intentionally used with dedicated PEFT methods.
VTAB-IK Vim-S
Commonsense reasoning Mamba 1.4B
Method
Natu.
Spec.
Struc.
Avg.
BoolQ
PIQA
SIQA HellaS
WinoG
ARC-e
ARC-c OBQA
Avg.
(X)A,(X)
76.6
83.9
60.8
71.5
63.1
73.5
42.7
57.7
61.6
60.4
32,9
37.4
53.7
(X)
76,9
84.1
60.6
71.6
63.9
73.8
43.2
57.9
62.1
60.4
32.4
37.4
53,9
superior performance. Additionl-scan and Partial-tuning show a superior trade-off between train- able parameters and performance. Furthermore, there was no significant performance improvement from VMamba-S to VMamba-B, indicating saturation, which is also consistent with Vim. These experimental results suggest that the phenomenon of performance saturation with PEFT Mamba for visual tasks, once the model size reaches certain threshold, is general regardless of variation.
A.8 COMPARISON WITH CONCURRENT WORK
We discuss the comparison with related studies conducted during the same period in addition SLL LoRA (Halloran al.) 2024). Galim et al. (2024) proposed SDLoRA, which selectively updates the channels and states SSM. Unlike SDLoRA, our work emphasizes comprehensive benchmarking and exploration. Furthermore, Galim et al. (2024) pointed out the limitations of expressiveness when applying Prefix-tuning to SSM. In contrast, our proposed Affix affix-tuning avoids this issue by allowing tokens to be inserted at arbitrary positions. In fact, Affix-tuning shows competitive performance with LoRA on language tasks (table
A.9 DIRECTION OF THE FUTURE WORK
Our goal is to explore PEFT for Mamba: specifically, how to tune and where to tune. By using simple architecture PEFT methods, we aim minimize extraneous influences, allowing evaluation results to be universally informative for various future tuning strategies. The proposed Affix-tuning and Additional-saai are also designed as simple as possible, although we believe they possess signif- icant novelty and usefulness. Based our findings, future work should be able to focus on effective PEFT research without searching a lot. To verify this, we conducted an experiment by replacing LoRA with the latest DoRA (Liu et al. 2024a). As shown in Table higher accuracy than LoRAp.X), which already surpasses the accuracy of state-of-th oftte-o-thhearn art dedicated PEFT for Transformers. We hope that exploration will contribute to the future evolution of PEFT for Mamba.
were able to achieve even
B IMPLEMENTATION DETAILS
B.1 LANGUAGE TASKS
We follow the fine-tuning setup of Liu al. (2024a): : Hu al. (2023) for commonsense reasoning tasks. Each model is fine-tuned with about 140,000 data for three epochs with batch size of 16. A linear learning rate scheduler is used with warmup period of 100 iterations.
As to the learning rate, we use suitable values for each method. For SLL LoRA on Mamba, we follow the original learning rate per model size (Halloran et al. 2024). We also roughly tuned learning rate for our PEFT methods obtained through the ablation studies in Appendix A.4 A.4 The configurations shown in Table
22

---

Published as a conference paper at ICLR 2025
Table 8: The learning rate configurations for language tasks.
Method
Mamba 130M
Mamba 370M
Mamba 790M
Mamba 1.4B
Full
5e-5
Se-5
5e-5
5e-5
Additional-scan
5e-3
le-3
le-3
le-4
Affix-tuning
lc-4
Ic-5
le-5
lc-5
LoRAp(X)
le-3
5c-4
5c-4
lc-4
Method
Pythia 160M
Pythia 410M
Pythia IB
Pythia 1.4B
Full
5c-5
5c-5
5e-5
5c-6
LoRA
le-5
lc-4
le-5
Ic-5
For the hyperparameters of each PEFT method, we use the optimal values obtained through the ablation studies on the "TAB-IK dataset. For Pythia (Biderman et al., 2023), We apply LoRA with r 8 to dense, query wwe.kkklllleeeeee.. value, dense_4t -ense_4hto__h and dense_h_to._ _o_th_4hhhh layers.
B.2 HYBRID PEFT SEARCH
In the first step, explore only whether to each PEFT method. As mentioned in the Sec- tion 3.4 the first step of our search verifies the effectiveness of each PEFT method and seeks the preferable combination with minimum trainable parameters. We rely the following hypothesis; effective PEFT methods work positively even when the number of trainable parameters is small, while when the number of parameters increases beyond certain level, the pre-trained model col- lapses and is negatively affected. As (able(]]soowe shows, simply combining PEFT or increasing the rank of LoRA decreases performance. Therefore, the hyperparameters of each PEFT method are as its trainable parameters become minimum. In addition, the proposed regularization for Partial-tuning PEFT methods is applied so as not to corrupt the pre-trained model. this step, a combination of PEFT methods that work positively is selected greedy while keeping the total degrees of freedom of the PEFT methods within the degrees of freedom that do not destroy the pre-trained model. We search for 100 trials using TPE algorithm (Bergstra et al. 201T) about whether to use each PEFT method with the fixed hyperparameters shown in step 1 of Table 10 As a result, it turns out that the combination of eight PEFT methods, CLS-token-tuning A A-tuning, Affix-tuning, LoRA(out proj), LoRA(in proj). LoRA(dt proj), (X), and LoRA,((). is the best.
In the second step, the hyperparameters for the eight methods searched. In addition, search dimension is added, which selects a PEFT method that should removed or decides not to remove any at all. This is intended eliminate less important PEFT methods and allocate more parameters to the more significant ones. The specific search space is shown in step of Table T0
This two-step approach reduces the search space only 2 for the first step and (N;+1) IpEr, dp for the subsequent steps, compared to a simultaneous search space with Nat I Pli dp- Nall is the number of PEFT methods, and (5 Nan) is the number of methods chosen in the previous search step. The Pau the of all the hyperparameters and P. Pan) the subset used the chosen methods. Also, we represent the search space size along parameter p as dp.
Since optimizing over all 19 tasks computationally expensive, we optimize over the average accu- racy of 5 tasks in the representative subset of VTAB- 1k (Zhai et al.) 2019): Caltech101, EuroS Pets, Camelyon, and Resise45. By processing five tasks parallel on one 100 GPU, one trial can be completed in around 20 minutes, with minimal dependency on the type and size of the applied PEFT methods.
The detailed algorithm is provided in Algorithm | algorithm, top-1 accuracy as the objective function, and the number trials is set to N 100.
In our experiments, use TPE as the search
23

---

Published as a conference paper at ICLR 2025
Algorithm HybridPEFT Search Algorithm
1: Input: Set of PEFT methods M, Hyperparameter space H. Training hyperparameter space T, Default hyperparameters (Adefault, fdetaut)), Objective function Number of iterations N, Search algorithm S
2: Output: Optimal combination of PEFT methods and hyperparameters C*, h²,
3: Step I: Search for PEFT Method Combinations using
4: Initialize set of active PEFT methods C <-
5: Initialize set of observations Ocombi + 0
6: for iteration i
N do 7: Sample PEFT method combination C₁ using S based on Ocombi
8: Evaluate performance f(C,, hdctautt tdctauls)
9: Add (C,, f(C, /ddeaults (deeault)) )) to Combii
if performance improves then 11: Update C <- C2
12: end if
13: end for
14: Step 2: Search for Hyperparameters using S
15: Initialize set of active PEFT methods C <- C
16: Initialize best hyperparameters lt* <- 0, 1t t-
Initialize set of observations Ohyper
18: for iteration j j= to N do
then 20: Use default hyperparameters (hy,tz) + (hodffants tdefault) )
if'
Initialize "not remove"
22: else
23:
Sample hyperparameters (h,, (j) using based on Onyper
24:
Sample PEFT method to remove from {C "noo_remove") using S based on Onyper
25: end if
26: if is "eoveee then
27:
Evaluate performance f(C, hy bj) 28: Add ((h))4z) (C, hig 12)) to Ohyper
29: else
Evaluate performance f(C h2,ttt 31: Add ((h,,4)) f(C (r,h,, (4) to Onyper
end if
33:
if performance improves then
34:
Update hi* + hj, 1" < lj if is "not remove" then
35:
Update C* C-
37:
else
38:
Update C* + Crr
39: end if
40: end if
41:
end for
42: Return h²,
24

---

Published as a conference paper at ICLR 2025
Method
Rank
Avg.
LoRA(in proj out proj)
8
71.33
LoRA(in proj + out proj)
16
70.68
LoRA(in proj out_proj)
32
69.77
LORA(ALL)
8
71.02
LORA(ALL)
16
70,42
LoRA(ALL)
32
68.74
(X)A
64
71.52
Table 9: Average performance of VTAB-IK when combining multiple LoRA.
The search space and the fixed hyperparameters in our Hybrid PEFT search of the first step and the second step. In the first step, the combination of CLS-token-tumin tuning. A-tuning, Affix- fffix-tuning, LoRA(out proj), LoRA(in proj), LoRA(dt proj), LXRAp(X), and LoRAp is the best.
Table 10:
Step
Step 2
boolean search
fixed
hyperparameter search
fixed methods to use
use not
hyperparameters
CLS-tokeen-uuning
wd=le-3
CLS-token-tuning
[le-4, 5e-3]
Bias-tuning
wd=le-3
wd [1e5, le-2]
Pos-embed-tuning
wd=le-3
A-tuning
Ieee-4, 5e-3]
D-uning
wd=le-3
wd [le5, le-2]
A-tuning
wd=le-3
Affix-tuning
n
Convid Id-tuning
wd=le-3
Ir [le-4, 5e-3]
Prompt-tuning
n=1
wd [le-6, 1c-3]
Aftix-tuning
n=1
LoRA(out proj)
144, 16]
Additional-scan
n=1
s s
ParallelAdapter
1=8, s=0.1
Ir |le-4, 5e-3]
LoRA(embed)
1=8, s=0.1
wd [ le-3]
LoRA(x pRoj)
1-8, s=0.1
LoRA((..
T[.766)
LoRA(dt proj)
s=0.1
s [1e-2, 1]
LoRA(out proj)
r-8, s=0.1
Ir [le-4. 5e-3]
LoRA(in proj)
s=0.1
[1e-6, 1c-3]
(d) (B)
1-4, s=0.1
"LORA(d) projj
764
14 s0011
s [1c-2, 1]
(C
s=0.1
Ir [1e-4, 5e-3]
(X)
8 sm1.1
wd [1e-6, 1e-3]
(Z)A,
8 8 s01
LORA,(Xi
r [4. 16]
[1c-2,
Ir [le-4, Sc-3]
wd 11e-6, Ic-3]
LoRA,(B)
12]
s 1le-2,
Ile-4, 5e-31
[1e-6, Ie-3]
remove CLS-ttoke-ttunnng
remove A-tuning
remove Affix tuning
remove LoRA(out.pro))
select
remove LoRA(in proj)
remove LoRo(dt.proo)
remove (X)
remove LoRA,.()
not remove
25


Reviews:
[{"review_id": "mr44YnNcCR", "reviewer": "Reviewer_hvVY", "summary": "This paper presents a comprehensive and detailed exploration of parameter-efficient fine-tuning (PEFT) for Mamba. By applying and refining existing PEFT methods, introducing novel PEFT approaches（Partial-tuning and Additional-scan）, and exploring optimal algorithmic combinations, it achieves outstanding performance in both image and language tasks. This work provides valuable insights for efficient fine-tuning in Mamba.", "questions": "Please further explain the advantage and effectiveness of the proposed method compared with the baselines.\nPlease further explain the novelty of the proposed method.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper not only improves existing PEFT algorithms to suit Mamba but also proposes targeted, innovative algorithms, providing the community with a broader selection of PEFT methods.\n2. The experiments comprehensively showcase the performance of each PEFT algorithm, offering a framework for optimal algorithmic combinations.\n3. The extensive experimental workload demonstrates thorough exploration of the algorithms.", "weaknesses": "1. The improved and original PEFT algorithms did not achieve the best results. In the VTAB-1k benchmark, the performance of the improved Partial LoRA is nearly identical to that of LoRA, while Additional-scan underperforms ParallelAdapter with comparable training parameters. Compared to traditional PEFT algorithms, these methods lack strong competitive advantage.\n2. Although LoRA demonstrates outstanding performance on the VTAB-1k benchmark, its comparison with the modified Partial LoRA on language tasks is missing, making it unclear whether Partial LoRA is indeed the optimal PEFT method.\n3. The algorithmic novelty is limited, as the concept of Partial-tuning appears relatively straightforward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fucGmfR1O4", "reviewer": "Reviewer_U29m", "summary": "This paper explores the application of parameter-efficient fine-tuning (PEFT) in the Mamba model. It begins by systematically analyzing the suitability of existing PEFT methods for the Mamba model, then adapts certain PEFT methods to better align with the characteristics of State Space Models (SSMs), and finally investigates hybrid PEFT search strategies. Extensive experiments were conducted across image and text modalities, providing thorough benchmarks for various PEFT methods and demonstrating that the Mamba model exhibits distinct fine-tuning behavior compared to Transformer models.", "questions": "1. Table 1 shows that Adapter-based and LoRA-based methods perform best. Did the authors try SequentialAdapter, or, as in the original LoRA paper, apply LoRA across multiple locations simultaneously in the model?\n2. In Table 1, the Hybrid approach shows no significant advantage over individual PEFT methods (e.g., ParallelAdapter or LoRA) despite having far more parameters (117,236). The performance of Hybrid is not markedly better than LoRA(in_proj), which has only 1,483 learnable parameters.\n3. Beyond parameter efficiency, computational efficiency is also crucial for PEFT methods. Did the authors consider comparing the computational efficiency of different methods (e.g., in terms of FLOPs or running time)?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper focuses on a novel and competitive model architecture, SSMs, systematically analyzing the applicability of existing PEFT methods for the Mamba model, offering insights for designing efficient fine-tuning schemes for SSMs.\n2. The paper adapts existing PEFT methods specifically for SSMs, proposing new approaches tailored for Mamba, achieving good performances.\n3. The extensive experiments, covering both image and text modalities, provide comprehensive benchmarks for all methods and reveal differences in PEFT performance between the Mamba and Transformer models.", "weaknesses": "1. The paper covers too many aspects, including various PEFT methods, individual analyses, and hybrid architecture search. The authors should concentrate on one aspect and provide valuable conclusions for each part instead of describing each in detail, as this could reduce readability and divert focus.\n2. The paper should consider including the most advanced works within each PEFT category, such as GPS[1] and SPT[2] for partial-tuning methods, and SSF[3] in addition to LoRA-based methods for reparameterization approaches.\n3. Experiments should involve larger-scale models, where PEFT methods are particularly valuable. Evaluation across different model scales is crucial for PEFT methods. Notably, \"scale\" here refers to model parameter size rather than training data size (e.g., a model trained on ImageNet-21K in Section 4.5 is not necessarily larger than one trained on ImageNet-1K; model parameter size is more relevant in this context).\n3. Presentation quality requires improvement, including spelling errors (e.g., \"Linear Probing\" instead of \"Linear Proving\" in Table 1’s ViT-S section) and inconsistent heights in sub-tables within Table 2.\n\n[1] Zhang Z, Zhang Q, Gao Z, et al. Gradient-based Parameter Selection for Efficient Fine-Tuning[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 28566-28577.\n\n[2] He H, Cai J, Zhang J, et al. Sensitivity-aware visual parameter-efficient fine-tuning[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11825-11835.\n\n[3] Lian D, Zhou D, Feng J, et al. Scaling & shifting your features: A new baseline for efficient model tuning[J]. Advances in Neural Information Processing Systems, 2022, 35: 109-123.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6tD04aQKDL", "reviewer": "Reviewer_Czw3", "summary": "This paper explores parameter-efficient tuning methods such as the prominent methods like Adapter, LoRA, Prompt-tuning, and Partial tuning for the Mamba architecture. The authors highlight that these methods have not been studied in Mambas compared to the extensive research conducted on Transformers. The primary focus of this paper is on the Vision Mamba (Vim) architecture, with a minor portion addressing language models with Mamba compared with the Transformer counterparts Pythia. The study compares Vim and Vision Transformer models using PEFT methods, which address some key architectural and methodological choices like layer selection for LoRA and prompt placement in Prompt-tuning, for example. The authors present a two-step PEFT search method that hybridizes various PEFT options to offer a near-optimal solution (in terms of performance). Finally, the authors present a new PEFT method called Additional-scan, which increases the state dimension of the SSM, and an additional initialization method for the state matrix A is also presented. The experimental results employing Vim-S pre-trained on ImageNet-1K to demonstrate the effectiveness of the diverse PEFT options; Mamba-language models compared with Pythias are also presented.", "questions": "- Could the authors provide additional results using Vim-Base or larger-scale architectures?\n- How was the data utilized in the hybrid search method? Was a held-out set created for the searches?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow.\n- Searching for diverse PEFT options for Mamba is informative and would be a valuable contribution for those who employ Mamba architectures. \n- Studying extensive PEFT options based on a baseline yields convincing results.", "weaknesses": "#### #  Main concerns\n- The authors' claim in line 22 that PEFT is more effective for Mamba than for Transformers lacks adequate rationale and supporting evidence. There is no clear intuition as to why Mamba would benefit more from PEFT; the experiments appear biased due to an unfair comparison - specifically, Transformers were tested with only a limited range of PEFT options, while Mamba underwent more comprehensive testing (as illustrated in Table 1).\n- The authors primarily tested Vision Mamba rather than the broader Mamba architecture, yet they frequently used \"Mamba\" as an abbreviation for Vision Mamba. This reviewer is concerned this may be misleading, especially as the result table for the language models offers only limited experiments. The authors should clarify their terminology to ensure readers understand that Vision Mamba is the specific variant.\n- The authors presented a hybrid PEFT approach as an optimal-like solution, which is identified through their proposed search method. However, its conclusion does not specify a practical, generalizable option. This reviewer feels It's important to note that the effectiveness of the identified hybrid approach may not be universal, and the search process itself would be resource-intensive for a new architecture.\n\n#### # Missing/wrong details in preliminary\n- Section 2.1, titled *Mamba*, should provide foundational knowledge about Mamba as a preliminary, but it mainly presents on S4 or early SSMs. Since Mamba has slightly distinct notations and a signature architecture, the authors should revise this section to include more comprehensive details about Mamba.\n- As the experiments involve the Vision Mamba architecture, the reviewer suggests including the original architecture, which has subtle design differences (e.g., the definition of B).\n- The discretization presented in Eq.(2) is not the zero-order hold discretization as implied but rather the bilinear one. The authors should correct this detail.\n\n####  # Some concerns about experiments\n- The experiments used a limited set of baseline models. Although the PEFT options explored were diverse and met the experimental standards of a scientific study, only the Vim-S baseline is used (since the PEFT methods are more likely to be applied to larger architectures, Vim-S may not be a proper choice as a baseline), where this reviewer feels the main comparison focuses on Vision Mambas versus Vision Transformers.\n- Vision Transformers are not thoroughly explored compared to the fully examined Mamba models. For instance, ViT-S in Table 1 has only a few options compared to Mamba's more comprehensive use of PEFTs.\n- Some experimental comparisons lacked control:\n  - In language models, for instance, Pythia 1.4B and the counterpart Mamba 1.4B had significantly different baseline performances (44.8 vs. 53.0), skewing the reported improvements, as higher starting accuracy inflates gains.\n\n#### # Minors\n- There is a related work that covers a similar topic, including PEFTs in Mamba, which suggests that this paper is not the first to study PEFT for the Mamba architecture as mentioned in lines 71-74. This reviewer understands that the authors could not include the related work due to its publication date being after this submission. However, it is suggested to compare this study with that work in future revisions to provide context and emphasize differences.\n  - Parameter-Efficient Fine-Tuning of State Space Models, NeurIPS 2024 workshop (https://arxiv.org/abs/2410.09016)\n\n- Typo in Table 1: Linear 'Proving'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kgyXaAFsU0", "reviewer": "Reviewer_jdYa", "summary": "This paper explores the PEFT methods for the Mamba. By modifying and adapting commonly used PEFT methods on Transformer, the paper proposes novel PEFT techniques tailored to the Mamba architecture. Experiments demonstrate that Mamba outperforms Transformer when applying PEFT methods. The paper also showcases how to effectively combine multiple PEFT methods to improve performance and introduces a new framework.", "questions": "see Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1 The authors extend PEFT methods from Transformer to the Mamba architecture and propose several Mamba-specific PEFT methods.\n\n2 The authors conduct experiments not only in the vision domain but also in the language domain.", "weaknesses": "1 Why wasn’t Vim-B selected for comparison experiments with ViT-B? The authors seem to have not provided an explanation in the paper.\n\n2 In Table 1, the PEFT methods selected by the authors are relatively outdated, such as FACT, which originates from AAAI23.\n\n3 Conducting PEFT experiments on Vmamba, another Mamba-based vision model, is necessary to demonstrate the generality of the proposed methods.\n\n4 In section 3.4, the mixed PEFT search strategy proposed by the authors lacks sufficient theoretical support, especially regarding the principles behind why different combinations of PEFT methods can improve performance, which the authors have not sufficiently explained.\n\n5 For the vision experiments, the authors only consider smaller models without discussing the applicability of these methods to larger-scale models.\n\n6 For the new architecture—Mamba—the training duration of these PEFT methods is worth considering and comparing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)