MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba

Authors: Masakazu Yoshimura, Teruaki Hayashi, Yota Maeda
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: parameter-efficient fine-tuning, PEFT, Mamba, State Space Model, SSM

Abstract:
An ecosystem of Transformer-based models has been established by building large models with extensive data. 
Parameter-efficient fine-tuning (PEFT) is a crucial technology for deploying these models to downstream tasks with minimal cost while achieving effective performance. Recently, Mamba, a State Space Model (SSM)-based model, has attracted attention as a potential alternative to Transformers. While many large-scale Mamba-based models have been proposed, efficiently adapting pre-trained Mamba-based models to downstream tasks remains unexplored.
In this paper, we conduct an exploratory analysis of PEFT methods for Mamba. We investigate the effectiveness of existing PEFT methods for Transformers when applied to Mamba. We also modify these methods to better align with the Mamba architecture. Additionally, we propose new Mamba-specific PEFT methods that leverage the distinctive structure of Mamba. Our experiments indicate that PEFT performs more effectively for Mamba than Transformers. Lastly, we demonstrate how to effectively combine multiple PEFT methods and provide a framework that outperforms previous works. To ensure reproducibility, we will release the code after publication.

Decision: accept (poster)
Reviewer scores: 5, 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1729698208875

thread_type:
official_review

<thread_object>
creator: reviewer_jdya

<content>
summary:
This paper explores the PEFT methods for the Mamba. By modifying and adapting commonly used PEFT methods on Transformer, the paper proposes novel PEFT techniques tailored to the Mamba architecture. Experiments demonstrate that Mamba outperforms Transformer when applying PEFT methods. The paper also showcases how to effectively combine multiple PEFT methods to improve performance and introduces a new framework.

strengths:
1 The authors extend PEFT methods from Transformer to the Mamba architecture and propose several Mamba-specific PEFT methods.

2 The authors conduct experiments not only in the vision domain but also in the language domain.

weaknesses:
1 Why wasn’t Vim-B selected for comparison experiments with ViT-B? The authors seem to have not provided an explanation in the paper.

2 In Table 1, the PEFT methods selected by the authors are relatively outdated, such as FACT, which originates from AAAI23.

3 Conducting PEFT experiments on Vmamba, another Mamba-based vision model, is necessary to demonstrate the generality of the proposed methods.

4 In section 3.4, the mixed PEFT search strategy proposed by the authors lacks sufficient theoretical support, especially regarding the principles behind why different combinations of PEFT methods can improve performance, which the authors have not sufficiently explained.

5 For the vision experiments, the authors only consider smaller models without discussing the applicability of these methods to larger-scale models.

6 For the new architecture—Mamba—the training duration of these PEFT methods is worth considering and comparing.

questions:
see Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652448145

thread_type:
official_review

<thread_object>
creator: reviewer_czw3

<content>
summary:
This paper explores parameter-efficient tuning methods such as the prominent methods like Adapter, LoRA, Prompt-tuning, and Partial tuning for the Mamba architecture. The authors highlight that these methods have not been studied in Mambas compared to the extensive research conducted on Transformers. The primary focus of this paper is on the Vision Mamba (Vim) architecture, with a minor portion addressing language models with Mamba compared with the Transformer counterparts Pythia. The study compares Vim and Vision Transformer models using PEFT methods, which address some key architectural and methodological choices like layer selection for LoRA and prompt placement in Prompt-tuning, for example. The authors present a two-step PEFT search method that hybridizes various PEFT options to offer a near-optimal solution (in terms of performance). Finally, the authors present a new PEFT method called Additional-scan, which increases the state dimension of the SSM, and an additional initialization method for the state matrix A is also presented. The experimental results employing Vim-S pre-trained on ImageNet-1K to demonstrate the effectiveness of the diverse PEFT options; Mamba-language models compared with Pythias are also presented.

strengths:
- The paper is well-written and easy to follow.
- Searching for diverse PEFT options for Mamba is informative and would be a valuable contribution for those who employ Mamba architectures. 
- Studying extensive PEFT options based on a baseline yields convincing results.

weaknesses:
#### #  Main concerns
- The authors' claim in line 22 that PEFT is more effective for Mamba than for Transformers lacks adequate rationale and supporting evidence. There is no clear intuition as to why Mamba would benefit more from PEFT; the experiments appear biased due to an unfair comparison - specifically, Transformers were tested with only a limited range of PEFT options, while Mamba underwent more comprehensive testing (as illustrated in Table 1).
- The authors primarily tested Vision Mamba rather than the broader Mamba architecture, yet they frequently used "Mamba" as an abbreviation for Vision Mamba. This reviewer is concerned this may be misleading, especially as the result table for the language models offers only limited experiments. The authors should clarify their terminology to ensure readers understand that Vision Mamba is the specific variant.
- The authors presented a hybrid PEFT approach as an optimal-like solution, which is identified through their proposed search method. However, its conclusion does not specify a practical, generalizable option. This reviewer feels It's important to note that the effectiveness of the identified hybrid approach may not be universal, and the search process itself would be resource-intensive for a new architecture.

#### # Missing/wrong details in preliminary
- Section 2.1, titled *Mamba*, should provide foundational knowledge about Mamba as a preliminary, but it mainly presents on S4 or early SSMs. Since Mamba has slightly distinct notations and a signature architecture, the authors should revise this section to include more comprehensive details about Mamba.
- As the experiments involve the Vision Mamba architecture, the reviewer suggests including the original architecture, which has subtle design differences (e.g., the definition of B).
- The discretization presented in Eq.(2) is not the zero-order hold discretization as implied but rather the bilinear one. The authors should correct this detail.

####  # Some concerns about experiments
- The experiments used a limited set of baseline models. Although the PEFT options explored were diverse and met the experimental standards of a scientific study, only the Vim-S baseline is used (since the PEFT methods are more likely to be applied to larger architectures, Vim-S may not be a proper choice as a baseline), where this reviewer feels the main comparison focuses on Vision Mambas versus Vision Transformers.
- Vision Transformers are not thoroughly explored compared to the fully examined Mamba models. For instance, ViT-S in Table 1 has only a few options compared to Mamba's more comprehensive use of PEFTs.
- Some experimental comparisons lacked control:
  - In language models, for instance, Pythia 1.4B and the counterpart Mamba 1.4B had significantly different baseline performances (44.8 vs. 53.0), skewing the reported improvements, as higher starting accuracy inflates gains.

#### # Minors
- There is a related work that covers a similar topic, including PEFTs in Mamba, which suggests that this paper is not the first to study PEFT for the Mamba architecture as mentioned in lines 71-74. This reviewer understands that the authors could not include the related work due to its publication date being after this submission. However, it is suggested to compare this study with that work in future revisions to provide context and emphasize differences.
  - Parameter-Efficient Fine-Tuning of State Space Models, NeurIPS 2024 workshop (https://arxiv.org/abs/2410.09016)

- Typo in Table 1: Linear 'Proving'

questions:
- Could the authors provide additional results using Vim-Base or larger-scale architectures?
- How was the data utilized in the hybrid search method? Was a held-out set created for the searches?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730672591303

thread_type:
official_review

<thread_object>
creator: reviewer_u29m

<content>
summary:
This paper explores the application of parameter-efficient fine-tuning (PEFT) in the Mamba model. It begins by systematically analyzing the suitability of existing PEFT methods for the Mamba model, then adapts certain PEFT methods to better align with the characteristics of State Space Models (SSMs), and finally investigates hybrid PEFT search strategies. Extensive experiments were conducted across image and text modalities, providing thorough benchmarks for various PEFT methods and demonstrating that the Mamba model exhibits distinct fine-tuning behavior compared to Transformer models.

strengths:
1. The paper focuses on a novel and competitive model architecture, SSMs, systematically analyzing the applicability of existing PEFT methods for the Mamba model, offering insights for designing efficient fine-tuning schemes for SSMs.
2. The paper adapts existing PEFT methods specifically for SSMs, proposing new approaches tailored for Mamba, achieving good performances.
3. The extensive experiments, covering both image and text modalities, provide comprehensive benchmarks for all methods and reveal differences in PEFT performance between the Mamba and Transformer models.

weaknesses:
1. The paper covers too many aspects, including various PEFT methods, individual analyses, and hybrid architecture search. The authors should concentrate on one aspect and provide valuable conclusions for each part instead of describing each in detail, as this could reduce readability and divert focus.
2. The paper should consider including the most advanced works within each PEFT category, such as GPS[1] and SPT[2] for partial-tuning methods, and SSF[3] in addition to LoRA-based methods for reparameterization approaches.
3. Experiments should involve larger-scale models, where PEFT methods are particularly valuable. Evaluation across different model scales is crucial for PEFT methods. Notably, "scale" here refers to model parameter size rather than training data size (e.g., a model trained on ImageNet-21K in Section 4.5 is not necessarily larger than one trained on ImageNet-1K; model parameter size is more relevant in this context).
3. Presentation quality requires improvement, including spelling errors (e.g., "Linear Probing" instead of "Linear Proving" in Table 1’s ViT-S section) and inconsistent heights in sub-tables within Table 2.

[1] Zhang Z, Zhang Q, Gao Z, et al. Gradient-based Parameter Selection for Efficient Fine-Tuning[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 28566-28577.

[2] He H, Cai J, Zhang J, et al. Sensitivity-aware visual parameter-efficient fine-tuning[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11825-11835.

[3] Lian D, Zhou D, Feng J, et al. Scaling & shifting your features: A new baseline for efficient model tuning[J]. Advances in Neural Information Processing Systems, 2022, 35: 109-123.

questions:
1. Table 1 shows that Adapter-based and LoRA-based methods perform best. Did the authors try SequentialAdapter, or, as in the original LoRA paper, apply LoRA across multiple locations simultaneously in the model?
2. In Table 1, the Hybrid approach shows no significant advantage over individual PEFT methods (e.g., ParallelAdapter or LoRA) despite having far more parameters (117,236). The performance of Hybrid is not markedly better than LoRA(in_proj), which has only 1,483 learnable parameters.
3. Beyond parameter efficiency, computational efficiency is also crucial for PEFT methods. Did the authors consider comparing the computational efficiency of different methods (e.g., in terms of FLOPs or running time)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730954561839

thread_type:
official_review

<thread_object>
creator: reviewer_hvvy

<content>
summary:
This paper presents a comprehensive and detailed exploration of parameter-efficient fine-tuning (PEFT) for Mamba. By applying and refining existing PEFT methods, introducing novel PEFT approaches（Partial-tuning and Additional-scan）, and exploring optimal algorithmic combinations, it achieves outstanding performance in both image and language tasks. This work provides valuable insights for efficient fine-tuning in Mamba.

strengths:
1. The paper not only improves existing PEFT algorithms to suit Mamba but also proposes targeted, innovative algorithms, providing the community with a broader selection of PEFT methods.
2. The experiments comprehensively showcase the performance of each PEFT algorithm, offering a framework for optimal algorithmic combinations.
3. The extensive experimental workload demonstrates thorough exploration of the algorithms.

weaknesses:
1. The improved and original PEFT algorithms did not achieve the best results. In the VTAB-1k benchmark, the performance of the improved Partial LoRA is nearly identical to that of LoRA, while Additional-scan underperforms ParallelAdapter with comparable training parameters. Compared to traditional PEFT algorithms, these methods lack strong competitive advantage.
2. Although LoRA demonstrates outstanding performance on the VTAB-1k benchmark, its comparison with the modified Partial LoRA on language tasks is missing, making it unclear whether Partial LoRA is indeed the optimal PEFT method.
3. The algorithmic novelty is limited, as the concept of Partial-tuning appears relatively straightforward.

questions:
Please further explain the advantage and effectiveness of the proposed method compared with the baselines.
Please further explain the novelty of the proposed method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732612808639

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reply to all Reviewers

comment:
Thank you for carefully reviewing our paper and providing valuable feedback. Bellow are for all Reviewers.

# Some updates of the results
We made some corrections regarding the results of the paper, as there were some mistakes and issues with the training settings. One correction is that the accuracy of Affix-tuning in Table 1 was incorrectly reported as too low; the correct values were in Appendix Table 4. Additionally, the significantly lower accuracy of Affix-tuning (wo_proj) and Prompt-tuning (wo_proj) compared to Affix-tuning and Prompt-tuning is due to the learning rate being too small for the embedding module. Other PEFT parameters were either over-parameterized with DNN or fine-tuned from already trained values, which allowed us to train them sufficiently within 100 epochs. However, the parameters in the embedding module are not over-parameterized and require tuning from scratch. Therefore, to converge the training within 100 epochs, the learning rate needed to be increased by 10 times. We have updated Table 1 and Table 4 with the improved values.

# Experiments with Vim-B
We received a concern about the lack of evaluation on larger models from three reviewers. So we will discuss it here.
To address the concern regarding the lack of experiments with Vim-B, we conducted additional experiments. The results of Vim-B were not included initially because the model information was not publicly available at the time of our experiments (this August).
Below are extracts of the results for comparison between Vim-S and Vim-B.

| base model | method |  #Params (K)  | Natural | Specialized | Structured | Average |
|--------|------|------------|------------|------------|------------|-----------|
| Vim-S | Full | 25,450 | 59.4 | 68.7 | 34.4 | 47.1 |
| | Linear | 9 | 62.5 | 77.3 | 32.0 | 52.8 | 
| | Affix-tuning (w/o proj) | 230 | 63.9 | 77.7 | 34.2 | 54.3 | 
| | Affix-tuning | 117,000 | 75.8 | 83.3 | 58.9 | 70.3 | 
| | Additional-scan | 672 | 74.6 | 82.7 | 56.4 | 68.7 | 
| | ParallelAdapter | 663 | 76.1 | 84.0 | 60.0 | 71.0  | 
| | LoRA(out_proj) | 2,663 | 76.4 | 84.0 | 60.1 | 71.1 | 
| | LoRA(in_proj) | 1,483 | **76.6** | **84.1** | 60.2 | 71.3 | 
| | LoRA$_p$(X) | 1,778 | **76.6** | 83.9 | **60.8** | **71.5** | 
| Vim-B | Full | 96,906 | 43.9 | 69.5 | 27.8 | 42.5 |
| | Linear | 18 | 64.5 | 79.2 | 32.2 | 54.0 | 
| | Affix-tuning (w/o proj) | 460 | 69.2 | 82.2 | 50.8 | 64.2 | 
| | Affix-tuning | 463,000 | 76.4 | 83.5 | 55.6 | 69.1 | 
| | Additional-scan | 690 | 75.5 | 82.4 | 55.8 | 68.7 | 
| | ParallelAdapter | 1,787 | 76.6 | 83.9 | **60.4** | **71.3** | 
| | LoRA(out_proj) | 5,916 | 76.7 | 84.0 | 60.1 | **71.3** | 
| | LoRA(in_proj) | 2,987 | 76.7 | **84.3** | 59.4 | 71.0 | 
| | LoRA$_p$(X) | 3,557 | **77.0** | 83.9 | 59.9 | **71.3** | 

From Vim-S to Vim-B, there is little improvement, whereas ViT-B improved from ViT-S. Because this phenomenon is not observed in vanilla Mamba for language tasks (see Table 3), it seems it is not due to PEFT methods for Mamba. This suggests that large Vim/Mamba models might face potential challenges when fine-tuning with limited data. Note that even with Vim-B, PEFT still significantly outperforms full fine-tuning and achieves equivalent or higher accuracy than state-of-the-art PEFT with ViT-B, indicating that PEFT for Vim remains effective.

We also conducted experiments with VMamba-S and VMamba-B [1] to check the generalizability. The results are shown in the reply to the reviewer jdYa and added to the appendix. VMamba made various improvements over the original Mamba. It eliminated the state dimension (in other words, reduced the state dimension to 1), introduced a hierarchical structure similar to CNNs, replaced causal 1D convolutions with 2D convolutions, removed the Gated MLP structure, and substituted half of the Mamba blocks with MLP blocks. Due to these changes, there are some differences in trends, but the proposed method works also well with VMamba, achieving higher performance on the VTAB-1K dataset than Vim as the original VMamba performs better on ImageNet1K evaluation.

[1] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024b.
</content>
</thread_object>

</thread>


Metareview:
metareview:
a) This paper explores the application of parameter-efficient fine-tuning (PEFT) for Mamba. It analyzes the suitability of existing PEFT methods for Mamba, adapts certain PEFT methods to better align with the characteristics of State Space Models (SSMs), and investigates hybrid PEFT search strategies. Extensive experiments on image and text modalities for various PEFT methods show that Mamba exhibits distinct fine-tuning behaviour compared to Transformers.

b) The paper improves existing PEFT algorithms to suit Mamba but also proposes targeted algorithms, providing the community with a broader selection of PEFT methods. The extensive experiments, covering both image and text modalities, provide comprehensive benchmarks for all methods and reveal differences in PEFT performance between the Mamba and Transformer models.

c) The algorithmic novelty of the paper is limited, as the proposed approaches are not very different form standard PEFT. For some results (e.g. PEFT is more effective for Mamba than for Transformers), authors provide empirical evidence but not theory, a rationale or intuition about the results. 

d) Overall, I consider that the paper deserves publication because: 
PEFT methods for Mamba architectures are still under-explored. The authors provided additional experimental results in their rebuttal, which addressed most of the concerns raised by reviewers. 
The paper includes extensive experiments covering various PEFT methods, different Mamba architectures, models of different scales, and different tasks, which can be have a strong impact for further research in the field.
However, I still think authors should work on further improving the presentation of the paper, for instance clarifying the contribution and providing a better understanding of the empirical results obtained, as suggested by reviewers.