Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mamba
Authors: Donghyun Lee
Yuhang Li
Ruokai Yin
Shiting Xiao
Priyadarshini Panda
Venue: ICLR 2026 Poster (2026)
Keywords: Brain-inspired computing
Mamba
Fine-tuning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: State Space Models (SSMs) have emerged as powerful alternatives to attention-based Transformers, with Mamba demonstrating impressive efficiency and scalability. As these models grow increasingly larger, the need for Parameter-Efficient Fine-Tuning (PEFT) methods becomes critical to adapt pre-trained Mamba to downstream tasks without prohibitive computational costs. However, previous approaches simply apply traditional Transformer-tailored PEFT methods without addressing the unique temporal processing dynamics of SSMs. To address this limitation, we propose ***Memba***, a membrane-driven PEFT approach specifically designed for Mamba. ***Memba*** introduces Leaky Integrate Membrane (LIM) neurons as bio-inspired gating mechanisms that naturally accumulate membrane potentials over time, enhancing selective information retention. By strategically combining LIM neurons with Low-Rank Adaptations (LoRA) and cross-layer membrane transfer, our approach significantly improves Mamba's temporal modeling capabilities. Extensive experiments across language and vision tasks demonstrate that ***Memba*** achieves substantial improvements over existing PEFT methods.

Published as a conference paper at ICLR 2026
Memba: MEMBRANE-DRIVEN SERAAMETER-EFFICIEE FOR MAMBA
Donghyun Lee' Yuhang Li Ruokai Yin?
Shiting Xiao² Priyadarshini Panda
EElettrccal and Computer Engineering, University of Southern California "Electrical Engineering, Yale University {donghyun. 1ee. 1eeusc.edu
ABSTRACT
State Space Models (SSMs) have emerged as powerful alternatives to attention- based Transformers, with Mamba demonstrating impressive efficiency and scal- ability. As these models grow increasingly larger, the need for Parameter- Efficient Fine-Tuning (PEFT) methods becomes critical to adapt pre-trained Mamba to downstream tasks without prohibitive computational costs. How- ever, previous approaches simply apply traditional Transformer-talloree PEFT methods without addressing the unique temporal processing dynamics of SSMs. To address this limitation, we propose Memba, a membrane-driveer PEFT approach specifically designed for Mamba. Memba introduces Leaky Inte- grate Membrane (LIM) neurons as bio-inspired gating mechanisms that nat- urally accumulate membrane potentials over time, enhancing selective infor- mation retention. By strategically combining LIM neurons with Low-Rank Adaptations (LoRA) and cross-layer membrane transfer, our approach signif- icantly improves Mamba's temporal modeling capabilities. Extensive experi- ments across language and vision tasks demonstrate that Memba achieves sub- stantial improvements over existing PEFT methods. The code is available at https://gitiie com/ WetelllgenttComeee Laa--a--------- Landa/Memis
INTRODUCTION
State Space Models (SSMs) (Gu et al., 2021b;a; Fu et al., 2022) have emerged as powerful alternatives to Transformer (Vaswani et al., 2017) architectures, offering linear computational complexity with respect to sequence length while maintaining competitive performance. SSMs share functional similarities with recurrent architectures, including Long Short-Tern Memory (LSTM) (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Unit (GRU) (Chung et al., 2014), through evolving hidden states, though they employ different mathematical foundations based on state space theory (Gu et al., 2020). Recent advancements, particularly Mamba (Gu & Dao, 2023; Dao & Gu, 2024), have demonstrated remarkable success across language modeling (Pióro et al., 2024; Wang et al., 2024), computer vision (Liu et al., 2024c; Zhu et al., 2024), and other domains (Wang al., 2025; Quan & Li, 2024; Ota, 2024; Hu et al., 2024) by introducing selective SSMs with data-dependent parameters. As these models scale, Parameter-Efficies with minimal trainable parameters. While PEFT techniques have shown success in Transformer-bast models (Hu et al., 2022; Houlsby et al., 2019), their application to SSMs remains limited. Recent works (Yoshimura et al., 2024; Halloran et al., 2024; Ham et al., 2024) have begun exploring PEFT for Mamba, but simply transfer Transformer- -ailloee methods without addressing the unique temporal processing dynamics of SSMs.
Fine-Tuning (PEFT) methods become crucial for adaptation
Although Mamba is meticulously designed based on state-space theory (Gu et al., 2020), current architecture lacks the sophisticated gating structures found in traditional recurrent networks such as LSTM (Hochreiter Schmidhuber, 1997) and GRU (Chung et al., 2014), relying instead on a single linear transformation. Traditional recurrent networks incorporate multiple trainable gates to manage memory retention and forgetting over time. In contrast, Mamba's 's simplified gating mechanism lacks temporal selectivity, structured memory, and nonlinear control capabilities. We believe that this limitation, shared with earlier SSMs such as S4 (Gu et al., 2021a), can hinder the model's ability to adaptively capture task-specific temporal information during fine-tuning. Furthermore,

---

Published as a conference paper at ICLR 2026
Original SSM
Saliency Map
Language
Vision
74
Input Image
SSM
Accuracy (50
Mamba-790M
Accuracy (%)
Vim-S
78.68
56.7
50.7
70-
SSM
SL
Skan
LoBk
Mendaa lours)
Affie
Sran
Lotd
Mombe (ourn)
Memba
(a)
(b)
Figure Overview of Memba architecture and performance comparison. (a) Architecture and saliency map comparison between original SSM and Memba on Pathfinder dataset image. The pink lines in architectures represent gating branches, and the green dashed circle indicates the target path to be identified. (b) Performance comparison language (commonsense reasoning) and vision (VTAB- 1k) tasks using Mamba- 790M and Vim-S architectures respectively. We compare Memba with SLL LoRA, Additional-scan, Affix-tuning, and LoRA in (Yoshimura al., 2024).
recent studies (Yoshimura et al., 2024; Ham et al., 2024) have revealed that directly fine-tuning the state-space components often degrades Mamha's ' performance, suggesting that modifying temporal processing mechanisms during adaptation is particularly challenging. This phenomenon raises fundamental question: how can we effectively incorporate temporal adaptation during fine-tuning without disrupting the balanced dynamics of pre-trained SSMs?
In this work, we propose Memba, which introduces a membrane-driven PEFT technique to enhance SSM gating. At the core of our approach is the Leaky Integrate Membrane (LIM) neuron, novel mechanism that leverages the inherent hidden states of neuronal membrane potentials to strengthen the selective gating capabilities of SSM. Unlike traditional gating with foed-forward layers (Hochreiter Schmidhuber, 1997; Chung et al., 2014), temporal chunked LIM neurons naturally accumulate the membrane potential over time, providing sophisticated yet computationally efficient solution for temporal processing without requiring additional learnable parameters complex recurrent structures. Furthermore, we design the LIM neuron with continuous flow of membrane information across layers, where each neuron transfers averaged membrane states initialize neurons in subsequent layers, creating uninterrupted temporal processing throughout the network. Our approach combines LIM neurons with strategically placed LoRA input and output projections, creating comprehensive PEFT method specifically tailored for Mamba models. Consequently, Memba achieves superior performance existing works (Yoshimura et al., 2024; Halloran et al., 2024) Mamba fine-tuning while requiring only a fraction of the trainable parameters.
To demonstrate the effectiveness of membrane-driven gating in SSM architectures, Figure presents an overall comparison between a standard SSM model and the proposed Memba. In Figure (a)), Memba exhibits sharper saliency focused along the true path, while the original SSM's attention remains diffused, highlighting the inherent benefit of membrane dynamics in promoting selective information flow. Figure (bb) shows the fine-tuning performance comparisons across language and vision tasks, where Memba consistently outperforms existing PEFT methods. The main contributions of our work are as follows:
We propose Memba, a membrane-driven PEFT approach that enhances Mamba's gating mechanisms, effectively introducing temporal adaptation without modifying the core state- space components.
We introduce a temporal chunked LIM neuron with cross-layer membrane propagation. This LIM neuron efficiently processes long sequences while preserving temporal information through evolving membrane potentials.
Through extensive experiments across language (commonsense reasoning) and vision (vi- sion adaptation) tasks, we demonstrate that Memba consistently achieves state- oftee-of-t--rrr the-art performance compared to existing PEFT methods.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORKS
2.1 STATE SPACE MODEL
State Space Models (SSMs) present promising direction in sequence modeling that combines the parallel processing advantages of Transformer (Vaswani et al., 2017) with the recurrence properties of Recurrent Neural Networks (RNNs) (Hochreiter & Schmidhuber, 1997; Chung et al., 2014). (Gu et al., 2021a) introduces structured state space sequence models (S4), which leverages efficient parameterization of continuous-time state space models for sequence modeling, enabling parallel computation. Subsequent architectures, including the Diagonal State Space (DSS) (Gupta et al., 2022), the Gated State Space (GSS) (Mehta et al., 2022), and Hungry Hungry Hippos (H3) (Fu et al., 2022), enhance the expressivity for better performance while simplifying the implementation complexity. Most recently, Mamba (Gu & Dao, 2023; Dao Gu, 2024) introduces selective SSMs with data-dependent parameters, enabling dynamic adaptation to input context. This innovation has led to competitive performance across several domains (Pióro al., 2024; Li et al., 2024; Wang et al., 2025; Quan & Li, 2024; Ota, 2024), establishing SSMs viable alternatives to Transformer architectures for sequence processing. Given its promising performance and efficiency, Mamba shows significant potential to serve backbone for pre-trained foundation models (Gu & Dao, 2023; Ham et al., 2024; Hatamizadeh & Kautz, 2024), making efficient adaptation techniques increasingly important for future downstream applications.
2.2 PARAMETER-EFFICHEE FINE-TUNING
PEFT methods address the computational and storage challenges of adapting large pre-trained models to downstream tasks. These approaches modify only small subset of model parameters while keeping the majority frozen. We differentiate the PEFT algorithms into four categories (Han et al., 2024): additive, selective, parameterized, and hybrid fine-tuning. Additive fine-tuning (Houlsby et al., 2019; He et al., 2021; Pfeiffer et al., 2020; Mahabadi et al., 2021) introduce new trainable components while preserving the original weights. including adapters that insert modules between layers, and prompt-base braset-based methods (Li & Liang, 2021; Liu et al., 2021; Lester et al., 2021; Liu et al., 2024b) that augment inputs with learnable tokens. Selective fine-tuning identifies and updates only a critical subset of the original parameters, either through unstructured approaches based on importance metries (Guo et al., 2020; Sung et al., 2021; Xu et al., 2021) or structured methods targeting specific components (Zaken et al., 2021; He et al., 2023). Reparameterized fine-tuning transforms the optimization space, primarily through low-rank techniques like LoRA (Hu et al., 2022; Zhang et al., 2023) that decompose weight updates into smaller matrices, and its derivatives that incorporate quantization (Dettmers et al., 2023) or direction-magniteee decoupling (Liu et al., 2024a). Hybrid fine-tuning combines multiple paradigms leverage their complementary strengths, integrating various strategies to achieve optimal performance-efficien trade-offs (He et al., 2021; Zhang et al., 2024; Hu al., 2023).
While PEFT methods have been extensively studied in Transformer architectures, their application to SSM models like Mamba remains relatively unexplored. (Halloran al., 2024) investigates the appli- cation of LoRA to Mamba's 's state space components, while (Yoshimura et al., 2024) comprehensively evaluates various PEFT techniques for SSMs, revealing unique challenges posed by their selective scanning mechanisms. (Ham et al., 2024) demonstrates that targeting projectors rather than state space components yields superior performance through their diagonal-based adaptation approach, focusing primarily on transfer learning. These initial studies indicate that although conventional PEFT methods can be applied to Mamba with reasonable success, there remains significant room for approaches that better leverage the architectural uniqueness of SSMs. The distinctive temporal processing characteristics of SSM architectures create opportumities for specialized adaptation tech- niques that enhance information flow control capabilities without modifying the core state-space components. Our work addresses this opportunity by introducing a bio-inspired mechanism designed to incorporate temporal dynamics into the gating mechanism.
3 PRELIMINARIES
Mamba (Gu& & Dao, 2023) architecture addresses the quadratic computational complexity of transform- by employing SSM with selective mechanisms. The standard continuous-time linear time-invariant
3

---

Published as a conference paper at ICLR 2026
SSM is defined by:
h (1()) Ah(1) + Bx((), v(() Ch(1) + Da(();
where x(t) E is the input sequence, h(t) E R is the hidden state with N being the state dimension that controls the model's representational capacity, and y(t) E R is the output. The parameters A E (N(x)), B E R ( C E (R(1) and D E R define the dynamics of the system. For practical implementation with discrete inputs like tokens, Mamba employs Zero-Order Hold (ZOH) discretization to derive equivalent discrete parameters with step size parameter A:
= cxp(AA) (A), = (A (cxp(AA) I)AB, C =
(2)
This discretization transforms the continuous system into its discrete counterpart called as selective scan (Scan):
h, Ah;-1 B
V1 CC= D.
The key innovation in Mamba lies in its selective paramcterization mechanism, where A, B, and C become input-dependent, consequently making A, B, C, and D input-dependent as well. This is achieved through a structured parameter matrix that computes these values dynamically based on input context, enabling adaptive behavior while maintaining linear computational complexity with respect to sequence length. Critically, Mamba employs multiplicative gating mechanism that combines the output of selective sean with transformed input:
gt yt @ (I+),
where (-)) is SiLU activation and @ represents element-wise multiplication. This gating mechanism plays a crucial role in controlling information flow through the network. However, while the selective scan in the SSM branch effectively handles temporal processing with evolving hidden states, (Yoshimura et al., 2024; Ham et al., 2024) have shown that fine-tuning this component directly leads suboptimal results. We address this limitation by introducing temporal adaptation capability the gating branch during fine-tuning. Detailed notations for the original Mamba architecture are provided Appendix A.
METHODOLOGY
In this section, we propose Memba, membrane-driven PEFT approach for Mamba models that introduces temporal processing in the gating branch during fine-tuning. We first present the overall architecture of the Memba block, then introduce its three components.
4.1 OVERALL ARCHITECTURE
We develop Memba by integrating three main components: ! Leaky Integrate Membrane (LIM), which provides bio-inspired temporal processing; 2 optimal placement of Low-Rank Adaptations, which strategically modifies key projection layers; and ④ cross-layer membrane potential transfer, which maintains temporal coherence across network depth. These modifications build upon the original Mamba architecture illustrated in Figure
When processing an input tensor Xinput BXLD with B, L, and D representing batch size, length of input sequence, and feature dimension respectively. Memba first applies normalization and projection, then divides the resulting tensor into two parallel pathways:
XSSM Xgare Split(War (W (Mapt)))
(5)
where Wi is the linear layer for input projection ("in.proj") doubling the channel dimension, RMS represents RMS normalization, and the Split represents channel-wise division by two. The XssM E x processes information through the selective scan, while the Xgate E BBxLDD leverages our LIM mechanism. We detail the LIM mechanism in Section 4.2.
Y ssM = Scan
gale = a (WWet gate out (LIM( Wa in
(6)
Here, Scan represents the selective sean computation shown in equation indicates SiLU activation, and geate Waate are the linear projections before and after the LIM neuron, respectively.
4

---

Published as a conference paper at ICLR 2026
Frozen
Learnable
Datallow
---? Membrane flow
LoRA
Fimhedding
I PRMS Norm
RMMS Norm
Linceniian in)
SSM
D LLinear
Liniaattaa in)
SSM
Linear
DDLI
P
Membranc Transfer
Figure 2: Overview of Memba architecture. On top of original Mamba architecture including embedding, normalization, linear layers, and SSM, our Memba is designed with ① Leaky Integrate Membrane (LIM), 2 Low-Rank Adaptations (LoRAs) on input and output projection, and 3
membrane transfer across layers.
W and out reduce the computational overhead of LIM neuron operations through dimensionality reduction. SSM and gate are combined through multiplicative gating to produce the final output:
Y out W out Y sSM @ Y gute),
(7)
where W out is the linear layer for output projection ('out-projj This architecture modification enhances temporal adaptation through the gate path without directly altering the dynamics of the selective scan computation.
4.2 KEY COMPONENTS Memba
① LEAKY INTEGRATE MEMBRANE NEURON
Implementation To bring the temporal flow to Mamba's 's gate branch, introduce the Leaky Integrate Membrane (LIM) neuron shown Fig- The LIM neuron is inspired by the Leaky Integrate-and-Fint (LIF) neuron, described in de- tail in Appendix B. Rather than processing each token as an individual step, which would be ex- pensive for long sequences, we adopt a chunking strategy for practical implementation. Given an input sequence of length L, we partition into T equal-sized chunks [X[1), X .... [[]], where each chunk X E R L(j for E [T/T] tokens, with B and D representing batch size and fea- ture dimension respectively. Here, L.] denotes the floor function, ensuring each chunk contains the same number tokens for consistent pro- cessing. For this uniformity in chunk size, trim any remainder tokens when the sequence length not evenly divisible by the number of chunks.
Leaky Integrate Membrane (LIM)
Tokens
(B D)
LIM
Reset
Mem,
T Chardes
X2
Mem; MV
Men, M
Mem N
X4
(1,2, 2.., T} contains [
Sequence
Mem,
Memt
Memg
Mom,
Output = Concat(Mem,, Memo, Mem Mem.)
Figure 3: Overview of Leaky Integrate Membrane (LIM).. Each token chunk is processed with LIM dynamics, and membrane outputs are concatenated to form the final sequence representation. In this figure, the input contains L tokens split into T 4 chunks, with each chunk (X,, X2, X,, X,) containing tokens.
The key innovation in our LIM approach lies in maintaining membrane continuity across chunk boundaries while processing chunks sequen- tially. For each chunk (corresponding to one step), we process the input tokens using the leaky integrate membrane dynamics with reset:
uli + 1l' r(ru(i]] w X()),,
(8)
0 if > Van
(xx)
(9)
x
otherwise

---

Published as a conference paper at ICLR 2026
Pabertal
Chunk #1 Chunk #2
Chunk #3
Chunk #4
-8.02
Flat
LIM
-0.08
-6.30
-2.44
1.50
200
250
eequ indce
(a) Chunked & Flattened Inpul
(b) Membraae distribution
(c) Saliency Map
Figure Membrane-driven temporal processing in Memba. (a) The input image is divided into spatial chunks and flattened into a sequential representation. (b) Membrane distribution and (c) saliency map through the LIM neuron show how the LIM neuron tracks main features while progressively decreasing baseline potentials across chunks, demonstrating adaptive temporal attention. We provide more visualizations Appendix I.
where uli]' E [XXLL/T)xDD is the membrane potential in (-th layer i-th chunk, E (0, 1] is the leaky factor for membrane potential leakage, W is the weight of 1-th layer, and r(.) is the reset function which sets values above threshold 0 while preserving others. Note that and Va, key parameters for deciding the membrane distribution, and we analyze the effects of T and VnA in Section 5.3. After processing all chunks sequentially, we concatenate the outputs reconstruct the full sequence representation of length L, zero padding if necessary to restore any trimmed tokens. The detailed algorithm of LIM neuron shown in Appendix C.
Cembrane-Driven Temporal Processing The LIM neuron is designed to retain the temporal infor- mation with selective memory through membrane dynamics. We analyze the membrane potential behavior of the LIM neuron as shown in Figure 4. The input is divided into four colored chunks, flat- tened into sequence, and processed through the LIM neuron to generate membrane potentials. This visualization reveals two key characteristics of our approach. (1) Critical path features, highlighted in purple and pink boxes, generate pronounced peaks in the membrane potential, demonstrating the model's 's selective attention to tesk-relevant information. (2) We observe gradual decrease in baseline membrane potential across chunks, indicating progressive forgetting of memory as context accumulates. This downward trend aligns with SSM's natural behavior of retaining recent tokens while forgetting earlier ones. Unlike the linear gate in the original Mamba, which delivers uniform sensitivity, our membrane-batt approach in Memba naturally modulates temporal responsiveness.
The LIM neuron not only enables efficient processing of long sequences but also provides theoretical advantages for model optimization and generalization as follows:
Theorem 1. Let L(y) be a diie-differentiable loss function, and let = y be the output of a standard Mamba block. When augmented with our LIM mechanism, the effective output becomes ÿ: = Yt @ g(u)), where u, is the membrane potential. The expected loss satisfies:
E[C(ý)) = L(ye @ g(,))) + R(ye, ū.) + +[1E,l1))
(10)
ū, with E[E,] = 0. and R is a bounded regularization term satisfying 3>xmxxx >-maxx where Amax the maximum eigenvalue of the Hessian of L and y depends on the model oulputs and gate sensitivity.
E: = where ū, = Ej
R(y,, ūt) S
This theorem reveals LIM's dual effect: the mean membrane component provides temporal context in- tegration through leaky dynamics, while the fluctuation component introduces bounded regularization that adapts to model sensitivity and output magnitude. The smoother loss landscape geometry ob- served in Appendix D.6 represents empirical evidence of how this theoretical regularization manifests in practice. See Appendix D for complete derivation.
2 PLACEMENT OF LOW-RANK ADAPTATIONS
To identify the optimal application points for our LIM neuron, we conduct an ablation study on the main components of 1emmaa-130M We evaluate applying LoRA to different projectors on commonsense reasoning tasks. "All" refers to simultancously adapting four key projectors in original Mamba: input ("in-proj"), output ("out-proj") time-scale "dtop""""), and selective state (" ["x-pro j") as shown in Appendix A. We then systematically exclude individual components, denoted by "-dt", "-x", "-out". and "iin". For example, in "-dt", LoRAs applied to in.proj, out-projj and x.proj.

---

Published as a conference paper at ICLR 2026
The results in Table 1 clearly show that input and output projectors are most critical for fine-tuning with the LIM neuron. Excluding either leads to performance drops of 1.2% and 0.8% respectively, suggesting these projectors act as crucial informa- tion bottlenecks in the Memba architecture. We fur- ther compare the accuracy of full fine-tuning against Memba with LoRA applied to both in-proj and jut_proo on 790M and 1.4B architectures in Fig- ure 5. Notably, our approach achieves higher perfor- mance than full fine-tuning while using fewer train- able parameters. Full fine-tuning often suffers from overfitting due to the relatively large number of train- able parameters compared to the size of downstream task datasets (Ham et al., 2024).
Table Ablation study on the impact of applying LoRA different projection com- ponents in Memba- 130M.
All
dt
-X
out
-in
Avg. Acc. (56
43,9
43.9
43.7
43.1
42.7
Acc. drop (%)
0.0
0,2
4
0.8
1.2
52
53
Full
Membba Memmba790M
out
in+out
Full
Memba-1 1.4B
out
in+uul
3 CROSS-LAYER MEMBRANE TRANSFER
55
Averaged Accuracy (%)
Figure 5: Performance comparison be- tween full fine-tuning and Memba with Lo- RAs applied to in-proj and out-proj across 790M and 1.4B models.
As large models scale in depth, maintaining tempo- ral context across layers becomes challenging, yet crucial for effective sequence modeling. To address this, implement cross-layer membrane potential transfer technique that propagates temporal informa-
tion throughout the network hierarchy without increasing computational cost. After processing all chunks within a layer and obtaining membrane potentials [u'[1], u' [2].... u' [T]}, we compute the average membrane state, ú' across all chunks. This averaged membrane potential ū' serves as the initial state for the first chunk of the subsequent layer:
ú'
Σ Jul. T
1
u [1] u'
(11)
By transferring this compressed temporal context, we enable each layer to begin processing with summary of the temporal dynamics captured by the previous layer. This mechanism creates hicrarchical flow of temporal information through the network, allowing deeper layers to build upon the representations learned by earlier layers. The use of averaged membrane potentials helps prevent information loss that might occur if only the final state were propagated, ensuring that the model maintains sensitivity to patterns that emerged at different points in the sequence.
5 EXPERIMENTS
In this section, we evaluate the proposed Memba approach on both language and vision tasks. For language tasks, we fine-tune pre-trained Mamba on 8 commonsense reasoning benchmarks. For vision tasks, we fine-tune two different architectures, Vim (Zhu al., 2024) and VMamba (Liu et al., 2024c), on the Visual Task Adaptation- -eraation--encmmakk (VTAB)- 1k dataset (Zhai et al., 2019). We also present ablation studies to analyze the Memba architecture for a better understanding. Our experimental setup follows the framework established in MambaPeft (Yoshimura et al., 2024), and the experiments are implemented by A A10 GPUs.
5.1 LANGUAGE TASK
Experimental Details To evaluate Memba, we begin by loading pre-trained Mamba models (Gu & Dao, 2023) trained on the Pile dataset (Gao et al., 2020), apply the Memba architecture modifications, and then fine-tune Memba reasoning tasks. We evaluate performance across eight individual benchmarks: BoolQ. PIQA, SIQA, HellaS wag. WinoGrande, ARC-Challenge, ARC-Easy, and OpenbookQA. For fine-tuning, follow the setup of Ldd-tdapter (Hu et al., 2023), and for evaluation, we use the 1m e framework (Gao al., 2024). Detailed hyperparameters provided in Appendix E.
combined dataset of approximately 170k examples from commonsense
7

---

Published as a conference paper at ICLR 2026
Table Performance comparison between Memba and prior methods on commonsense commonsense reasoning
datasets. Reported values are accuracy percentages (%). Bold and underlined entries indicate the best and second- best performance, respectively. HS and WG refer the HellaS wag and WinoGrande datasets. All results are from (Yoshimura et al., 2024), except for SLL LoRA (Halloran al., 2024).
Model
Method
(Params(())
BoolQ
PIQA
SIQA
HS
WG
ARC-e
ARC-
OBQA
Avg.
Pythia 160M
Full
100
61.3
62.9
37.1
30.7
50.6
41.5
24.3
27.8
42.0
61.0
36.3
52.0
38.2
28.0
Full
100
56.1
65.3
38.7
35,3
$2.0
46.4
25.7
32.8
43.8
SLL LoRA
1.45
56.1
63.3
38.2
34.6
51.6
43.5
23.6
30.6
42.7
Additional-som
30.0
Affx--tniig
64.64
59.7
64.3
38.2
35.2
51.9
42.9
24.0
29.0
43.2
Mamba 130M
LoRA proj)
2.23
53.5
62.9
38.2
33.8
53.1
46.4
23.7
30.8
42.8
LoRA (X)
2.67
61.7
64.0
39.5
34,3
52.2
43.5
25.3
29.4
43.7
Memba (in. (in_)
3.95
52.4
30.0
43.5
Memba (our (our_proj)
3.10
58.4
64.9
38.8
34.4
51.8
50.0
24.2
30.0
44.0
(in+out.proj)
58.8
51.6
47.7
24.7
31.2
Pythia 410M
Full
100
35.0
68.4
42.1
40.8
53.9
50.8
26.7
30.0
46.0
LoRA
61.3
54.9
28.6
45.7
Full
100
58.1
69.9
41.9
45.7
53.8
52.7
29.7
33.4
48.2
SLL LoRA
2.30
59.5
69.6
42.2
44.1
$4.9
50.6
26.3
30.8
47.3
Additional
Aflix-tuning
68.88
61.2
68.4
39.6
46.2
55.4
48.2
28.2
30.6
47.2
Mamba 370M
LoRA (in proj)
2.07
55.4
41.0
44.7
54.1
52.4
28.3
33.4
47.2
LoRA, (X)
2.67
60.8
68.8
42.1
44.7
56.2
50.4
27.4
32.2
47.8
Memba (in proj)
59.1
42.7
45.1
$4.0
55.2
33.0
48.3
Momba (uul_proj)
2.88
58.0
70.0
42.5
45.4
$4.9
$5.4
26.7
31.2
48.0
(rroj)
58.7
34.0
Pythia IB
100
$5.0
70.2
42.5
47.5
54.4
54.1
29.7
33.2
48.3
LoRA
0.41
60.0
69.3
40.9
45.3
53.6
49.8
27.2
31.0
47.1
62.0
72.1
54.0
55.9
57.7
35.2
51.6
SLL LoRA
3.1
60.7
72.0
42.4
54.7
$6.9
55.3
29.4
34.2
50.7
Additionalscaa
0.33
63.0
$4.9
32.6
50.7
Aflix-tuning
69.99
61.0
72.5
41.0
54.9
55.6
$4.6
29.6
33.8
50.4
Mamba 790M
LoRA proj)
61.7
56.7
33.8
50.7
LoRA, (X)
1.75
59.9
72.2
44.2
52.8
58.0
53.7
30.8
34,8
50.8
Memba (in_proj)
2.61
62.2
43.8
34.4
57.5
61.0
34.0
52.1
Memba (out.proo)
2.04
$7.9
72.0
44.3
55.2
56.7
60.4
31.2
34.0
51.5
Momba proj)
34.3
Pythia 1.4B
58.6
71.1
42.7
53.6
55.1
58.5
29.9
34.8
50.5
LoRA
0.44
60.1
71.3
42.5
50.1
58.9
$7.6
29.6
33.6
50.5
Full
61.4
73.3
43.9
56.9
59.0
59.7
34.0
35.4
$3.0
SLL LoRA
4.64
59.7
73.5
43.1
$6.9
60.7
59.7
31.7
36.0
52.7
Additional
0.26
63.0
73.5
42.8
57.5
60.5
60.9
32.4
37.4
53.5
LoRA (inuproj) LoRA, (X)
1.13
62.6
73.6
43.7
55.6
59.7
58.3
31.7
35.6
$2.6
Mamba 1.4B
1.36
63.1
42.7
57.7
61.6
60.4
37.4
53.7
Memba (in-proj)
2.02
63.1
74.0
43.0
58.5
61.7
63.8
33.2
37.4
$4.3
Memba (out proj)
1.58
62.6
74,3
43.5
58.6
60.6
64.3
32.3
37.0
34.1
Memba (intoul projj
2.68
64.4
74.3
43.4
58.6
60.7
64.2
33.0
37.4
$4.5
Results The overall results of Memba on language tasks are shown Table 2. Our comparison baselines are Transformer-basee architecture Pythia (Biderman al., 2023) and previous Mamba fine-tuning approaches, including SL..I LoRA (Halloran et al., 2024) and MambaPEFT (Yoshimura al., 2024). We present three variants of our approach for each model size: LoRA applied to input projections (in.proj), output projections (out proj), and both. We observe that our Memba consistently achieves state-of-the-ar performance. Notably, with the Mamba 790M model, our in.pro joutt.pro variant demonstrates substantial 1.5% absolute improvement over the best results reported MambaPEFT (Yoshimura et al., 2024), highlighting the effectiveness of our membrane-based approach for larger models. The iso-parameter cases are shown in the Appendix G.2.
5.2 VISION TASK
Experimental Details We further investigate the effectiveness of Memba on vision tasks by fine- tuning two pre-trained Mamba architectures: Vim (Zhu et al., 2024) and VMamba (Hatamizadeh & Kautz, 2024), both initially trained on ImageNet- Ik (Deng et al., 2009). Since the original VMamba architecture removes the gate branch, utilize Vanilla-VMambe which preserves the gating
8

---

Published as a conference paper at ICLR 2026
Table 3: Performance comparison between Memba and previous works VTAB-IK datasets. Values shown are accuracy percentages (%) Bold and underlined values represent the best and second-best performance respectively. represents our implementation, and other results are from (Yoshimura et al., 2024).
Model
Method
#Params(K)
Natural
Specialized
Structured
Avg.
Scratch
21,704
10.66
$6.12
24.83
26 20
Full
21,704
51.79
72.29
45.27
53.47
VIT-$
LoRA
628
73.60
82.22
57.61
68 68
Adaptformer
333
73.63
83.15
57.80
68.97
Adapter+
122
74.68
83,57
58.82
69.87
Scratch
25,450
8.33
49.87
28.16
25.42
Full
25,450
59.35
68.74
34.39
50.08
LoRA (emhed)
45
64.66
77.53
43.83
58.60
LoRA (x_pruj)
2,540
74.41
81.92
54.88
67.77
LoRA pdo.proo)
2,442
75.35
83.05
57.12
69.30
LoRA (out_proj)
2,663
76.42
83.96
60.08
71.12
LoRA (in.proj)
1.483
76.58
84.08
60.16
71.25
Vim-S
LoRA, (Z)
1,778
76.15
84.26
59.72
70.94
LoRA, (X)
1.778
76.64
83,89
60.84
71.52
LoRA (in-oul-proo)
709
75.69
84.42
59.43
70.68
Hybrid (w/ proj)
117,236
77.00
84.41
61.55
72.05
Hybrid (w/o proj)
1,044
76.85
84.42
61.06
71.80
Memba (in proj)
2,064
76.91
85.10
60.70
71.81
Memba (out, proj)
3,244
76.92
85.32
61.18
72.06
Memba (in-out pproj
4,718
77.07
85.66
61.70
72.40
LoRA (in (in-proj)i
3,993
77.76
86.05
63.44
73.48
LoRA (out proj)f
2,396
77.43
64.33
73.73
LoRA (in+out projff
6,389
77.31
85.81
63.30
73.20
Memba (in proj)
5,591
77.66
86.06
63.92
73.64
Membu (out proj)
3,993
77.77
85,98
64.87
74.07
Memba cintoul_prrji
7,987
77.14
86.14
63.93
73.48
mechanism. We evaluate performance on the VTAB-Ik image classification benchmark (Zhai et al., 2019), which comprises Natural, Specialized, and Structured domains. For fine-tuning, we adopt the DeiT (Touvron et al., 2021) training framework across all domains. Detailed hyperparameters are provided in Appendix E.
Results Table 3 presents our Memba results on Vim (Zhu et al., 2024) and VMamba (Hatamizadeh Kautz, 2024) architectures, compared against ViT (Dosovitskiy et al., 2020) and previous PEFT approaches from MambaPHFT (Yoshimura et al., 2024). As in our language experiments, we evaluate variants with LoRA applied input projections (in_proj), output projections (out-pro j), or both. Our Memba outperforms previous PEFT methods on both Vim-S and Miailla-MMaba--- architectures. Notably, with Vim-S, our out-proj variant achieves 72.40% average accuracy across all domains, surpassing the previous best result of Hybrid method while using only 28% of the trainable parameters. Per-task performances each categories are shown in Appendix F.
5.3 ANALYSIS
Contribution of Key Components To quantify the impact of each component in Memba, we conduct an ablation study examining combinations of our three key innovations: ! LIM neurons, @ LoRA, and membrane transfer. Table 4(a) presents these results. For language tasks, we use the 130M parameter Mamba model on commonsense reasoning, while for vision tasks, we use the Vim-S architecture on the Specialized category of VTAB-IK, which includes Camelyon, FuroSAT, Resisc45, and Retinopathy datasets. Our results show that while LoRA enables effective fine-tuning of Mamba models, adding LIM neurons and membrane transfer further boosts performance.
Impact of Membrane Parameters In LIM neuron, two key hyperparameters control the membrane dynamics: the leaky factor (T) and threshold (Vv.) in equation 8 and equation The leaky factor determines how much previous membrane potential is retained, while the threshold governs the reset mechanism. To understand their influence, we conduct experiments with various parameter combinations using the Vim-S architecture the Specialized category of VTAB- 1k, shown in

---

Published as a conference paper at ICLR 2026
Table 4: Ablation studies of (a) key components of LIM and (b) membrane parameters.
① LIM
2 LoRA
Membraan Transer
Mamba- 1 130mm
Vim-S
(Via = 1)
1/2
1/3
1/4
1/5
Averaged Acc.(%)
85.7
85.2
85.2
85.0
43.3
84.0
X
43.8
85,3
Veh 1/2)
0.5
1
2
3
44.3
85.7
Averaged Acc.(%)
85.4
85.7
85.3
85.2
(a) Accuracy comparison between combinations of key components of LIM neuron.
(b) Impact of membrane parameters vision tasks. т and Via. are leaky factor threshold respectively.
Table 4(b). For the leaky factor, higher values of T yield better performance, confirming that stronger retention of previous states benefits temporal processing. For the threshold, VvA = provides optimal performance, indicating lower values trigger excessive resets that disrupt information flow, while higher values reduce reset frequency, hindering the model's ability to filter irrelevant information. These results demonstrate that balanced membrane dynamics are critical for effective temporal processing. Additional extensive ablation studies are presented Appendix G.
LIMITATIONS
The LIM neuron accumulates membrane dynamics with a reset function, which inherently introduces recurrent computation and consequently incurs additional computational overhead. Although the selective scan operation in SSMs also involves recurrent computation, Gu et al. (Gu & Dao, 2023) address this efficiency challenge through specialized hardware kernel modifications. Similarly, our LIM algorithm can be integrated into optimized CUDA kernels in future implementations. The widely- used SpikingJelly framework (Fang et al., 2023) demonstrates that custom CUDA kernels for LIF neurons achieve up to 30x speedups through operator fusion, where element-wise operations (leaky decay, addition, reset) are fused into a single kernel launch. Our chunk- recurrent operations can directly leverage this optimization approach, the sequential dependency between chunks does not prevent intra-chunk parallelization. We expect this would achieve negligible computational overhead comparable the selective scan kernel.
7 CONCLUSION
We introduce Memba, membrane-drrive PEFT approach for Mamba models that integrates Leaky Integrate Membrane (LIM) neurons with strategic Low-Rank Adaptations (LoRAs). Without mod- ifying state-space components, our method enhances temporal adaptation capabilities exclusively through the gating branch, preserving the tuned dynamics of pre-trained SSMs. Experiments across language and vision tasks demonstrate Memba's consistent improvement over existing PEFT meth- ods. Our approach represents important step toward specialized adaptation techniques for SSMs, opening possibilities for effective fine-tuning these architectures across diverse applications.
ACKNOWLEDGEMENT
This work was supported in part by CoCoSys, JUMP2.0 center sponsored by DARPA and SRC, the National Science Foundation (CARBER Award, Grant #2312366, Grant #2318152), the DARPA Young Faculty Award, the DoE MMICC center SEA-CROGS (Award #DE-SC0023198) and the Global Industrial Technology Cooperation Center (GITCC) program.
REFERENCES
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pp. 2397-2430. PMLR, 2023.
10

---

Published as a conference paper at ICLR 2026
Anthony N Burkitt. A review of the integrate-ane and--ire neuron model: I. homogeneous synaptic input. Biological cybernetics, 95:1-19, 2006.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3553 2014.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, arXiv preprint arXiv:2405.2 20600, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36: 10088-10015,, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:20IO.I 201.09999.11 2020.
Wei Fang. Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 9440): 1480, 2023.
Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprini 222::22212.1405.. 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.000 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https: /ttp:///eeooo/ooo wwt/recordeeeeeeoooooo 12608602.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.0077 2023.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, arXiv preprint arXiv:2111.00396 2021a.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572-585, 202
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning.
arXiv preprint 2Xix:2012.074633 2020.
Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are effective as structured state spaces. Advances in Neural Information Processing Systems, 35:22982-2298 2022.
John Timothy Halloran, Manbir S Gulati, and Paul Roysdon. Mamba state-space models can be
strong downstream learners. 2024.
Seokil Ham, Hee-Seon Kim, Sangmin Woo, and Changick Kim. Parameter efficient mamba tuning via rrojectortaareteer diagonal-centric linear transformation. arXiv preprint 44X1122411.15224, 2024.
11

---

Published as a conference paper at ICLR 2026
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-cffiiint fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 2024.
Ali Hatamizadeh and Jan Kautz. Mambavision: hybrid mamba-transformer vision backbone. arXiv preprint arXiv:2407.0000 2024.
Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivit--aware visual parameter-efficient fine-tuning In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11825- 1835, 2023.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Sorg-Kiikaatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. arXiv preprint drXiv:2110.0436,, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference machine learning, pp. 2790 2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Bjom Ommer. Zigma: Zigzag mamba diffusion model. arXiv e-prints, pp. arXiv- 2403, 2024.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ec-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy La-Wei Lee. Llm-adapters: An adapter family for parameter-efffcient fine-tuning of large language models. arXiv preprint arXiv:2304.0 03993313333,,,.. 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint (tth::/0440899)) 2021.
Kunchang Li, Xinhao Li, Yi Wang, Yinan Hc, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. In European Conference on Computer Vision, pp. 237-255. Springer, 2024.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101 1000000, 2021.
Shih-Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-
Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning. 2024a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning be comparable fine-tuning universally across scales and tasks. arXiv preprint 2 2021.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 5:208-215, 2024b.
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. Advances in neural information processing systems, 37:103031
2024c.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 1099):1559-1671, 1997.
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint (0ti:22/06004489) 2021.
Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. arXiv preprint arXiv:2206. /3947, 2022.
12

---

Published as a conference paper at ICLR 2026
Toshihiro Ota. Decision mamba: Reinforcement learning via sequence modeling with selective state spaces. arXiv preprint arXiv:2403.19925, 2024.
Jonas Pfeiffer Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00227, 2020.
Maciej Pióro, Kamil Ciehiera, Krystian Król, Jan Ludziejewski, Michal Krutul, Jakub Krajewski, Szymon Antoniak, Piotr Mitos, Marek Cygan, and Sebastian Jaszczur. Moe-mamba: Efficient selective state space models with mixture of experts. arXiv preprint arXiv:2401.04066 2024.
Changsheng Quan and Xiaofei Li. Multichannel long-term streaming neural speech enhancement for static and moving speakers. IEEE Signal Processing Letters, 2024.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784))607-617,, 2019.
Jimmy TH Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04990 2022.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193-24200 2021.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint (tl::2011.04000 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357, PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. Mambabyte: Token-free selective state space model. arXiv preprint arXiv:2401. 13660, 2024.
Zihan Wang, Fanheng Kong, Shi Feng, Ming Wang, Xiaocui Yang, Han Zhao Daling Wang, and Yifei Zhang. Is mamba effective for time series forecasting? Neurocomputing, 119112,,78, 2025.
Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable fine-tuning. arXiv preprint arXiv: 05687 2021.
Masakazu Yoshimura, Teruaki Hayashi, and Yota Maeda. Mambapeft: Exploring parameter-effiiient fine-tuning for mamba. arXiv preprint arXiv:2411.03855 2024.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-modes arXiv preprint arXiv:2106.10/99, 2021.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto. Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint 1etp::1900.087,,.. 2019.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine- tuning. arXiv preprint arXiv:2303. 10512, 2023.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model, 2024.
13

---

Published as a conference paper at ICLR 2026
A
ORIGINAL MAMBA ARCHITECTURE
SSM
Mamba Block
Causal Conv ld
Linear (s prof
Selective Sean
Linear
Embedding
RMS Norm
Linear (in pro)
Linear
Figure 6: Overall architecture of the original Mamba block.
To facilitate understanding of our LoRA placement strategy in conjunction with the LIM neuron, we present the overall architecture of the original Mamba block in Figure The SSM core, enclosed in the purple dotted box, follows the computations described in Equations equation I, equation 2, equa- tion 3, where Z represents the gating values. We highlight the four linear layers used for LoRA insertion in blue: in_ in.proj, j.proj, át dt.proj, and out out_proj. The ablation results reported in Table correspond the impact of selectively applying LoRA to each of these layers combination with the LIM neuron.
B LEAKY SITEGRATE-AND-EEEE NEURON
The Leaky Integrate-and-FF (LIF) neuron (Burkitt, 2006) has emerged as an important component for energy-efficient computation in SNNs (Maass, 1997; Roy al., 2019). The LIF neuron processes temporal information through its membrane potential dynamics as follows:
uli + 11' ruli]' + w'r(uli)-)),
(12)
r(uli)) {1 o otherwise
[ if ufil' > Vaa
(13)
where, uli]' is the membrane potential in I-th layer timestep i, (0, 1] is the leaky factor for membrane potential leakage, W is the weight of 1-th layer, and f(-) is the LIF function with threshold ViA. When the membrane u(i]' is higher than Vaa the LIF function generates spike and the membrane potential is reset to O. The LIF neuron exhibits two core characteristics: (1) Leaky Integration and (2) Reset Mechanism.
The term ru[i]' models the leaky accumulation of membrane potential in biological neurons' mem- branes. This leaky mechanism enables forgetting old, unnecessary information while focusing on new inputs, mimicking how biological neurons naturally attenuate stale signals. The reset function r(-) is analogous to the fire-and-reset mechanism in biological neurons, where the neuron discharges its potential upon crossing threshold Via. In biological neurons, this generates spike.
C THE DETAILS OF LIM ALGORITHM
We present the details of the LIM process, which involves efficient processing of sequential data through temporal chunking and cross-layer membrane potential transfer.
14

---

Published as a conference paper at ICLR 2026
Algorithm Leaky Integrate Membrane
Input: Input sequence tensor Xinput E BxL number of chunks T, leak factor т, threshold
Vin linear projection f()), previous membrane potential Uprev BB
Output: Output membrane Unmput E BXLDD averaged membrane Upev E <RBxLL/TI&D
I
L/TJ
# Chunk size using floor function # Calculate remainder
f- L - IXT
if # 0 then
X <
X inpul
Trim sequence to be divisible by T
L 6 --r
end if
Uprev None then Meureent <
else
Ucurrent
Uprev
# Transfer membrane state from previous layer
end
for i 0 to T do
start idx t ixx
end idx (i+ x
X, f X :, sstattrtidd
add_idx.
# Extract current chunk # Leaky integrate dynamics # Apply Reset with threshold Vth
eeurent T. Ueurrent + f (X;)
Mcureent Reset Ucurrent)
Hist Uuurentt
end for
Mourpat + concatenate( Ulist: dim = 1)
Combine all chunks
Uprev mean Ulist: dim 1)
# Average the membrane for transfer to next layer
r + 0 then
Duutput -pad(usupu)) (guupput)
# Pad to original length
end
return Moutput: Uprev
D THEORETICAL DERIVATION
D.1 DECOMPOSITION OF LIM SIGNAL
In our LIM mechanism, the membrane potential evolves according the dynamics defined in equa- tion
u, r(ru((111 + WX,)
(14)
where r(.) is the reset function from equation 9.
We decompose the membrane potential u, into component and fluctuation component:
U ū; + Ex, where [[[,,] 0
(15)
The mean component u, Elu,] captures the expected membrane potential, which represents the temporal integration of input history. The fluctuation component E; = u, ū, represents deviations from this expected behavior.
The reset function r(-) enforces an upper bound by resetting values exceeding Vaa zero, ensuring u, S Via element-wise. For the lower bound, we that the membrane potential can take negative values when ru,-1 WX, < 0. However, since E (0. 1] and inputs bounded (|[x,ll Xmma). the recursive relation |nl S r/u,-1/ + [X] yields, by geometric series convergence:
Boundedness of membrane potential:
W Xmxx
-M, S u, S VnA for all t, where M,
(16)
1
Defining Umax = max(Via, Ma), the fluctuations are bounded:
||ex|| || S 2
(17)
15

---

Published as a conference paper at ICLR 2026
The decomposition u, u, + E, enables to analyze how membrane dynamics affect the loss function: the mean component i2 provides stable temporal context integration, while the bounded fluctuation component e, introduces controlled variability that, show in the following sections, acts as an adaptive regularization mechanism.
D.2 MULTIPLICATIVE GATING ANALYSIS
In our architecture, the membrane potential influences the output through multiplicative gating:
yr = Y: @ g(u))
(18)
where yt is the output from the selective sean, g the gating function applied element wise (typically SiLU activation), and @ denotes element wise multiplication.
Using the decomposition of u, from the previous section:
g(u:) = g ü + ex)
(19)
We apply element-wise Taylor expansion of g around ú,; g (u,) + g (u) o E, + (m,)
(w,) o (((e,11))
(20)
where g'(ū,) and g denote element-wise derivatives, and ef represents element-wise squaring. Therefore, the gated output becomes:
yr = g(ú) y Oolguu 2 о ((, o (((E,1)))
(21)
D.3 ANALYSIS OF EXPECTED LOSS
We analyze the expected loss adopting local perturbation perspective: for given input neighbor- hood, we treat the selective sean output y: as approximately constant and examine how variations in the membrane potential u, affect the loss. This justified by the architectural separation between the SSM branch and the gate branch, which process information through distinct temporal dynamics.
We expand around yī Yı g (.),, which represents the output when the membrane potential is its local mean value. Note that this is not exactly Ely;] due to the nonlinearity of but serves as natural reference point that isolates the effects of membrane fluctuations.
Let dy; y:- yf. From the multiplicative gating analysis:
gy, y: (ū,))) o + y (u Oz O(
(22)
The Taylor expansion of the loss function around ył gives:
((ý.) L(y? 5y:)
(23)
-(y?) VL(Y)) Ty: Sy. b + 0(115y:11)
(24)
2
Taking expectation over the membrane fluctuations under the local analysis framework, where y: is treated as locally fixed:
B((():) [(yi) T
(25)
Since y, and ü, locally constant, and Eje,] 0 by definition of the fluctualctttiion decomposi- tion:
E(5y,] yr g' (..) Ele,] y: 2 (m.) E(e?)
(26)
(u.) o E
(27)
16

---

Published as a conference paper at ICLR 2026
For the second-order term, we assume independent fluctuations across dimensions with B(e,, &yof, which is reasonable given the independent reset behavior in each dimension. The dominant contribution comes from the first-order term in dy::
Bjöy, V (yy))yy = Bl(y, o (',) ) (yy()))) o ('u) Σ (veag (Tr,) (V
(28)
(29)
We define the regularization term to include both the bias correction and the quadratic regularization:
R(y,, m.) = V ( @ 0 + 2
1 E (wnsd (Tn,)) (",
(30)
Therefore:
B(c(ý:) = Cly, o ((.)) + R(y,, u.) + ((1e,,1))
(31)
D.4 BOUNDEDNESS THE LIM Loss
To establish the theoretical foundation for our regularization analysis, we first demonstrate the boundedness E[C():) by showing that both components of the gated output yı + g(u)) remain bounded.
Boundedness of y: (Mamba output): The HiPPO theory (Gu et al., 2020) ensures that Mamba's hidden state remains bounded through its initialization strategy. Specifically, HiPPO initializes the state matrix with all negative real parts of eigenvalues, making the discretized system matrix A = exp(AA) contractive with spectral radius less than 1. This contractivity property ensures that the recurrent dynamics (equation 3) cannot grow unboundedly. Even with Mamba's input- dependent discretization parameter A,, the constraint A, > preserves the contractivity property since exp( A) maintains the same spectral properties as exp(A) when A, 0. Therefore, for bounded inputs || ||xe|] I Xmaxx we have hh] < for some constant H, which ensures ||y: S M
for some bound M.
Boundedness of g(u:): The gating function is bounded by the design of our LIM mechanism. Since the membrane potential is bounded within -M, u, Vi, established in Section D.1, and the activation function g(-) (typically SiLU) is continuous on this compact interval, we have (u,)) S Gmax for some constant Gmax.
Final boundedness:
Since both components are bounded, the gated output satisfies J @ g(u))l < M Gmax- For Lipschitz continuous loss function L with Lipschitz constant Lip- we obtain:
E ((())) S C(0) + Lüp M Gmax
(32)
Therefore, the expected loss under our LIM mechanism is bounded, providing the theoretical founda- tion for our subsequent regularization analysis.
D.5 BOUNDING THE REGUL TERM
We now establish the bound on the regularization term R(y,, ūz) as stated in Theorem 1.
From the analysis in the previous section, the dominant contribution to R(ye, úz) comes from the quadratic term:
R(y.. ū.) ~ 2 "(nnd
(33)
Bounding the diagonal Hessian elements by the maximum eigenvalue Amaxi
[V'((y))lss & Amax (V2
(34)
Therefore:
& Amaax (V= Σ (Vragg (mr
(35)
17

---

Published as a conference paper at ICLR 2026
The term y in Theorem captures the combined effect of model outputs, gate sensitivity, and fluctuation variance. We can bound:
L(urs (vung S 9c?
(36)
where y is a dimensionless constant and e² B[[lec|17] [|] E.
Let Amax m&Xt Amax (yy()))).
Therefore:
R(y,, 0e) S Amaxx 78² = Amax e²
(37)
This establishes the bound stated in Theorem
D.6 Loss LANDSCAPE VISUALIZATION
To qualitatively verify theorized regularization effect, Figure presents comparison of loss landscapes between Mamba fine-tuned with standard LoRA and our Memba approach. The land- scapes are visualized for models fine-tuned the Diabetic Retinopathy dataset from the VTAB-IK benchmark. In both plots, the red star indicates the original pre-trained model, while the cyan dot marks the converged solution after fine-tuning, The visualization reveals two key findings that align with our theoretical analysis: (1) Memba achieves a lower overall loss (minimum error 0.2459 vs. 0.281 I), demonstrating its better optimization capability; and (2) Memba produces a smoother, more convex loss landscape with more gradual contour transitions. The smoother geometry suggests LIM helps avoid sharp minima, potentially contributing to its improved generalization performance and stability during fine-tuning.
Original model
Original model
0.3961
Min 0.2811
Min erron: 2429
Direction (8)
50
Direction 2 (B)
0.3614
25
25
0.3441
0.3094
-25
0.2920
-50
Original SSM
Memba
0.2747
-75
-75
0.2573
-100
50
100
-50
50
100
Direction
Direction 1 (a)
Figure 7: Loss landscapes of original SSM and Memba.
18

---

Published as a conference paper at ICLR 2026
E HYPERPARAMETER DETAILS
E LANGUAGE TASK
For commonsense reasoning evaluation, we adopt the fine-tuning setup from (Hu al., 2023; Yoshimura et al., 2024). We train on a combined commonsense reasoning dataset containing approxi- mately 170K examples for epochs with batch size of 16. Our LoRA configuration applies rank-64 adapters to the projection components (in.prej and out-proj, denoted as Win and W Wor)., and rank-32 adapters to the gate components (W and o For the LIM neuron, we configure chunks with a threshold of 1.0 and a leaky factor of 0.5. We employ model-specific learning rates: 1e-3 for Memba- 130M, 5e-4 for Memba-370M and Memba-790Ms and le-4 for Memba-1 1.4B. All models are trained using linear learning rate scheduler with 100 warmup steps.
E.2 VISION TASK
the VTAB-1K benchmark, which comprises 19 diverse datasets with 1,000 training examples each. Our training protocol consists of 100 epochs with a cosine learning rate scheduler, epochs of warmup, learning rate of Ie-3, weight decay of le-4, and batch size of 32, For parameter-efficient fine-tuning, we employ architecture--peeific LoRA configurations. The Vim (Zhu al., 2024) architecture uses ranks of 32 for Wi.. 96 for W Wou.s and 16 for gate-related linear layers (Waate and W The VMamba (Liu al., 2024c) architecture employs ranks of 64 for both Win and V out, with rank 16 for gate components. For both architectures, we configure the LIM neuron with 4 temporal chunks, threshold of 1.0, and a leaky factor of 0.5.
For vision evaluation,
E.3 S4 IMPLEMENTATION
We train the Memba-Sa architecture from scratch on the Long Range Arena (LRA) benehmarks (Tay et al., 2020), which include ListOps, Text, Retrieval, Image, and Pathfinder tasks, in Section G.4. These benchmarks are specifically designed to evaluate model performance on long-range sequence processing. Table 5 provides the experimental configuration for our S4 implementation. For the gate modification component, employ 32-rank for gate components (W and Wg ), with chunks. Additionally, we set the LIM threshold to 1.0 and use leaky factor 0.5.
Table 5: Experimental settings on training S4 architecture. #layers: the number of layers. d_features: dimension of feature maps. state: dimension of state. Ir: learning rate. adopt the S5 (Smith et al., 2022) training setup for other hyperparameters.
ListOps
Text
Retrieval
Image
Pathfinder
#layers
d f-feature
128
128
128
512
256
d_state
64
4
4
64
64
batch size
50
16
32
50
64
Ir
0.003
0.003
0.002
0.005
0.005
epoch
40
30
250
200
19

---

Published as a conference paper at ICLR 2026
F INDIVIDUAL TASK ACCURACY FOR VTAB-IK I BENCHMARK
We provide the details of individual task performance on VTAB-IK benchmark as shown Table 6. The Table 3 shows the averaged values of each category (Natural, Specialzed, and Structured).
Table 6: Per-task performances of VTAB-IK benchmark in Table 3. We separate three tables according to baseline architecture: ViT-S, Vim-S, and Vanilla- VMamba-S. Arg. and Overall Arg. represent average accuracy of each category and the average accuracy of overall tasks. Bold and underlined values represent the best and second-best performance respectively.
Speciallzdd
CIA 00
C3000101
102
Comelyyn
Rolte45
Cl Count
Ble
Ohenol Arg.
DTD
P05
DM0
20pe
d8ye
.0g
Method
Scracta
24.
73.6
71.6
LaKN"
5533
55.3
1/.33
9022
90.2
15.
5976
562
30.3
72.
8222
56.
31.
4976
29.4
351
50.6
18.7
Adaptiomer
56.0
55.01
90.5
90.5
83.2
79.3
5410
Ailatter-
36.6
89.2
66.3
91.1
90.
68.2
400
94.7
848
94.2
82.9
72.4
83.6
76.8
99.7
48.4
50.5
87.8
31.9
32.4
33.1
38.8
69.0
(a) VYT-S architecture pre-trained with ImageNet-1K
Naternl
Specinlaed
Stretored
Oveeall Moos
CI
Cley Coun
ANDRB.BL
SVHN
Sua307
Clar-bic
AMORB
Mode
Meihad
Scrrch
11
12.1
5.0
16.3
1.6
8.5
62.3
51.6
48.8
53.3
22.5
40.99
18.6
11.3
18.3
26.2
25.4
Pull
$1.6
K3.0
69.5
11.1
45.0
88.4
149
19.4
66.5
99.1
63.3
57.1
68.7
48.88
60.4
18.8
55.0
0.3
11.3
31.0
24.1
34.4
50
LoRAicisted
51.7
$4.9
39.2
15.5
83.0
33.4
393
64.7
71.5
91.2
71.5
70.0
77.5
42.9
31.0
34.9
73.0
54.9
41.0
19.7
26.2
43.8
38.6
LoRAva pent
90.4
$1.9
65.7
33.1
633
86.0
54.5
40.5
71.1
LoRhiia
63.4
90.3
68.2
91.0
91.0
18.6
43.
76.
94.7
95.0
83.8
72.9
94.1
54.0
61.3
48.6
90.0
13.5
52.6
31.5
39.5
60.2
71.3
61.3
00.6
68.1
96.5
903
18.4
432
1N-2
85.6
98.
81.5
12.
84.3
82.4
60.
48.2
81.9
11.8
93.4
114
96.5
59.7
700
LaRA
91.
88.0
436
78-
810
84.
62.9
48.6
33.1
60.
71.5
35.
51.6
221
63.1
91.4
63.3
91.4
913
88.4
43.
76.9
35.
96.5
84.
15.3
853
86.
63.2
50.3
82.
12.8
32.4
35.4
40.1
61.
23.1
Mensball (ooer.gggg)
63.2
21.1
68.
213
M14
19.
77.1
873
M15
785
857
M8
02.
833
15.1
53.5
36.7
803
63.7
71.4
(b) Vim-S architecture pre- trained with ImageNet-IK.
Specialied
CIPAIC- 141
Cabua.01
Canntyo
Cle COULE
WOORS-EL Boo
Ovonall Mfwan
SWHM
Mrom
DMMLA
Mrno
67.3
20.4
92.5
M7.
92.4
44.5
37.8
W6-7
95.6
85.3
76.7
86.0
K5
64.1
S6.1
W5.8
86
54.5
15.1
67.4
23.
LoRAVou pro))
62.3
89.8
20.2
92.0
92.5
91.7
43.7
77.4
85.2
95.2
85.5
76,3
86.1
10.1
62.5
54.7
83.7
88.3
56.
37.4
41.9
84.2
23.7
proj)
22.
91.8
53.7
33.2
600
Moshal
91.
90.6
63.5
64.9
Moskelia poo...gg)
61.3
57.8
68.4
933
922
928
445
77.1
87.9
95.0
85.1
767
86.
38.1
639
56.6
83.7
90.4
54.1
35.
403
63.9
23.5
(c) Vanilla-MMamba- VMamba-S architecture pre-trained with ImageNet-1KG
G ADDITIONAL ABLATION STUDY
In this section, we conduct comprehensive ablation studies to provide deeper insights into Memba's design choices and performance characteristics. We systematically examine five key aspects: (1) the impact of chunk count LIM neuron performance, (2) accuracy comparisons under co-parrmettr conditions, (3) parameter allocation within the LIM neuron, (4) performance comparison with traditional recurrent gating mechanisms, and (5) computational overhead analysis through inference time measurements.
G.1 NUMBER OF CHUNKS
The number of chunks in the LIM neuron determines the computational iteration of membrane dynamics. Table 7 presents accuracy comparisons across different numbers of chunks for both language and vision tasks. For language tasks, we employ the Memba-1301 architecture on commonsense reasoning benchmarks. For vision tasks, utilize the Vim-S architecture evaluated on Camelyon, EuroSAT, Resisc45, and Retinopathy datasets from the VTAB- 1K benchmark. Across both task domains, LIM with 4 chunks achieves optimal performance while maintaining reasonable
20

---

Published as a conference paper at ICLR 2026
inference time. The reported inference time corresponds to LIM neuron processing only. Notably, inference time does not scale proportionally with the number of chunks, as increasing the chunk count results in smaller sequence lengths per chunk.
Table 7: Accuracy comparison across different numbers of chunks. #chunk represents the number of chunks in the LIM neuron. For language and vision tasks, we use Memba- 130M and Vim-$ architectures, respectively. Inference times measured during vision task evaluation.
#chunk
Language (%)
Vision (%)
Inference time (ms)
2
43.9
85.60
1.5
4
44.3
85.66
1.7
6
44.1
85,43
1.8
43.7
85,36
2.0
G.2 CCO-PARAMETER ACCURACY COMPARISON
To verify the effectiveness of Memba under fair comparison conditions, evaluate accuracy with the same number of learnable parameters as previous works, particularly MambaPeft (Yoshimura al., 2024), shown in Table 8. To match the parameter count, apply LoRA only the output projection (W oon)) and reduce the rank of gate components (Waale and W 16. This configuration matches the parameter count of MambaPeft's "LoRA (in proj)" setting, and our Memba consistently achieves better performance across all architecture sizes.
Table 8: Performance comparison between Memba and MambaPeft (Yoshimura al., 2024) monsense reasoning datasets. Reported values are accuracy percentages (%). Bold and underlined entries indicate the best and second-best performance, respectively. HS and WG refer to the Hel- laSwag and WinoGrande datasets.
Model
Method
#Params(*)
BoolQ
PIQA
SIQA
HS
WG
ARC-e
ARC-c
OBQA
Avg.
Full
100
56.1
65.3
38.7
35.3
$2.0
46.4
25.7
32.8
43.8
Additional
57.8
41.3
42.7
Mamba 130M
Affix-tunigg
64.64
59.7
64.3
38.2
35.2
51.9
42.9
24.0
29.0
43.2
LoRA (roj)
2.23
53.5
62.9
38.2
33.8
53.1
46.4
23.7
30.8
42,8
LoRA, (X)
2.67
61.7
64.0
39.5
34.3
43.5
25,3
29.4
43.7
Memba
2.23
56.3
64.5
39.1
34.4
$1.6
48.0
24.0
29.4
43.4
Full
100
58.1
69.9
41.9
45.7
53.8
52.7
29.7
33.4
48.2
Additicnal-scan
0.47
61.9
69.3
41.2
45.3
54.9
49.5
28.4
31.4
47.7
Mamba 370M
Affix-tuning
68.88
61.2
68.4
39.6
46.2
55.4
48.2
28.2
30.6
47.2
LoRA (in (in.proj)
2.07
55.4
68.6
41.0
44.7
54.1
$2.4
28.3
33.4
47.2
LoRA, (X)
2.67
60.8
68.8
44.7
56.2
50.4
27.4
32.2
47.8
Memba
2.07
56.2
69.2
42.1
44.8
55.7
56.0
28.3
33.4
48.2
Full
100
62.0
72.1
54.0
57.7
312
35.2
31.6
Additional-scan
0.33
63.0
71.9
41.9
54.2
57.1
54.9
30.0
32.6
50.7
Mamba 790M LoRA (in.proj)
Affix-tuning
69.99
61.0
72.5
54.6
30.4
1.47
61.7
71.9
44.0
50.8
56.7
56.3
30.5
33.8
50.7
LoRA, (X)
1.75
59.9
72.2
44.2
52.8
58.0
53.7
30.8
34.8
50.8
Membu
1.47
58.8
72.2
44.1
54.7
58.3
61.5
31.1
34.0
51.8
100
61.4
73.3
56.9
59.7
34.0
53.0
Additional-scan
0.26
63.0
73.5
42.8
57.5
60.5
60.9
32.4
37.4
53.5
Mumba 1.4B
LoRA (in-proj)
1.13
62.6
73.6
55.6
58.3
31.7
35.6
52.6
LORA, (X)
1.36
63.1
73.5
42.7
57.7
61.6
60.4
32.9
37.4
53.7
Memba
1.13
64.1
73.9
43.4
58.3
60.8
65.1
32.3
37.6
54.5
G.3 PARAMETER ALLOCATION ANALYSIS
To provide deeper insights into Memba's parameter efficiency, Table 9 presents a detailed breakdown of parameter allocation across different model sizes. Unlike traditional PEFT methods that distribute parameters across various projection layers, Memba strategically allocates all trainable parameters to the gate projections gWgale and Wgale) within the LIM neuron. For instance, the Kaamba790M configuration, Memba allocates 4.72M parameters input gate projections and 7.08M parameters
21

---

Published as a conference paper at ICLR 2026
to output gate projections, totaling 11.80M parameters, identical to the baseline LoRA (in-proj) method. This targeted allocation enables Memba to enhance temporal processing capabilities while maintaining parameter efficiency. Notably, Memba consistently outperforms baseline methods across all model sizes: achieving 1.1% improvement over LoRA p(X) for Mamba-790M (51.8% vs 50.8%) and 0.8% improvement for Mamba-1. 1.4B (54.5% 53.7%), demonstrating that strategic parameter placement in the gating mechanism is more effective than conventional projection-based adaptations.
Table 9: Parameter allocation and performance comparison for Memba. The "#Params" column represents the total number of trainable parameters, "Was WWellte shows the parameters dedicated to the gate modules the LIM neuron, and "Others" represents all remaining parameters excluding the gate modules.
Model
Method
# Params
W
W
Others
Avg. Acc. (%)
LoRA (in proj)
2.95M
42.8
Mambu-1300
LoRA p(X)
3.54M
43.7
Memba
2.95M
LISM
1.77M
43.4
LoRA (in proj)
7.87M
47.2
Mambu-370M
LoRA p(X)
9.44M
47.8
Membu
7.87M
3.15M
4.72M
48.2
LoRA (in (in.proj)
11.00M
50.7
Mamba-7900 790M
LoRA L(X)-p(X)
14.16M
50.8
Memba
11.80M
4.72M
7.08M
51.8
LoRA (in.proj)
15.73M
52.6
Mamba- 1,4B
LoRA.p(X)
18.87M
53.7
Memba 15.73M
6.29M
9.44M
54,5
G.4 COMPARISON WITH TRADITIONAL RECURRENT GATING MECHANISMS
Our LIM neuron is designed to enhance Mamba's temporal adaptation capabilities through recur- rently evolving membrane potentials. To evaluate this approach against established alternatives, compare LIM with traditional recurrent architectures by replacing LIM with LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Chung et al., 2014) in the gate path.
Table 10 presents a comprehensive comparison across two distinct scenarios: vision adaptation tasks using the Vim-S architecture VTAB-1k, and sequence modeling tasks the Long Range Arena (LRA) benchmark (Tay et al., 2020) using S4 (Gu et al., 2021a) trained from scratch. Our LIM neuron consistently outperforms both LSTM and GRU across domains while offering two significant advantages: (1) zero learnable parameters and (2) lower inference latency. The superior performance of LIM stems from its temporal processing driven by membrane dynamics, whereas LSTM and GRU rely on parameter-hean heavy gating units that increase computational cost with trivial improvement to temporal modeling capability.
Table 10: Comparison of LIM with other recurrent units in terms of performance, parameter count, and latency. We report average accuracy (%) on two benchmarks: VTAB (using the Vim-S architecture) and Long Range Arena (using the S4 architecture). # Params denotes the number of learnable parameters per unit given a hidden dimension H, excluding the input and output projection layers (W gule and W ) which are applied consistently across all methods. Latency refers to inference time per batch measured the $4 architecture using an A100 GPU.
Vim-S
S4
Gate
# Params
Latency (s)
VTAB
ListOps
Text
Retrieval
Image
Pathfinder
Avg.
LSTM
8/12 + Al1
71.59
62.10
88.00
91.37
89.60
96.75
85,57
0.189
GRU
6H +3H
71.68
61.80
87.97
91.70
89.02
96.81
85.46
0.188
LIM (ours)
72.33
62.05
89.60
91.52
89.88
97.12
86,04
0.150
G.5 INFERENCE TIME ANALYSIS
The proposed Memba incorporates the LIM neuron, which introduces inevitable recurrent computa- tion for membrane accumulation and reset mechanisms. To quantify this computational overhead
22

---

Published as a conference paper at ICLR 2026
compared to previous PEFT methods on Mamba, provide inference time comparison based on Table 4(a). Table demonstrates how the LIM module affects both accuracy and inference time. Since membranc transfer has negligible impact inference time, exclude its influence from analysis. We conduct experiments on language and vision tasks using the Memba-1300 and Vim-S architecture, with inference times measured on single-batch evaluation.
Memba, which integrates LIM, LoRA, and membrane transfer, achieves higher performance on both language and vision tasks. However, this comes with 8.8% inference time overhead compared to Mamba with LoRA. This overhead stems from the iterative operations required for membrane accu- mulation the LIM neuron. Nevertheless, the 8.8% increase represents modest computational cost, and our chunked sequence processing approach in the LIM neuron effectively balances performance gains with inference efficiency.
Table 11: Inference time analysis with different Memba components. The table shows the impact of each component accuracy and computational overhead. For language and vision tasks, we use Memba- 130M and Vim-S architectures, respectively. Inference times are measured during vision task evaluation on single batch.
LIM
LoRA
Membrane Transfer
Language (%)
Vision (Si)
Inference Time (8)
43.8
85.3
0.517
44.3
85.7
0.563
H COMPREHENSIVE MEMORY AND LATENCY ANALYSIS
To provide a complete understanding of the computational requirements of Memba, we present comprehensive GPU memory and inference latency comparisons across baseline methods, model sizes, and task domains. For language tasks, we measure GPU peak memory during fine-tuning and per-sample inference latency on the ARC-Easy benchmark, shown in Table 12.
Table 12: GPU memory and inference latency comparison on language tasks (ARC-Easy benchmark).
Latency represents per-sample inference time.
Model
Method
Learnable Param
GPU Peak Memory (GB)
Lateney
SL. LoRA
1.45
2.62
0.015
Additional-scan
2.17
0.014
Mamba-130M
Affx-tuning
0.17
2.49
0.013
LoRA (in+out)
3.53
2.51
0.014
Memba
3.95
2.82
0.020
SLL LoRA
2.30
5.00
0.027
Additional-scun
0.47
4.15
0.023
Mamba-370M
Affi--tnning
0.16
4.50
0.021
LoRA (in+out)
2.28
5.00
0,024
Memba
3.67
5.80
0.044
LoRA
3.10
8.20
0.028
Additional-scam
0.33
6.56
0,023
Mamba-790M
Aflxxtuning
0.11
6.97
0.021
LoRA (in+out)
2.32
8.03
0.024
Memba
9.21
0.045
SLL LoRA
4.64
12.49
0.028
Additional-scan
0.26
9.65
0.024
Mamba- 1.4B
Affix-tuning
0.09
9.71
0.021
LoRA (in+out)
1.80
0.025
Memba
0.045
Memba incurs approximately 12-14% additional GPU memory overhead compared to LoRA (in+out), which is required for storing membrane potentials across layers to enable cross-layer membrane transfer. We observe that inference latency appears to be largely independent of hidden dimension size. Mamba-370M, 790M, and 1.4B exhibit similar latency despite having different hidden dimensions, as they share the same number of layers. In contrast, Mamba- 130M shows noticeably lower latency due its shallower architecture with fewer layers. Regarding the latency overhead introduced by
23

---

Published as a conference paper at ICLR 2026
recurrent operations in the LIM neuron, discussed in the Limitation section (Section 6), this latency can be substantially reduced through CUDA kernel optimization.
Table 13: GPU memory and inference latency comparison on vision tasks (Caltech 101 dataset in VTAB-1k benchmark).
Model
Method
Learnable Param (%)
GPU Peak Memory (GB) Latency (s)
LoRA (in+out)
13.98
7.68
0.517
Vim-S
Memba
0.563
For vision tasks in Table 13, we observe similar patterns on the VTAB- Ik benchmark using the Vim-S architecture. The memory overhead approximately doubled compared to language tasks because Vim-S uses bidirectional SSM with two gate paths, requiring storage of membrane potentials for two LIM neurons. The inference latency increase remains modest. These resource requirements represent reasonable trade-off given the substantial performance improvements demonstrated in Tables and 3.
VISUALIZATION OF TEMPORAL PROCESSING
(a) Input
(b) Saliency Map
(c) Membrane distribution
0.02
0.10
-0.14
0.07
-0.00
-010
-0.1
100
350
200
250
Seauence Index
0.07
0.02
-010
-0.14
30
100
350
200
230
0.67
+0.00
-0.06
-2.14
-C.18
50
200
1.50
200
250
Sequ
Figure 8: Membrane-driven temporal processing in Memba. Given input image (a), we present saliency maps (b) and associated membrane distributions (c) throughout the sequence processing.
To understand Memba's temporal processing, we analyze membrane dynamics on the Pathfinder task. Figure 8 shows the relationship between visual attention and membrane activity during sequence processing. Task-critical path segments, highlighted in purple and pink, trigger pronounced mem- brane responses, demonstrating how the LIM neuron selectively amplifies relevant visual elements. Additionally, the membrane baseline declines across temporal chunks, indicating adaptive forgetting as new information accumulates.
24


Reviews:
[{"review_id": "nMWo2zprrh", "reviewer": "Reviewer_wqXu", "summary": "This manuscript proposes Memba, a membrane-driven Parameter-Efficient Fine-Tuning (PEFT) approach tailored for Mamba models, addressing the limitation that traditional Transformer-focused PEFT methods fail to account for Mamba’s unique temporal processing dynamics. Memba integrates Leaky Integrate Membrane neurons with strategic Low-Rank Adaptations (LoRA) on input/output projections and cross-layer membrane transfer, enhancing selective information retention without modifying Mamba’s core state-space components. Extensive experiments on commonsense reasoning and VTAB-1k demonstrate that Memba outperforms existing PEFT methods  across various model sizes while maintaining parameter efficiency, with theoretical analysis confirming LIM’s role in stable temporal integration and adaptive regularization.", "questions": "Will this method or idea be applied to other linear models, like MoR or Gated networks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The LIM neuron brings biologically inspired temporal processing into Mamba’s gating, filling the gap left by its simpler gates and helping the model learn what to remember or forget over time.\n2. The paper backs this up with clear analysis (like loss decomposition and bounded regularization), explaining how LIM adapts over time and smooths the loss landscape.\n3. The writing and figure are both well illustrated.", "weaknesses": "1. The performance depends on tuning things like the number of chunks (T), the leak factor (τ), and the threshold (Vth), so LIM likely need task-specific tuning.\n2. The theoretical analysis shows that the LIM neuron acts as a regularizer to smooth the loss landscape. However, the paper fails to explain the causal link between this optimization-level effect and the claimed practical benefits of enhanced temporal modeling and selective attention.\n3. The evaluation's focus on classification-style benchmarks, instead of long-sequence generative tasks, fails to fully validate the method's claimed improvements to Mamba's core temporal modeling capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PMjr7B6DDl", "reviewer": "Reviewer_yM88", "summary": "This paper introduces Memba, a parameter-efficient fine-tuning (PEFT) strategy desginated for Mamba, which is under well expolored. Memba incorporates Leaky Integrate Membrane (LIM) gating value adapters in SSM. LIM mainly features the chunked LIM neurons desgin and layerwise membrane potential propagation. The authors present empirical results across language and vision tasks, and compelling ablation and analysis experiments to demonstrate the effectiveness of the LIM adapters.", "questions": "1. From Line 198-206, there are duplicate description of the three components, which could be shortened or simpified.\n\n2. In the area of vision tasks, can LIM and the original SSM part take different scan orders of the same image? Does this setting help or impair the models?\n\n3. The output values of LIM in the figures are negative, are these values collected before or after SiLU gate?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. PEFT for Mamba is a quite novel topic and the temporal-wise gating LIM seems to be a good design for finetuning the gate values in SSM.\n\n2. The experiments are wide-ranging—spanning both language and vision tasks. The analysis results are also abundant and convincing. In addition to experiments, the theoretical analysis of loss boundaries is also interesting, enhancing the insight of the design, with the lower loss.\n\n3. The proposed LIM is efficient, with minimal additional parameters compared to LoRA adapters, enabling its wide potential of applications on various SSM models.", "weaknesses": "1. How is the proposed by-pass of SSM in low rank adapters training related to the biological term, Membrane? The paper does not include any references or tutorials about the membrane mechanism or other background knowledge about it, therefore it is somehow confusing why the proposed mechanism resembles a membrane.\n\n2. The average membrane potentional values in the figures show steps of decrease across the temporal chunks, and the authors attribute this phenomenon into a \"forget\" manner of LIM. However, as some earliear papers have pointed out, the temporal modeling behavior of SSM should be like a local filter to keep the most recent tokens and forget the much earlier ones. Figure 4. and Figure 8. demonstrate the replicated pattern of spikes across chunk 1 to chunk 4, indicating that the \"forget\" effect is rather weak. Even though other chunks do not have a same activated area like chunk 1, the LIM still outputs a high response gate value to emphasize this area, which is not fully reasonable. \n\n3. Although the novelty of this paper is pretty decent, Mamba-like models still fall behind the conventional attention based Transformers or linear attention architectures, especially large scale models. I suspect a little about the contribution of the proposed method on larger models and more challenging tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ft9Tm3uJwb", "reviewer": "Reviewer_H6sZ", "summary": "The paper proposes Memba, a new parameter-efficient fine-tuning (PEFT) method for Mamba. It introduces a PEFT module inspired by Leaky Integrate Membrane (LIM) neurons and adds cross-layer membrane transfer. The combination of LIM and LoRA outperforms existing PEFT methods for Mamba in both vision and language domains.", "questions": "- Please read the major weakness.  \n- In Fig. 4, why is the downward trend in membrane potential good?\n- Although the authors say the speed of Memba can be imporved by CUDA implimentation, is it really posible? if the number of chunk is small, I think parallel scan like Mamba can not improve the latency.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The use of LIM neurons for temporal gating of the output of SSM is innovative and outperforms existing methods. The motivation of the cross-layer membrane potential transfer is also reasonable.\n- Extensive experiments on both language (commonsense reasoning) and vision (VTAB-1k) tasks prove that Memba outperforms existing PEFT methods, including LoRA and other recent SSM-specific PEFTs.", "weaknesses": "#### Major Weakness\n- From Table 7, it seems the computational cost in terms of speed is more than the increase in parameter counts. Hence, the speed of Memba seems slower than other PEFT methods. How about memory consumption? In my opinion, small memory consumption is more important than inference speed, since it enables finetuning of large models with limited GPUs.  \n- I understand that LIM processes data recurrently in fixed-size chunks. Since the number of image tokens is predetermined, that part seems straightforward. However, how is language handled? In particular, during inference, which I assume is performed in an auto-regressive manner, does the number of tokens within a chunk change as each new word is generated? If so, does that mean caching can't be used, and all tokens must be reprocessed at every step?\n\n\n#### Minor Weakness\n- In Fig. 1, it is difficult to understand what the authors want to tell with the two saliency maps.  \n- Typo:  \n    - L028) INTODUCTION -> INTRODUCTION  \n    - L214) LIM mechanis -> LIM mechanism  \n    - L485) seperate -> separate  \n    - L742 & L744) Equation equation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zdpci6kWcg", "reviewer": "Reviewer_VB1b", "summary": "This paper introduces Memba, which combines LIM with LoRA and cross-layer membrane transfer, aiming to improve parameter-efficient adaptation. The proposed Memba demonstrates superior performance over existing PEFT methods on language (commonsense reasoning) and vision (VTAB-1k) tasks, emphasizing parameter efficiency and temporal modeling capabilities.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The membrane-driven PEFT method is conceptually novel, as it creatively integrates LIF modules into SSM fine-tuning, addressing a gap in temporal adaptation for Mamba models. \n\nThe chunked processing and cross-layer transfer efficiently handle long sequences without adding learnable parameters.", "weaknesses": "This work is actually a generic PEFT method, not exclusive to Mamba, as it involves adding LIM to LoRA's hidden layers. Therefore, I would like to see experimental results on Transformers.\n\nI find the absence of comparisons with advanced PEFT methods for Transformers (e.g., recent adapters or prompt-based techniques) or other SSM variants (e.g., Mamba-Adapter) reduces the claim of general superiority.\n\nWhile Memba improves performance, the 8.8% inference overhead (Table 11) is non-trivial for real-time applications, and the paper underdiscusses trade-offs, such as energy efficiency or scalability on edge devices. Notably, spike networks (SNNs) are primarily beneficial for their low energy consumption, but the motivation for using LIM in this work does not seem to be energy efficiency. Therefore, I am puzzled as to why the authors chose to use LIM instead of other RNN-style modules, which would seem to be a more straightforward choice.\n\nThe LIM neuron is not entirely similar to the LIF used in SNNs, as it discards binary outputs. In this regard, this module actually shows similarities to traditional RNN gates, but the ablation (Table 10) only briefly compares them. A more rigorous analysis of novelty versus established gating mechanisms is needed.\n\n\n\nThe vision tasks using VTAB-1k fail to demonstrate the performance superiority, as previous work has shown that different PEFT methods perform comparably on this dataset [1]. Therefore, semantic segmentation and object detection tasks are imperative.\n\nGPU memory and latency comparisons among different methods are missing.    \n\nRefs:    \n[1] Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition. CVPR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)