Mamba State-Space Models Are Lyapunov-Stable Learners
Authors: John Timothy Halloran
Manbir S Gulati
Paul F Roysdon
Venue: Accepted by TMLR ()

Abstract: Mamba state-space models (SSMs) have recently outperformed state-of-the-art (SOTA) Transformer large language models (LLMs) in various tasks and been widely adapted.  However, a major concern for stable learning in recurrent-based deep models (such as SSMs) is the sensitivity of their recurrent dynamics. Despite widespread adaptation, the sensitivity of Mamba’s recurrent dynamics under common fine-tuning methods–e.g., mixed-precision fine-tuning (MPFT) and parameter-efficient fine-tuning (PEFT)–remains unexplored. Empirically,
we show that Mamba LLMs are extremely stable to changes introduced by combinations of MPFT and PEFT, in stark contrast to Transformer LLMs, which we demonstrate may drastically diverge from their respective full-precision counterparts under different
combinations of MPFT and PEFT (despite the near-ubiquitous adaptation of these fine-tuning frameworks for attention-based models). The demonstrated robustness of Mamba LLMs are due to their recurrent dynamics, which we prove are guaranteed to be stable using
dynamical systems theory (in particular, Lyapunov stability). We conclude by using MPFT and PEFT to novelly study Mamba LLMs’ in-context learning (ICL) abilities on natural language tasks, thus supplementing other recent work.

Published in Transactions on Machine Learning Research (08/2025)
Mamba State-Space Models Are Lyapunov-Stable Learners
John T. Halloran
halloranjid@ecrdos.com
Leidos
Manbir Gulati Leidos
Paul Roysdon
Leídos
Reviewed on OpenReview: https: // openreview. form? id=wzsY0ys3dl
Abstract
Mamba state-space models (SSMs) have recently outperformed state-of-the-arl (SOTA) Transformer large language models (LLMs) in various tasks and been widely adapted. However, major concern for stable learning in recurrent-based deep models (such SSMs) is the sensitivy of their recurrent dynamics. Despite widespread adaptation, the sensitivity of Mamba's recurrent dynamics under common fine-tuning methods e.g., mixed-precision fine- tuning (MPFT) and parameter-efficient fine-tuning (PEFT) remains unexplored. Empirically, we show that Mamba LLMs are extremely stable to changes introduced by combinations of MPFT and PEFT, in stark contrast to Transformer LLMs, which we demonstrate may drastically diverge from their respective full-precision counterparts under different combinations of MPFT and PEFT (despite the near-ubiquitous adaptation of these fine- tuning frameworks for attention- models). The demonstrated robustness of Mamba. LLMs are due to their recurrent dynamics, which we prove are guaranteed to be stable using dynamical systems theory (in particular, Lyapunov stability). We conclude by using MPFT and PEFT to novelly study Mamba LLMs' in-context learning (ICL) abilities on natural longuage tosks, thus supplementing other recent work.
1 Introduction
Mamba(Gu & Dao, 2024a) state-space models (SSMs) have been shown to greatly outperform comparable attention-based LLMs (Biderman et al., 2023) across large number of standard natural language bench- marks. Subsequently, pretrained Mamba models have been widely adapted across a large number of data modalities (Liu et al., 2024; Li & Chen, 2024; Quan & Li, 2024; Li et al., 2024), tasks (Xie et al., 2024; Wang et al., 2024a), and architectures (Anthony et al., 2024; Park et al., 2024; Lieber et al., 2024). However, despite such widespread adaptation and subsequent research threads (Dao & Gu, 2024; Park et al., 2024; Wang et al., 2024b), no cxisting work has sought to understand the stability of lcarning Mamba SSMs common LLM fine-tuning frameworks, such as mixed-precision fine-tuning (MPFT). For recurrent- models, such as Mamba SSMs, the sensitivity of recurrent dynamics to small changes (e.g., those introduced by mixed-precisson) is primary concern for stable learning (Bengio et al., 1994; Pascanu et al., 2013).
deep
To determine the stability of time-varying Mamba models, we leverage theory from dynamical systems. Under the framework of Lyapunov stability, we show that small input changes within the SSM layer of both Mamba and the more recent Mamba-2 (Dao Gu, 2024) models do not lead exponentially deviating outputs. Furthermore, we extend these theoretical results account for deviations in input and latent Mamba states produced by combinations of MPFT and PEFT.

---

Published in Transactions on Machine Learning Research (08/2025)
We empirically validate these theoretical results; for large number of randomly generated SSM layers, we show that manually adjusting initial latent and input states produces maximum deviations in the output states which exponentially decrease over discrete time. Furthermore, by expanding previous divergence performance metrics (Dettmers et al., 2022; Dettmers & Zettlemoyer, 2023; Dettmers et al., 2024) and evaluating combinations MPFT and PEFT, we show that fine-tuned Mamba LLMss do not substantially deviate performance compared to full- full-precision full fine-tuning- This demonstrated in stark contrast. to comparable Trausformer-baset LLMs, which we show may drastically differ from their full-precision full fine-tuning counterparts, exhibiting large deviation spikes. Over two widely used fine-tuning datasets and two widely used natural language benchmarks, attention-based LLMs Pythia (Biderman et al., 2023) and OpenELM (Mehta et al., 2024) exhibit 9 and 4 large deviation spikes, respectively, whereas Mamba LLMs exhibit zero. Thus, Mamba LLMs are significantly more stable to performance deviations under MPFT and PEFT, whereas Transformer LLMs are susceptible to large deviation spikes.
Lastly, we use stable MPFT and PEFT to explore the ICL capabilities of instruction tuned Mamba and Manba-2 models on natural language lasks, thus complementing recent. studies which evaluated Mamba ICL synthetic tasks (Park et al., 2024; Lee et al., 2024). While the ICL capabilities of pretrained Mamba models lag behind those of comparable Transformer
and Mamba-2 models only achieve 38% and 82%, respectively, of the performance improvements (relative to zero-shot) of Pythia mrodes-instruucton tuned Mamba and Mamba-2 models improve to 81 51.5% and 132% of the TCL improvements achieved by Pythia. Thus, while pretrained Mamba models' ICL capabilities lag behind comparable Transformers, we show that instruction-tunin using MPFT and PEFT greatly narrows this gap.
Our major contributions summarized as follows:
We derive bounds on the Lyapunov exponents of both Mamba and Mamba-2 models' SSM equations. Using these bounds, we theorctically show that small input changes within the MambaBlock do not lead to exponentially devialing outputs.
Building on the aforementioned bounds, we theoretically show that changes MambaBlock latent and input states stemming from MPFT and PEFT do not lead to exponentially deviating outputs.
We empirically demonstrate the above theoretical results:
Directly introducing changes to the inputs states of 100 random MambaBlock, we confirm that the resulting output states do not deviate from the unperturbed outputs over discrete time (the maximum deviation exponentially decrcases).
For MPFT and PEFT, we measure the performance deviation between FP32 full fine-tuned and MPFT/LLRA Mamba models and compare to attention-based models. Across two fine- tuning datasets, two widely used natural language benchmarks, several model sizes, and a large number of MPFT/PEFT configurations, we show that training Mamba LLMs is significantly more stable than comparable Transformer LLMs.
We expand performance divergence measures previously used to study MPFT and PEFT for Trans- former LLMs (Dettmers et al., 2022; Dettmers & Zettlemoyer, 2023; Dettmers et al., 2024). In addition to showing Mamba models are drastically more stable than comparable attention based LLMs, the expanded divergence metric also demonstrates that Transformer LLMs are susceptible to large deviation spikes.
Using stable MPFT and PEFT, we complement recent studies by examining the ICL capabilities of Mamba/Mamba-2 Mamba-2 models evaluated on natural language tasks. We show that instruction tuning allows SSMs to perform ICL competitively with comparable Transformer LLMs on natural language tasks.
Terminology. When describing a particular foundation model or result, we use the term "Mamba model" refer one of the original models relcased in Gu Dao (2024a) and "Mamba-2 model" refer models released in Dao & Gu (2024). Despite subtle architectural differences, these two SSMs share the same state-space equations and design scheme of storing the majority of SSM parameters in large memory buffer.
2

---

Published in Transactions on Machine Learning Research (08/2025)
We thus synonymously use the term MambaBlock to refer to the SSM layer of both Mamba and Mamba-2 models.
2 Background
Mixed-precision and parameter-efficien fine-tuning. In the current era of extremely large loundation models, both MPFT and PEFT have become ubiquitous tools for the rapid adaptation of Transformer-basel LLMs towards specific applications. PEFT using adapters (He et al., 2021) allows a large pretrained model to be efficiently adapted for a particular downstream task by freezing the full model and training only a small number of extra parameters. Arguably the most widely used such PEFT method is LoRA (Hu et al., 2021), which injects trainable low-rank matrices into Transformer layers to approximate weight updates.
To further decrease the computational demands necessary for LLM fine-tuning and inference, MPFT via mixed-precision (i.e., FP16 or BF16) (Kalamkar et al., 2019; Micikevicius et al., 2018) and quantized low- precision (Dettmers ct al., 2024) have proven effective strategies to reduce GPU memory and runtime requirements without deleterious effects on downstream performance (Dettmers et al., 2024; Wu el al., 2020). Additionally, mixed preseisinnsiionn approaches have paved the way for hardware-aware optimizations within the self-attention module (Dao et al., 2022), greatly mitigating the quadratic complexity of Transformer LLMs. Together, PEFT and MPFT have created rich ecosystem with which varying combinations of these approaches may be used to meet the computational constraints of given training system.
State-space Models. Structured state-space sequence (S4) models (Gu et al., 2022; Fu ct al., 2023) are SSMs which leverage linear time-invariant (LTI) systems to combine the computational advantages of Transformers- i.e. highly parallelizable training- and recurrent neural networks (RNNs)-i.e. subquadratic autoregressive inference using recurrency. Within the $4 layer, an input signal discretized and LTI parameters representing the input's latent dynamics are learned. Owing to the S4 block's latent dynamics being LTI, the S4 block's output may be thus compactly represented as single convolution between the input and an SSM convolution kernel (a matrix whose entries products of LTI learnable parameters resulting from unrolling the statc-space equations). However, despite hardware elficiency and deeepenndncc-modeelin improvements, LTI-based S4 models remained inferior to Transformers of comparable parameter-sizes for natural language tasks, even when augmenting S4 layers with attention-layers for hybrid architectures (Gu & Dao, 2024a).
Innovating on these previous S4 approaches, Mamba utilizes vorying parameters to model latent dynamics, thus broadening the ability to capture nuanced changes evolving discrete-time. Without LTI dynamics, however, the input-output representation via the SSM convolution kernel no longer applicable, thus voiding previous hardware-aware $4 optimizations (Fu et al., 2023). To enable hardware efficiency with timo-varying SSM parameters, (Gu & Dao, 2024a) thus introduced extensively customized CUDA kernels which implement highly parallelized prefix sums to compute recurrnt states. Subsequently, Dao & Gu (2024) considered the unrolled state-space equations and leveraged tensor contractions (i.c., cinsum notation (Rogozhnikov, 2022)) to efficiently calculate Mamba variables. The resulting Mamba-2 foundation models contained significantly larger latent-variable dimensions than the Mamba models of (Gu & Dao, 2024a), while maintaining efficiency on modern GPU accelerators.
In-context learning. ICL provides an adaptable alternative to fine-tuning. Rather than fine-tune the LLM directly, ICL augments a prompt with n relevant examples (called shots) preceding the query of interest. Given sufficiently large models and pretraining data (Brown et al., 2020; Wei et al., 2022), Transformer LLMs have proven adept at learning new concepts on the fly provided such few-shot prompting. However, it is worth noting that ICL inference time increases dramatically as the number of shots grows (due to self-attention'ss quadratic complexity) and PEFT (when possible) is known to produce more accurate downstream lcarning results (Brown et al., 2020; Liu et al., 2022).
3 Mamba state-space models
For latent-variable dimension d and maximum input sequence length T, the MambaBlock defines state-space parameters A, B,, Cr, A, Rdxd for te (1,...,
......... T). The matrix A, controls the discrete step-size. Given
3

---

Published in Transactions on Machine Learning Research (08/2025)
input sequence u Ur R the following linear mapping through latent states 2 2T IR is used to produce the output , Yr Rd.
32 A, + B
Y: C,a,,
(2)
where A, softplus(Linear( (Ai)) Raxd A, exp (,A)) and B, A- In practice, A, B,, Ca and A, diagonal matrices.
Due to the time-variance of Equations 1 and 2, previous SSM stability results longer applicable (discussed further in Appendix E... The Mamba foundation models were pretrained in full FP32 precision and, subsequently, official implementations have cautioned against training in reduced precision (Gu & Dao, 2024b; Huggingface, 2024). Thus, how sensitive the MambaBlock's recurrent dynamics are remains an open question. We next answer the latter using theory from dynamical systems.
3.1 Stable dynamics in the MambaBlock
For Mamba's discrete dynamical system in Equations 1 and 2, define
at F0(22-1,,
(3)
where I denotes the time-varying parameters described in Section 3. For input sequence ui...., and initial latent state voctor fo, we thus write
FT Fo(Fo Fe( u1)) F u1).
The rate of divergence between two scalar e-close inputs to discrete dynamical system bounded by the system's s maximal Lyapunov exponent Asaz (Mikhaeil et al., 2022). Given Auaz and two initial values (æo. un) and (xo &, u1 + E), the maximum deviation between these points grows as (Laffargue et al., 2013; Sayama, 2015):
max P (mo, un) F5" (mro t, + +)]l O(Ie| exp (Naasx)).
Thus, when Auaz O, nearby trajectories exponentially separate and, when Aaaz & 0, nearby trajectories ultimately converge to the same fixed point periodic cycles.
The maximal Lyapunov exponent is defined as
T
de:
Amax
lim T-
log
1=0
where Ill2 denotes the spectral norm for matrices. For an arbitrary MambaBlock, we prove the following:
Theorem 1. Let (#1-1, Uz) be the latent state and inpul at an arbilrary time I E (1,...,T) within a 1, +)) produce deviations which are exponentially non-inereasing over discrete-time. That is, max FF (#t-1. (2: + e, u. +s] O(s exp (NC)), for some scalar < O.
MambaBlock. Then small changes (xx-1 114+6, +
The proof of Theorem is available in Appendix A, where the maximal Lyapunov exponent for arbitrary MambaBlock is first proven to be non-ponn non-positive. The main result subsequently follows.
Thus, the latent states of Mamba and Mamba-2 models are stable under small input changes. However, variables ; are the primary outputs for such models, particularly for LLM applications. We next show that, given Theorem 1, Mamba and Mamba-2 output variables are also stable.
Theorem Assume (x:-1 + + ut +) produce deviations which are exponentially non-increasing over Siscreletime. Then small changes to the output yt. are also exponentially non-increasing over discrele time.
The proof of Theorem 2 is available in Appendix Thus, by Theorems and 2, the latent and output states of both Mamba and Mamba-2 models stable changes encountered during recurrency.
4

---

Published in Transactions on Machine Learning Research (08/2025)
3.1.1 Divergence bounds for automatic mixed- preed-preciioon and PEFT
For an arbitrary MambaBlock, let Ep...., By and U1...., Uy be the trajectory of latent and input states under full-precision fine-tuning. During a forward pass, automatic mixed-precisson (AMP) saves time and memory by computing forward activations in half- colff-reeiiion (FP16 or BF16). During backward pass, AMP computes gradients in hall-precision and up-casts full-precision prior to updating. MFPT thus introduces changes the imputs u....., ur (which are passed through Swish) and latent states Zo..... 2y through A, (which is passed through a softplus), up/downcasting of parameter updates, and the aforementioned changes to the input states.
Potentially in concert with MPFT, PEFT via LoRA induces changes by learning low-rank matrices A, B,, C2
a. = (A, + A.)a,1 + (B, B,)u,
y. = (C, C,)a,
Let vb...., dh and u'...., u'y be the trajectory of latent and input states under MPFT and/or PEFT. We thus have the following two thcorems:
Theorem 3. Let Cº max lu, - u/l, i.e., the marimum, positive scalar-differene between the tth MPFT/PEFT and full fine-tuning input state. Similarly, let ct max l s'i and e* max U U (where we define un = 0). We thus have
max 1Fg (wo, ui) - (2/6 u')l Ole" exp (T(),,
for some scalor & O. Thus, differences introduced by AMP and/or PEFT for Mamba models do not exponentially compound over discrete-time.
Thcorem 4. Let y, be the trajectory of output states produced by zo...., I' and u/.................. u'y under MPFT and/or oor PEFT. Then we thuus have
max YT - y'rl E O( exp (T()),
for some scalar < < O. Thus, small changes to the output yr resulting from MPFT and/or PEFT are exponentially non-increasing over discrete time.
The proof of Theorem 3 is available in Appendix C. Theorem 4 may be derived by directly applying the proof tochnique Thcorem 2 to the results of Theorem 3. Thus, the latent and output states of both Mamba and Mamba-2 models are stable to changes encountered during AMP and MFPT.
4 Experiments
4.1
Non-divergent MambaBlock dynamics
We explore the implications of Theorem 2 for arbitrary MambaBlocks, For n random trials and various e, we randomly initialize the inputs (i.e., 4ı for I = 1,..., TT) and parameters (i.e., A, B., Ct, A,) of a NambaBlock with T 2048 and latent-variable dimension d = 64.
For each trial and E, we first calculate the MambaBlock outputs y1....,yr (where y: = FF uy)). We then perturb the initial states (wo u, +) (x/) un{) and calculate the subsequent perturbed model outputs vi...., Y' (where y' (2 u/) (x each time point, i.c., max ly+ v/l = max (m). un) (2 + E, + e)]. Figure shows both the mean maximum deviation per time averaged across trials, as well as the standard deviation, for e = 0.1.
u/ + +)). The maximum deviation is then calculated for
As we can see, the maximum divergence starts out proportional to the change in initial conditions, but quickly decays and converges the original output trajectory. Thus, changes to MambaBlocks inputs do not compound over discrete time. Additional experiments for (0.01, 0.001) are available in Appendix D and demonstrate the same dynamics.
5

---

Published in Transactions on Machine Learning Research (08/2025)
E = 0.1, 100 random trials
# Abcoluze Deviation)
0.10
0.08
0.01
0.02
0.00
0
500
1000
1500
2000
a Absolute Deviation)
0.10
0.08
0.06
0.04
0.02
0.00
500
2000
1500
2000
Figure 1: For 100 random MambaBlocks, initial latent and input states (æp, Up) are deviated by 8. The resulting maximum deviation in output states is computed and shown to exponentially decrease over discrete time. denotes the maximum absolute deviation at discrete time & averaged over all trials, while denotes the respective standard deviation at each discrete time.
4.2 Non-divergent Mamba MPFT and PEFT
We explore the implications of Theorem 4 for combinations of MPFT and PEFT on pretrained Mamba models. We focus on utilizing LoRA (Hu et al., 2021), which is arguably the most widely used PEFT framework for LLMs. Using the Alpaca dataset (Taori et al., 2023), Mamba 160M, 410M, and 790M models are fine-tuned for three epochs with a maximum sequence length of 512. We denote the targeting of all linear layers (ALL) for LoRA as ALL LoRA, the targeting of subset of linear layers (SLL) for LoRA as SLL LoRA (i.c., A,, B,, and C, parameters), and no adapters as Full (i.e., full fine-tuning).
single Nvidia A10G GPU (24 GB total memory). To further limit extraneous nunerical effects, the same batch size is used for all FP32, FP16, and BF16 experiments for a given model size. While this leads to hardware underutilization (i.e., non-saturated GPU memory for mixed- -peccippeeisioo and LoRA experiments), this is necessary to guarantce no divergence is due to differences in parameter update schedules. For comparison, two Transformer-basedd LLM families of similar parameter counts are fine-tuned using the same experimental setup: Pythia (sizes 160M, 410M, and 1B) and OpenELM (Mehta et al., 2024) (sizes 270M and 450M). The training recipe for all models was adapted from Tunstall al. (2023), with the StaWW_tordh optimizer and a cosine annealing schedule. Further experimental details are available in Appendix F.
Each fine-tuning
occurred
All models were evaluated using the LM evaluation harness from Eleuther AI (Gao et al., 2023). Model performance is measured as percent accuracy using the MMLU (Hendrycks et al., 2020) and Wino- grande (Sakaguchi et al., 2021) datasets. The difference model performance reported as the mean divergence (i.c., absolute difference) between the original full-precision and respective mixed-procision model, averaged over {0, 3, 5)-shot percent. accuracy. Thus, divergence greater than one denotes an average difference greater than one entire percentage of accuracy. We thus refer to divergence greater than unity as large deviation spike.
Mean divergence results for all models are displayed in Figure ((a) and Figure 2(b) for MMLU and Winogrande, respectively. Across mixed-precisionn and adapter settings, Mamba displays smaller divergences than both Pythia and OpenELM models. E.g., for Winogrande evaluations, Alpaca finc-tuning with Mamba models is an average 2.9 and 2.4 times smaller in mean divergence than Pythia and OpenELM models, respectively. For MMLU evaluations, Alpaca fine-tuning with Mamba models is average 2.6
6

---

Published in Transactions on Machine Learning Research (08/2025)
Mean Per-shot FP32 Divergence
1.0
LORA
0.5
0.0
Mamba
Pythia 160M
OpenELM 270M
Mamba 370M
Pythia 410M
OpenELM 450M
Mamba 790M
Pythia 1B
130M
(a) Mean full-precision (FP32) divergence in MMLU performance.
Mean Per-shot FP32 Divergence
3-
FP16 Full
BF16 Full
FP16 ALL LORA
2
BF16 ALL LORA
FP16 SLL LORA
BF16 SLL LORA
1-
Mamba 130M
Pythia 160M
OpenELM 270M
Mamba 370M
Pythia 410M
OpenELM 450M
Mamba 790M
Pythia 1B
Mean full-precision (FP32) divergence Winogrande performance.
Figure 2: Mean full- fill-precision (FP32) divergence in performance for Mamba, Pythia, and OpenELM models. Models are fine-tuned over the Alpaca dataset using different combinations of MPFT and PEFT. Full fine-tuning (i.e., no PEFT adapters) is denoted as Full.
times smaller in mean divergence than both Pythia and OpenELM models. Importantly, Mamba models do not exhibit large deviation spikes after fine-tuning, in contrast to both Pythia and OpenELM models.
We note that the results in Figure 2 display the fine-tuning of 72 LLMs and 576 few-shot evaluations. In Appendix H, these experiments expanded to include the LIMA fine-tuning dataset (Zhou et al., 2024) well as the standard deviations for all mean deviation results. Across all experiments, Mamba models are significantly more stable for MPFT/PER compared to Transformer-basel LLMss. F.g., for MMLU evaluations, LIMA fine-tuning with Mamba models is an average 7 and 3.3 times smaller in mean divergence than Pythia and OpenELM models, respcctively. For Winogrande evaluations, LIMA fine-tuning with Mamba models an average 4.1 and 3.3 times smaller in mean divergence than Pythia and OpenELM models, respectively. Across the two fine-tuning datasets, two benchmarks, and six evaluated MPFT /PETT combinations, Pythia and OpenELM produce 9 and 4 large deviation spikes, respectively, while Mamba produces zero
4.3 Hyperparameter tuning robustness Mamba models
We empirically demonstrate that Mamba models are robust to the choice of PEFT hyperparemters. Using the training recipe of Tunstall al. (2023) as basis, we conduct an extensive hyperparameter search across the learning rate, LoRA dimension, and number of warmup steps. From the pardesian-product of these three parameters, 150 hyperparameter configurations were sampled and used to fine-ture Mamba 370M over the Openhermes dataset (Teknium, 2024), which consists of 242,000 supervised samples. For comparison, Pythia 410M is similarly fine-tuned using the same set of 150 hyperparameter configurations.
The MMLU 5-shot performance for each of the 150 Mamba and Pythia fine-tuned models is displayed Figure 3. Pythia 410M is capable of higher performance than Mamba 370M, where the average accuracy for the former and the latter are 26.5% and 24.8%, respectively. However, Mamba 370M much more robust. to the choice of hyperparameters, with difference of 1.5% between the minimum (23.3%) and maximum
7

---

Published in Transactions on Machine Learning Research (08/2025)
MML 5-Shet Acc.
2000
LoRA Dim
16
32
64
128
256
0.26
1000
0.24
0.24
LoRA Dim
0
10-7
10-6
10-5
10
10-3
10-7
10
10-3
10-4
10-3
Learning Rate
Learning Rate
(a) Mamba 370M
(b) Pythia 410M
Figure Fine-tuning hyperparameter search for OpenHermes. Fach point is different hyperparameter configuration. SLL LoRA was used for both models. The z-axis is the learning rate, the y-axis is resulting MMLU 5-shot performance, bubble size is the LoRA dimension, and the color is the number of warmnu steps E {0, 1k, 2k3.
(24.8%). In contrast, Pythia 410M fine-tuned models display large performance difference of 4.7% between the minimum (22.9%) and maximum (27.6%).
4.4 Stable instruction tuning improves Mamba ICL for natural language tasks
With both stable MPFT and PEFT, we determine the impact of instruction tuning Mamba and Mamba-2 models on ICL for natural language tasks. We instruction fine-tuned Mamba and Mamba-2 pretrained models using ALL LORA, the OpenHermes dataset, and the training recipe described in Section 4 (with BF16). Zero and few-shot performance are evaluated using the same five standard natural language benchmarks used in Section 4.3. ICL performance is reported as the average improvement percentage of {1, 3, 5)-shot versus 0-shot (AIPSS). For comparison, Pythia pretrained models are instruction inn-tuned using the same training recipe and ALL LoRA (i.e., all Pythia linear layers are adapted).
o-shoot ICL Improvement
2.88
2.78
2.88
1448
4
370M
370M
3
410M
-2
-2
0-Shot
1-Shot
3-Shot
5-Shot
0-Shot
1-Shot
3-Shot
5-Shot
0-Shot
1-Shot
3-Shot
5-Shot
(a) Mamba Pretrained
(b) Mamba-2 Pretrained
(c) Pythia Pretrained
96 Improvement
2.80
6
2.78
2,80
5
1.38
5
1.48
790M
4
780M
10
3
3
2
130M
2
130M
2
160M
1
-2
0-Shot 1-Shot
3-Shot
5-Shot
0-Shot
1-Shot
3-Shot
5-Shot
0-Shot
1-Shot
3-Shot
5-Shot
(d) Mamba ALL LoRA
(c) Mamba-2 ALL LoRA
(f) Pythia ALL LoRA
Figure 4: Instruction tuning narrows the ICL gap between Mamba and Pythia, and creates gap from Pythia to Mamba-2 models. ALL LoRA models were instruction tuned on the OpenHermes (Teknium, 2024) dataset for one epoch. Performance is reported as the average improvement percentage {1, 3, 5)-shot versus 0-shot over five standard natural language benchmarks.
Figure displays AIPSS for pretrained and instruction fino-tuned Mamba and Pythia models. As previously noted, pretrained Mamba models do not display similar ICL, ability comparable Pythia models the
8

---

Published in Transactions on Machine Learning Research (08/2025)
evaluated standard NLP benchmarks. In particular, Mamba 2. 2.8B, the largest pretrained Mamba model, displays inconsistent zero-shot improvements as the number of shots increase. While pretrained Mamba-2 models display significantly better ICL ability than Mamba models, Mamba-2 models smaller than 780 million parameters struggle.
However, after instruction funing, all Mamba models larger than Mamba 130M consistently improve in ICL performance as the number of shots increase. Similarly, the majority of Mamba-2 models larger than Mamba-2 130M greatly improve in ICL performance. Thus, while pretrained Mamba and Mamba-2 models are only capable of 38% and 82% (respectively) of the AIPSS compared to similar pretrained Pythia models, instruction tuned Mamba and Mamba-2 models are capable of 81.5% and 133% of the AIPSS relative to similarly fine-tuned Pythia models.
In addition to Pythia, we compare instruction tuned Mamba models' ICL abilities to other state-of-the Transformer LLMs of comparable sizes and pretraining token counts: OpenELM (Mchta. et al., 2024) (sizes 270M, 450M, and 1. 1B), TinyLlama 1.1B (Zhang et al., 2024), and OLMo 1. (Groeneveld et al., 2024). We additionally group models by parameter count, to understand ICL as emergent behavior of fine-tuned Mamba models. To further explore the emergence of this ability for Mamba SSMs, include recent Mamba-specifie PEFT methods (Yoshimura et al., 2025). Displayed in Appendix G, pretrained SSMs and Transformers of parameter counts 270 million and less display slight or detrimental ICL abilities (i.e., few-shot, performance is worse than zero-shot). For models of greater than 450 million parameters, the majority of SSMs and Translormers display positive ICL abilities, with the exception of Mamba 1.4B.
Instruction tuning greatly smooths TCT. performance across both parameter classes. Thus, while instruction tuned SSMs and Transformers of 160 million paramcters or fewer continue to display slight or detrimental ICL abilities, all models of 270 million parameters and greater show positive ICL abilities, with Mamba-2 7B closely matching (or exceeding) the ICL ability of the top performing Transformer LLM per-shot. Furthermore, in terms of an emergent ability (Wei et al., 2022), ICL emerges for instruction tuned Mamba and Mamba-2 SSMs of size 370 million and greater, while no clear trend presents itself for pretrained models.
5 Conclusions and Future Directions
Using dynamical systems theory, we've shown that the recurrent dynamics of Mamba SSMs are stable to small input perturbations. We've extensively confirmed this result, showing that Mamba training is significantly more robust to changes introduced during MPFT and PEFT than Transformer-baset LLMs across widely-used fine-tuning datasets and benchmarks. Furthermore, by including many more few-shot evaluations in our divergence metric than previous work, our stability studies uncovered that Transformer LLMs are susceptible to large deviation spikes during MPFT alone, as well as when MPFT is combined with PEFT. As MPFT and PEFT are two of the most common LLM fine-tuning frameworks, and as lower precision formats (e.g., NF4 and FP8) become more widely used for fine-tuning, we advocate for more extensive stability analysis (e.g., including several few-shot evaluations when calculating divergence) to expose large deviation spikes.
In order to mitigate large deviation spikes, we've shown that, unlike Transformer LLMs, Mamba models are extremely stable and do not exhibit this phenomena. Furthermore, in addition to stable MPFT and PEFT, we've shown that a core LLM reasoning ability (ICL) of Mamba models significantly improves with instruction tuning; i.e., we've ve shown that while pretrained Mamba models' ICL capabilities lag behind those of pretrained L.Mss on natural language tasks, instruction tuned Mamba models' rival (and surpass, in many Mamba-2 cases) the ICL capabilities of instruction tuned Transformers. Thus, Mamba models are extremely stable learners with the potential to display the ICL capabilities of attention- models once instruction tuned.
There are several avenues for future work. In particular, adapting Manba's CUDA kernels to support more aggressive low-precision PEFT methods (Dettmers et al., 2024) would further decrease the hardware needed to train Mamba models, while providing additional speodups and testing the limits of the derived stability results. Furthermore, our theoretical contributions open the door for several follow up studies. E.g., additional results building off Theorems 3 and may tackle generalization error bounds and privacy fine-tuning (by
9

---

Published in Transactions on Machine Learning Research (08/2025)
considering a language modeling framework and calculating upper bounds under distributions of E), and Mamba-specific decoding schemes (where new algorithms may exploit the derived deviation bounds ensure next-token predicitions lie within a deviation tolerance from the true optimal decoding).
6 Relations to Previous Work
MPFT divergence for LLMs. Previous work has studied the performance difference for attention- based LLMs under low-precision inference (Dettmers al., 2022; Dettmers & Zettlemoyer, 2023) and fine-tuning (Dettmers et al., 2024). However, these works only mcasured divergence using single-shot performance (i.e., zero-shot in (Dettmers el al., 2022; Dettmers & Zeitlemoyer, 2023), 5-shol for MMLU and zero-shot otherwise in (Dettmers et al., 2024)). In contrast, we report the difference in model performance as the mcan divergence between the full-precision and mixed-precision models, averagod over the accuracy of four different few-shot evaluations. Thus, in contrast to previous work, our inclusion of multiple shots: a) uncovers significant. divergence in widely used LLMs even at widely used mixed-precisiooss (FP16 and BF16), and (b) more rigorously tests mixed- prrccssion training's effect on one of LLMs' most important reasoning abilities (ICL). We are not aware of any previous works which have attempted to understand the performance impact of MPFT on Mamba models.
Lyapunov stability and RNNs/SSMs. Lyapunov exponents have previously been considered (Mikhaeil et al., 2022; Vogt et al., 2022) for classic RNN structures (e.g., vanilla RNNs, LSTMs, GRUs, PLRNNs, etc.), to determine when such models exhibit chaotic dynamics and the impact on the exploding/ phenomena'.
sxploding/vannishigg vanishinng gradient
For S4 (Gu et. al., 2022) SSMs, Goel et al. (2022) used Hurwitz matrices to characterize the numerical stability of linear time-invariant (LTI) $4 models. Similarly, Orvieto et al. (2023) used the LTI property to derive the cigenvaluc decomposition of S4-like models, providing stability conditions by bounding the corresponding eigenvalues (further discussed in Appendix E). However, such analysis is not applicable to Lime-varying models, such as Mamba, nor does it characterize the effects of sensitive dependence on initial conditions (e.g,, divergence of two E close inputs). To the best of our knowledge, no previous works have used Lyapunov exponents to explore the effects of mixxed-reciisoon on recurrent neural models or Mamba architectures.
In-context learning. Recent work has sought. to understand the in-context learning (TCL. capabilities of Mamba LLMs when trained from scratch for specific tasks (Park et al., 2024; Lee et al., 2024). Another line of recent work has sought to understand how hybrid Transformer models (Wang et al., 2024b). However, to the best of our knowledge, no existing works have either thcoretically explored the effects small input changes (c.g., due to mixod-procision)) have on Mamba's 's recurrent dynamics or empirically explored such effects rewmstream impact on fine-tuning.
models may be directly distilled from
As previously noted, recent works (Park et al., 2024; Lee et al., 2024) have studied Mamba's 's ability to perform ICL by training Mamba models for specific tasks. Such tasks include logistic regression, decision trees, and learning other simple function classes, following the work of Garg el al. (2022). We emphasize that, in this set up, relatively small Mamba models-22 33 million and 90 million parameters for Lee et al. (2024) and Park et al (2024), respectively-are trained from scratch for every evaluated task. Indeed, Park et al. (2024) notes that subsequent work is necessary to understand Mamba's ICL capabilitics for language modeling using standard natural language benchmarks, as well as for larger model sizes. Thus, our study of both the pretrained and instruction tuned ICL capabilities of Mamba/ Mamba-2 LLMs for natural language tasks are complimentary to previous works.
Failure Cases and Limitations
While we explored the use of LoRA for Mamba models, many other PEFT adapters exist (Liu et al., 2022; Li & Liang, 2021; Houlsby et al., 2019; Lester ct al., 2021). Furthermore, while mixed-precision using FP16 and
We note that this continues a long line of research exploring RNNs scnsitivity initial couditions and their subscquent ability to produce chaotic output (Ribeiro al., 2020; Laurent von Brecht, 2017; Bertschinger & Natschläger, 2004; Bertschinger et. al., 2004), although previous work did leverage Lyapunov exponents.
10

---

Published in Transactions on Machine Learning Research (08/2025)
BF16 were explored, lower- lower-precision methods exist (Dettmers al., 2024). Both are interesting directions for future work. Finally, our timing and memory usage experiments using Alpaca did not consider the largest. two Mamba models (1.4B and 2.8B) due to their exceeding A10G memory capacity for FP32 full fine-tuning.
Broader Impact Statement
The Mamba models considered are all LLMs, and thus have the same potential positive and negative societal impacts as other LLMs (e.g., hallucinations). Furthermore, fine-tuning is known to possibly erode existing LLM guardrails, and thus the described methods may be adapted for this fine-tuning use case (as is the case for all PEFT and MPFT methods). However, MPFT/PEFT shown to improve the quality of Mamba models for downstream applications, which may be adapted for all positive LLM applications in society (e.g., personal assistants, task automation, code completion, etc.). Finally, the described methods decrease the computational constraints required to train and inference Mamba SSMs, which has implications for green ML (e.g., decreased CO2 emissions, positive climate change impact, etc.).
Acknowledgments
We thank Leidos for funding this research through the Office of Technology. This manuscript has been approved for public release 24-LEIDOS-0515-7755
We thank (NeurlPS) Reviewer Uruo, (TMLR) Reviewer z2sE, and (TMLR) Reviewer SpRc for helpful comments and feedback, which spurred the authors to derive and add several additional theoretical results. This work dedicated to the memory of Bella Halloran; thank you for supporting this one final paper, you are missed.
References
Quentin Authony, Yury Tokpanov, Paolo Glorioso, and Beren Millidge. Blackmamba: Mixture of experts for stale-space models. arXiv preprinl arXiv:2402.01791, 2024.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions neural networks, 5(2):157-166, 1994.
Nils Bertschinger and Thomas Natschläger. Real-time computation at the edge of chaos in recurrent neural networks. Neural computation, 16(7):1413-1436, 2004.
Nils Bertschinger, Thomas Natschläger, and Robert Legenstein. At the edge of chaos: Real-time computations and self-organized criticality in recurrent neural networks. Advances in neural information processing systems, 17, 2004.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning (ICML), pp. 2397-2430. PMLR, 2023.
Tom Brown, Benjamín Mann, Nick Ryder, Mclanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neurol information processing systems, 18:1877-1901, 2020.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient. algorithms through structured state space duality. In Forty-first International Conference on Machine Learning (ICML), 2024.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-effciint exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.
Tim Dettmeers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference Machine Learning, pp. PMLR, 2023.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Newral Information Processing Systems, 35:30318 1033332, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, Advances in Neural Information Processing Systems, 36, 2024.
Danicl Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with stale space models. In International Conference on Learning Representations, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou, framework for few-shot language model evaluation, 2023. URL https: https://zenodo. org/reecordd/112566666
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What transformers learn in-context? case study of simple function classes. Advances in Neural Information Processing Systems (NewrIPS), 35: 30583-30598, 2022.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-space models. In International Conference on Machine Learning, pp. 7616 7633. PMLR, 2022.
Dirk Groeneveld, Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafford, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arxiv preprint arXiv:2402.0088 00838, 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024a. URL
Albert Gu and Tri Dao. Mamba Precision Guidance. httppp///ggtaao.co////tttteepepe
precision', 2024b. "Accessed: 2024-04-25'.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR), 2022.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Herg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-offiiient transfer learning. In International Conference on Learning Representations (ICLR), 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Paraeeter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790 2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 206666666660665 2021.
Huggingface. Mamba PEFT. 'ntops://nuggingfaae peft-finetuning' 2024. "Accessod: 2024-04-25'.
Dhiraj Kalamkar, Dheovatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjoe, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of 16 for deep learning training. arXio preprint arxiv: 905. 12322, 2019.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Tanguy Laffargue, Khanh-Dang Nguyen Thu Lam, Jorge Kurchan, and Julien Tailleur. Large deviations of Iyapunov exponents. Journal of Physics A: Mathemalical and Theoretical, 46(25):254002, 2013.
Thomas Laurent and James von Brecht. A recurrent neural network without chaos. In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2017.
Ivan Lee, Nan Jiang, and Taylor Berg- Kirgpiirkaa.rick. attention required for icl? exploring the relationship between model architecture and in-context learning ability. In The Twelfth International Conference on Learning Represenlations (TCI.R; 2024.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-effiiient prompt tuning. arXiv preprint arXie:2104.080/0 2021.
Kai Li and Guo Chen. Spmamba: State-space model is all you need in speech separation. arXiv preprinl 24t::2404.00033 2024.
Lincan Li, Hanchen Wang, Wenjie Zhang, and Adelle Coster. Stg-mamba: Spatial-temporal graph learning via selective state space model. arXiv preprint arXi::2408.12/18, 2024.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. ar preprint arXiv:2101.00190 2021.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-manlee language model. Xix preprint Xrx::22033 198887 2024.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel, Few-shot parameter- sarameter-effcienn fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint axXiv:2401.1066,, 2024.
Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al. Openelm: An efficient language model family with open-source training and inference framework. ar preprint arXiv:2404.14619, 2024.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations (ICLR), 2018
Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with rmns. Advances in Newral Information Processing Systems, 35:11297 11312, 2022.
Antonio Orvicto, Samuel Smith, Albert Gu, Anushan Fernando, Caglar Gulcchre, Razvan Pascanu, and Sohamn De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pp. 26670 26698. PMLR, 2023.
Jongho Park, Jacseung Park, Zheyang Xiong, Nayoung Lee, Jacwoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? comparative study on in-context learning tasks. International Conference Machine Learning (ICML), 2024.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. Proceedings of the 30th International Conference on International Conference on Machine Learning- Volume 28, pp. III 1310, 2013.
Changsheng Quan and Xiaofei Li. Multichannel long-term streaming neural speech enhancement for static and moving speakers. arXio preprint arXin:2403.07677 2024.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Antônio H Ribeiro, Koen Tiels, Luis A Aguirre, and Thomas Schön. Beyond exploding and vanishing gradients: analysing rmn training using attractors and smoothness. In International conference on artificial intelligence and statistics (AISTATS), pp. 2370 2380. PMLR, 2020.
Alex Rogozhnikov. Einops: Clear and reliable tensor manipulations with einstcin-like notation. In International Conference on Learning Represenlations, 2022.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd scheraa challenge at scale. Communications of the ACM, 64(9))99 106, 2021.
Hiroki Sayama. Introduction to the modeling and analysis of compler systems. Open SUNY Textbooks, 2015.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. /ttps://github.. com/ tatta--lab/stanfodd__pppp 2023.
Teknium. Openhermes. http:s//wanguugggreor/r//r/////p 2024-04-25'.
2024. Accessed:
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint orXiv:2810.16944 2023.
Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, and Guillaume Lajoie. On Iyapunov exponents for rans: Understanding information propagation using dynamical systems tools. Frontiers in Applied Mathemalics and Statislies, 8:818799, 2022.
Chloe Wang, Oleksii Tsepa, Jun Ma, and Bo Wang. Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. arXiv preprint arXhv:2402.00789 2024a.
Junxiong Wang, Daniele Paliouta, Avner May, Alexander M Rush, and Tri Dao. The mamba the Ilama: Distilling and accelerating hybrid models. In Workshop on Efficient Systems for Foundation Models I1@ ICML2024, 2024b.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXtv:204.00900,, 2020.
Jianhao Xie, Ruofan Liao, Ziang Zhang, Sida Yi, Yuesheng Zhu, and Guibo Luo. Promamba: Prompt-mamba for polyp segmentation. arXiv preprint ar Xiv: 13660, 2024.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-contexi learning implicit bayesian inference. In International Conference Learning Representations (ICLR), 2021.
Masakazu Yoshimura, Teruaki Hayashi, and Yota Maeda. Mambapeft: Exploring parameter-officient fine- tuning for mamba. In The Thirteenth International Conference on Learning Representations, 2025.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint ar Xiv:24401 02388, 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information. Processing Systems, 36, 2024.
14

---

Published in Transactions on Machine Learning Research (08/2025)
A Mamba stable dynamics proof
Recall the state-space parameters and equations for the MambaBlock; A, B,, C., A, E Rdxd for L e (I....,?] T [T]. Given an imput sequence uI,.... ur R2, the following linear mapping through latent states $1...., WT E
Rd is used to produce the output y Yr Rd:
2; A + B,u,
(4)
C,a,,
where A. softplus(Linear(A)) (A,) R30 dxd Ar exp (A,A), A (A-)B, and R is the of non-negative real numbers. In practico, A, Br, Ct and A diagonal matrices.
Furthermore, recall the following definitions:
Fo(0:-11 Uz)
where denotes the aforementioned time-varying parameters. For input sequence u,, Ur and initial latent. state value To, we thus write
TT Fo(Fo(... Fa u1))) F F
We first prove that, given two scalar e-close inputs MambaBlock their deviations do not grow exponentially the number of recurrences increases (Lemma 1). The main result in the paper is subsequently proved.
Lemma 1. For inpul (æo, un) to a MambaBlock, small changes (æo + E, + + produce deviations which are exponentially non-increasing over discrete-time That is, max [Fg (xo, u1) F (xo + E, E, + =) e ([e) exp (NC)), for some scalar { 0.
Proof. Firstly, note that within the MambaBlock, A is stored in log-space followed by negative exponen- tiation prior to use. Thus, A Rg dxd, where Roo is the set of non- non-positive real numbers.
Recall that for the maximum deviation, we have:
max FF (zo, u1) Fgy (2o + un + e)] e O(ee] exp (N (NDasx)).
where the maximal Lyapunov exponent Apax is defined as:
T
ox,
Amax lim
log
T-
oæ.-11
1-
and ||l| denotes the spectral norm for matrices.
Thus, to complete the proof, it suffices to show that Auax & 0. Recall that A and A. are diagonal. From Equation 4, we thus have
T
da:
Anax lim
log
T-100 T
0x
= lim
1
log
| r-0
exp A
A
T- T
lim
log
exp
A A
T
t-
15

---

Published in Transactions on Machine Learning Research (08/2025)
Let i be the dimension which corresponds to the output of the spectral norm, i... j A. ((.))))A((..)) We thus have
T
Aaa
lim
log
exp
A)
T
!=U
lim
log cxp
A i
T-x
t=0
T
1
A[i, lim T-
A [i
i is non-positive and limr- $ Ei= A.[i, > O, since A.(i, Rz0 Vt. Thus, Amax & O.
Theorem. Let (x uz) be the latent state and input at an arbitrary time t e [1, within a MambaBlock: Then small changes (#t-1 + e, us +e) produce deviations which are exponentially decreasing over discrele-tine time, i.e., max FF (zo. u1) F (xo +euu, u, E)] e O(E) exp (NC)), for some scalar Ç < 0.
Proof. Let r(t) be function that maps time values such that 7(t) [1,T o and 7(t) 1, 1, + 1) 2, .....,7(lL). T I. Then Br C+g), Ar()) define new MambaBlock with inpuls (,())....) (,,(())) and Applying Lemma 1 to this MambaBlock with (z-(0-1: (r(1))
subsequent recurrent states completes the proof.
B Mamba stable outputs proof
Theorem. Assume (xt-1 E, Ut + E) produce deviations which are exponentiolly non-increasing over discrete- time. Then small changes to the output yt also exponentially non-increasing over discrete time.
Recall that oT = FJ (2o, un). Furthermore, recall from Equations 1 and 2, C, where C, is diagonal.
Proof.
Let
Yr Yr = G% (No, ui) = Cr#r = Cr (vo, un).
Consider E-close inputs (#t-1, uz) and (xt-1 + E, u, + E), and their respective outputs y, and y/- Assume (x,-1 E, ut + E) produce deviations which are exponentially non-increasing over discrete-time. That is, max (7=(1111 uz) EN(x; (x,-1 + u, +e] O(Je] exp (NC)), for some scalar ( & 0.
We thus have
max ly - y'l max G.2 (I,-1, u.) GD (x:-1 + u, + +e)
max C F. (B--,, FN FN (# u. =)]
max FFN (z,-1. u,) F2 (z,-1 + u, +
where proportionality follows due the diagonality of Cy and the vector-absolute value. Thus,
max I (#t-1. u:) G. (x--1 e, un + +a)] O(le] exp (NC))
16

---

Published in Transactions on Machine Learning Research (08/2025)
C Mamba stable AMP and/or PEFT dynamics proof
For an arbitrary MambaBlock, let To....,, 2T and U1,..., ur be the trajectory of latent and input states under full-precision fino-tuning. During a forward pass, automatic mixed-procision (AMP) saves time and memory by computing lorward activations in half-precision (FP16 or BF16). During backward pass, AMP compules gradients in half-precision and up-casts full- cill-precision prior to updating. MFPT thus introduces changes the imputs u,...., ur (which are passed through Swish) and latent states Zu.-.., By through A, (which is passed through a softplus), up/ up/downcasting of parameter updates, and the changes to the aforementioned changes to the input states.
Potentially in concert with MPFT, PEFT via LoRA induces changes by learning low-rank matricos A, B,, Č
d. (A, + A,)a,-1 + (B, + B,)u,
8: = (C, + C,)=,
Let z6...., Th and u'...., p'y be the trajectory of latent and input states under MPFT and/or PEFT. We thus have the following
Theorem. Let et max juu u/], i.e., the mazimum, positive-scalar difference between the tth MPFT and full finettniig input state. Similarly, let ct max |x, x'1l and max UT=o G] (where we define un 0). We thuu have
max |F (zo, u1) - (2'22 u'il E Ole* exp (T()),
for some scalor S < O. Thus,, differences introduced by AMP and/or PEFT for Mamba models do nol exponentially compound over discrete-time.
Proof. Consider the e-vector definition of the Lyapunov exponent: given Amax e Rd, and two initial values (0o, un) and (2o + e, un + e), the maximum deviation between these points grows as:
max IF (mo, un) F (ma e, + +e] O(llell, exp (T (m)))))
where ||l| denotes the L2-norm for vectors.
Let f(-) be function which maps a vector the of its elements, e.g., for 2 E Rd, f(x)= (F1....,,.)) Thus, note that for a² max([a] Va f(e)), ||e||2 & a² vi and O(| exp (T>ax)) Olaa exp (T>aa)))
Let & = UTolu - u/ x. - x/} (where we define un = 0), and ex = max[lal Va E, Vee we thus have
max 1P (æo, u1) (20 + +,, ui + +e]] O([|e|2 exp (a))a))) O(E* exp (T>axx))
Furthermore, (u, - u/, x. - 2t} E E, and thus
max F (æo, u1) F3 (xo - wo, u - u/)l= OlE exp (T\nax))
Let (t) be function that maps time values such that r(t) (1,T-1) and r(() 1, [(++1) 2,...,7(1+7))
T - t. Then Braa) Craj Ar(0) define new MambaBlock with inputs ur((), .... u-(++2) and subsequent
recurrent states 2r(t)- [T(()) T(++T)) Vt f1...., T). {u - u x x1} S E, so that
max Ff-'t (I=(() 1, 1, Fd "1'' " O(c* exp ((T t)>nax))
Thus, we have
max 1F (#o, u1) - Fg (2/, n/)] E O(e" exp (T (TAna))),
where Anax & 0 by Lemma 1.
17

---

Published in Transactions on Machine Learning Research (08/2025)
D MambaBlock Maximum Deviation results
D.1
varying initial conditions
0.01, random trials
0.008
0.006
0.002
0,000
500
1000
1500
2000
(Max Absolute Devistion
0.030
0.006
0,004
0.000
500
1000
1500
2000
Figure 5: Perturbing the initial states of random MambaBlocks produces output state deviations which exponentially decrease over discrete time. μ denotes the maximum absolute deviation at discrete time averaged over all trials, while a denotes the respective standard deviation at each discrete time.
0,001, 100 random trials
0.001
0.000
2000
MMax Abvolute Devistion)
0.003
0,001
0,000
5000
2000
Figure 6: Perturbing the initial states of random MambaBlocks produces output state deviations which exponentially decrease over discrete time. j denotes the maximum absolute deviation at discrete time averaged over all trials, while a denotes the respective standard deviation at each discrete time.
18

---

Published in Transactions on Machine Learning Research (08/2025)
E
Previous SSM stability results
Previous SSM stability results (Gocl et al., 2022; Orvicto et al., 2023) consider the following linear time- invariant (LTI) equations used in S4 models,
8t A + Bu,
(5)
Y1 Ca
(6)
which can be unrolled to produce
22 Σ A A'y
(7)
By eigendecomposition,
A (PAP- PA'P-
In (Orvieto et al., 2023), the derived sufficient condition for stability is thus |os| & 1, for A diag([ag. a,,..., aul). Relatedly, in (Goel et al., 2022), stability is enforced by constraining A to be Hurwitz, in which case the real part of each diagonal entry is negative, i.e., Re(a;) 0, for A = diag [aa 01...., (dd)).
Both results critically rely on the LTI property of Equations 5 and 6 to derive Equation 7. However, Mamba's state-space equations are linear time-varying (LTV), and thus Equation does not hold. Thus, previous SSM stability results do not directly apply Mamba models.
F
Experimental Details
Table 1: Learning rate and LoRA dimension values
Mamba size
Mamba-2 size
Pythia size learning rate Mamba/Pythin Pythha LoRA
Mamba- LoRA r
130M
130M
160M
1.0e-5
370M
370M
410M
5.0e-5
16
16
790M
780M
1.0e-6
32
1.4B
1.3B
1.
5.0e-6
64
64
2.8B
2.7B
2.8B
5.0e-7
1 128
64
All model checkpoints were evaluated on all benchmarks and few-shot settings using the LM evaluation harness from Eleuther Al (Gao et al., 2023), version 0.4.2. Pythia and Mamba Huggingface checkpoints were used for all inference and fine-tur thne-tnnigg experiments, e.g., EleutherAA/pythhaatBe for the smallest. respective models. All fine-tuning experiments were run using package versions Transformert 4. 4.40.0 O. devo, Accelerate 0828.0, o, TRL 0.8.1, PyTorch 2.2. 22211i and PEFT 0. 0. All Mamba-2 models were run using mamba-ssm +2.2.2 using Huggingface checkpoints, e.g,, state-spaces/nambee for the smallest model.
and stateesspaces//ammaaaaa
For MPFT, Flash Attention 2.0 (Dao et al., 2022) flash_ attn 2.5.7 was used for Pythia models. For FP16 and BF16 inference results, Flash Attention 2.0 was used for both Pythia and OLMo models. For OLMo results, the 336B-token checkpoint was used by specifying seriisioosstts80000ttttoeee
All Alpaca and OpenHermes fine-tuning experiments used the following training recipe (adapted from (Tunstall et al., 2023)): AdamW torch optimizer, cosine annealing schedule, no gradient. accumulation, maximum norm 1.0 for gradient clipping, and no warmup steps. Training epochs used for all Alpaca and OpenHermes experiments were three and one, respoctively. For both Pythia and Mamba models, the learning rate and LoRA dimension were scaled to improve performance of smaller models (per-model values listed in Table 1).
19

---

Published in Transactions on Machine Learning Research (08/2025)
For SLL LoRA, targeted Mamba layers were (x_proj, embeddings, in in_proj, out_proj]; x_proj is large MambaBlock memory buffer which, when targeted by LoRA, adapts parameters A, B,, and C.. Pythia targeted SLL LORA layers were (dense, embed_in in, query Beery__ey__ee _ea_uuuue,,,,, dense_h, h_to___t_____h_____ 4h, dense 4h to, chosen to balance performance across model sizes.
All experiments were run using a single-GPU Nvidia A10G (24 GB total memory). For Pythia, Mamba, and Mamba-2 ALL LoRA experiments in Figure 4, all models followed the same training and PEFT recipes, save for Mamba-2 2. 7B which required LoRA dimension of 64 to fit in A10G memory.
The Alpaca dataset is freely available for download at https:/ alpaca under open-source license CC-by-NC 4.0. The OpenHermes dataset is freely available for download at. attps://huggingfaces httpa////etttt///aa///////aeeeee 5 under open-source license MIT, Apache 2.0, CC.
co/dataasetstttaa-laa
20

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: Pretrained model performance. Model checkpoints were evaluated on all benchmarks and few-shot settings using the LM evaluation harness from Eleuther Al (Gao et al., 2023). LAMBADA zero-shot is more effective for the model sizes considered (further discussed in (Xie et al., 2021; Brown et al., 2020)) and thus excluded from few-shot performance averages. Highlighted in bold is the lop-performing few-shot learner per benchmark and model grouping.
Model
N-shot
LAMBADA
LAMBADA
HellaSwag
PIQA
Arc-E
Arc-C
WinoGrande O-shot incr.
ppl
acc
Mean %
16.0
44.3
35.2
61.7
48.0
21.3
52.6
Mamba
35.1
61.6
130M
23.1
35.2
35.0
65.1
49.1
24.0
51.0
-0.4
21.4
36.2
34.9
61.9
19.1
23.7
50.0
-1.2
16.3
43.9
33.3
64.9
47.4
24.2
62.6
Mamba-2
20.6
37.9
31.9
61.1
16.9
23.1
51.3
-2.0
130M
26.5
34.9
34.6
64.4
48.6
24.8
51.7
0.2
38.2
32.7
30.2
61.8
43.4
23.6
5T.0
Pythia 160M
47.2
30.6
62.2
49.3
63.7
24.7
30.5
61.9
11.8
22.9
51.3
0.1
30.4
62.6
23.1
8.1
55.6
16.5
69.5
51.9
27.8
55.3
Mamba 370M
9.7
49.8
43.9
69.3
57.4
26.5
54.7
-0.3
69.5
11.4
48.6
46.2
69.4
58.3
28.0
55.9
1.5
8.0
46.9
70.3
26.7
55.4
Mamba-2
9.8
50.3
46.4
70.5
56.5
26.8
54.2
0.0
370M
11.3
48.5
46.6
70.2
59.0
26.9
54.3
1.0
12.5
46.6
46.7
70.3
58.5
28.2
53.3
1.5
T0.8
31.6
40.6
66.3
62.0
24.1
633.4
Pythia
12.3
17.1
10.5
68.0
53.8
25.6
52.1
410M
14.4
43,2
40.9
67.9
55.1
26.9
54.0
4.2
14.6
44.1
40.8
68.1
54.6
26.6
53.4
3.5
Mamba
7.1
$5.9
54.5
72.4
63.0
30.2
56.9
1.3
8.1
54.5
54.2
72.3
63.5
31.4
57.1
2.2
8.8
52.9
54.6
72.6
64.4
31.9
57.2
3.1
6.9
61.7
51.9
72.0
61.0"
25.5
60.2
Mamba-2 780M
T2.4
8.6
53.3
54.7
72.5
62.8
32.3
57.8
2.5
55.2
72.1
62.8
32.2
56.8
2.2
7.0
56.3
47.2
70.7
67.0
27.0
53.4
Pythia
8.0
51.8
47.3
70.7
57.1
28.2
53.4
1.0
71.2
10.9
48.4
47.3
71.4
58.7
28.4
53.1
1.9
5.0
64.9
59.2
74.1
65.5
32.9
61.3
Mamba 1.4B
5.8
60.6
58.2
74.7
64.5
33.0
60.9
-0.5
6.6
58.9
58.9
73.6
66.1
34.5
00.9
0.7
59.0
74.1
5.0
65.6
60.0
73.2
602
I3.I
6I.I
Mamba-2 1.3B
6.0
60.1
59.4
73.1
65.6
35.3
59.4
1.0
6.7
58.6
60.1
73.4
35.4
61.9
2.5
7.0
58.6
60.2
73.7
66.5
35.9
61.4
2.7
6.I
61.7
52.1
70.3
60.5
23.5
57.4
Pythia 1.4B
7.0
56.3
52.1
71.4
62.0
29.5
57.5
7.9
54.4
52.6
70.9
63.9
31.1
56.8
2.9
8.0
52.8
71.0
57.8
4.2
69.1
66.1
75.2
69.6
36.4
63.3
Mamba 2.8B
65.6
75.6
37.1
5.5
62.8
65.5
75.3
70.8
38.1
65.1
1.7
5.7
62.5
66.1
76.1
70.9
38.1
64.6
2.0
4.1
66.6
Mamba-2
4.8
65.1
65.9
75.1
70.0
38.6
65.1
1.3
66.8
75.2
71.9
64.1
5.7
62.3
67.1
75.3
70.7
41.2
65.9
3.6
5.0
64.7
59.3
73.3
6.2
32.5
50.8
Pythia 2.813
59.4
73.8
6.2
59.1
59.9
74.7
67.4
34.9
60.8
2.9
60.2
74.5
67.1
35.0
61.3
3.1
21

---

Published in Transactions on Machine Learning Research (08/2025)
Table 3: Instruction tuned model performance. Model checkpoints were evaluated on all benchmarks and few-shot settings using the LM evaluation harness from Eleuther Al (Gao et al., 2023). LAMBADA zero-shot more effective for the model sizes considered (further discussed in (Xic et al., 2021; Brown et al., 2020)) and thus excluded from few-shot performance averages. Highlighted bold the top-performing few-shot learner per benchmark and model grouping.
Model
N-shot
LAMBADA LAMBADA HellaSwag
PIQA
Arc-E
Arc-C
WinoGrande
O-shot incr.
ppl
acc
acc
1
Mean %
12.9
46.5
35.1
64.2
48.7
25.5
51.7
Mamba 130M
35.0
64.2
22.3
35.3
34.8
64.2
50.2
24.5
50.6
-0.8
23.6
35.9
34.7
61.7
19.8
21.6
50.2
-0.9
15.2
44.5
35.1
64.3
47.2
24.7
52.2
Mamba-2
21.9
36.1
31.5
61.3
16.8
21.0
50.8
-1.7
130M
26.9
34.7
65.1
29.0
33,8
34.5
61.88
18.7
25.1
51.3
0.4
30.2
36.1
30.0
62.2
44.7
23.6
50.3
Pythia 160M
44.5
30.4
62.0
50.5
66.7
25.5
30.3
62.8
15.2
22.8
19.8
-0.3
30.5
62.9
7.2
56.0
46.3
69.2
55.3
27.7
56.0
Mamba 370M
9.3
49.9
45.7
68.7
57.
28.3
55.4
0.5
15.7
11.0
48.3
45.7
70.1
59.3
29.1
54.5
1.9
7.6
46.8
69.3
5272
27.0
56.0
Mamba-2
9.9
48.3
46.0
69.6
55.7
28.8
55.2
2.1
370M
11.8
46.3
46.3
70.1
59.0
29.1
54.5
3.6
13.3
46.7
40.9
67.4
62.7
25.4
63.4
Pythia
17.2
10.4
10.5
68.4
53.0
25.7
53.0
0.5
410M
21.1
37.4
40.9
67.7
55.7
27.1
52.6
2.3
21.5
38.2
40.7
67.8
55.7
27.3
53.8
2.8
72.8
Masbba
6.3
56.6
54.9
72.7
64.6
31.7
56.3
1.2
7.0
35.6
54.7
72.4
65.3
33.2
57.5
2.7
7.5
54.9
72.9
65.6
35.8
57.2
3,2
1.9
63.1
65.8
71.7
611
30.6
58.2
Mamba-2
72.7
780M
7.8
52.7
51.9
73.5
65.0
34.6
57.8
3.6
8.6
52.8
54.8
73.4
64.6
34.0
58.0
3.1
7.7
36.6
47.3
70.3
67.1
26.7
53.4
Pythia
8.8
52.0
47.4
7O.7
57.5
28.8
53.6
1.8
71.4
10.6
48.8
47.4
71.5
58.9
28.4
53.0
2.0
4.6
64.8
59.3
74.3
66.2
35.1
62.3
Mamba 1.4B
6.1
59.3
58.4
74.1
67.4
36.6
61.8
1.0
74.5
68.3
59.9
1.9
63.0
60.I
73.8
6..0"
31.8
6I.3
Mamba-2 1.3B
6.1
58.2
59.2
74.2
67.0
35.0
60.1
0.5
T3.7
7.2
56.5
59.9
73.5
68.5
36.7
60.7
2,2
5.2
63.6
52.9
71.1
6172
30.3
58.2
Pythia 1.4B
6.2
57.4
52.7
71.7
62.2
30.6
56.9
0.2
7.0
56.1
53.1
71.1
64.5
32.8
56.8
2.3
7.1
55.5
63.3
71.2
63.8
57.5
1.0
67.7
66.4
75.6
68.4
36.6
61.2
4.8
63.3
65.9
76.2
70.9
39.4
64.6
2.4
2.8B
5.3
62.1
65.7
75.8
71.3
39.1
65.4
2.4
5.4
61.9
66.2
77.2
71.4
40.4
66.
3.9
3.8
67.5
76.0
65.3
Mamba-2
4.5
63.8
66.7
76.0
71.8
41.5
67.1
2.8
5.0
62.3
67.3
76.2
73.3
44.4
66.0
5.3
61.8
67.4
76.4
72.4
44.5
65.0
5.0
64.7
59.3
74.0
647
33.3
50.2
Pythia 2.8B
59.5
T4.0
6.1
59.2
59.9
75.0
67.5
34.9
60.0
2.9
60.4
74.5
67.0
35.1
61.2
22

---

Published in Transactions on Machine Learning Research (08/2025)
F.1 Hardware throughput and memory-utilizann improvements given PEFT and MPFT
With stable dynamics and observed divergences smaller than comparable Transformers, show that MPFT and PEFT may be used to significantly increase GPU-training throughput for Mamba SSMs. To demonstrate such improvements, we utilize the previous fine-tuning settings for the Alpaca dataset. However, we now adjust the batch size to maximize throughput per MPFT and PEFT configuration.
For each MPFT and PEFT configuration, the average tokens-per-second (ATPS) is calculated as the total tokens used for fine-tuning divided by total training time, and the mazimum memory-per-tookee (MMPT) is calculated the maximum GPU memory utilization incurred (over the entire fine-tuning run) divided by the total number of tokens in each mini-batch Results plotted in Figure
Both throughput and memory utilization improve as the number of Mamba parameters increases in Figure 7. Compared to the full-precision full fine-tuning of Mamba 790M (the largest model supported by an A10G's memory capacity), evaluated MPFT and PEFT combinations result in an average 2.15 times more training tokens-per-sccond while reducing per-token memory utilization by an average 62.7%. Across all model sizes, evaluated MPFT and PEFT combinations result in an average 1.74 times more training tokens-per-seett per-second while reducing per-token memory utilization by average 47.2% compared to respective full-precision fine-tuned runs. Furthermore, while full fine-tuning is longer possible single A10G for Mamba models greater than 790 million parameters, MPFT and PEFT allow training Mamba models up to 2.8 billion parameters on GPUs with as little as 24 GB onboard memory.
1e4
Tokens/second 3
FP32 Full LORA
BF16 ALL LORA
FP16 SLL LoRA
BF16 SLL LoRA
1
Mamba 130m
Mamba 370m
Mamba 790m Model
Mamba 1.4b
Mamba 2.8b
(a) Average cokens-per-secom 1.
max-memmrr/token (MB)
1e1
2.0
FP32
FP16
1.5
BF16 ALL LORA
FP16 SLL LoRA
1.0
BF16 SLL LORA
0.5
0.0
Mamba 130m
Mamba 370m
Mamba 790m Model
Mamba 1.4b Mamba 2.8b
(b) Maximum memory--ee-ttoenn
Figure 7: Timing and memory usage calculated Mamba model-sizes and PEFT combinations. Each model was trained using the Alpaca dataset dataset for three epochs and maximum sequence length 512. For each PEFT combination, the batch size was tuned to maximize GPU occupancy. Full fine-tuning exceeds available GPU memory (24 GB) for models greater than 790 million parameters.
G ICL as an emergent ability of Mamba SSMs
We study the emergent behavior (as a function of model size) of Mamba/Manba- SSMs' ICL abilitics on natural language tasks by comparing a larger number of Transformer- LLMs of varying sizes. We compare to OpenELM (Mehta et al., 2024) (sizes 270M, 450M, and 1. 1.1B), TinyLlama 1.1B (Zhang al.,
23

---

Published in Transactions on Machine Learning Research (08/2025)
2024), and OLMo 1.2B (Grocneveld et al,, 2024). To limit the emergent effects both parameter size and pretraining token counts, we did not evaluate models greater than 2.8 billion parameters and chose open-source checkpoints as close as possible to the 300 billion total pretraining tokens used for Mamba, Mamba-2, and Pythia models. Thus, pretraining token counts for OpenELM, TinyLlama, and OLMo models were 429 billion, 503 billion, and 336 billion, respectively. We note this potentially biases ICL. performance in favor of the newly evaluated Transformer-based LLMs, and that direct comparisons between Mamba, Mamba-2, and Pythia are the most fair (as these three classes of models were all pretrained on the same dataset for the same number of total pretraining tokens).
We repeat the experiments from Figure 4, where we evaluate the pretrained and instruction tuned ICL capabilities of all models. To undersland the critical role of parameter counts, we group all models into two classes: LLMs containing 450 million parameters or less, and LLMs containing greater than 450 million parameters. ICL. performance measured by AIPSS is displayed Figure 8.
From Figure 8, it clear that pretrained SSMs and Transformers of parameter counts 270 million and less display slight detrimental ICL abilities (i.e., few-shot performance is worse than zero-shot). For models of greater than 450 million parameters, the majority of SSMs and Transformers display positive ICL abilities, with Mamba 1. 1.4B being an outlier in terms of poor performance. With the exception of TinyLlama at 1-shot performance and Mamba-2 2. 7B for 3- and 5-shot performance, the majority of other pretrained models cluster together.
Instruction tuning greatly smooths ICL performance across both parameter classes. While instruction tuned SSMs and Transformers of 160 million parameters or fewer continue to display slight or detrimental ICL abilitios, all parameters of 270 million and greater show positive ICL abilitics. For instruction tuned models of greater than 450 million parameters, all SSMs and Transformers show positive ICL abilities, with Mamba-2 2. 2B greatly outperforming all other models (both SSM and Transformer) in this class.
Thus, in terms of ICL as a function of SSM model size, while no clear trend presents itself for pretrained models, ICL emerges for instruction tuned Mamba and Mamba-2 SSMs of size 370 million and greater.
H Expanded divergence results: Alpaca and LIMA fine-tuning, MMLU and Winogrande benchmarks, Mean and Standard Deviation divergences
We exlend the non-divergent Mamba fine-tuning results from Section 4. Recall that the following MPFT and PEFT configurations are considered to fine-tune each considered LLM:
1. Full fino-tuning in FP32
2. Full fine-tuning in FP16
3. Full fine-tuning in BF16
ALL LoRAfine-tuning FP32
ALL LoRAfine-tning FP16
6. ALL LoRAfine-tuning in BF16
7. SLL LoRAfinettmingg in FP32
8. SLL LoRAfine-tuming in FP16
9. SLL LoRAfine-tunnng in BF16
In addition to the Alpaca dataset (Taori et al., 2023), we also fine-tune all models using the LIMA dataset (Zhou et al., 2024). Models are trained using LIMA for 5 epochs, while all other settings follow the fino-tuning recipe used for Alpaca (described in Appendix F).
24

---

Published in Transactions on Machine Learning Research (08/2025)
Konta 2.40
Mamba 2.40
1.4443.43
Vertal 1:3
Teytian
Openk.M
Manos Las
Marbs
Tepuama
Docnc L8
Vero 700
Reroz D00
0-Shot
1-Shot
3-Shot
5-Shot
0-Shot
1-Shot
3-Shot
5-Shot
450M < Parameter total, Pretrained
(b) 450M Parameter total, ALL LoRA
operiii Pytii 1100
Opanaak
Marba
Monsa 131
Vumoo23300
No Sctent et imprremmnn
opentil 270
Narbee 2 130M
Open M 7729
Roma
Runzr
Runc/ 1366
0-Shot
1-Shot
3-Shot
5-Shot
0-Shot
1-Shot
3-Shot
5-Shot
(c) Parameler total & 450M, Pretrained
(d) Parameter total & 450M, ALL LoRA
Figure 8: Instruction tuning improves Mamba-2 ICL performance past Transformer LLMs. ALL LoRA models were ruction fine-tuned on the OpenHermes dataset for one epoch. Performance is reported as the average improvement percentage of {1, 3, 5)-shot versus O-shot over five standard natural language benchmarks: HellaSwag, PIQA, Arc-E, Are-C, and WinoGrande. Mamba* and Mamba' refer to recent Mamba-1 PEFT methods, ie., LoRAp(X) and Additional-Scan, respectively, from Yoshimura et al. (2025).
25

---

Published in Transactions on Machine Learning Research (08/2025)
For natural language benchmarks, in addition to MMLU, evaluate each fine-tuned model using Wino- grande (Sakaguchi et al., 2021). Recall that, for each benchmark, divergence between mixedd-recision fine-tuned model is measured between its full-precision counterpart and averaged over {0, 1, 3, 5j-shot performance. addition to the average divergence, we also include the standard deviation of divergence. Thus, in total, 144 LLMs were fine-tuned, 576 MMLU evaluations were conducted, and 576 Winogrande evaluations were conducted.
Figure H displays results for Alpaca and MMLU, Figure H displays results for Alpaca and Winogrande, Figure H displays results for LIMA and MMLU, Figure H displays results for LIMA and Winogrande. Summary statistics for all experiments are presented in Table H. While OpenELM exhibits large deviation spikes for both Alpaca benchmark evaluations and Pythia exhibits large deviation spikes for all four evaluations Mamba does not exhibit a single large deviation spike on any benchmark for all considered model sizes and MPFT/PEF'T configurations (i.e., 18 total configurations excluding the full-precision baselinos). Furthermore, Mamba models are significantly more stable for MPFT/PEFT compared to Transformer-basse LLMs. E.g., for MMLU evaluations, Alpaca fine-tuning with Mamba models is average 2.6 times smaller in mean divergence than both Pythia and OpenELM models, while LIMA fine-tuning with Mamba models is an average 7 and 3.25 times smaller in mean divergence than Pythia and OpenELM models, respectively.
Table 4: Summary of divergence results for Alpaca and LIMA fine-tuning datasets, MMLU and Winogrande benchmarks, and Mamba, OpenELM, and Pythia models. For each deviation summary statistic per fine- tne-tuning dataset and benchmark, the lowest deviation is highlighted in bold.
(Fine-tuning dataset), Benchmark Architecture
Large deviation spikes Avg mean divergence Std mean divergence
Pythia
0.37
0.41
(Alpaca, MMLU)
OpenELM
0.37
0.32
Mamba
0
0.14
0.08
Pythia
0.72
0.58
(Alpaca, Winogrande)
OpenELM
3
0.59
0.37
Mamba
0.25
0.09
Pythia
0.28
0.34
(LIMA, MMLU)
OpenELM
0
0.13
0.15
Mamba
0.04
0.03
Pythia
3
0.45
0.45
(LIMA, Winogrande)
OpenELM
0.36
0.18
Mamba
0.11
0.12
Mean Per-shot FP32 Divergence
FP16 Full
1.5
BF16 Full
FP16 ALL LORA
BF16 ALL LORA
1.0
FP16 SLL LORA
BF16 SLL LORA
0.5
0.0
Mamba 130M
Pythia 160M
OpenELM 270M
Mamba 370M
Pythia 410M
OpenELM 450M
Mamba 790M
Pythia 1B
(a) Mean full- full- precision (FP32) divergence in MMLU performance.
Figure 9: Alpaca fine-tuning, MMLU evaluation. Mamba, Pythia, and OpenELM models are fine-tuned over the Alpaca dataset using different combinations of MPFT and PEFT. Full finc-tuning (i.c., no PEFT adapters) denoted as Full.
26

---

Published in Transactions on Machine Learning Research (08/2025)
Mean Per-shot FP32 Divergence
FP16 Full
BF16 Full
FP16 ALL LORA
BF16 ALL LORA
2
Mamba 130M
Pythia 160M
OpeNELM
Mamba 370M
Pythia 410M
OpenELM 450M
Mam 790M
Pythia 1B
270M
(a) Mean full-precision (FP32) divergence in Winogrande performance.
Figure 10: Alpaca fine-tuning, Winogrande evaluation. Mamba, Pythia, and OpenELM models are fine-tuned over the Alpaca dataset using different combinations of MPFT and PEFT. Full fine-tuning (i.e., no PEFT adapters) denoted as Full.
Mean Per-shot FP32 Divergence
FP16 Full
BF16 Full
2.0-
FP16 ALL LoRA
BF16 ALL LORA
1.5
FP16 SLL LORA
BF16 SLL LoRA
1.0
0.5
0.0
Mamba
Pythia 160M
OpenELM 270M
Mamba 370M
Pythia 410M
OLeEELM 450M
Mamba 790M
Pythia 1B
130M
(a) Mean full-precision (FP32) divergence in MMLU performance.
Figure 11: LIMA fine-tuning, MMLU evaluation. Mamba, Pythia, and OpenELM models are fine-tuned over the LIMA dataset using different combinations MPFT and PEFT. Full fine-tuning (i.e., no PEFT adapters) is denoted as Full.
Mean Per-shot FP32 Divergence
FP16
BF16 Full
BF16 ALL LORA
2
FP16 SLL LORA
BF16 SLL LoRA
1
Mamba 130M
Pythia 160M
OpenELM 270M
Mamba 370M
Pythia 410M
OpenELM 450M
Mamba 790M
Pythia 1B
Mean full-precision (FP32) divergence in Winogrande performance.
Figure 12: LIMA fine-tuning, Winogrande evaluation. Mamba, Pythia, and OpenELM models are fino-tuned over the LIMA dataset using different combinations of MPFT and PEFT. Full fine-tuning (i.c., PEFT adapters) is denoted as Full.
27


Reviews:
[{"review_id": "Uj1e2ovO4q", "reviewer": "Reviewer_z2sE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n+ Overall this paper is easy-to-follow and clearly presented.\n\n+ The use of Lyapunov exponents to establish formal stability is a novel and solid perspective.\n\n+ The empirical studies are extensive, covering multiple model sizes, datasets, and fine-tuning schemes.\n\nWeakness: \n\nMajor concerns:\n \n- While this paper systematically analyzes the stability of Mamba from a perspective of Lyapunov exponents, its generalization error upper bound is not analyzed, which negatively impact the soundness of this work.\n\n- While Sec.2.1.1 discusses the consequences of automatic mixed-precision, it lacks a theoretical analysis. \n\n- Besides, more experimental evidence on this aspect is also needed.\n\n- While this paper presents good experimental analysis by numerical comparison, it lacks a in-depth analysis from the feature space perspective.\n\n- Can this paper discuss some possible application or user cases?\n\n- The related work section need to be reorganized. For each aspect and research line, it should cover an independent subsection. \n\n- This paper lacks a failure case and limitation discussion.\n\n- The ICL gains of Mamba after instruction tuning are strong, but the pre-tuning ICL results are weak. This raises the question of architectural adequacy: is the ICL capability truly a learned emergent behavior or a by-product of alignment? \n\n- It’s unclear how much the performance of instruction-tuned Mamba-2 (surpassing Pythia) is due to stability versus other factors like parameter count or tuning schedule.\n\nMinor comments for improvement:\n\n- The abstract should not contain citations.\n\n- In the introduction, the authors directly start to mention Mambaa-2 models. But a clear definition is missing.\n\n- Fig.2 should be on top of the page.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper systematically studies the property of the emerging selective space model (Mamb) from the perspective of the dynamical system theory.\n\nSpecifically, a theoretical analysis of recurrent stability via Lyapunov exponents is conducted. It shows that Mamba models exhibit non-positive Lyapunov exponents, making their recurrent dynamics stable under small perturbations.\n\nExperiments on multiple datasets and settings show its effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness mentioned above.\n\nMajor concerns:\n \n- While this paper systematically analyzes the stability of Mamba from a perspective of Lyapunov exponents, its generalization error upper bound is not analyzed, which negatively impact the soundness of this work.\n\n- While Sec.2.1.1 discusses the consequences of automatic mixed-precision, it lacks a theoretical analysis. \n\n- Besides, more experimental evidence on this aspect is also needed.\n\n- While this paper presents good experimental analysis by numerical comparison, it lacks a in-depth analysis from the feature space perspective.\n\n- Can this paper discuss some possible application or user cases?\n\n- The related work section need to be reorganized. For each aspect and research line, it should cover an independent subsection. \n\n- This paper lacks a failure case and limitation discussion.\n\n- The ICL gains of Mamba after instruction tuning are strong, but the pre-tuning ICL results are weak. This raises the question of architectural adequacy: is the ICL capability truly a learned emergent behavior or a by-product of alignment? \n\n- It’s unclear how much the performance of instruction-tuned Mamba-2 (surpassing Pythia) is due to stability versus other factors like parameter count or tuning schedule.\n\nMinor comments for improvement:\n\n- The abstract should not contain citations.\n\n- In the introduction, the authors directly start to mention Mambaa-2 models. But a clear definition is missing.\n\n- Fig.2 should be on top of the page.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "C8QRYZGXUK", "reviewer": "Reviewer_SpRc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. State-space models such as Mamba are important due to their efficiency compared to their transformer counterpart. However, their overall performances are weaker than transformers. The strongest contribution of this paper is to showcase the robustness and stability of Mamba during fine-tuning phase over transformer models.\n\n2. This paper uses dynamical system theory to provide fundamental reasons why Mamba models are robust under fine-tuning, as input perturbation won't cause the error to exponentially grow, as $\\lambda_{\\max}\\leq 0$ and the error is in the form of $\\epsilon\\cdot \\exp(N \\lambda_{\\max})$ where $N$ is the number of applications of the model $F_\\theta$. \n\n3. Experiments are comprehensive, in the sense that various modes of fine-tuning are tested, including all parameters, FP, BF, parameter-efficient ones (LoRA, SLL LoRA) and the metric is the mean per-shot FP-32 divergence. Grid search over the set of hyperparameters (learning rate, LoRA dimension, warm ups) is performed to show the stability of Mamba. Finally, multi-shots performance improvements over zero-shot ICL after instruction-tuning has been tested, to show the improved performance of Mamba models under tuning.\n\nWeaknesses:\n\n1. The theory is explanatory and mainly applies existing results from dynamical systems, therefore it lacks novelty. However, it should be noted that the main purpose of the paper is to explain stability behavior of Mamba, rather developing new theory for dynamical systems.\n\n2. I found the writing of the paper, in particular the core message, not super clear before the conclusion section. What is the major question posed in this paper that authors try to solve? Is that the stability behavior of Mamba for fine-tuning? What is the main takeaway from authors study? Should we widely adapt Mamba models for fine-tuning and instruction-tuning? These questions become more obvious in the conclusion section, but are unclear before that. For example, when reading through the paper, I feel the contents are quite segmented and a bit incoherent. For example, the theory suggests that Mamba models are stable under input perturbations, then the experimental sections show that the performance of Mamba models are generally more stable in the sense that they have smaller deviations among the choices of hyperparameters and in comparison to FP fine-tuning when using LoRA-type fine-tuning? How do these two parts relate to each other beyond using different precisions? It feels like through experiments, Mamba models are more robust and stable beyond using different precision for fine-tuning, but the authors do not very explicitly spell out this part well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The goal of this paper is to understand the robustness of models based on SSMs such as Mamba, both theoretically and empirically. Theoretically, this paper identifies that prior works analyze SSMs via time-invariance property, but Mamba has time-varying parameters, so prior analysis is not enough. To address this issue, authors use theory from dynamic systems to prove the robustness of Mamba models, by showing the maximal Lyapunov exponent $\\lambda_{\\max}\\leq 0$, this shows that the input perturbations won't cause exponentially compounded error over time. Next, they perform extensive experiments to show the robustness of fine-tuning for Mamba models compared to transformer-based models (Pythia and OpenELM). Experiments include: various modes of fine-tuning, such as full-precision, mixed-precision, all parameters, LoRA, SLL LoRA. Mamba models exhibit much constrained divergence behavior of different fine-tuning procedures compared to Pythia and OpenELM, particularly obvious on Winogrande. Further, authors show that Mamba models, while achieve lower 5-shot MMLU accuracy, are more stable under choices of hyperparameters. Finally, authors empirically verify that under instruction tuning, Mamba exhibits improved ICL ability compared to Pythia.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper could be significantly improved if it emphasizes more on the main topic -- stability of fine-tuning for Mamba models -- in the introduction and in particular the authors could try to tie the theory and experiments on mixed precision fine-tuning more tightly. At its current form, the paper feels a bit fragmented (e.g., the consequences of the theory results could be further emphasized).\n\nFinally: I'm not an expert in Mamba fine-tuning experiments, I would weigh in the opinions of the reviewers that are more familiar with this aspect.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "unPl5uybvf", "reviewer": "Reviewer_qxVf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The empirical findings of this paper should be of interest to the TMLR audience. However, the way in which they are reported is a weakness of the paper. The authors do not show error bars in their main figures, instead deferring measures of variability to tables and separate bar plots in the Appendices. This makes it challenging to assess the statistical strength of their claim that \"training Mamba LLMs is significantly more stable than comparable Transformer-based LLMs\". I also have some concerns about the theoretical results presented, as I believe they are amenable to a simpler proof. I elaborate on these and other issues under **Requested Changes** below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper observes that Mamba state space models are Lyapunov-stable, and argues that this provides an explanation for their experiments showing that Mamba models fine-tuned either using full precision, mixed precision, or low-rank updates all behave robustly.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors emphasize the fact that previous SSM stability results do not apply to Mamba because it is not an LTI system, and thus put forward a proof of stability based on computation of the maximum Lyapunov exponent. I have two concerns regarding this result (questions 1 and 2 here). First, it risks feeling like it borders on tautological, given that the Mamba designers presumably chose the particular parameterizations that lead to this in order to guarantee stability of the linear dynamics. In particular, a key part of the author's argument for stability is the fact that Mamba chooses a parameterization such that $A_{ii} \\leq 0$, which is fundamentally what guarantees stability (see below). I imagine the authors will disagree with this assessment, and I would ask them to elaborate on the reasons why. \n\n2. Second, unless I am mistaken the claim follows directly without the need to invoke the general machinery of Lyapunov stability, as this is still a linear system, time-variant though it may be. As the authors remark, all matrices of interest are in practice chosen to be diagonal, and even if the parameters are time-varying one can write down a formal solution to the dynamics for each component so long as the parameters depend on time (and perhaps the input signal) but not on the hidden state (note that this is consistent with the fact that they write $\\partial x\\_{t}/\\partial x\\_{t-1} = \\bar{A}\\_{t}$). In particular, one is interested in the state dynamics $x\\_{t+1} = \\bar{A}\\_{t} x\\_{t} + \\bar{B}\\_{t} u\\_{t}$, which has formal solution $x\\_{T+1} = \\prod\\_{t=0}^{T} \\bar{A}\\_{t} x\\_{0} + \\sum\\_{t=0}^{T} \\prod\\_{t'=t+1}^{T} \\bar{A}\\_{t'} \\bar{B}\\_{t} u\\_{t}$. Here, the order of matrices in the products does not matter given that they are assumed to be diagonal, but one can always fix the convention $\\prod\\_{t=0}^{T} \\bar{A}\\_{t} = \\bar{A}\\_{T} \\bar{A}\\_{T-1} \\cdots \\bar{A}\\_{0}$. Then, given the assumption that $\\bar{A} = \\exp(\\bar{\\Delta}\\_{t} A)$ for diagonal matrices $\\bar{\\Delta}\\_{t}$ and $A$, and that $\\bar{B}\\_{t}$ is likewise diagonal, for each $i \\in [d]$ one will have terms like $\\prod\\_{t=1}^{T} (\\bar{A}\\_{t})\\_{ii} = \\exp( A\\_{ii} \\sum\\_{t=1}^{T} (\\bar{\\Delta}\\_{t})\\_{ii} )$. Under the assumptions stated in Appendix A (*but not in the main text, this must be rectified*), $A\\_{ii} \\leq 0$. Then, one has $\\prod\\_{t=1}^{T} (\\bar{A}\\_{t})\\_{ii} \\leq \\exp( T \\zeta_{i} )$, where $\\zeta_{i} = A\\_{ii} \\min\\_{t} (\\bar{\\Delta}\\_{t})\\_{ii} \\leq 0$, using the assumption that $(\\bar{\\Delta}_{t})\\_{ii} \\geq 0$ for all times $t$. One would then get a global sensitivity bound in terms of $\\zeta = \\max\\_{i \\in [d]} \\zeta\\_{i}$ that is exponentially decaying in the time lag since a perturbation was introduced. Can you clarify what I'm missing here - what is the obstacle to this approach if in practice one only needs to consider the diagonal case? \n\n3. The figures in the main text do not include error bars, and standard deviations are reported only in Appendix G in separate bar plots. This makes it very challenging for the reader to assess the statistical significance of results, and as such I ask that the authors update all of their main-text figures to include error bars. This is critical for the legibility of the results, and therefore for acceptance.\n\n4. In equation (4-6), the authors adopt a particular format for storing the parameters in the memory buffer matrix $W$, and then performs low-rank updates to this format of $W$. Though choosing this format is advantageous from the perspective of computational efficiency, it is not clear to me whether this format is in some sense optimal for low-rank fine-tuning, i.e., whether standard matrix rank is an appropriate measure of the complexity of an update given the underlying parameters one fundamentally wants to target. Can you provide more justification here, particularly as it is this formatting that leads to the claimed result that LORA induces dependencies between the different Mamba weights?  \n\n5. Can you compare your results to those of [Yoshimura et al. (2025)](https://openreview.net/forum?id=UAKnJMIBwf) on parameter-efficient fine-tuning for Mamba models? \n\n6. A tangential comment - the authors might find [this recent paper by Haiping Huang's group](https://journals.aps.org/pre/abstract/10.1103/PhysRevE.111.034308) on using Lyapunov exponents to measure stability of vanilla RNNs to weight perturbations to be of interest.", "broader_impact_concerns": "None, other than that I wonder whether the authors overstate somewhat the implications of their work for reducing emissions.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)