Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models
Authors: Hung-Yueh Chiang
Chi-Chih Chang
Natalia Frumkin
Kai-Chiang Wu
Mohamed S. Abdelfattah
Diana Marculescu
Venue: ICML 2025 poster (2025)
Keywords: State Space Models
Model quantization
Primary area: deep_learning->large_language_models
TL;DR: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models

Abstract: State Space Models (SSMs) are gaining attention as an efficient alternative to Transformers due to their constant memory complexity and comparable performance. Yet, deploying large-scale SSMs on cloud-based services or resource-constrained devices faces challenges. To address this, quantizing SSMs using low bit-width data types is proposed to reduce model size and leverage hardware acceleration. Given that SSMs are sensitive to quantization errors, recent advancements focus on quantizing a specific model or bit-width to improve their efficiency while maintaining performance. However, different bit-width configurations, such as W4A8 for cloud service throughput and W4A16 for improving question-answering on personal devices, are necessary for specific scenarios.
To this end, we present Quamba2, compatible with \textbf{W8A8}, \textbf{W4A8}, and \textbf{W4A16} for both \textbf{Mamba} and \textbf{Mamba2}, addressing the rising demand for SSM deployment across various platforms. We propose an offline approach to quantize inputs of a linear recurrence in 8-bit by sorting and clustering for $x$, combined with a per-state-group quantization for $B$ and $C$. To ensure compute-invariance in the SSM output, we offline rearrange weights according to the clustering sequence. The experiments show Quamba2-8B outperforms several state-of-the-art SSMs quantization methods and delivers 1.3$\times$ and 3$\times$ speedup in the pre-filling and generation stages and 4$\times$ memory reduction with only a $1.6$% accuracy drop on average. The code and quantized models will be released at:

Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models
Blung-Yuel Chiang' Chi-Chih Chang² Natalia Frumkin'
Kai-Chiang Wu Mohamed S. Abdelfattah
Diana Marculescu
Abstract
1. Introduction
Despite State Space Models (SSMs) are emerg- ing as an efficient alternative to Transformers, de- polying SSMs on both cloud and edge devices is challenging due to the limited resources. Model quantization reduces model size and leverages hardware acceleration, and recent efforts on SSM quantization have focused on optimizing a par- ticular model or bit-width. However, distinct bit- widths are essential for different scenarios, like W4A8 for boosting barge-batch decoding speed, and W4A16 16 for enhancing generation speed in short-prompt single-user applications. Quamba2, compatible with W8A8, W4A8, and W4A16 for both Mambal and Mamba2 back- bones, addressing the growing demand for SSM deployment. Based channel order preserving and activation persistence of SSMs, we propose an offline approach to quantize inputs of the linear recurrence in 8-bit by sorting and clustering for input x, combined with a per-state-group quanti- zation for input-dependent parameters B and C. To ensure compute-ing compute-invariance invariance in the SSM output, we rearrange weights offline according the clus- tering sequence. We show that Quamba2-8B out- performs two state-of-the-ee the-art SSM quantization methods and delivers 1.3x and 3x speed-ups in the pre-filling and generation stages, respectively, while offering 4x memory reduction with only 1.6% average accuracy drop. The evaluation MMLU shows the generalizability and robust- ness of our framework. The code and quantized models are released at the link.
State Space Models (SSMs) (Gu et al., 2020; Smith et al., 2023; Gu & Dao, 2024; Dao & Gu, 2024), offering constant memory complexity. are emerging as efficient alternatives to Transformers (Vaswani, 2017) in various areas such as lan- guage modeling (Wang et al., 2024; Waleffe et al., 2024), vi- sion (Zhu et al., 2024a; Liu al., 2024a; et al., 2025), and audio (Goel al., 2022: Saon al., 2023). Some studies expand the model sizes and demonstrate their performances par with Transformers of the same scale (Lieber et al., 2024; Team et al., 2024; Waleffe et al., 2024). However, the increased size and computational demands of large SSMs limit the hardware options and raise deployment costs.
applicactioon.... We present
Post-training quantization (PTQ) offers an attractive solu- tion efficient deployment by eliminating the need for fine- tuning large models. PTQ methods reduce the bit- width of pre-trained weights and activations to lower-bit formats (such as 8-bit), cutting down memory use for weight storage and leveraging advanced hardware units. Recent studies (Xu et al., 2025; Chiang et al., 2025) reveal that quantiza- tion techniques that are effective Transformers struggle with SSMs due to the sensitivity of linear recurrence to quantization-induced errors. This prior work introduces PTQ algorithms tailored for SSMs to bridge the perfor- mance gap between low and half-precision models. How- ever, they either do not explore diverse bit-widths (Chiang et al., 2025) or fail to achieve satisfactory performance at lower bit-widths (Xu et al., 2025), like W4A8.
Specific bit-width setups are crucial for certain scenarios. For instance, W4A8 enhances cloud service throughput with large-batch inputs (Lin et al., 2024b), whereas W4A16 16 improves the efficiency of short prompt applications (Lin et al., 2024a). As a result, current SSM-based quantiza- tion methods may underperform on edge devices or fail to maximize throughput on cloud services. Moreover, a recent study (Zhao al., 2024a; Kumar al., 2025; Gong et al., 2024) reveals that heavy quantization of model weights and activations (e.g.. W4A4) impairs model generalization multi-step reasoning tasks.
'Chandra Family Department Electrical and Computer Engineering, The University of Texas at Austin 'Department of Electrical and Computer Engineering, Cornell University 'Department of Computer Science, National Yang Ming Chiao Tung University. Correspondence wwwpueehhhhhnnttutteeeee
Hung- Yuch Chiang
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To address these issues, we present Quamba2, robust and

---

Quamba2
Table I: (Supported models.) Our framework supports W8A8, W4A8, and W4A16 for both Mambal (Gu & Dao, 2024) and Mamba2 (Dao & Gu, 2024).
Onn Nara
embedding
heed
7P26
VHAAS VH4A
system
som block
WEAB
16
128
14-
2 X. Smallar
4% Smaller
Through (atokknuzked)
12
Models Mambal Mambu2 W8A8 W4A8 W4A16
Bitwidth
10-
Methods
Momory
-2-
MambuQuant (Xu et )
13 TPS
Quamba (Chiang al.)
/
Quamha2 (Ours)
/
/
FP16
WSA8
WBAB
W4A8
W4A8/16
A5000
Nano 8G
(Ours) H2T (Durs)
(a) Memory
(b) Throughput
Table 2: (Supported bit-width.) Quamba2 supports head- to-toe (H2T) 4/8-bit from the embedding layer, SSM blocks, to the final output layer (i.e. Im head).
Figure I: (Quamba2-8B memory and throughput.) The
head-to-toe (H2T) quantization enables the deployment of Mamba2-8B on the edge platform such as Nvidia Nano 8G. Quamba2 delivers 3x throughput on Nyidia A5000 and 13 lokens-per-secee (TPS) on Nvidia Nano 8G.
Methods
Embed.
SSM blocks
Im head H2T
MamhaQuant (Xu et al.)
16-bit
4/8-bit
16-hit
x
Quamba (Chiang et al.)
16-bit
8-bit
16-bit
zation (PTQ) (Zhu et al., 2024b; Zhou al., 2024). QAT requires additional training efforts to adapt models to low bit-width. PTQ is an attractive option for large language models (LLMs) since it eliminates the need for training. Our study falls under PTQ and provides W8A8, W4A8, and W4A16 16 bit-width configurations, delivering memory and latency reduction across platforms.
Quamba2 (Ours)
4/8-bit
4/8-bit
4/8-bit
scalable post-training quantization framework for selective SSMs. As shown in Table and our framework supports head-to-toe W8A8, W4A8, and W4A16 for both Mambal (Gu & Dao, 2024) and Mamba2 (Dao & Gu, 2024), meeting the demands for SSM deployment on cloud and edge plat- forms. Based on channel order preserving and activation persistence of the SSM computation, as shown in Figure 2 and 3, we employ an offline cluster-aware weight reordering approach to group SSM heads and channels with similar value ranges, allowing them share a quantization scaling factor and boost quantization precision. For selective SSM deputdeependett// parameters (B, C), we identify state persis- tence in activations and apply quantization per state group. Our sort-and-cluster and per-state-group quantization meth- ods improve quantization accuracy, closing the accuracy gap in half-precision models. In Figure and the rest of our experiments, we show that Quamba2-8B surpasses several leading SSM quantization methods, achieving up 1.3x and 3x higher speeds in prefilling and generation, respec- tively, and offering a 4x memory reduction, with only 1.6% accuracy loss across six zero-shot tasks. Additionally, we tested Quamba2 on MMLU (Hendrycks et al., 2020), a large multitasking dataset, demonstrating the generalizabil- ity and robustness of our framework.
PTQ and weight reordering for Transformers. Post- training quantization (PTQ) techniques are generally clas- sified into two categories: weight-only quantization (e.g., W4A16) and weight-activation quantization (e.g.. W8A8) (Zhu et al., 2024b). Weight-only quantization (Frantar et al., 2023; Lin et al., 2024a) minimizes weight storage, while activation quantization (Zhao et al., 2024b; Ashk- boos et al., 2024b) optimizes throughput with low bit-width operations. Reordering weights is frequently used to en- hance quantization precision (Zhao et al., 2024b; Yuan et al., 2024) or efficiency (Lin al., 2024b) of Transformers, but its use and its subsequent effectiveness in SSMs is unclear. Our study shows that the selective State Space Duality (SSD) computing (Dao & Gu, 2024) preserves channel order be- tween input and output, with activated channels and states consistent over time.
PTQ and mixed-precision for SSMs. Xu et al. (2025) and Chiang et al. (2025) highlight that standard quantization techniques for Transformers are not effective for SSMs and propose PTQ algorithms tailored for SSMs. Despite this, these strategies do not offer a variety of bit-width configura- tions (Chiang et al., 2025) and struggle perform well re- duced bit-widths such W4A8 (Xu et al., 2025). Moreover, Zhao et al. (2024a) show that 4-t models lose generalizabil- ity, and Kumar et al. (2025) indicate the best performance under memory constraints for a bit-width of 6-8, with worse results for a bit-w of 4. Also, previous mixed--peeisste research focuses soly on Convolutional Neural Networks
2. Related Work
Model quantization. Representing model weights and activations in low bit reduces the cost of storing and loading parameters and benefits from advanced low bit- bit-width com- puting units (i.e,, Tensor Cores). Quantization methods are generally divided into two categories: Quantization-awan training (QAT) (Liu et al., 2024b; Dettmers et al., 2024; Yu et al., 2025; Tang et al., 2024) and post-training quanti-
2

---

Quamba2
Channel order preserving
Oueouts
A2
CAiath Chust
States
State persistance
Mlab
Goa Stafts
Blaa
Sdea
Ape
Koloa
Am
Kodia
Ghoally Glasd
Glack Geeesh Gleal
Ippanss
Channel persistence
Eorted head
Diagonal Input Output
Low-Rank Input State
x xand Activated Channela
Low-Rank Block: State State
Low-Rank Block: State Output
Activated State
Activated State
Figure 2: (SSD flows with sorted heads and the activation persistence.) We sort the head channels prior to applying quantization scaling factors. The orange blocks on the right indicate the activated channels with higher values in the input and output SSD heads. The SSD performs channel-wise calculation thereby retaining the channel order between input x and output which we call channel order preserving. The blue and green blocks represent the activated states of input-dependent parameters B and C. Our study shows that activated channels and states remain consistent across time steps and input samples, a property we denote as channel persistence and state persistence.
State porsistaric
Channel
Channel order preserving
State porsiatence
SsD
Silan
Dims
Thken
Taken
Dims
Tiken
DOms
Sorted x activation
(b) activation
(c) activation
(d) C activation
(c) B group maximum
(1) group maximum
Figure (Channel order preserving and activation persistence.) We show the activations in the last block of Mamba2-8B. For an input with tokens, demonstrate that the remains sorted by the maximum of the calibrated channel (a). The SSD calculation is channel-wise,
the output channel order y matches the input order (b). For B and C, the activated states remain consistent over time steps t (c-d) and input samples (e-f). We leverage the observations and design our techniques, sort-and-cluster and per-state- group quantization, to increase the quantization precisions for x (a), B, and C (c-f).
(CNNs) (Wang et al., 2019; Dong et al., 2019) and Trans- formers (Zhao al., 2021). We aim to fill the missing point of low wit-w and mixed-precision SSMs. Our framework provides W8A8, W4A8, and W4A16 for both Mambal (Gu & Dao, 2024) and Mamba2 (Dao & Gu, 2024) with practical speed-up and memory reduction, addressing the growing demand for the deployment of SSMs both in the cloud and on the edge. We evaluate Quamba2 on a large and challenging multitasking dataset, MMLU (Hendrycks al., 2020), show the robustness of our framework.
Quantization. We focus on symmetric uniform quantiza- tion to approximate floating-poine weights and activations with discrete N-bit signed integers (i.e., INT8 or INT4) due to its hardware compatibillity The general symmetric uniform quantization function is defined as
X -
X = Clamp
-2 22 12--12N--111) 11)
(I)
where =Max(x))/(22-11 1). X represents the quan- tized weights activations, X is the input matrix in floating point, and s is the scaling factor (i.e., quantization step) that is determined by the target bit-width N (N {3, in our setting). The static scaling factor is pre-calibrated and fixed during inference.
3. Background
3.1. Model Quantization
Notations. We follow the notation in Chiang al. (2025). We use X to represent the doating-poin matrices, and X to represent their quantized matrices with their point scaling factors 8,. For operators, we use f(-) to represent the quantized version of the function f(-) (i.e., the weights are quantized in the function f).
3.2. Selective State Space Models
The selective SSM (Gu Dao, 2024; Dao & Gu, 2024) transforms the time-invariant SSM (Gu al., 2020)
3

---

Quamba2
Reshape
Sort each head
Disentangling the head embedding
#Channels
#Heads #Channels
#Heads
#Heads
#Hoocs
Calibrated channel
#Chennels
t-SNE
(b)
(c)
(c-1) x with unsorted heads
#Chennets
#Channels
#Hoods
Cluster and reorder the heads
#Hoads
#Hoeds
*Channels
(-SNE
(c-2) with sorted heads (Ours)
(c)
(d)
Cluster the channels
each group
Figure 4: (Sort-and-cluster) We leverage the channel-persistent property in SSMs to sort the channel with the calibrated maximum (a-c). The sorted heads disentangle the embedding, shown in (c-1) and (c-2), enabling the clustering on the heads. We cluster the sorted heads into m groups (m = 8 in (d)), and reorder the weights offline to match the clustering results. Then, we apply the clustering again in each head group to cluster the channels into groups (n = in (e)). For each group, a scaling factor is calculated, resulting m n factors used to quantize It to 8-bit.
time-varying system. The system dynamies is defined by
Mamba2. Dao & Gu (2024) establish a theoretical link, Structured State Space Duality (SSD), between selective SSMs and self-attention. They also introduce efficient algorithm that utilizes matrix multiplication units contem- porary hardware to perform linear recurrence calculations. Mamba2 simplifies block design by removing sequential lin- ears where Ft; Be, Ct. and A, are produced parallel with single projection layer such that (Iz, Br, C., A,) = F(u.), where u, is the block input at the time step The modified block design is better suited to tensor parallelism (Shoeybi et al., 2019) in the context of larger models.
h4 = Adu-1 + BaT V1 -C + Dr,
(2)
where (A,, Br, C+) are input-dependent. Å, and B, are discrete parameters of and B. The discretization function for A, and B, with a given input-dependent A, defined as A, = Ax)(A,A), B. (A,4) (cxp(A,A) - 7) A,B, B. (A, D) are trainable parameters, and D is an optional residual parameter. An optional residual branch is applied to the SSM output such that y1 (i) before the output projection. We follow Dao & Gu (2024) and abstract the selective SSM computation at the time step t with the function
3.3. Quantizing Selective SSMs
SSM input parameters. The SSM defined in Equation 3 receives input in the form of (Á,, Br, Ct. Recent ef- forts (Xu et al., 2025; Chiang et al., 2025) show that the SSM block is extremely sensitive to quantization-induced errors in Øy due to the linear recurrence mechanism in Mambal (Gu Dao, 2024). Our work indicates that the phenomenon persists in Mamba2 (Dao & Gu, 2024). To address this issue, propose hort-and-cluste to quantize the input 2, with 8-bit. Our method groups the channels across the heads with the same value range to create smoother landscape in the group, and therefore increases the quantization precision.
SSM(A,, Br. (z).
(3)
Optional 2, and D are omitted in the function. We omit the subscript & represent the computation for the entire sequence. The abstract SSM block is shown in Figure 5.
Mambal. Gu & Dao (2024) presents selective SSMs in which the parameters B, C, and vary with input (i.e., time-varying), allowing the model to selectively prioritize or ignore inputs based on their content. The interaction with the input 2, is specified B, (x,)),, C, Fo(xx), A, softplus(F (z:)), where F3 and Fo are linear transformations mapping to B, and C, The func- tion FA involves two sequential projection layers, formu- lated as FA = Proj(Projæz)) + bias. The æ, is calculated from the input of the block u, with projection layer at the time stept. t.
SSM outliers. Prior studies Transformers (Dettmers et al., 2022; Xiao et al., 2023) have detected channel- persistent outliers. A common method for outlier climi- nation applying the Hadamard transform (Ashkboos et al., 2024b; Liu et al., 2024c). In SSM quantization (Xu et al., 2025; Chiang et al., 2025), online Hadamard matrices trans-

---

Quamba2
the use of unsupervised clustering into m groups. For each group of heads, we apply the clustering algorithm again to group channels into groups. The scaling factor is calcu- lated for every group, leading to total of m x n scaling factors, which are then utilized to quantize D2 to 8-bit pre- cision. The detailed sort-and-cluster process is shown in Figure 4. We find that m = 4 and n = 4 provide sufficiently good results throughout all experiments. The कe in Figure refers to the activation applied with sort-and-cluster.
Residual Input..1
RMSNorm
H[Cut Proji
y
Sequence Transform
HadQuant
RMMNNorm
Projection
y;
4
INT8
SSM
FP16
A
C.0 B
Offline Reordering
Per-state-group quantization. Dao & Gu (2024) relax the number of state group size and introduce a Multi-input SSM where Br, C2 matrices are shared across all channels of the input 2t, akin to grouped-ques et al., 2023) in Transformers. Our findings indicate that the activated states (with larger numerical values) are the same across time steps t and input samples. In Figure 3 (c-f), we visualize the activation distribution of B and C in the last block of Mamba2-8B. The number of groups in B and C is set 8, where each group has 128 channels. Figure 3 (c-d) shows that only a few groups are activated with larger val- ues. For example, in Figure 3 (c-f), group six in B is mostly activated, while group seven in both B and C' has minimal variations. Thus, we apply per-state-group quantization to B and C, where each group utilizes a single scaling factor. The B and Cf in Figure refer to the activations applied with per-state-grouy quantization. The per-state- group quan- tization largely increases the quantization precision in the groups where the value range is small, e.g. group seven in both B and C. We show that per-state-grous quantization is key to mitigating the performance gaps with the FP16 model for Mamba2-8B.
Conv
T--- f---
[In Proj)/f
attention (Ainslie
RMSNorm
Figure 5: (Quamba2 precision.) The detailed precision mapping of W4A8 and W8A8 Quamba2. We reorder the weights offline to match the sorting and clustering indices of zy. and apply per-state-growt group quantization on B and CỤ
form the input of output projection into smoother space, enhancing the quantization precision. Although the fast Walsh-Hadamare transform (FWHT) can be executed in par- allel with a nlogn complexity (Dao, 2024b; Sloane, 1999), we adhere to Xu et al. (2025) and Chiang et al. (2025) to quantize the output projection input, with the aim of mini- mizing online Hadamard transform overheads.
4. Proposed Method: Quamba2
4.1. Quantizing SSM Parameters
Our method is based on two findings in SSM activations: channel persistence and state persistence, together with a computational property of SSM: channel order preserving. The notation follows the definition from Equation
4.2. System and Framework Design
Cluster-aware weight reordering. We create a new chan- nel and head sequence in sort-and-cluster, where the heads within the same cluster are grouped and their channels are arranged by the pre-calibrated maximum. To produce the activations with the sorting and clustering orders, clustering and sorting indices to reorder offline the input projection, causal convolution, normalization, and output projection in the block. The output column of input projec- tion weights and the channel of causal convolution weights are reordered. As SSD computing maintains channel order (see Figure 2 right), we reorder normalization weights and apply fused Hadamard quantization. Finally, input rows of the output projection are rearranged using the same in- dices to keep the output the same. The offline aluster-aware weight reordering is depicted in Figure 5.
Sort-and-chuser. We observe the persistence of the chan- nel magnitude and the preservation of channel order in the SSM input and output y, shown in Figure 2. Although x is sensitive to quantization-indueed errors in Mamba2 (Dao & Gu, 2024), with the findings of Chiang et al. (2025) still applicable. Chiang et al. (2025) overlook the persistence characteristic and arder-preserving of the SSM channel. In contrast, we leverage these two properties to first sort the head channels and group both heads and channels. Specifi- cally, we first obtain the channel maximum from a calibra- tion dataset. In Figure 3 (a), we visualize the x sorted by the offine calibrated channel maximum of the last block of Mamba2-8B. x remains sorted input with an online t-token sample. The sorted x disentangles the head embedding, allowing head grouping. Figure 4 (cl-c2) shows that heads with similar characteristics are closely grouped, leading to
Offline Hadamard matrix fusion. Hadamard matrices have the computational property H. = nII, where n denotes n-dimensional square matrices. We therefore fuse offline the Hadamard matrices into the input and output lin-
5

---

Quamba2
Table 3: (SSD latency.) We profile SSD latency of Mamba2- 8B in milliseconds (ms) across sequence lengths with differ- ent input bit-width. We set batch size to eight.
Table 4: (Quamba2 model size in GB.) We profile the model size in GB of different width configurations for Mambal and Mamba2 in our framework.
Inputs
L 256
512
1024
2048
Models
Size
FP16
W8A8 W4A8 W4A16
FP16
0.82
1.61
3.51
7.22
Mambal
2.8B|
5.3 GB
2.8 GB
1.5 GB 1.5 GB
Int8 (Ours)
0.76
1.47
2.97
6.07
Mamba2 2.7B| 5.2 GB
2.7 GB 1,4 GB 1,4 GB
Speedup
1.08x
1.10x
1.18x
1.19x
8B 15.7 GB 7.9 GB 4.0 GB 4.0 GB
car projections. For the output projection, the Hadamard matrices are multiplied at both sides of the weight matrix, such that W H out = H,, W our H,,- We fuse a Hadamard ma- trix at the input side of the input projection weight, such that w W H., Thus, pairing Hadamard matrices in input/output projections with online Hadamard quantization results in compute-invariance invariance (Ashkboos et al., 2024a;b), yielding an identical block output. The offline Hadamard matrix fusion is shown in Figure 5. We apply the 4-bit/8-bbi quantization on the weights after matrix fusion.
promises generalizability of Transformers. We extend such analysis to verify SSM robustness and generalizability on MMLU (Hendrycks et al., 2020) dataset. Our findings in- dicate that while full W4A8 quantization maximizes pre- filling speedup, suffers from a notable generalization gap (-5.8% on MMLU vs. -2.1% on LAMBADA). In con- trast, full W4A16 quantization demonstrate robustness but comes at the cost of increased prefilling latency. To address this, we introduce mixed- -recidiireision support in our frame- work. We automatically search salient blocks based on their performance sensitivity and assign them a higher precision. Our W4A (14A(8/16) (8/16)-mixxe) -mixed SSM achieves 2.9% accuracy im- provement on MMLU while incurring only a 10% increase in prefilling latency.
Efficient 4-bit/8-bit Mamba blocks. Our framework ac- commodates W8A8, W4A8, and W4A16 projection kernels, a W8A8 causal convolution kernel, 4-bit and 8-bit embed- ding kernels, and 8-bit selective scan and SSD kernels. For projection layers, we reorder the weights and their per-group scaling factors (Lin et al., 2024b; Frantar et al., 2024; Zhang et al., 2024) to maximize the Tensor Core loading through- put. The output scaling factors are fused to the input scaling factors such that Y = Stsed W X where Sfused 8x/sy. We implement W4A8 and W4A16 mattmul-transooss fused kernels for the Mambal block. For sequence transforma- tions, we load the 8-bit activations and 8-bit cached states to reduce memory pressure, thus improving latency, as shown in Table 3. In the forward Hadamard transform, the seal- ing factor 8y is integrated, making 7 8x H,,y thereby avoiding extra computational load during quantization. The efficient kernels of our framework provide generic speed-up and memory reduction, addressing the increasing demands for the deployment of SSM on the cloud and on the edge.
5. Experiments
5.1. Experimental Setup
We provide framework design details in Appendix C.
Evaluations. We use LM-EVAL (Gao et al., 2023) to evaluate Quamba2 and baselines on six zero-shot down- stream tasks: LAMBADA (Paperno al., 2016), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC (Clark et al., 2018) and WinoGrande (Sakaguchi et al., 2020), and show the average accuracy over five runs in each table. To compare with MambaQuant (Xu et al., 2025) !, we aver- age the accuracy across five datasets: ARC-easy, ARC- challenge, PIQA, WinoGrande and HellaSwag. The full evaluation is in Appendix Section A, where we follow the evaluation protocol in Mambal (Gu & Dao, 2024), and re- port the accuracy for LAMBADA, WinoGrande, PIQA, and ARC-easy, and accuracy normalized by sequence length for HellaSwag and ARC-challenge. To show the generalizabil- ity and robustness, we evaluate the 8B models on MMLU (Hendrycks al., 2020), a large multitask test consisting of multiple-choice questions from various domains.
Head-to-toe quantization. Quantizing from embedding to the output head (i.e., Head-to-toe quantization) brings ad- ditional memory and latency reduction, which is necessary on edge computing platforms with limited memory capacity. As shown in Figure 1, our head-to-toe (H2T) quantization enables the deployment of Mamba2-8B on Nano 8G. Specif- ically, we employ per-token quantization to the embedding layer, and per group quantization to the weight of the head. As shown in Table 2, we implement the CUDA kernels and support the b-bit/8-bit embedding layer and 4-bit/8-bit out- put head. Therefore, our framework achieves generic 4x memory reduction.
In our W8A8 setting, we compare our frame- work with the latest quantization methods for SSM, Mam- baQuant (Xu et al., 2025) (W8A8, W4A8) and Quamba (Chiang et al., 2025) (W8A8) on zero-shot downstream
Baselines.
Improving robustness via -mixem Zhao et al. (2024a) demonstrate that applying W4A4 to all blocks com-
As of this writing, Mamba authors have not released their kernel implementations yet.
6

---

Quamba2
Table 5: (Mamba2-8B latency.) Time--er-output--oken (TPOT) and finett--iistttokenn (TTFT) on Nvidia A5000 GPU and Orin Nano 8G are measured milliseconds (ms) with one batch. TTFT is profiled with 1024 tokens. W4A8 and W4A16 dequantization overheads slow down TTFT compared to W8A8 and FP16. (OOM: out-of-memory)
Table 6: (Zero-shot evaluation.) We compare our fram- work with Quamba (Chiang et al., 2025) and MambaQuant (Xu el., 2025) on the average accuracy on ARC-E, ARC- PIQA, WinoGrande and HellaSwag.
Mambal 1.4B
Mamha2 2.7B
Bitwidth
Methods
2.8B
8B
A5000
Orin Nano 8G
Methods
Bitwidth
FP16
58866%
62.2%
62.4%
70.8%
TPOT
TTFT
TPOT
TTFT
Quamba
|57.3%
61.5%
57.3%
67.0%
FP16
22.73
197.80
00M
OOM
W8A8
MambaQuant
| 58.33%
62.1%
Quamba
W8A8
14.12
124.01
0OM
OOM
Quamba2 (Ours)
57.5%
61.8%
12.1% 69.9%
Quamba2
(Ours)
W8A8
W4A8
12.61
7.43
122.33
140.78
OOM
79.91
2088.03
OOM
W4A8
MambaQuant
54.3%
58.5%
W4A16
7.58
209.19
78.77
2316.23
Quamba2 (Ours)
56.7%
61.0%
61.4%
69.4%
W4A16
Quamba2 (Ours)
57.5%
61.9%
62.3%
70.2%
tasks. In the Quamba setting (Chiang et al., 2025), we applied the Hadamard transform to the output projection in- put and implemented percentile clipping on the input SSM, establishing our W8A8 Mamba2 baseline for latency and ac- curacy. We also provide the latency for W4A8 and 1.
Table 7:
(Five-shot evaluation of Quamba2-8B on MMLU.) We evaluate W4A8, W4A16, and [4A[X]- mixed on MMLU, large multitasking dataset. We set the A8:A16 ratio to 3 1 for the model and search the best precision configuration for each layer, which outperforms the handcrafted (HC) counterparts.
5.2. Latency and Model Size
We test all methods on the A5000 for cloud applications and on the Orin Nano 8G for edge applications. Timt-per-output- token (TPOT) and time-to-first first-tokenn (TTFT) are measured for batch size of one, recorded in milliseconds (ms). TTFT is profiled with 1024 input tokens. The results are shown in Table 5 and Figure 1. In the W8A8 setting, head-to-toe quantization of our framework improves the TPOT latency for Mamba2-88 --B by 1.80x (22.73 ms vs. 12.61 ms). outper- forming Quamba 1.61 x (Chiang et al., 2025) (22.73 ms vs. 14.12 12 ms). In the W4A8 configuration, Quamba2 achieves 3.89x less memory use, 1.39x prefilling, and 3.05x faster generation speed for Mamba2-8B on A5000. W4A8 and W4A16 dequantization overheads slow down TTFT com- pared to W8A8 and FP16. However, the 4-bit weights bring latency benefits in memory-bound generation stage. Our approach allows Mamba2-8B deployment on Nano 8G with a speed of generating 13 tokens per second, whereas FP16 and W8A8 fail, as illustrated in Figure and Table 5. For the SSD kernel, we load the 8-bit activations (z, A, B. C, 2) to reduce memory pressure and improve the latency by 1.18x, shown in Table 3. The bit- witth configurations with their practical latency and memory reduction enable the deployment on wide range of devices and applications.
LAMB (0-shot)
MMLU (5-shot)
W4A(X) (A8:A16)
Bitwidth
Method
TIFT
FP16
70.9%
47.0%
197.80
W4A8
68.8%
41.2%
56:0
140.78
W4A16
70.6%
45.3%
0:56
209.19
Mixed
HC-last HC-first
68.3%
42.1%
42:14 14
Mixed
68.9
43.1%
42:14
158.36
Mixed
Auto
69.1%
44.0%
42:14
Mambal (Gu & Dao, 2024). In contrast to Quamba (Chiang et al., 2025), when applied to Mambal, our approach uti- lizes Hadamard transforms on input and output projections to increase quantization precision, thus enhancing accu- racy for Mambal. As illustrated in Table 6, our techniques sort-und-claster and per-state- clipping in Mamba2 (Dao & Gu, 2024). Our framework performs head-to-toe quantization, outperforming Quamba in latency and memory usage (refer to Table 5) for both W8A8 Mambal and Mamba2. Quamba2 also outperforms MambaQuant in W4A8 Mambal and delivers real speedup computing platforms. Moreover, our framework supports W8A8, W4A8, and W4A16 precisions for both Mambal and Mamba2 with satisfactory accuracy and latency.
quantization surpass
5.3. Zero-shot Evaluation on Downstream Tasks
5.4. Evaluation on Large Multitasking Dataset
We present the average accuracy for Quamba2 over five datasets: ARC-easy, ARC-challenge, PIQA, WinoGrande, and HellaSwag. allowing a fair comparison with Mam- baQuant (Xu et al., 2025) !. The full evaluation is in the Appendix, where we follow the evaluation protocol in
We evaluate 14 and W4A8 Quamba2-8B in the MMLU dataset (Hendrycks et al., 2020), a large multitasking dataset, covering 57 subject ranges at different difficulty levels. Our study shows that previous quantization methods may over- look the generalizability of now-bit-widd models. W4A8
7

---

Quamba2
Table 8: (Ablation study on W4A8 Quamba2-8B.) The accuracy on Lambada dataset is reported. (PerSG: per-state- group quantization for B and C, SnC: sort-and-cluste for æ, PerG: per-group weight quantization, GPTQ: Frantar al. (2023), and Had: Hadamard transforms)
Table 9: (Ablation study on W4A16 Quamba2-8B.) The accuracy on Lambada dataset is reported. The Hadamard transform effectively reduce the quantization error from weights. (PerG: per-group quantization, GPTQ: Fran- al. (2023), and Had: Hadamard transforms)
Weights
B/C
Weights PerG GPTQ
Size
Bitwidth
Had.
Ace.
Size
Bitwidth
Had.
Acc.
PerG GPTQ
PerSG
SnC
FP16
71.2%
FP16
71.2%
V
fail
v
64.7%
V
53.8%
8B
4
[9.66%
8B
W4A16
W4A8
V
/
v V
55.1
69.2%
V
60.7%
71..2%
V
V
68.8%
Table 10: (Ablation study on W4A8 Quamba2.) We experiment on quantizing the embedding and output head in addition W4A8 blocks. The accuracy on Lambada dataset is reported.
strikes a balance between prefilling and generation speed but falls short in MMLU generalization, whereas W4A 16 main- tains a better generalization despite an increased prefilling latency, as shown in Table 7. We handcraft two mixed- precision models that replace the last 14 layers and the first 14 layers with W4A16 denoted as HC-last and HC-first in the table, respectively. However, they show marginal im- provement on MMLU dataset. To this end, we employ an evolutionary search approach to find the layer-wise preci- sions. The resulting sexxed-reeision model mitigates the loss of generalizability (+2.9%) on the MMLU dataset.
W4A8 blocks
4-bit Im_dead
+ 4-i embed.
size
FP16
+ both
130M
43.7%
37.6%
37.0%
33.4%
33.4%
370M
53.1%
50.5%
50.3%
46.2%
46.6%
2.7B
69.5%
65.8%
66.1%
66.0%
65.7%
8B
70.9%
68.5%
68.3%
69.0%
68.8%
6.3. Quantizing the embedding and output head
6. Ablation Studies
In Table 10, we perform an analysis of quantizing the embed- ding and output head in addition to W4A8 blocks. Larger models show more resilience to quantizing both the em- bedding layer and the output head, as the accuracy on the Lambada dataset remains nearly unchanged. This finding is particularly useful for deploying large models to edge de- vices with limited memory budgets, as they are more robust to head-to-toc quantization.
6.1. Ablation study on W4A8
We conduct an ablation study on the W4A8 Quamba2-8B in Table 8. In the W4A8 setting, it is essential to apply the Hadamard transform to the input of the output projection. Due to the sensitivity of the SSM to countization-inducee errors, even with per-group quantization and GPTQ (Frantar et al., 2023) (second-order information) applied on top of the Hadamard transform, the results remain unsatisfactory. Our methods per-state-grous quantization (PerSG) and sort- and-cluster (SnC) address this issue in SSMs by quantizing the xr B, and C' in 8 bits with minimal accuracy drop.
7. Conclusion
We introduce Quamba2, a robust post-training quantization framework tailored for selective State Space Models, com- patible with W4A8, W4A16, and W8A8 on Mambal and Mamba2. Using channel order preservation and activation persistence observed in SSMs, we propose sort-and-cluss and por-state-group quantization techniques for 8-bit activa- tion quantization. Experiments demonstrate that Quamba2 surpasses previous methods, offering significant reductions in latency and memory for both cloud and edge applications. addressing deployment challenges for emerging SSM-based applications on various platforms.
6.2. Ablation study on weight-only quantization
We study the impact of each component in the case of W4A16 Quamba2-8B in Table 9. Our analysis indicates that the use of the Hadamard transform in the input of the out projection crucial to narrowing the performance gap in W4A16. The table demonstrates that the Hadamard trans- form combined with per-group weight quantization yields greater accuracy than GPTQ (Frantar et al., 2023). Specif- ically, the Hadamard transform eliminates outliers in half- precision activations, thereby avoiding the amplification of guantization errors from 4-bit weights by large outliers.
8

---

Quamba2
Impact Statement
question answering? try are, the A12 reasoning challenge. CoRR, 2018.
This paper aims to enhance the efficiency machine learn- ing and expand the accessibility of large language models. We find that the accuracy degradation not negligible. De- spite this, the performance trade-off is acceptable given the significant improvements in latency and resource efficiency. Our work enables large language models to be deployed on resource-limited devices. As positive feature, our method may push the development of privacy-centric on-device ap- plications, where sensitive data can be processed locally without relying on cloud services. However, our work may also present challenges such as increased device resource us- age and potential security vulnerabilities if the local devices are compromised.
Dao, T. Causal depthwise convid in cuda with py- torch interface, 2024a. URL Dao ATLab/cause coovid.
Dao, T. Fast hadamard transform in cuda, with py- torch interface, 2024b. URLntttss://githbb com/
Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 O:: 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems (NeurIPS), 15:30318-30033 2022.
Acknowledgments
This work was supported in part by the ONR Minerva pro- gram, NSF CCF Grant No. 2107085, iMAGiNE the Intel- ligent Machine Engineering Consortium UT Austin, UT Cockrell School of Engineering Doctoral Fellowships, NSF CAREER Grant No. 2339084, and Taiwan's 's NSTC Grant No. 111-2221-B-A499 148-
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Dong, Z., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. Hawq: Hessian aware quantization of neural networks with mixed-precision. In ICCV, 2019.
References
Ainslie, J., Lee- Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
Frantar, E., Ashkboos. S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. International Conference on Learn- ing Representations (ICLR), 2023.
Ashkboos, S., Croci, M. L., Nascimento, M. G. d., Hoefler, T., and Hensman, J. Slicegpt: Compress large language models by deleting rows and columns. In International Conference on Learning Representations (ICLR), 2024a.
Frantar, E., Castro, R. L.,, Chen, J., Hoefler, T., and Alis- tarh, D. Marlin: Hixed-precisio auto-regressive paral- lel inference on large language models. arXiv preprint arXiv:2408. 1144008.77333 11733, 2024.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated lms.. Advances in Neural Information Processing Systems (NeurIPS), 2024b.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.000277. 2021.
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourd AAAI Conference on Artificial Intelligence (AAAI), 2020.
Gao, L. Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L.,, Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 2023. URL attps:://enoss org weeeorrss1025555
Chiang, H.-Y., Chang, C.-C., Frumkin, N., Wu, K.-C., and Marculescu, D. Quamba: A post-training quantization recipe for selective state space models. In International Conference on Learning Representations (ICLR), 2025.
Goel, K., Gu, A., Donahue, C., and Ré, C. It's raw! au- dio generation with state-space models. In International Conference on Machine Learning (ICML), 2022.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved

---

Quamba2
Gong, R., Yong, Y., Gu, S., Huang, Y., Lv, C., Zhang, Y., Tao, D., and Liu, X. Llme: Benchmarking large language model quantization with versatile compression toolkit. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.
Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems (MLSYS), 2024a.
Lin, Y., Tang, H., Yang, S., Zhang, Z., Xiao, G., Gan, C., and Han, S. Qserve: W4a8kv4 quantization and sys- tem co-design for efficient I1m serving. arXiv preprint arXiv:2405 04532, 2024b.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788. 2024.
Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q. and Liu, Y. Vmamba: Visual state space model. In Advances in neural information processing systems (NeurIPS), 2024a.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. In Conference Language Modeling (COLM), 2024.
Liu, Z., Oguz, B., Zhao, C., Chang, E.,, Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V. LIm-qat: Data-free quantization aware training for large language models. The 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024b.
Gu, A., Dao, T., Ermon, S., Rudra, A., and Ré, C. Hippo: Recurrent memory with optimal polynomial projections. In Advances in neural information processing systems (NeurIPS), 2020.
Guo, 7., Zhang. X., Mu, H.,, Heng, W., Liu, Z., Wei, Y., and Sun, Single path one-shot neural architecture search with uniform sampling. In The European Conference Computer Vision (ECCV), 2020.
Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant-IIn quantization with learned rotations. arXiv preprint arXiv:2405.164000 2024c.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M.. Song, D., and Steinhardt, J. Mcasuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.00 03300, 2020.
LY, B. L. Cuda hgemm, 2024a. URL httpt::/tggieeeeee com/ BeaaBBuuce-Lee-e LY/ccda aadd__heem.,
LY, B. L. Cuda hgemv, 2024b. URL httpts://gtthbbb
com/ Bet///ruueeeeee Leeeee-Le-LYYY Luduuaeeeem
Kumar, T., Ankner, Z., Spector, B. F., Bordelon, B., Muen- nighoff, N., Paul, M., Pehlevan, C., Ré, C., and Raghu- nathan, A. Scaling laws for precision. In International Conference Learning Representations (ICLR), 2025.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computa- tional Linguistics, (ACL). The Association for Computer Linguistics, 2016.
Kwon, W. Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, E., Zhang, H., and Stoica, I. Ef- ficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Princi- ples (SOSP), 2023.
Rajpurkar, P., Jia, R., and Liang, Know what you don't know: Unanswerable questions for squad. arXiv preprint 10X6v:880.003880 2018.
Lee, K., Chang, M.-W., and Toutanova, K. Latent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv: 003000 2019.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. In The Thirly-Fourth AAAI Conference on Artifi- cial Intelligence (AAAI), 2020.
Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., and Qiao, Y. Videomamba: State space model for efficient video understanding. In European Conference on Com- puter Vision (ECCV), 2025.
Saon, G., Gupta, A., and Cui, X. Diagonal state space aug- mented transformers for speech recognition. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP),20S)
Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi- gos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev- Shwartz, S., al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv: 19887, 2024.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. arXiv preprint 1909: 2019.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M,, Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, Awq:
10

---

Quamba2
Sloane, N. J. A library of hadamard matrices, 1999. URL http:/ //wp:/wwww.. nei 1 loane com/ hadamard/ weed//indx..iie hnmee. html.
Yu, Z.., Kojima, T., Matsuo, Y., and Iwasawa, Y. Slender- mamba: Fully quantized mamba in 1.58 bits from head to toe. In Proceedings of the 31st International Conference on Computational Linguistics (COLING), 2025.
Smith, J. T., Warrington, A., and Linderman, S. Simplified state space layers for sequence modeling. In International Conference on Learning Representations (ICLR), 2023.
Yuan, Z., Niu, L., Liu, J., Liu, W., Wang, X., Shang, Y., Sun, G., Wu, Q., Wu, J., and Wu, B. Rptq: Reorder- based post-training quantization for large language mod- els. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Tang, S., Ma, L., Li, H., Sun, M., and Shen, Z. Bi-mamba: Towards accurate 1 1-bit state space models. arXiv preprint arXiv:2411.11843, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? In Korhonen, A., Traum, D. R., and Mårquez, L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL), 2019.
Team, J., Lenz, B., Arazi, A., Bergman, A., Manevich, A., Peleg, B., Aviram, B., Almagor, C., Fridman, C., Padnos, D., et al. Jamba-1.5. 1.5: Hybrid ceaefformer-mmmbbe models at scale. arXiv preprint arXiv:2408. 12570, 2024.
Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Qiao, F,, Majcher, P., Springer, P.. Hohnerbach, M., Wang, J., and Gupta, M. CUTLASS, January 2023. URL https: //tp://ggtttt//// com/ com/NVIDIAA /W/.IIVIIIIIAAAA cutlass.
Zhang, Y., Zhang, P., Huang, M., Xiang, J., Wang, Y., Wang, C., Zhang, Y., Yu, L., Liu, C., and Lin, W. Qqq: Qual- ity quattuor-bit quantization for large language models. arXiv preprint arXiv:2406. 09904, 2024.
Zhao, C., Hua, T., Shen, Y., Lou, Q., and Jin, H. Auto- matic mixed-precision quantization search of bert. arXiv preprint 1iti:22112.149388 2021.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Zhao, J.. Lu, W., Wang, S., Kong. L., and Wu, C. Qspec: Speculative decoding with complementary quantization schemes. arXiv preprint arXiv:24100 2.. 11305, 2024a.
Zhao, Y., Lin, C.-Y., Zhu, K., Ye, Z., Chen, L.,, Zheng, S., Ceze, L.,, Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate Ilm serving. Proceedings of Machine Learning and Systems (MLSYS), 2024b.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Waleffe, R., Byeon, W., Riach, D., Norick, B., Kor- thikanti, V., Dao, T., Gu, A., Hatamizadeh, A., Singh, S., Narayanan, D., et al. An empirical study of mamba- based language models. arXiv preprint 20Xiv:2406.07887, 2024.
Zhou, Z., Ning, X., Hong, K., Fu, T., Xu, J., Li, S., Lou, Y., Wang, L.., Yuan, Z., Li, X., et al. A survey on efficient inference for large language models. Transactions Machine Learning Research (TMLR), 2024.
Wang, J., Gangavarapu, T., Yan, J. N., and Rush, A. M. Mambabyte: foken-free selective state space model. In Conference on Language Modeling (COLM), 2024.
Zhu, L., Liao, B., Zhang, Q. Wang, X., Liu, W., and Wang, X. Vision mamba: Efficient visual representation learning with bidirectional state space model. In International Conference on Machine Learning (ICML), 2024a.
Wang, K., Liu, Z., Lin, Y., Lin, J., and Han, S. Haq: Hardware-awan aware automated quantization with mixed pre- cision. In CVPR, 2019.
Zhu, X., Li. J.. Liu, Y., Ma, C., and Wang, W. A survey on model compression for large language models. Trans- actions of the Association for Computational Linguistics (TACL), 2024b.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning (ICML), 2023.
Xu, Z., Yue, Y., Hu, X., Yuan, Z., Jiang, 7., Chen, Z., Yu, J., Xu, C., Zhou, S., and Yang, D. Mambaquant: Quantizing the mamba family with variance aligned rotation methods. In International Conference on Learning Representations (ICLR), 2025.

---

Quamba2
A. Full Results for Six Zero-shot Downstream Tasks
Table II, we follow the evaluation protocol Mamba (Gu & Dao, 2024), and report the accuracy for LAMBADA (Papemo et al., 2016), WinoGrande (Sakaguchi et al., 2020), PIQA (Bisk al., 2020) and ARC-easy (Clark al., 2018), and the accuracy normalized by the sequence length for HellaSwag (Zellers et al., 2019) and ARC-challenge (Clark et al., 2018). Given the slight variation in accuracy across runs, present the average accuracy frame work outperforms Quamba (Chiang et al., 2025) in the Mambal backbone, providing with more quantization flavors such as W8A8, W4A8 and W4A16 for different use cases. Our method also outperforms Quamba in the Mamba2 backbone, where we apply the clipping technique to Mamba2, by a large gap in the average accuracy.
five runs in each table. Our
Table 11: (Zero-shot accuracy.) We evaluate our framework on six common sense tasks and report the average of five runs. Our framework surpass previous baseline, Quamba (Chiang et al., 2025), in average accuracy on both Mambal and Mamba2 backbones, with supporting more quantization flavors.
Model
Size
Methods
Bitwidth
LA
HS
PIQA
Arc-E
Arc-C
WG
Avg.
FP16
64.9%
59.1 %
74.2%
65.5%
32.8%
61.5%
59.7%
Quamba
W8A8
61.4%
58.3%
72.7%
64.0%
32.3%
58.8%
57.9%
1.4B
W8A8
62.3%
58.6%
73.1% %
64.0%
32.2%
58.5%
58.
Quamba2
W4A8
61.5%
57.6%
72.0%
63.0%
32.2%
58.7%
57.5%
(Ours)
W4A16
63.6%
58.1% %
72.6%
64.3%
32.4%
60.5%
58.5%
Mamba
FP16
69.1%
65.9%
75.6%
69.2%
35.8%
63.0%
63.1%
Quamba
W8A8
65.4%
65. %
74.2%
68.9%
35.9%
62.6%
62.0%
2.8B
W8A8
65.7%
65.4%
74.5%
68.9%
36.7%
61.8%
62.2%
Quamba2
W4A8
63.5%
64.9%
74.2%
68.2%
35.3%
62.2%
61.4%
(Ours)
W4A16
66.0%
65.3%
74.6%
69.2%
36.6%
63.6%
62.6%
FP16
65.6%
59.9%
73.3%
64.1% %
33.3%
60.8%
59.5%
Quamba
W8A8
49.8%
58.5%
71.2%
61.9%
32.1% %
58.1% %
55.2%
1.3B
W8A8
62.0%
59.2%
72.5%
63.4%
32.7%
60.0%
58.3%
Quamba2 (Ours)
W4A8
61.0%
58.8%
72.4%
62.7%
32.6%
59.1% %
57.7%
W4A16
64.3%
59.2%
72.6%
63.8%
33.1%
60.3%
58.9%
FP16
69.5%
66.6%
76.4%
69.5%
36.4%
64.2%
63.8%
Quamba
W8A8
52.4%
60.4%
71.6%
62.9%
33.7%
58.0%
56.5%
Mamba2
2.7B
W8A8
66.1%
65.5%
74.4%
68.4%
37.1% %
63.7%
62.5%
Quamba2
W4A8
65.6%
65.1%
74.7%
68.1%
36.1 %
62.8%
62, 1%
(Ours)
W4A16
68.8%
65.6%
75.5%
68.6%
36.6%
64.9%
63.3%
FP16
70.9%
77.7%
79.7%
76.0%
48.0%
72.0%
70.7%
Quamba
W8A8
54.0%
74.6%
77. %
73.5%
44.2%
65.5%
64.8%
8B
W8A8
69.8%
77.8%
79.1%
75.9%
46.9%
69.0%
69.8%
Quamba2 (Ours)
W4A8
68.8%
77.1%
79.1%
75.0%
46.0%
68.7%
69. 1%
W4A 16
71.2%
76.8%
79.1%
75.2%
45.9%
70.8%
69.8%
12

---

Quamba2
B. Evaluation Results on Generation Tasks
We evaluate Mamba2-8B with all bit-widths on the generation-base tasks Natural Questions (NQ) (exact match) (Lee al., 2019) and S2 (FI) (Rajpurkar al., 2018) on the open-source LM-EVAL (Gao et al., 2023). We show the results in Table 12. The W4A16 model closely matches the FP16 model, whereas the W4A8 and W8A8 mod- els, with 8-bit SSM states, preserve the meaningful generation outputs. We show that the searched W4AX also improves the generation scores and outperforms the W4A8 model. This result reveals an interesting observation that cached SSM states are redundant, which can care- fully quantized to 8 bits. Our framework supports 8-bit SSM states for W4A8 and W8A8 models and improves their generation speeds with large batch-size inputs, the cached states are the major memory and latency bottlenecks. Please refer to Section E for more details.
Table 12: (Generation tasks.) We evaluate
Mamba2-88 8B with different precisions on the generation tasks.
Bit-width
NQ
SquadV2
FPI6
17.2
51.9
W8A8
15.0
43.6
W4A8
14.2
45.9
W4A16
16.6
50.7
W4AX
14.9
47.4
C. Implementation and Evaluation Details of Quamba2 Framework
Quantization setup. The calibration set is constructed by randomly sampling 512 sentences from the Pile dataset (Gao al., 2021), where fixed the random seed in the sampling process. We collect the static scaling factors for each operator based on the absolute maximum value observed from the calibration set to quantize the activations and cached SSM states in both W4A8 and W8A8 settings. The same scaling factors applied in all our experiments.
Implementation. We implement our framework based on CUTLASS (Thakkar et al., 2023), vLLM (Kwon et al., 2023). Our 4-bit and 8-bit matrix multiplication (matmul) kernels adapted from (Xiao al., 2023; Frantar al., 2024; Zhang al., 2024; LY, 2024b;a). We implement W4A8 and W4A16 fused tramuul-ranssose kernels for the Mambal architecture. We apply GPTQ (Frantar et al., 2023) to the projection layers in the 4-bit weight settings. Quantization is integrated and adapted to the CUDA kernels of both the fast Hadamard transform (Dao, 2024b) and causal convolution (Dao, 2024a). Furthermore, the selective scan and SSD kernels (Gu & Dao, 2024; Dao & Gu, 2024) are modified to accommodate inputs with quantized weights, activations, and their scaling factors.
Latency and model size profiling. We evaluate all methods on the A5000, widely used GPU for Al workloads with 24GB of memory, emulating the setting for cloud applications. For edge applications, we profile all methods on the Nvidia Orin Nano 8G. We perform a few warm-up iterations and then report the average latency of the next 100 iterations. We report the size of the model that includes all quantized parameters and buffers for calculation.
D. Details for Mixed Precision Quamba2
In Table 7, outline the generalizability issue when utilizing the precision of W4A8 only. We show that our W4AX mixed-precision models mitigate accuracy degradation while incurring only marginal latency overhead. Figure 6 visualizes the detailed layer-wise bit-width configuration of Quamba2-889 11-22----4AX.-
The handerafted mixed-precision models. We explored types of handcrafted (HC) configurations, referred to as HC_first and HC_last, where we apply W4A16 blocks at the beginning and end of the network, respectively. Handcrafted configurations only deliver marginal improvements in the average accuracy (approximately 1 1% on MMLUU, and still fall behind in the upper bound scenario, where all blocks utilize the precision of W4A16, as shown in Table 7.
The automated W44.X models. We implement evolutionary search identify the best mix precision levels (Guo et al., 2020). We set the population size 40 and the number of generations to 5. In each generation, the top performing half of the candidates are retained, with 10 mutation and crossover operations applied, respectively, to generate new candidate precision configurations. The search algorithm identifies the sensitive blocks and assigns W4A16 to these blocks. This automated approach searches the best mix- mnix-precssoon configurations and balances between the precision and performance. Our W4AXX models addresses the performance gaps in the MMLU dataset, as shown in Table 7, compared to naive mixed peeeciiio and pure W4A8 models.
13

---

Quamba2
HC first
Auto
W4A16
W4A8
0
3
6
12
15
18
21
24
27
30
33
36
39
42
45
48
51
54
Layer Index
W4A16
W4A8
0
3
6
9
12
15
18
21
24
27
30
33
36
39
42
45
48
51
54
Layer Index
W4A16
W4A8
0
3
6
9
12
15
18
21
24
30 Layer Index
27
33
36
39
42
45
48
51
54
Figure 6: (The layer-wise bit-width for Shamba2-8B-W4AR We search the bit-width for Quamba2-8B- ---AAXX (the last row red), which outperforms the handcraft counterparts shown in the first (HC.first) and the second (HC last) rows.
Analysis on W4AX latency and accuracy trade-off. In Figure 7, we show our W4AX models outperform naive handcrafted models MMLU (Hendrycks et al., 2020) sive-shot accuracy, and place the Pareto-frontier of prefilling latency ttime-to-firs-tooknn TTFT) trade-off. In this experiment,
FP18
46
adjusted the ratios of W4A16 and W4A8 (e.g. 1:2) in Quamba2-8B and used evolution- search to find the mixed precision configuration. As shown the figure, the searched W4AX models in differ- ent ratios improve the accuracy the 5-shot evaluation on MMLU compared to W4A8, introducing marginal pre-filling latency overheads (i.e., 140.7 vs. 158.3 ms). Moreover, the automatic designed W4AX models by our search algorithm are above naive handcrafted W4AX models in accuracy. This finding highlights the challenges of designing mixed-precision models for SSMs, as well as the limits of generalization (Zhao et al., 2024a; Kumar al., 2025) of low-bit SSMs on large-scale datasets. We expect more advanced search algorithms address the generalization issue in the future.
MMLU 5-5hot Acc.
45
(1:21
WAA10
45 3
44.0
W4AX (Auto)
43
(1:4)
AuDo
UN:ffirstt (1:3)
W4AX (HIC)
42
42.1
W4A8
W4A16
41.2
TP16
WGAB
140
150
160
170
180
190
200
Time-to-first-token (msec.)
Figure 7: (Pareto front analysis for mixed-precision mod- els.) Our W4AX models by searched (Auto) outperform naive handcrafted (HC) models in MMLU accuracy and prefilling latency trade-off.
E. Investigating Memory and Latency with Large Batch Sizes
The cached state sizes. Although the constant state nature of SSMs, the cached states grow linearly with respect to the input batch size. We show theoretical memory brcakdowns versus batch size Figure (a). As the batch size increased, cached states occupied most of the total memory, making state loading and updating the bottleneck during generation. Our framework (W4A8) compresses and updates the states with 8-bit, thus decreasing overall memory usage and generation latency for cloud applications with large batch sizes.
14

---

Quamba2
embedding
conv state
Im_ head others
projection
5sm state
1.6
30
Nano
1.4
8-bit States (Ours)
25
1.2
Memory (GB)
FP16
Latency (ms)
20
1.0
15-
0.8
FP16 States
10
0.6
S-bit States
5
0.2
b=1
b=8
bw64
b-128
b
bag
bw64
bw128
b=1
beg
bw
1=
150 Batch Size
FP16
W4A16 (Ours)
W4A8 (Ours)
(a) Memory breakdown vs. batch size the generation stage
(b) State update latency vs. batch size
Figure 8: (Large batch inputs.) The cached states grow linearly with respect to the input batch size. For batch size of 128, half-precision cached states use most of the memory (a), making state loading and updating the bottleneck during generation. Our framework (W4A8) compresses the states to 8-bit, thereby reducing the total memory and generation latency (b) with large batch size inputs for bloudbbased applications.
Quantizing cached SSM states. We reduce generation latency by quantizing the cached SSM states to 8-bit for W4A8 and W8A8 models. Since the cached SSM states follow the head reordering and channel grouping indices from the SSM input x (ref. Figure 4), we apply the same m head and n channel groups to quantize each SSM state before caching them in memory. This finding eliminates the need for additional online reordering of SSM states and only requires calibrating the SSM quantization scales. Our approach introduces dstate m n floating-poie pronngg--onnttoo scales with minimal latency overhead, while significantly reducing the state update latency, as shown in Figure (b).
120
120
100
100
0.5
80
80
60
ostata
60
40
70
B B E 1 a
8 E E E 15
headdim
88
nhead
nhead
g8
(a) SSM states
(b) SSM states after weight re-reordering
(c) Apply grouping to SSM states
Figure 9: (SSM states.) The states are quantized before cached in memory. We apply the same m head and n channel groups from the SSM input æ to SSM states (b-c).
15

---

Quamba2
The roofline model. We show the roofline model of A5000 GPU in Figure 10 (w-bitXa-bll in the figure), and profile A5000 with different batch sizes in Table 13.
the generation latency (i.e, time-perrouuppu-tooen TPOT) of Mamba2-8B When the input batch size is small (e.g., b=1 in the table), the generation is memory-bound and therefore loading 4-bit weights (e.g.. W4A8 and W4A16) improves the roofline model. As the batch size increased (e.g., b=64 in the table), the W4A 16 models are bounded by hardware performance in terms of trillions of operations per second (TOPS). In contrast, the W4A8 and W8A8 models leverage 8-bit computation and deliver better TOPS. The ultimate TOPS of W4A8 is lower than W8A8 due to the extra steps for dequantizing weights from 4-bit to 8-bit (e.g. b=256 in the table). Our framework supports W8A8, W4A8, and W4A16 that are at the frontier of the roofline model satisfy the deployment needs of most applications for both Mambal and Mamba2.
Table 13: (Mamba2-8B time-per--uutputttoken latency on A5000 24GB.) We compress the cached SSM states with 8-bit, enabling larger batch size inputs under the same memory constraints. We report latency in millisec- onds (ms). OOM denotes out-of oft-oo-memory.
different bit-width)
Dequant Overhead
Performance (TOPS)
bsize=256
beize-84
beize=32
Dequant
Overhead
Bitwidth
b=1
b=32
b=64
b=128
b-256
INT8xINT8
INT4XINT8
FP16
22.73
35.74
49.63
OOM
OOM
haizees
FP16XFP16
INT4xFP16
W8A8
12.61
23.83
30.82
44.85
79.65
W4A8
7.43
15.05
24.65
44.54
85.26
Computation Intensity (Ops/Element)
W4A16
7.58
20.58
38.48
74.25
OOM
Figure 10: (Rooftine model of the generation.)
Batch size vs. time-to-last-token latency across bit-widths. Figure 11 shows the lo-st-o-lst- (TTLT) of Mamba2-8B quantized with different bit-widths (e.g., W8A8, W4A8, and W4A16) supported by our framework on a A5000. We vary the batch size of the input from 1 to 64, and profile the end-to-end latency of pre-fflling 2024 tokens and generating 2048 tokens (i.e., TTLT). The latency is estimated for the batch sizes that empirically do not fit A5000 and is represented with dashed lines with unfilled markers. We show that the W4A8 Mamba-8B model is suited for most latency- sennsivveee applications, serving with general batch sizes (i.e., range from to 64) on both cloud and edge devices. In contrast, W4A16 16 serves as a better option for personal applications (i.e., batch size equal to one) on mobile platforms as features higher average accuracy (ref. Table 11 and 7). For large batch size (i.e. greater than 128), the W8A8 model delivers the highest performance in terms of latency. Our framework supports all options on the frontier of the roofline model, as shown in Figure 10.
60
Batch Size
40
30
FP16
20
WBAB
W4A8
10
W4A16
OOM (Est.)
20
40
60
80
100
120
Time-to-last-token (sec.)
Figure 11: (Batch size vs. time-to-last-token)) W4A8 is suited for most applications serving with general batch sizes among all supported bit-widths.
F. Accuracy-latency Trade-off
Accuracy vs. latency across backbone models. Figure 12 illustrates the average accuracy across six zero-shot tasks (y-axis) versus latency (x-axis, in log-scale) on cloud-based A5000 GPU (a) and an Orin Nano 8G (b). We profile TTLT in seconds (sec.), with 2K imput tokens and 2K generated tokens the A5000 GPU. For the Orin Nano 8G, we profile the TTLT with prefilling of 512 input tokens and 512 generated tokens. For QuaRot (A et al., 2024b), we use the official implementation and profile latency for Llama2 (Touvron et al., 2023). We profile Llama3 (Grattafiori et al., 2024) and use the official implementation from (Lin et al., 2024b) to quantize it to W4A8KV4. We note that the latencies and memory denoted with dashed lines and circles are merely estimated. For example, FP16 Llama2 13B is too large for the A5000's 24GB GPU memory, and W4A4 Ilama2 13B also exceeds the capacity of Orina Nano. Quamba2
16

---

Quamba2
models the Pareto frontier and offer the best trade-off between average accuracy and latency, well smallest memory footprints, outperforming other low bit-width SSM and Transformer baselines.
Lama2 (FP16) ["buaroon al.]
Ulema3 (FP16) [Grettafion al.
75
75
Mambaa
136 Io0
Wamba2 BB (OOM)
Namba (FP16) Deo)
Avg. Acc. (%6
Quambeek BB
SE
Avg. Acc. (96
Utirn Jafl
Uaro
BB
Mamba2 (FP:6) [Dac Gu]
Liamae
(WGMKY4;
(OOM)
03
2.70
65
2 20
Uama3 [WaADKY4] (Lin et el.]
[Ch ang
1:38
Storage
1.38
Storage
Quamba2 (W4AS, Ours)
1140
1:40
53
Quambs
85
95
365
55
Quaroas
26
46
86
186
WBA8
W444
COM (Est.)
16
32
64
128
256
512
16
32
128
256
512
Time-to-last-token (sec.)
Time-to-lastttoken (sec.)
(a) A5000 (2k+2k)
(b) Orin Nano 8G (512+512)
Figure 12: (Pareto front analysis for accuracy latency.) Quamba2 models (green) on the Pareto front over other low wit-width SSM (red) and Transformer (purple) baselines, while also featuring lower memory footprints as evidenced in the size of the circle.
17


Reviews:
[{"review_id": "FUberObTUk", "reviewer": "Reviewer_zREv", "summary": "This work introduces Quamba2, a novel post-training quantization (PTQ) framework designed for State Space Models (SSMs), particularly the Mamba1 and Mamba2 architectures. The work addresses the challenge of efficiently scaling SSMs for deployment in cloud and edge computing environments by optimizing low-bit-width quantization techniques (W8A8, W4A8, and W4A16). Unlike previous methods that struggle with quantization-induced errors in SSMs, Quamba2 leverages channel order preservation and activation persistence to improve quantization accuracy. The framework employs a sort-and-cluster technique for input processing and per-state-group quantization for model parameters, ensuring computational consistency while minimizing accuracy loss. Experimental results demonstrate that Quamba2-8B achieves up to 3× generation speed-up, 4× memory reduction, and only a 1.6% average accuracy drop compared to full-precision models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. We address the reviewer’ concerns and responses below. References [1-3] are from the review.\n\n> The paper's novelty is somewhat incremental.\n\nOur SSM quantization framework **introduces novel observations**, **open-sources low bit-width kernels**, and **explores mixed-precision** strategies to improve the generalization, which are distinct from *prior SSM quantization* work [4, 5]. We summarize the key novelty and contributions of our work:\n- We identify three key properties in SSMs: channel order, channel persistence, and state persistence.\n- Based on these, we propose sort-and-cluster and per-state-group quantization, validated on large-scale datasets.\n- We explore  W4AX mixed-precision quantization to boost robustness and generalization of low-bit SSMs.\n- Our framework supports W4A8, W4A16, W4AX, and W8A8, enabling flexible deployment and speedups across platforms.\n\n> Outperform only two previous methods.\n\nTo the best of our knowledge, the most **up-to-date**, **peer-reviewed**, and **strongest** *SSM baselines* were included by the Jan 30 deadline. We compare against and outperform *two and only two* latest state-of-the-art *SSM quantization* methods, MambaQuant and Quamba, which were accepted to ICLR 2025 on Jan 22. We will change “several state-of-the-art” to “two latest state-of-the-art” in our manuscript.\n\n> Compared with 1-bit Bi-mamba\n\nWe compare our framework against 1-bit Bi-mamba on Mamba2 2.7B in terms of storage, performance, GPU hours, and cost versus the average accuracy of PIQA HS WG ARC-e ARC-c in Table R1. We estimate the A100 GPU hours for a fair comparison.\n\n|Table R1|Storage|Tokens|GPU Hrs|Cost|Avg Acc| \n|-|-|-|-|-|-|\n|Bi-mamba-2.7B-W1A16| 0.55 GB |105B|7822|$32070|52.68|\n|Quamba2-2.7B-W4A8|1.4 GB|0.18M|0.05|$0.205|61.36|\n\nAlthough QAT-based Bi-Mamba offers better storage efficiency, it requires several orders of magnitude more tokens, GPU hours, and cost (more than 100,000$\\times$) compared to our PTQ framework.\n\n> The paper doesn't clearly specify the batch sizes for all accuracy evaluations. Statistical significance testing is absent.\n\nWe use a batch size of 16, a fixed random seed, and report the average accuracy over five runs (Section 5.1) for all experiments. We include standard deviation in Table R2 for Quamba2-8B-W4A8. Notably, our method outperforms all baselines on Mamba2 beyond the reported deviations.\n\n|Table R2|LA| HS|PIQA|Arc-E|Arc-C|WG|Avg|\n|-|-|-|-|-|-|-|-|\n|Quamba2-8B-W4A8|68.6±0.10|77.1±0.23|79.1±0.21|75.0±0.24|46.0±0.39|68.7±0.28|69.1±0.17|\n\n> The ablation studies focus primarily on the 8B model.\n\nWe conduct the same ablation study for Quamba2-2.7B-W4A8, and show the results in Table R3. We note that the group sizes of B/C for 2.7B are one, making it equivalent to per-tensor quantization. Our framework in the last row outperforms all other settings.\n\n|Table R3|Weights||Had.|B/C|x|Acc.|\n|-|-|-|-|-|-|-|\n||PerG|GPTQ||PerSG|SnC||\n|FP16|-|-|-|-|-|69.5%|\n|W4A8|✓|||-||fail|\n||✓||✓| - ||39.8% |\n||✓|✓|✓|-||51.2%|\n|W4A8 (Ours) |✓|✓|✓|-|✓|65.6%|\n\n> Quamba2 better than non-SSM quantized models on edge?\n\nWe include the 4-bit Llama-3-8B [7] in Table R4 for reference. Due to time constraints, we report only the Time-To-Last-Token (TTLT) in seconds on an A5000 GPU, using 2K input tokens and 2K generated tokens, along with the average accuracy across six zero-shot tasks. We will include latency results on the Nano 8G in the next version of our manuscript.\n\n|Table R4|Bit-width|Avg Acc|TTLT (2k+2k)|Memory|\n|-|-|-|-|-|\n|Llama-3-8B|FP16|70.4%|48.52|15.4G|\n|Llama-3-8B-QServe|W4A8KV4|69.0%|24.63|5.7G|\n|Mamba2-8B|FP16|70.7%|47.3|15.7G|\n|Quamba2-8B (ours)|W4A8|69.1%|15.4|4.0G|\n\n> Explain the search space and constraints. The 3:1 ratio of W4A8/A16 seems predetermined.\n\nOur search space consists of N^2 configurations, where N denotes the number of layers. We fix the W4A8/A16 ratio and search for the precision for each layer to achieve the best accuracy. We will include more configurations in our manuscript.\n\n> Evolutionary search for mixed-precision seen in other domains. Cite relevant on hardware-aware NAS targeting bit-width selection.\n\nOur work to address the performance gap for low bit-width SSMs on large datasets by mixed-precision with generic speedups, which are different from CNNs [2, 3], Transformers [6], and prior SSM quantization work [4, 5]. We will cite and discuss the relevant work in our final version. \n\n[4] Xu, et al. \"MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods.\" ICLR (2025).\n\n[5] Chiang, et al. \"Quamba: A Post-Training Quantization Recipe for Selective State Space Models.\" ICLR (2025).\n\n[6] Zhao, et al. \"Automatic mixed-precision quantization search of bert.\" IJCAI (2021).\n\n[7] Lin, et al. \"QServe: W4A8KV4 Quantization and SystemCo-design for Efficient LLM Serving.\" MLSYS (2025).", "claims_and_evidence": "While the claims regarding the technicality and results are well-supported, some claims remain problematic in my opinion.\n\n1. The paper claims Quamba2 outperforms \"several state-of-the-art SSM quantization methods,\" but there are comparisons to only two previous methods (MambaQuant and Quamba).\n\n2. While the paper demonstrates deployment on Nvidia Nano 8G, it is not clear whether Quamba2 can enable real-world performance for edge applications better than non-SSM quantized models.\n\n3. The paper mentions evolutionary search for mixed precision configurations but doesn't thoroughly explain the search space and constraints", "methods_and_evaluation_criteria": "Mostly yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "While the experimental design is generally sound and appropriate for the claims made, it would be better if the authors can add more statistical rigor, and clearly report the experimental parameters.\n\n1. The paper doesn't clearly specify the batch sizes for all accuracy evaluations, which could affect performance metrics. Statistical significance testing is absent when comparing different methods, making it difficult to determine if observed differences are meaningful or just noise.\n\n2. The ablation studies focus primarily on the 8B model; additional ablations on smaller models would strengthen generalizability claims.\n\n3. The 3:1 ratio of W4A8:W4A16 seems predetermined rather than being a result of optimization", "relation_to_broader_scientific_literature": "The paper's novelty is somewhat incremental. While Quamba2 introduces techniques like sort-and-cluster and per-state-group quantization specifically for SSMs, these methods largely adapt existing quantization approaches to the unique properties of State Space Models. The core insights about channel persistence and state persistence in SSMs are intriguing, but the paper primarily builds upon and extends two recent works (MambaQuant and Quamba) rather than presenting fundamentally new quantization paradigms. Core approaches like weight grouping, per-channel/group quantization scaling, Hadamard transforms to smooth activation distributions, and mixed-precision configurations are all established techniques in the Transformer quantization literature. The mixed-precision configuration approach, while practical, employs standard evolutionary search methods seen in other domains.", "essential_references_not_discussed": "1. This paper does not cite some of the relevant literature in quantization of Mamba models (when there are not a lot of works yet on this topic). For example, [1] explores binary (1-bit) quantization for state space models and addresses some of the same challenges with linear recurrence sensitivity. While more extreme than Quamba2's focus, this work provides important context on the quantization limits of SSMs, and it would be important to compare the performance-efficiency trade-off with this work.\n\n2. The paper uses evolutionary search for mixed precision but does not cite relevant prior work on hardware-aware neural architecture search that specifically targets bit-width selection, such as HAQ [2] or HAWQ [3], which pioneered similar approaches for Transformer models.\n\n[1] Tang, S., Ma, L., Li, H., Sun, M., & Shen, Z. (2024). Bi-mamba: Towards accurate 1-bit state space models. arXiv preprint arXiv:2411.11843.\n\n[2] Wang, K., Liu, Z., Lin, Y., Lin, J., & Han, S. (2019). HAQ: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).\n\n[3] Dong, Z., Yao, Z., Gholami, A., Mahoney, M. W., & Keutzer, K. (2019). HAWQ: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).", "other_strengths_and_weaknesses": "Please see the points above.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please clarify the novelty and the experimental design choices as explained above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NCRdGsvc3l", "reviewer": "Reviewer_hMgy", "summary": "This paper proposed a quantization scheme for SSMs such as mamba2. By leveraging the channel order preserving and activation persistence characteristics of SSMs, author mainly utilized two existing techniques, reordering and Hadamard rotation, to alleviate the quantization difficulty for SSMs and improved the resulting performance. Because not all the additional quantization-related components are done by offline processing, author also profiled the model inference and verified real end-to-end speed-up. Furthermore, author discussed the speed/accuracy trade-off between different quantization configurations, e.g. W8A8, W4A8, W4A16, and mixed precisions. Lastly, author demonstrated that for model size greater than 2.7B, quantizing embeddings and lm_head to 4bit has almost no accuracy penalty, so it can be used to further reduce model size for deployment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review and positive feedback. We address the reviewer’s questions below.\n\n> a breakdown table for GPU hours.\n\nWe provide the detailed breakdown of GPU hours for Quamba2. We report the GPU hours on A5000 for offline clustering, scale calibration, quantization, and weight packing in Table R5.\n\n| Table R5 | Group & Reorder  | Scale Calib. | GPTQ | Weight Packing | Total (hr) |\n|-|:--:|:--:|:--:|:--:|:--:|\n| Quamba2-2.7B| 0.04  | 0.05  | 0.07     | 0.01   | 0.17 |\n| Quamba2-8B | 0.05  | 0.17  | 0.10     | 0.03   | 0.35 |\n\n> show the impact of SSD quantization on accuracy.\n\nWe show the impact of causal convolution and SSD scan on Mamba2-2.7B in Table R6. We quantize the weights to 4-bit with different activation bit-width settings, and report the accuracy on the Lambada dataset. We apply our quantization techniques to SSD inputs (B, C, x).\n\n| Table R6       | Inproj output | Conv output | SSD output + Had |Acc. |\n|:---:|:---:|:---:|:---:|:---:|\n| | (Conv input) | (SSD input) | (Outproj input)        |     |\n| | FP16  | FP16   | FP16      |  68.8%    |\n| | Int8     | FP16   | FP16   |  68.0%  |\n| | Int8     | Int8     | FP16   |  67.2%  |\n| | Int8   | Int8      | Int8      |  65.6%   |\n\n> The value of quantizing SSD scan and Conv layer is a bit unclear. \n\nThe SSD update is a latency and memory bottleneck during generation, especially as batch size increases, although the SSD scan remains minor during prefilling. We profile latency (µs) of the linear layer, causal convolution, and SSD update, along with memory usage (MB) during generation for Quamba2-8B W4A8 and W4A16. As shown in Table R7, cached SSM and convolutional states grow linearly with batch size, eventually exceeding model size and dominating both latency and memory in large-batch scenarios.\n\n| Table R7  |Model size (MB) | bsize | Linear (µs) | Conv (µs) | SSD (µs) | conv size (MB) | state size (MB) |\n|-----------|------------------|-------|-----------|-----------|-----------|------------|-------------|\n| Quamba2-8B-W4A8   | 4049            | 1     | 95.08   | 2.2       | 5.825    | 2.2        | 56          |\n|              |                     | 64   | 155.7   | 22.2     | **219.99**  | 140       | **3584**      |\n|              |                     | 128 | 254.37 | 53.2     | **466.65**  | 280       | **7168**      |\n| Quamba2-8B-W4A16 | 4056            | 1     | 96.9     | 1.79     | 6.22      | 4.4        | 112         |\n|             |                      | 64   | 170.59 | 25.2     | **397.84**  | 280       | **7168**      |\n|             |                      | 128 | 364.22 | 47.6     | **785.46**  | 560       | **14336**    |\n\nOur W4A8 models with 8-bit SSD inputs halve the cached state size and improve time-per-output-token (TPOT, i.e., generation) latency, as shown in Table R8. End-to-end TPOT latency is reported in milliseconds (ms), and OOM indicates an out-of-memory error.\n\n|Table R8| Bit-width| b=1  | b=32 | b=64 | b=128 | b=256 |\n|-|----------|------|------|------|-------|-------|\n|Mamba2-8B|FP16     | 22.73 | 35.74 | 49.63 | OOM   | OOM   |\n|Quamba2-8B| W8A8     | 12.61 | 23.83 | 30.82 | 44.85 | 79.65 |\n| | W4A8     | 7.43  | 15.05 | 24.65 | 44.54 | 85.26 |\n| | W4A16    | 7.58  | 20.58 | 38.48 | 74.25 | OOM   |\n\nThese justify our motivation of quantizing 8-bit SSD and the cached SSM states.\n\n\n> Experiments 5.1 \"input SSM\" refers to one, some, or all in (A, B, C, x)? Which percentile were used? explaining the motivation or impact would be nice.\n\nFor the Mamba2 baseline, we follow the setting in Quamba [5] and use their official implementation. Specifically, we apply the percentile clipping to the x input activation and per-tensor quantization for B and C for Mamba2 models. We list the clipping percentiles in Table R9.\n\n| Table R9 | x perc. |\n|--------|----------------|\n| 2.7B | 0.9995        |\n| 8B    | 0.999         | \n\nIn Table R10, we apply clipping to (B, C, x) for Quamba2-8B-W4A8, respectively, and compare to our proposed weight-reordering and clustering. We apply per-group GPTQ quantization to the weights and Hadamard transforms to the SSD output. We report the accuracy on the Lambada dataset.\n\n| Table R10 | Bit-width | B/C         | x           | Acc.     |\n|------------|------------|----------------|------------|-----------|\n|Mamba2-8B| FP16     | -                 | -            | 70.9%  |\n|Quamba2-8B| W4A8   |  Per-tensor | Clipping | 55.1% |\n| | W4A8 |  Clipping     | Clipping |  58.7% |\n| | W4A8 |  PerSG       | Clipping |  60.7% |\n| |  W4A8  |  PerSG       |   SnC     |  68.8% |\n\nWe sincerely thank you for your valuable input which prompted us to clarify the details of our work.\n\n[5] Chiang, et al. \"Quamba: A Post-Training Quantization Recipe for Selective State Space Models.\" ICLR (2025).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA, no theoretical derivations in this work.", "experimental_designs_or_analyses": "Yes, looks reasonable, no issues.", "relation_to_broader_scientific_literature": "improve upon previous SSM quantization works. doesn't seem to improve W8A8 accuracy compared to MambaQuant (Table 6) but this work enables 4bit options so that it can improve the generation speed of memory-bound cases (Table 5).", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength\n1. well written and easy to follow.\n2. covered most of the important quantization considerations and demonstrated real speed-up using custom kernels.\n3. promised to open source kernels.\n\nWeakness\n1. Even though most of the additional processing are offline, some of the steps sound a bit time-consuming than others, e.g. clustering. It would be beneficial to give reader a rough idea and a few examples of the GPU hours of the key steps, maybe a break down table similar to Table 8 or 9.\n\n2. The value of quantizing SSD scan and Conv layer is a bit unclear. Author mentioned the purpose to quantize SSD to 8bit is to reduce memory pressure. But general understanding is that SSD and Conv are very light in computation and constant memory with respect to sequence length, unlike KV cache. In fact, Table 3 shows that with input seq_len=1024, bs=8, INT8 SSD only saves ~0.5 msec out of ~3msec. Compared to Table 5 input seq_len=1024 bs=1, time-to-first-token for INT8 case is ~122 msec. The saving seems negligible. Maybe the author could elaborate more on SSD quantization and provides some justifications for doing so. And it would be helpful to add this option to one of the ablation tables (8 to 10) and show the impact of SSD quantization on accuracy. \n\n3. Experiments 5.1 Line 346, \"...percentile clipping on the input SSM, ...\" may need to be a little more specific. For example, \"input SSM\" refers to one, some, or all in (A, B, C, x)? which percentile were used? and maybe a simple sentence or two to explain the motivation or impact would be nice.", "other_comments_or_suggestions": "NA", "questions_for_authors": "please see \"Weakness\" above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O8g5kPcKB8", "reviewer": "Reviewer_pT2i", "summary": "The paper introduces Quamba2, a robust and scalable post-training quantization framework tailored for Selective State Space Models (SSMs), specifically Mamba1 and Mamba2. Quamba2 leverages structural properties unique to SSMs, such as channel order preservation and activation persistence, through novel techniques like sort-and-cluster and per-state-group quantization, significantly enhancing quantization accuracy. The framework supports various bit-width configurations, achieving notable speedups and a 4× reduction in memory usage, with minimal accuracy loss. Extensive experiments demonstrate Quamba2's effectiveness, outperforming state-of-the-art approaches in both speed and accuracy, while maintaining generalizability across diverse tasks, including the challenging MMLU dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive feedback and respond to the concerns as follows.\n\n> Limited evaluation of generalizability, primarily focused on MMLU.\n\nWe evaluate our framework on more tasks to show that the W4AX improves generalizability in Table R11. We include BoolQ (accuracy), and the generation-based tasks Natural Questions (NQ) (exact match) and SquadV2 (F1).\n\n| Table R11 | Precision | BoolQ | NQ   | Squadv2 |\n|---------------|---------------|--------|-------|-----------|\n| Mamba2   | FP16        | 76.5  | 17.2 | 51.9    |\n| Quamba2 | W8A8       | 68     | 15.0 | 43.6    |\n|                  | W4A8       | 64.6  | 14.2 | 45.9    |\n|                  | W4A16     | 71.2  | 16.6 | 50.7    |\n|                  | W4AX       | 73.2  | 14.9 | 47.4   |\n\n\nWe sincerely appreciate the reviewer’spositive feedback and will include the additional evaluations in the final version.", "claims_and_evidence": "Yes, the claims made in the submission are generally well-supported by clear and convincing evidence. The paper provides comprehensive experimental results demonstrating significant speedups and memory reductions across various bit-width configurations (W8A8, W4A8, W4A16), comparing directly to relevant baselines such as MambaQuant and Quamba, thus adequately supporting claims of superiority. Key conceptual contributions, including \"sort-and-cluster\" and \"per-state-group quantization,\" are clearly illustrated through ablation studies, justifying their effectiveness in addressing quantization-induced errors in Selective SSMs.", "methods_and_evaluation_criteria": "Yes, the proposed claims, including improved accuracy, speedup, and memory reduction via techniques like sort-and-cluster, per-state-group quantization, and cluster-aware weight reordering, are clearly supported by detailed experiments on multiple benchmarks (LAMBADA, HellaSwag, PIQA, ARC, and WinoGrande), latency profiling, and memory comparisons.", "theoretical_claims": "The paper does not present explicit theoretical proofs requiring validation.", "experimental_designs_or_analyses": "Yes, the experiments are methodologically sound and comprehensive: the authors clearly justified the benchmark choices, carefully detailed their quantization methods, and presented extensive latency and accuracy comparisons. The ablation studies effectively validate individual contributions of their techniques, such as sort-and-cluster and per-state-group quantization, which clearly isolate and demonstrate their empirical benefits. \n\nHowever, there exists a limited diversity in datasets when evaluating mixed-precision robustness (primarily relying on MMLU), suggesting a broader evaluation could strengthen the robustness claims.", "relation_to_broader_scientific_literature": "The paper’s key contributions connect closely to prior literature on efficient neural network quantization and compression, especially works focusing on selective SSM. Quamba2 extends previous findings by introducing novel techniques, e.g., sort-and-cluster and per-state-group quantization informed by SSM-specific propertie: channel order preservation and activation persistence, to significantly reduce quantization errors. Also, the paper leverages established methods such as Hadamard transformations and weight reordering to further enhance quantization precision.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n- Creative combination of existing ideas, e.g., quantization, weight reordering with novel SSM-specific insights.\n- Strong practical significance for efficient deployment of large SSMs.\n- Rigorous and clear experimental design and ablation studies.\n\nWeaknesses:\n- Limited evaluation of generalizability, primarily focused on MMLU.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oV0ZBY09qS", "reviewer": "Reviewer_v4mg", "summary": "The paper introduces Quamba2, a PTQ framework for State Space Models, which aims to reduce model size and improve computational efficiency while maintaining performance. Quamba2 supports multiple bit-width configurations (W8A8, W4A8, and W4A16) for different deployment scenarios, such as cloud services and edge devices. The framework leverages an offline sort-and-cluster approach for quantizing inputs and per-state-group quantization for input-dependent parameters, resulting in significant speed-ups and memory reduction with minimal accuracy loss. The paper demonstrates Quamba2's effectiveness through extensive experiments and evaluations on various tasks and datasets, including MMLU.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and thoughtful questions. We address the reviewer’s  concerns and responses below.\n> While emphasizing speed-ups, the paper does not quantify the computational cost of offline weight reordering and clustering, which could affect deployment practicality.\n\nWe provide the detailed breakdown of the GPU hours on A5000 for our offline clustering, scale calibration, quantization, and weight packing in Table R5. \n\n| Table R5 | Group & Reorder  | Scale Calib. | GPTQ | Weight Packing | Total (hr) |\n|--------------|-------------------------|-----------------|-----------|----------------------|-------------|\n| Quamba2-2.7B        | 0.04                      | 0.05             | 0.07     | 0.01                  | 0.17        |\n| Quamba2-8B           | 0.05                      | 0.17             | 0.10     | 0.03                  | 0.35        |\n\nWe note that this is a one-time cost. The quantized model can be deployed to various devices and applications.\n\n\n> The evaluation focuses only on Mamba1/Mamba2. It remains unclear whether Quamba2 generalizes to other SSM architectures (e.g., S4, DSS, DenseMamba).\n\nWe apply our method to the latest Jamba-1.6-Mini, a large SSM-based language model with 52 billion parameters (12B active / 52B total), and show the result in Table R12. Following the official guidelines [8], we quantize the Transformer and MoE blocks with *bitsandbytes* [9, 10], while keeping the Mamba blocks in half-precision. Next, we apply different precision levels to the Mamba blocks using our framework. We observe that the accuracy degradation after quantizing the Mamba blocks to low bit-width is approximately 1%. This demonstrates that our quantization framework enables effective low-bit quantization for large-scale Mamba-Transformer hybrid models, addressing a key limitation of existing LLM quantization techniques.\n\n|Table R12                         | Transformer | MoE   | Mamba  | Avg Acc |\n|-----------------------------------|-----------------|----------|------------|--------|\n|Jamba-1.6-Mini                  | W8A8        | W8A8  | FP16     | 78.3%   |\n|Quamba2-Jamba-1.6-Mini | W8A8        | W8A8  | **W8A8**   | 77.2%    |\n|                                           | W8A8        | W8A8  | **W4A8**   | 77.3%    |\n|                                           | W8A8        | W8A8  | **W4A16** | 78.0%      |\n\n\n\n> Offline weight rearrangement may complicate integration into existing inference pipelines, especially for dynamic or frequently updated models.\n\nOur framework only increases the common GPTQ quantization pipeline by 0.05 GPU hours (~3 minutes) on A5000, while largely boosting the accuracy by 13.7% for Quamba2-8B-W4A8, as shown in Table R13. \n\n| Table R13 | Group & Reorder  | Scale Calib. | GPTQ | Weight Packing | Total (hr) | Acc. |\n|--------------|-------------------------|-----------------|-----------|----------------------|-------------|---------|\n| GPTQ     | -                            | 0.17             | 0.10     | 0.03                  | 0.30        | 55.1%|\n| Ours       | 0.05                      | 0.17             | 0.10     | 0.03                  | 0.35        | 68.8%|\n\nNotably, our framework also supports LoRA modules for various downstream applications. Once the trained LoRA weights are fused into the main model weights, the same weight reordering and quantization pipeline is reused and applied.\n\n[8] https://huggingface.co/ai21labs/AI21-Jamba-Mini-1.6\n\n[9] Dettmers, et al. \"LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale\"ArXiv (2022)\n\n[10] Dettmers, et al. \"8-bit Optimizers via Block-wise Quantization\" ICLR (2022)", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper propose a new method to solve the quantization problem in SSMs.", "essential_references_not_discussed": "\"Post-training quantization for vision transformer. Z Liu, Y Wang, K Han, W Zhang, S Ma, W Gao.\"", "other_strengths_and_weaknesses": "Strengths:\n\n1. Quamba2 supports multiple bit-width configurations (W8A8, W4A8, W4A16), addressing diverse deployment needs (e.g., cloud throughput vs. edge-device efficiency).\n2. The offline cluster-aware weight reordering and per-state-group quantization effectively mitigate SSMs’ sensitivity to quantization errors, improving precision without fine-tuning.\n\nWeaknesses:\n\n1. While emphasizing speed-ups, the paper does not quantify the computational cost of offline weight reordering and clustering, which could affect deployment practicality.\n2. The evaluation focuses only on Mamba1/Mamba2. It remains unclear whether Quamba2 generalizes to other SSM architectures (e.g., S4, DSS, DenseMamba).\n3. Offline weight rearrangement may complicate integration into existing inference pipelines, especially for dynamic or frequently updated models.", "other_comments_or_suggestions": "Please see the above comments.", "questions_for_authors": "Please see the above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors debut Quamba2, a new post-training quantization (PTQ) method for Mamba state-space models (SSMs).  Unlike recent Mamba PTQ methods which may struggle at W4A8 precision--MambaQuant (which applies applies the Karhunen-Loeve Transformation and equalizes SSM-channel variance) and Quamba (which performs symmetric uniform quantization)--Quamba2 introduces sort-and-cluster technique for input processing and per-state-group quantization to preserve accuracy at lower precisions.  The paper identifies and leverages the channel order preserving property and activation persistence of SSMs.  Across both Mamba-1 (1.4B, 2.8B) and Mamba-2 (2.7B, 8B) models, zero-shot (averaged across the standard datasets used in the Mamba-1 paper), 5-shot (MMLU), and several precisions (W8A8, W4A8, W4A16), the authors show the effectiveness of their approach; they remain competitive with MambaQuant at 8-bit precisions, while outperforming both MambaQuant at 4-bit precisions.  Additionally, they perform extensive ablations across Mamba-2 model sizes, showing language modeling degradation decreases as model sizes grow.  This an overall interesting study on PTQ for Mamba SSMs, I look forward to the ensuing code release.

Reviewer hMgy raised concerns with the compute overhead of the clustering step, but the authors provided additional experiments during the rebuttal demonstrating the modest runtime (in GPU hours) of each Quamba2 PTQ step for Mamba-2-2.7B and Mamba-2-8B models.  The authors successfully addressed the other reviewer concerns, pointing to the novelty of their approach compared to very recently published methods MamabaQuant and Quamba.
Decision: Accept (poster)