Quamba: A Post-Training Quantization Recipe for Selective State Space Models
Authors: Hung-Yueh Chiang
Chi-Chih Chang
Natalia Frumkin
Kai-Chiang Wu
Diana Marculescu
Venue: ICLR 2025 Poster (2025)
Keywords: State Space Models
Model quantization
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a 8-bit static quantization method for selective state space models.

Abstract: State Space Models (SSMs) have emerged as an appealing alternative to Transformers for large language models, achieving state-of-the-art accuracy with constant memory complexity which allows for holding longer context lengths than attention-based networks. The superior computational efficiency of SSMs in long sequence modeling positions them favorably over Transformers in many scenarios. However, improving the efficiency of SSMs on request-intensive cloud-serving and resource-limited edge applications is still a formidable task. SSM quantization is a possible solution to this problem, making SSMs more suitable for wide deployment, while still maintaining their accuracy. Quantization is a common technique to reduce the model size and to utilize the low bit-width acceleration features on modern computing units, yet existing quantization techniques are poorly suited for SSMs. Most notably, SSMs have highly sensitive feature maps within the selective scan mechanism (i.e., linear recurrence) and massive outliers in the output activations which are not present in the output of token-mixing in the self-attention modules. To address this issue, we propose a static 8-bit per-tensor SSM quantization method which suppresses the maximum values of the input activations to the selective SSM for finer quantization precision and quantizes the output activations in an outlier-free space with Hadamard transform. Our 8-bit weight-activation quantized Mamba 2.8B SSM benefits from hardware acceleration and achieves a 1.72 $\times$ lower generation latency on an Nvidia Orin Nano 8G, with only a 0.9\% drop in average accuracy on zero-shot tasks. When quantizing Jamba, a 52B parameter SSM-style language model, we observe only a $1\%$  drop in accuracy, demonstrating that our SSM quantization method is both effective and scalable for large language models, which require appropriate compression techniques for deployment. The experiments demonstrate the effectiveness and practical applicability of our approach for deploying SSM-based models of all sizes on both cloud and edge platforms.

Published as a conference paper at ICLR 2025
QUAMBA: A POST-TRAININ QUANTIZATION RECIPE FOR SELECTIVE STATE SPACE MODELS
Hlung-Yueh Chiang"), Chi-Chih Chang* Natalia Frumkin
Kai-Chiang Wu?, Diana Marculescu'
Chandra Family Department of Electrical and Computer Engineering,
The University of Texas at Austin
2 Department Computer Science, National Yang Ming Chiao Tung University
3 Department of Electrical and Computer Engineering. Cornell University
(hungyueh. wttpuuuhhhhhinnmem nfrumkin, dianam) @utexas.edu; cc2 86 00roee1lrnlll..mme edu, kcwêcs nycu. edu.tw
ABSTRACT
State Space Models (SSMs) have emerged as an appealing alternative to Transform- ers for large language models, achieving sfat-oof-the-ar accuracy with constant memory complexity which allows for holding longer context lengths than attention- based networks. The superior computational efficiency of SSMs in long sequence modeling positions them favorably over Transformers in many scenarios. However, improving the efficiency of SSMs on request-intensive cloud-serving and resource- limited edge applications is still a formidable task. SSM quantization is possible solution this problem, making SSMs more suitable for wide deployment, while still maintaining their accuracy. Quantization is a common technique to reduce the model size and to utilize the low bit-width acceleration features on modern computing units, yet existing quantization techniques are poorly suited for SSMs. Most notably, SSMs have highly sensitive feature maps within the selective scan mechanism (i.e., linear recurrence) and massive outliers in the output activations which are not present in the output of token- token-mixing in the self-attention modules. To address this issue, we propose a static 8-bit per-tensor SSM quantization method which suppresses the maximum values of the input activations to the selective SSM for finer quantization precision and quantizes the output activations in outlier-free space with Hadamard transform. Our 8-bit weight-activation quantized Mamba 2.8B SSM benefits from hardware acceleration and achieves 1.72 lower generation latency on an Nvidia Orin Nano 8G, with only a 0.9% drop in average accuracy on zero-shot tasks. When quantizing Jamba, 52B parameter SSM-style language model, we observe only a 1% drop in accuracy, demonstrating that our SSM quantization method both effective and scalable for large language models, which require appropriate compression techniques for deployment. The ex- periments demonstrate the effectiveness and practical applicability of our approach for deploying SSM-based models of all sizes on both cloud and edge platforms. Code is released at https:: :eap:s//giite ///git/ghttta com/ http:/enaaccgcc////00 caooup//uamba
INTRODUCTION
State Space Models (SSMs) Gu & Dao 2023 Lieber et al. 2024b) have attracted notable attention due their efficiency in long sequence modeling and comparable performance Transformers (Zhang et al.] 2022; Brown et al.) 2020). Although Transformers have shown strong ability to capture the causal relationships in long sequences, the self-attention module within Transformers incurs a quadratic computation complexity with respect to the context length in the prefilling stage as well as a linear memory complexity (e.g., the K-V cache) in the generation stage. In contrast,
Equua contribution.
1 The work was done National Yang Ming Chiao Tung University

---

Published as a conference paper at ICLR 2025
16
Pythie-2.06
x
Nano
Namba-2.98
Weight & cache size (GB)
14
Pyttre-2.00
60
Nembe-liad-2.00 0B
Uama-2-70 (W006BK0)))
Time to last token
64
Quamba-2.8
3.3x
12
Rambo 2.65
Quamba-2.8g 85
Avg. Accuracy
55
370M
32
20
1.48
2.7%
1.2x
50
16
FP16
2.6x
1.2x
45
Mamba Had
1.2x
50
1k
8k
1k
2k
16k
TILT Latency (s)
Sequence length
Sequence length
(a) Pareto-optimality on Nano
Long input sequences on A5000
Memory usage
We demonstrate that (a) method achieves Pareto-optimality on the Nano 8G with 1K input tokens. Figure (b) shows latency speedups for long input sequences on the A5000, and Figure (c) shows the memory usage across devices comparing to Pythia 2.8B (Biderman al., 2023) and 4-bit Llama-2-7B (Touvron et al. 2023).
Figure 1:
SSMs, an attractive substitute to Transformers, perform sequence modeling with a recurrent neural network-like (RNN-like) linear recurrence module (i.e., selective scan) which has linear computation complexity in the prefilling stage and constant memory complexity in the generation stage.
Despite their computational advantages, deploying SSMs diverse hardware is challenging due to memory and latency constraints. Quantization, such as 8-bit integer (INT8), offers a solution by reducing model size and latency while preserving accuracy and enabling hardware acceleration. Quantizing SSMs is non-trivial as current post-training quantization (PTQ) techniques for Transform- ers (Xiao al. 2023 Dettmers al. 2022) fail handle the sensitive activations of the selective scan (i.e., linear recurrence) resulting in poor performance (Lieber et al. 2024b Zhao et al. (024)). Our study shows SSMs exhibit distinct outlier patterns compared Transformers (Ashkboos et al.. 2024b) Dettmers et al. 2022] Xiao et al. 2023) Zhao et al., (202)). We show that outliers appear in the SSMs output (i.e. the tensor). contrast, the input and output self-attention layers are relatively smooth and do not exhibit outlier issues. More comparisons can be found in Section M This underscores the need for specialized quantization methods for SSM-based models, as their lack leads to suboptimal memory usage and latency on deployed hardware.
We first analyze the input and output activations of the selective scan module to reveal the quantization sensitivity and outliers in SSM activations (ref. Figure 2 and Figure Specifically, show that quantizing SSMs is particularly challenging since SSM input and output activations present a causal relationship, making the input tensor (i.e., in Eq. [
sensitive quantization errors. Additionally, representing the large outliers using 8-bit precision in the output activations (i.e., y in Eq. 1. which
SSM
y
-
MSE (step)
2.00
2.75
inputh
R-
2.00
Input
8
B-
Norm. MSE (sample)
0.25
Fjle] Duambs (Dun)
A
B
C
x
1.00
0.3
2.5
Self-attention
038
0.00
K
h
1008
Self attention
SSM Quamba
(Durs)
(a) SSM vs. Self-attention
(b) Quantizing self-attention
(c) Quantizing SSM
(d) Quantization errors
Figure 2: We analyze the sensitivity of quantization emors for (b) self-attention layers and (c) SSMs input activations. Our study shows that the æ tensor causes huge errors at the output due to causal relationship of the linear recurrence, which is unique to SSMs. Self-attention layers are more robust to quantization errors. Our method (d) reduces the quantization error for the input sample. In Figure T2 highlight the smooth, outlier, and sensitive paths in SSMs and self-ate aff-attention layers.
2

---

Published as a conference paper at ICLR 2025
not present in the token-mixing output of self-attention modules, is difficult. To address this, propose Quamba, an 8-bit static per-tensor quantization method for selective SSMs that can leverage the low bit-width acceleration features modern computing units with minimal overhead. We suppress maximum values in input activations to SSMs, which is the most sensitive to the quantization error, for finer quantization precision. For the extreme outliers in output activations from SSMs, we use the Hadamard transform to smooth out the activations. Our quantized 8-bit 2.8B Mamba SSM achieves x.72x speedup in generation latency (i.e., time-per-oupputttoken TPOT) on Nvidia Orin Nano 8G while only incurring 0.9% accuracy drop in zero-shot tasks. We demonstrate our method achieves -rtitaloptimllttyy on the Nano 8G and lower the latency by 1.2 on A5000, as shown in Figure T While quantizing Jamba, 52B parameter SSM-style language model, we observe just a ~ 1% reduction in accuracy. The effectiveness and scalability of our quantization technique for SSM-based models, as well as the practicality of our approach for deploying SSM-based models of various sizes on cloud and edge platforms.
RELATED WORK
Model quantization.
Deploying large, weer-parameterized neural networks on resource-constrained devices is challenging, and researchers have developed model quantization (Han et al. 2015) Jacob al. 2018 Wang et al., 2019) as solution to this problem. Quantization techniques reduce the
data type precision (e.g. FP32 to INT4) to compress the model size and accelerate inference. The
quantization techniques are generally divided into two categories: Quantization-awal training (QAT) and post-training quantization (PTQ) (Zhu et al. [2023] Gholami et al.] [022] (Zhou et al. (024)) QAT (Liu et al. 2023 Dettmers et al., 2024) requires additional training efforts to adapt models low bit-width, exchange for better model performance. Our work falls under PTQ, which does not require training and can therefore be plug-and-play.
LLM post-training quantization. Post-training quantization (PTQ) techniques are generally broken down into two categories: weight-only quantization (e.g.. W4A16 and W2A16) and weight- activation quantization (e.g., W8A8 and W4A4) (Zhu et al. 2023). Weight-only quantization (Frantar et al. 2022 Lin et al. 2023) focuses on quantizing weight matrices (e.g., 4-bit or 2-bit) while keeping the activations in half- half- prrccisoennn floating point. However, although weight-only quantization reduces memory usage, it still requires costly floating-poit arithmetic for compute-intense
operations
(e.g., linear layers). To utilize low bit-width operations, [iioo et al. (2023); Zhao al. (2023);
Ashkboos et al. (2024b); Dettmers et al. (2022) study quantization for both weights and activations in Transformers. They address outliers in activations by using mixed fixeedprecisioo (Dettmers al. 2022). rescaling quantization factors (Xiao et al.] 2023), group quantization (Zhao et al. [203]]. and quantizing activations in an outlier-free space (Ashkboos et al., (024b). Unfortunately. these techniques target Transformers, do not generalize well to SSMs, and either fail to handle the sensitive tensors SSMs resulting in poor performance (Xiao et al. 2023 Dettmers et al. (2022) or introduce additional computational overhead to the input of the selective scan (Zhao et al. 2023 Ashkboos jt al. 2024b). Our research addresses this gap by examining SSM weight-activation quantization, aiming concurrently reduce memory and compute costs by harnessing hardware acceleration for integer operations.
State Space Models. In recent times, new wave of RNN-like models (Gu al. 2021 Smith et al. 2022: Peng et al., 2023) Gu & Dao 2023 Beck et al., 2024) has emerged, noted for their efficacy in modeling long-range dependencies and achieving performance comparable Transformers. State Space Models (SSMs) (Gu et al. 2021 Smith et al. 2022 Gu & Dao 2023) are a promising class of architectures that have been successfully applied to various applications, such as text (Gu Dao
2023 Wang et al. 2024), image (Zhu et al. 2024 Nguyen et al. 2022), video (Li et al. |2244] Nguyen al. 2022). and audio (Goel et al. 2022; Saon et al. [202)) Despite their successes, the challenge of deploying SSMs across ussurce-limitee hardware platforms remains largely unresolved, Our work addresses this challenge by proposing a quantization method specifically tailored for SSMs.
BACKGROUND
3.1
SELECTIVE STATE SPACE MODELS
State Space Models.
Inspired by continuous systems, discrete linear time-invariant (LTI) SSMs 2021 2022) map input sequences to output sequences y. Given discrete input
(Gu al. 2020
3

---

Published as a conference paper at ICLR 2025
signal xy at time step t, the transformation 2e + y/: through hidden state h is defined
he Ahi-1 + Brg, Vt Che + Dz,
(1)
where and B are discrete parameters. The discretization function for and with a given A is defined as A = Ax)(AA), B = (A (Ax)) - 1) AB R AB. This system uses as state transition parameter and B and C' projection parameters. A is the time-scale parameter that is used to discretize the continuous parameters A and B. D is an optional residual parameter. (A, B, C, D, A) are trainable parameters. A residual branch 2n applied the SSM output such that (z, before the output projection.
SSMs with selection. Gu & Dao (2023) improve SSMs with selection by letting their parameters B, C. and be input-dependent, allowing the model to selectively remember or ignore inputs based on their content. Specificallyy the interaction with input 2; is defined as B, (x C2 Fe(e)), A, = softplus(F2, (2++) where F3 and Fe are linear layers that map It + Bt, C.. FA two consecutive projection layers, such that FA (roj(Projæz:) bias. With the selection mechanism, the model has changed from iimeiinvariant to time-varying.
3.2 QUANTIZATION
We focus symmetric uniform quantization to approximate floating-pood point weights and activations with discrete 8-bit signed integers (i.e., INT8) due to its hardware compatibility. The general symmetric uniform quantization function is defined as
X = clammp (
max (IxI) N1 2N -2-11-1) $ 2N-1 1
(2)
where X represents the quantized weights activations in INT8, X is the input matrix in floating point, and s is the scaling factor (i.e. quantization step) that is determined by the target bit-width N (N in our setting). The static scaling factor is pre-calibrated on subset of data and is fixed during inference. We use the notation X to represent the floating point matrices, and X to represent their quantized matrices with their floating- moiting-point scaling factors 8z. For operators, we use f(-) to represent the quantized version of the function f(-) (i.e., the weights are quantized in the function J).
3.3 WALSH-HADAMARD TRANSFORM
Hadamard matrices. A Hadamard matrix is n-dimensiinal square matrix whose entries either +1 or 1, and the rows and columns are mutually orthogonal with the computational property H,H,, nI.- Hadamadd matrix is special category of Hadamard matrix, consisting of square matrices of size 2* and can be constructed as follows:
H2 H
H
1
1 -1
H
H2
=
H
H where H2
Walsh-Hadamard transform. The Walsh-Hadamard transform (WHT), a generalized class of Fourier transforms, has been applied to many related areas, such as LLM quantization (Ashkboos) al. 20.244) and efficient transfer learning (Yang et al.) (2024). due its favorable computational properties. We perform WHT to remove outliers from the output of the selective SSM. WHT projects discrete input sequence signal onto a set of square waves (i.e., Walsh functions). Its forward and inverse transform can be expressed in matrix form as I H and = H a. is the input discrete sequence signal, and ă denotes the WHT coefficients (i.e., sequence components) that describe the magnitude of the square waves. WHT is efficient since the transform matrices consist only of real numbers, +1 or -1, so no multiplications are needed. The fast Haddmardd compute the transformation with complexity of nlogn in GPU-friendly
transform (FWHT) (i.e,, parallelizable) fashion [Doo] 2024b). For input dimension n * 2* we factorize n 2mm, where is the size of known Hadamard matrix (Sloane 1999).
4

---

Published as a conference paper at ICLR 2025
Easy to Quantize
Quart
Hard to Quantize
Easy to Quantize
Quanda
Small
X
AlC. Wale
1.20
Cause Huge
Huge outulers in
Transform
1.00
Errors in
8.79
0:25
Time
8.00
Uimestop
(a) SSM Inpur X
(b) SSM Output y1
Figure 3: The primary difficulties in quantizing Mamba blocks lie in the precision of the activations input into and output from the selective SSM. Although inputs are numerically small, the quantization step is skewed by the maximum value, causing significant errors in the output SSMs after the linear recurrent system. In contrast, large outliers are observed in the outputs. We use Hadamard matrices to transform the outputs to an outlier-free free space.
QUAMBA: QUANTIZING MAMBA BLOCKS
4.1 PRELIMINARY STUDY
Theoretical error bound for SSM quantization. We consider discrete linear time-invariant (LTI) state space model at time step i defined as h(t)) = A(t)h(t - 1) + Bx((),, where the state matrix A(L) E RNXN, input matrix B E RNXP, implicit latent state h(() E RN11,, and input a(t) E R 1x1 We assume the norm the state matrix A can be bounded by an exponential function such that ||A(t)]le < a- e and the input matrix B is bounded by B S b. where 0 < < 1, > O, and 1 1 S T. In our proof, we utilize the spectral norm (1) -1- 11 = l12). Furthermore, we assume the system input contains quantization error 55 (t) E R Px1 such that .r(t) xxt) (t), where || (t)ll I c and > O. The system is initialized as h(0) = [0,0, (0,0.....00 .......".. For details of the proof, please check Section N.2 Appendix N
Theorem 4.1. The quantization error of (t)) T(() h(t) at time step t for the given discrete linear iineiinvariant model is bounded such that: ||A(t)la S eb (T---)) Consequently, the global quantization error (i.e., t =T) is bounded by: || S T-a'
Empirical analysis of SSM quantization. As shown in Figure 2 and Figure the main chal- lenge in quantizing Mamba blocks is the pre- cision of the SSM input activation æ, which is sensitive to quantization errors, and the output activation y- In Figure [ (a), the naive 8-bit quantization introduces large quantization er- rors, resulting in model collapse for all model sizes. We delve into the causal relationship be- tween (A, B, C, A, x)) and y as modeled by the linear recurrent system in Equation As shown in Figure and F (a), we find that æ is sensitive quantization errors and it leads to the largest errors in the SSM output y, although the values in the z tensor numerically small (ref: Figure 13). We conjecture the reason be- hind the phenomenon is that (B, C, A) are input- dependent (i.e., d-dependent). We note that this finding is specific to SSMs. As shown in Figure 2 (a), self-attention layers are more resilient to quantization errors and do not experience the same issues.
Residual,
Ingut..:
RMiSNorm
Sequence Transform
HadQuant
Intä Linear Projection
Y/=
Selective SSM
INT8
g
FP16
Causal
RMSNorm
Figure 4: The precision mapping and dataflow of Quamba. All scaling factors and element- wise op- erations (i.e., non- nin-linyarit and residual addition) are fused in the quantized operations.
5

---

Published as a conference paper at ICLR 2025
SSM outliers. Our study shows SSMs exhibit distinct outlier patterns compared to Transformers (Ashkboos et al. 2024b; Dettmers et al. 2022: Xiao et al., 2023; Zhao et al. 2023). We show that outhiers appear in the SSMs output (i.e. the tensor), which perform similar token-mixing function to self-attention layers. In contrast, the input and output of self-attention layers are relatively smooth and do not exhibit outlier issues. More comparisons can be found in Section [M] This highlights that different quantization methods are needed for SSM-based models. Therefore, we tailor two different quantization techniques: percentile-based quantization for the input activation æ and quantizing the output in an outlier-free space using the Hadamard transform. Our method recovers performance by improving the quantization precision for the inputs and outputs of the SSM and achieves better trade-off between accuracy and latency.
4.2 QUANTIZATION FOR SELECTIVE SSM
We aim to quantize the weight (A, D) bits, and the activations (Br, C., 4,, are) to 8 bits for selective SSMs. The quantized selective SSM takes 8-bit weights and activations input, well as their scaling factors, and outputs half precision Yr such that y/: = SSM B,, C., D, D., Tz, Sin).
(A, Br, C, D, Ae, Zz) are the quantized weights and activations, and their scaling factor 8in in floating point. (B, C₁ Σ2) depend on the input F to perform selection mechanism as B, = (T,)) C, Fz A, = softplus(Fa (z)) The weights and biases in linear layers Fg. Fc, and F& are also quantized to 8-bit integers. For simplicity, we omit the residual branch 2, in the discussion.
SSM inputs. Our findings show that x is highly sensitive quantization errors, which results in the largest errors in the SSM output y. Specifically, we found the quantization error of æ is dominated by outliers during the calibration. Although they are numerically small (S 10), as shown in Figure 13 the small amounts of outliers (S 0.1%) increase the quantization step (i.e.. scaling factors in Eq 2 and reduce the quantization precision for x. Clipping the values with a percentile max (Zhao et al. 2022, Li et al., 2019) is simple solution to restrict the activation range and has no additional overhead during inference. For example, using the 99th percentile to clip the top 1% the highest activation values prevents the activation range from being skewed by extreme outliers. We use percentile max to calculate the scaling factor for x: 8 = (max" ([=() / -1 1), where p is a parameter for percentiles. In our experiments, we found p 99.998 works well for Quamba.
SSM outputs. We perform WHT to remove the outliers from the SSM output. The output y is transformed to an outlier-free space using Hadamard matrix such that y" H,,y where is the token dimension y and the dimension of the squared Hadamard matrix H. We fuse the inverse Hadamard matrix into the output linear projection WH overhead and achieve compute-invariance (Ashkboos et al. 2024a(b) (i.e., the same output) by Mambaoupptt = W V W we collect the quantization scaling factor for n (i.e., transformed y). Therefore, the fused Hadamard quantization layer can be expressed as
out H, W out to avoid additional computation
W Woul HHHHH,y
W y" In the calibration stage,
H
max (I N1 12N-1122-1112 -N)- 8y = 2
JH = clamp
8y
where N represents the target bit-width We fuse the scaling factor sy in the forward Hadamard transform such that 7d *x H.y, so the quantization does not incur additional computational overhead. The fused Hadamard quantization layer is parallelizable and efficient on GPU with complexity of nllogn (Dao 2024b).
4.3 OTHER OPERATORS
Projection layers. Projection layers, which perform dense matrix multiplications, are the most over- parameterized and the most compute-intensive operators in the models. With quantized activations and weights, projection layers benefit from hardware acceleration (e.g., Tensor Cores) for 8-bit integers as well as reducing the memory costs by half. We implement the 8-bit linear layers with commercial libraries, except for the output projection, which produces half aref-precision outputs for the subsequent normalization layer.

---

Published as a conference paper at ICLR 2025
Fused causal convolution. Causal convolution applies X weight to perform the calculation in a depthwise convolution fashion only using tokens from previous time steps t - w to the current time step f, cach of which is dimensional vector. The operator is memory- bound, as typical depthwise convolution (Lu et al., 2021; Zhang et al. 2020), so applying quantization to the input and output activations and weights largely reduces memory pressure. We quantize the inputs and weights as 8-bit integers and fuse the quantization operator before writing the result to memory. The causal convolution operation is Tout- _ (Conv (Tin, W, W,))), 5
Su The SiLU (Elfwing al.
Sue. 2018 is fused into the convolution, as described by Gu & Dao 2023
Fused RMSNorM. We implement an operator that fuses the residual addition and static quantization with RMSNorm (hang & Sennrich 2019). We do not quantize the weights in RMSNorm, so the normalization is performed in half half- precision.... The fused operator takes as input a half- (Fout Zrex), where Zou is the output from the Quamba block and Fres is the residual. The operator returns a tuple (Jin, Zires) where the 8-bit Tin is the input to the next Quamba block. The operator can L
tuple
be expressed as (z//++1 11 =
=(es); sout + #T8s)
where the L is the layer
Suzi
number in the model, and the scaling factor Sou is pre-calibrated.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Model and datasets. We evaluate Quamba on the open-sourced Mamba family of SSMs (Gu & Dao 2023) and on Jamba (Lieber et al. 2024a), hybrid architecture composed of self-attention
SSMs, and Mixture of Experts (MoE). For zero-shot tasks, we use LM-EVAL (Gao et al. 2023), on which we evaluate baselines and Quamba on LAMBADA (Paperno et al., 2016), HellaSwag (Zellers et al.). PIQA (Bisk et al.] 2020). ARC (Clark et al. 2018) and WinoGrande (Sakaguchi ct al.] 20200p. Model accuracy each dataset and the average accuracy
reported. We follow the evaluate protocol with Mamba (Gu & Dao 2023). and report the accuracy for LAMBADA, WinoGrande, PIQA, and ARC-easy, and accuracy normalized by sequence length for HellaSwag and ARC-challenge. For perplexity, we evaluate the models using the testing set of WikiText2 (Merity al. 2016) and a randomly sampled subset from validation set of Pile dataset (Gao al. 2021
Quantization setup. The calibration set is constructed by randomly sampling 512 sentences from the Pile dataset (Gao et al.) 202T). We collect the static scaling factors for each operator based on the absolute maximum value observed from the calibration set and apply symmetric per-tensor quantization for weights and activations, except for the input to the SSM, where we use the 99.999th percentile (i.e., the p described Section to clip the maximum values. The same scaling factors are applied in all our experiments. Furthermore, clipping optimization algorithm can be applied to Quamba. We incorporate OCTAV (Sakr et al.] 2022). , referred to as Quamba-O in our work, which results in improvement in average accuracy. We note that our method does not require extra training efforts and can be suug-and-laay.
Implementation. We implement the INT8 linear layer using CUTLASS library Thakkar et al. 2023). We did not adjust the tensor core or thread configurations for each case, nor did we disable the PyTorch cuDNN auto-tuner, both of which highlight the robust acceleration achieved by our method. Quantization is integrated and adapted into the CUDA kernels of both the fast Hadamard transform
Dao 2024b) and causal convolution (Dao 2024ah. Additionally, the selective SSM CUDA kernel Gu & Dao 2023) is modified to accommodate inputs with quantized weights and activations, and their scaling factors. We evaluate all methods on the A5000, widely used GPU for AI workloads with 24GB of memory, emulating the setting for cloud applications. For the case of edge applications, we profile all methods on the Nvidia Orin Nano 8G. We perform a few warm-up iterations and then report the average latency of the following 100 iterations.
Baselines. our W8A8 setting, we compare Quamba with static quantization, dynamic quantiza- tion, and Mamba-PTQ (Pierro & Abreu 2024). We re-implement the state-of-the-arr Transformer quantization methods: W8A8 Smoothhuant (SmQ) (Xioo et al.) 2023) and QuaRot (Ashkboos et al. 2024b) for 8-bit neight-activation SSM quantization as additional baselines. These are denoted by Hammb--mmQ and Mamba- Had, respectively. We apply Hadamard matrices (Ashkboos et al. 2024b) to the activations and weights, and quantize them in 8-bit, as shown in Figure 8 For the
7

---

Published as a conference paper at ICLR 2025
FP16
---
WHAB
W4A4
OOM (Est.)
Memory
Ulama-2 2 (FP20) (Touwrnn et
L 2 (WAMEVA)) [Ashiboos el.)
Nembe (F (FP16) (Gu & (Deo]
MambeeHllr (W/S8)
Quarbba (WBAB)
13B
21
Avg. ACC. (96)
2,48
1.40
70
[sec)
(44)
Latency
Latency
(a) TTLT (512+ 512) on Nano 8G
(b) TTLT (2048 + 2048) on A5000
Figure 5: Pareto front analysis for accuracy latency on A5000 and Nano. Quamba models are on the Pareto front for average accuracy and latency when compared to other SSM and transformer -aased LLMs, while also featuring lower memory footprint evidenced in the figure (size of the circle).
re-implemented SmoothQuant (Xiao al., 2023), we apply smoothing factor 0.5 to all linear layers within Mamba. We find the low wit-width quantization methods for Transformers do not gener- alize well to SSMs and quantizing SSMs with wow-bit-width remains unexplored, we include some of the results in Section L For QuaRot (Ashkboos et al.] 2024b), we use the official implementation and profile latencies for W4A4KV4 Llama-2 (Touvron et al. 2023) on both A5000 and Nano.
5.2 PARETO FRONT ANALYSIS FOR ACCURACY vs. LATENCY
Figure iillustrates the average accuracy across six zero-shot tasks (y-axis) versus latency (x-axis, in log-scale) on Nano 8G edge GPU and A5000 cloud platform. In panel (a), we profile TTLT (fime-to-last-tokee)) in seconds, with 512 input tokens and 512 generated tokens on Nano 8G. On A,5000, we increase the input length to 2048 for both input and generated tokens, as shown in panel (b). We note that latency is merely estimated for models that do not fit within the 8G/24GB memory of the Nano/A5000 and is represented with dashed lines. Half-precision Llama-2 models are not included in panels (a) as they do not fit on the Nano. Our method on the Pareto front and offers the best trade-off between average accuracy and latency. outperforming half- prefisiec Mamba (Gu & Dao 2023) and Mamba- Han on both A5000 and Nano. Additional results for TTFT time-to-firs-token are included in Section C in Appendix.
5.3 MODEL SIZE AND LATENCY
Quamba reduces the size of the 2.8B model nearly by half (5.29 GB vs. 2.76 GB) by quantizing weights as 8-bit integers except for normalization layers, as shown the Table [ We profile the latency on an A5000 GPU and an Orin Nano 8G for cloud and edge deployment scenarios. Quamba enjoys the 8-bit acceleration and improves the latency by 1.27 with a 512 input context (i.e., time-to-firs--tor sime-per-uupputii token, TPOT). On the Orin Nano, Quamba improves the latency by 1.2x with a 512 input context and 1.72x in the generation stage. Figure 7 shows the snapshot of real-time generation on Nano 8G. Despite that having similar accuracy to Mamba-Had, Quamba delivers better speedup on both A5000 and Nano. Mamba-Had requires extra matrix transpositions and online Hadamard transforms to handle the SSM input activations, since the Hadamard matrices cannot be fused in the causal convolution (forward transform) and selective scan (backward transform). We show the details in Appendix H In contrast, we study the causal relationship between the input and output activations of SSMs and avoid the additional transpose and transforms by clipping the input outlier values (<10) to increase the quantization precision. In Figure [ (a), we show that Quamba is indeed Pareto-optimal and has a better trade-off between latency and accuracy than other approaches. Figure (b) shows the total time of the generation time, including the prefilling and generation time (i.e., the time to last token, TTLT). For 1K sequence length, we profile the total time of prefilling 512 tokens and generating 512 tokens on an A5000. Quamba improves the TTLT by 1.2x compared with Mamba. Compared with Pythia (Biderman et al. (023), more latency improvement is observed as the sequence length is increased, since SSMs do not require K-V cache for the generation.
TTFT) the A5000 GPU and 1.21 in the generation stage (i.e., L=1,
8

---

Published as a conference paper at ICLR 2025
Table Profiling latency for Mamba and Quamba 2.8B on Nvidia A5000 GPU and Orin Nano 8G. The latency is measured milliseconds (ms). We report the Mean + Std in Table on A5000.
A5000
Otin Nano &G
Method
Precisson
Generate
Pre Pre-illing
Generate
Pre ree-filligg
L=512
L=1024
L==048
L=512
I=1024
L=2048
Mamba-Smi
6.81
43.85
79.09
151.56
56.53
572.36
1123.91
2239.63
Mamba Had
W8AS
2.76
10.46
56.87
103.98
199.83
67.76
746.31
1453.91
2892.95
Quamba (Ours)
48.24
165.13
Mamba
FP16
5.29
9.86
61.19
102.29
184.16
103.56
730.16
1634,08
2756.28
Quamba Reduction (Ours)
1.91
1.21 x
1.27 x
1.21
1.12 x
1.72
1.20
1.38 x
1.17
Table 2: Zero-shot accuracy of W8A8 models on six common sense tasks. Quamba closes the performance gap and outperforms the same-sized Transformers, Pythia (Biderman et al. (203)). in accuracy. We apply a clipping optimization algorithm (Sakr et al. (0022) to Quamba, denoting it as Quamba-O. Bold is the best, and underline is the second best. The full table presents in Appendix A
Table 6
Model
Size
Methods
LAMBADA
HellaSwag
PIQA
Are-E
Are-C
WinoGrande
Avg.
FP16 SmQ
62.0%
41.8% 37.6%
72.0%
61.7%
27.4%
56.5%
53.6%
1.48
19.6%
66.9%
53.8%
25.2%
55.3%
43.1%
Pythia
FP16
65.2%
59.4%
74.1%
63.5%
30.0%
58.5
58.4%
2.8B
SmQ
59.7%
58.4%
73.2%
62.1%
28.4%
57.1%
$6.5%
FP16
64.9%
59.1%
74.2%
65.5%
61.5%
59.7%
Mamba-PTQ
43.8%
70.2%
54.3%
1.4B
Mamba-SmQ
50.8%
56.9%
71.9%
61.8%
31.4%
57.5%
55.1%
Mamba-Had Had
63.1%
58.7%
72.6%
64.1%
32.1%
58.1%
58.1%
Quamba (Ours)
62.6%
58.4%
72.7%
64.5%
33.4%
58.6%
58.4%
Mamba
Quamba-O (Ours)
63.8%
58.8%
73.7%
64.3%
32.3%
57.6%
58.4%
FP16
69.1%
65.9%
75.6%
69.2%
35.8%
63.0%
63.1%
Mamba-PTQ
51.4%
47.6%
70.2%
57.6%
2.8B
Mamba-Smi SmQ
46.6%
62.8%
73.3%
66.5%
35.0%
59.8%
57.3%
Mambu-Had
65.9%
65.6%%
74.3%
68 6%
36.8%
63.3%
62.4%
Quamba (Ours)
65.9%
65.3%
73.9%
68.7%
36.5%
62.9%
62.2%
Quamba-O (Ours)
66.9%
65.6%
74.6%
68.9%
35.9%
63.0%
62.5%
5.4 ZERO-SHOT EVALUATION
We evaluate Quamba and other methods six common-senme tasks in zero-shot fashion. The accuracy of each task and the average accuracy across the tasks are reported in Table Quamba 2.8B has only 0.9% accuracy drop compared to floating-poin Mamba 2.8B and outperforms Mamba-PTQ (Pierro & Abreu 2024) and Mamba-SmQ (Xiao et al.) 20233 in accuracy. Quamba achieves similar accuracy to Mamba-Had. but Quamba achieves better trade-off between accuracy and latency, as shown in Figure (a), We apply a clipping optimization algorithm (Sakr et al.) 2022) to Quamba, denoting the enhanced version Quamba-O. This improvement increases our average accuracy and outperforms Mamba-Har in both accuracy and latency.
5.5 QUANTIZING JAMBA: LARGE-SCALE HYBRID MAMBA- HAMBA-TRANSFUR TRANSFORMER LLM
Jamba (Lieber et al. 2024b) is hybrid transformer-mant language model with 52B parameters, built with self-attention. Mixture of Experts (MoEs), and Mamba blocks, making it the first large-scale Mamba-style model with number of parameters comparable to Mixtral (Jiang et af.) 2024). In Table we compare the LAMBADA OpenAl accuracy of Jamba's 's FP16 inference by combining off-the-shelf quantization methods with different quantization strategies. Applying LLM. int (Dettmers al. 2022) to self-attention and MoE preserves the model's accuracy, whereas jointly quantizing Mamba with LLM. int8 (Dettmers et al. 2022) degrades the model and fails to produce meaningful accuracy. In contrast, combine Quamba with LLM. int (Dettmers et al.) 2022) and as a result we achieve competitive accuracy (1.15 accuracy drop) with lower model footprint than FP16.

---

Published as a conference paper at ICLR 2025
Table Quantizing Jamba, a transformer-mant hybrid model. We combine off-the-shelf quan- tization methods with our method. The zero-shot LAMBADA accuracy is reported. We apply
SmoothQuant Xiao et al. 2023) and LLM. int 8 (Dettmers al. 2022 to self-attention and MoEs.
Self-attention
Mamba
MoE
Accuracy
FPI6
FP16
FP16
74.0%
L.M. int SmQ
FP16
LLM. int
73.9%
FPI6
LIM. int8
70.6%
LLM. int L
LLM. int
LLM, int
fail
SmQ
Quamba (Ours)
LIM. int8
68.7%
inL8
Quamba (Ours)
LLM. inL8
72.9%
6 ABLATION STUDY
6.1 QUAMBA ABLATION
In Table We conduct a performance analysis on each component in Quamba and report the average accuracy across six zero-shot datasets. Naive W8A8 quantization results in significant performance discrepancics across all sizes of models. We improve the performance of quantized models by constraining the quantization range of the SSM input using percentile clipping (+ In Per.). While addressing the large outlier in the SSM output using the Hadamard transform improves performance (+ Out Had.), the results remain unsatisfactory. Quamba integrates two techniques, thereby closing the performance gaps across all model sizes.
6.2 PERCENTILE-BANE BASED ACTIVATION CLAMPING
In T we conduct sensitivity analysis on the percentile maximum clipping for the input x to SSM. We test different percentiles (i.e., the described in 4.2) and report the accuracy LAMBADA dataset. The table shows more outliers in the larger models, while smaller amounts outliers are clipped in the smaller models. Therefore, clipping 0.001% (i.e., p= 99.999) of outliers in the model with 130m parameters produces best performance. In contrast, for the model with 2.8b parameters, clipping 0.1% (i.e., p 99.9) performs best the LAMBADA dataset (Paperno al. 2016).
Table 5: Ablation Study different per- centiles for Quamba on LAMBADA Paperno et al. 2016) dataset.
Table 4: Ablation study on Quamba. Avg. aceu- racy of zero-shot tasks is best for Quamba; other approaches are inferior individually.
Quamba
Size
FP16
W8A8 + In Per. Out Had. Quamba
Size
Quamba-O
p
99
99.9 99.99
99.999
130M
44.7%
37.4%
38.7%
41.8%
43.5%
370M
51.6%
40.6%
41.9%
47.8%
49.0%
130M
10.8%
36.4%
40.0%
40.6%
40.7%
1.4B
59.7%
42. 1%
47.4%
50.8%
58.4%
370M
20.4%
44.7%
49.9%
50.3%
53.2%
2.8B
63.0%
45.9%
48.5%
56.4%
62.2%
1.4B
44.4%
60.6%
62.6%
60.4%
63.8%
2.8B
58.9%
66.5%
66.3%
65.5%
66.9%
CONCLUSION
We investigate quantization methods for selective State Space Models and propose Quamba, methodology for successfully quantizing the weight and activations as 8-bit signed integers tailored for the Mamba family of SSMs. Our experiments show that Quamba maintains the original FP16 when accuracy compared with state-of-the-att counterparts, including current techniques for Transformers. The profiling results on a wide variety of platforms show that the low bit-width representation of devices, such as edge GPUs, but also
Quamba not only enables deployment to resource-constraie benefits from hardware acceleration with reduced latency. summary, our extensive experiments demonstrate the effectiivness of Quamba in addressing the real deployment challenges faced by many emerging applications based on SSMs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the ONR Minerva program, NSP CCF Grant No. 2107085, iMAGINE the Intelligent Machine Engineering Consortium UT Austin, UT Cockrell School of Engineering Doctoral Fellowships, and Taiwan's NSTC Grant No. 111-2221-E-A9914488-M
REFERENCES
Ghada Alsuhli, Vasileios Sakellariou, Hani Saleh, Mahmoud Al-Qutayri, Baker Mohammad, and Thanos Stouraitis. Number systems for deep neural network architectures: A survey, 2023, URL
https://ane //tpiv/aaxvvv.vo wttp://////////..ooo./// 05035
Salch Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint 1rxXiv2401115024 2024a.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian I. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms, arXiv preprint arXiv:2404.00 00456, 2024b.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04517, 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432-7439. AAAI Press, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jack Choquette. Nvidia hopper h100 gpu: Scaling performance. IEEE Micro, 43(3):9-117 (1,3):917,,0022 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, 2018.
Tri Dao. Causal depthwise convid cuda with a pytorch interface, 2024a. URL https: git http:///aaaaaaaaaaccuaaaaaaaaaaa catp://cssaacoooooa
AI
Tri Dao. Fast hadamard transform in cuda, with a pytorch interface, 2024b. URL https://
github. Battuuu.coo//aaAAaa//aaaaa
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 30318-30332 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36. 2024.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17..2333, 2022.
11

---

Published as a conference paper at ICLR 2025
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021. URL
ht.tps:/ http://awwvvvrgaaaaaaae abtp///////// 2002700027
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoft, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 12 2023.
Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pp. 291-326, Chapman and Hall/CRC, 2022.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-spast space models. In International Conference on Machine Learning, pp. 7616-7633, PMLR, 2022.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint ar arXiv:2312.00752 2023.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1487 1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, arXiv preprint arXiv:211 11.0036, 2021.
Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971- 35983, 2022.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint wrXiv:1510.0011 2015.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmeeic-onl, inference. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7704-2713, 2018.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.00 040000888 2024.
Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Man- sheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision. arXiv preprint arXiv:2411.04330 2024.
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jom Peters, and Tijmen Blankevoort. Fp8 quantization: The power of the exponent, 2024. URL httphhss///aiii..rrrge///oee /208
09225
Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. arXiv preprint arXhv:2403.06977, 2024.
Rundong Li, Yan Wang, Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan. Fully quantized network for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2810-2819, 2019.
12

---

Published as a conference paper at ICLR 2025
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shwaar,,, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Frez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A hybrid transformer-manba language model. CoRR, abs/2403.19887, 2024a. doi: 10.48500ARXIV..403. 19887 URL
2403 19887
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer- mamba language model. arXiv preprint arXiv:2403.19888 2024bb.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation- aware weight quantization for IIm compression and acceleration. arXiv preprint arXiv:2306. 00978, 2023.
Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient Ilm serving. arXiv preprint arXiv:2405.04532. 2024.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Date-free quantization aware training for large language models. arXiv preprint arXiv:2305. 2023.
Gangzhao Lu, Weizhe Zhang, and Zheng Wang. Optimizing depthwise separable convolution operations on gpus. IEEE Transactions on Parallel and Distributed Systems, 33(1):70-87, 2021.
Justus Mattern and Konstantin Hohr. Mamba-chat, GitHub, 2023. URL https:///tthubb. com/ havenhgg/mamaa-che
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint /009. 2016.
Daisuke Miyashita, Edward H Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint arXiv:1603.01033 2016
Eric Nguyen, Karan Goel, Albert Gu, Gordon W Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. $4nd: Modeling images and videos as multidimensional signals using state 2022.
spaces. arXiv preprint
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 18.18////VV/
P16- 1144. UtLpttts: /oi. org /0/../001100 18653/6311113 11t.1/.1/111111111111111 htt/////ppp1////////
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprini arXiv:2305. / 13048, 2023.
Alessandro Pierro and Steven Abreu. Mamba-ptqg ptq: Outlier channels in recurrent large language models. arXiv preprint arXiv:2407.12397 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. In The Thirty-Fourds AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8732-8740. AAAI Press, 2020.
Charbel Sakr. Steve Dai, Rangha Venkatesan, Brian Zimmer, William Dally, and Brucek Khailany. Optimal clipping and magnitude-awee aware differentiation for improved quantization-awee aware training. In International Conference on Machine Learning, pp. 19123-19138. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2025
George Saon, Ankit Gupta, and Xiaodong Cui. Diagonal state space augmented transformers for speech recognition. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Neil Sloane. A library of hadamard matrices. available at the website: http:///ww.. research. att. com/ njas/hadamard, 1999.
Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2288.0493: 2022.
Vijay Thakkar, Pradeep Ramani, Cris Cecka, Aniket Shivam, Honghao Lu, Ethan Yan, Jack Kosaian, Mark Hoemmen, Haicheng Wu, Andrew Kerr, Matt Nicely, Duane Merrill, Dustyn Blasig, Fengqi Qiao, Piotr Majcher, Paul Springer, Markus Hohnerbach, Jin Wang, and Manish Gupta. CUTLASS, January 2023. URL https: http/gttttuuccooc com Bw..mNNVIDDIE IIII/ cutlass.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 23Xi:22377009288 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipit: Even better IIm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.044: 043966 2024.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. Mambabyte: Token-free selective state space model. arXiv preprint arXiv:2401.13660 2024.
Kuan Wang. Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8612-8620, 2019.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. International Conference Machine Learning, pp. 38087-38099. PMLR, 2023.
Yuedong Yang, Hung-Yueh low-rank backpropagation for vision transformer adaptation. Advances in Neural Information Processing Systems, 36, 2024.
Yueh Chiang, Guihong Li, Diana Mareulescu, and Radu Marculescu. Efficient
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Märquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 4791 4890- Association for Computational Linguistics.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Pengfei Zhang, Eric Lo, and Baotong Lu. High performance depthwise and pointwise convolutions mobile devices. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 6792-6802, 2020.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXivv2205.01060 2022.
Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. arXiv preprint arXiv:2403. 14520, 2024.
14

---

Published as a conference paper at ICLR 2025
Lingran Zhao, Zhen Dong, and Kurt Keutzer. Analysis of quantization mlp-based vision models. arXiv preprint arXiv:2209.06383, 06383, 2022.
Yilong Zhao, Chien- Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving, arXiv preprint arXiv:2310. 2023.
Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang. Zhihang Yuan, Xiuhong Li, al. A survey on efficient inference for large language models. arXiv preprint arXiv:2404. 1 14294, 2024.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. survey on model compression for large language models. arXiv preprint 1xx1:23308.07633: 2023.
15

---

Published as a conference paper at ICLR 2025
A ADDITIONAL W8A8 RESULTS FOR ZERO-SHOT ACCURACY
Table 6: Zero-shot accuracy of W8A8 models on six common sense tasks. Quamba closes the performance gap and outperforms the same-sized Transformers, Pythia (Biderman et al. [202)). in accuracy. We apply clipping optimization algorithm (Sakr et al. 2022 Quamba, denoting it as Quamba-O. Bold is the best, and underline is the second best.
Model
Size
Methods
LAMBADA
HellaSwag
PIQA
Arc-E
Arc-C
WinoGrande
Avg.
FP16
62.0%
41.8%
72.0%
61.7%
27 37.4%
56.5%
53.6%
1.4B
SmQ
19.6%
37.6%
66.9%
53.8%
25.2%
55.3%
43.1%
Pythia
FP16
65.2%
59.4%
74,
63.5%
30.0%
58.5 %
58 4%
2.8B
SmQ
59.7%
58.4%
73.2%
62.1%
57.1%
56.5%
44.2%
35.3%
64.5%
48.0%
24.3%
51.9%
44.7%
dynamic
38.6%
34.1%
60,2%
41.5%
24.6%
51.9%
41.8%
130M
static
24.0%
31.8%
58.1%
37.5%
24.4%
48.5%
37.4%
Mamba-PTQ
4.3%
27.7%
56.5%
51.1%
Mamba-SmQ
41.1%
34.4%
63.0%
43.6%
23.6%
51.9%
42.9%
Mamba- Had
40.7%
35.2%
62.0%
44.1%
24.0%
49.7%
42.6%
Quamba (Ours)
40.6%
35.0%
63.0%
46.5%
23.0%
53.1%
43.5%
Quamba- O (Ours)
40.7%
34.5%
61.3%
44.2%
25.8%
51.4%
42.9%
FP16
55.6%
46.5%
69.5%
55.1%
28.0%
55.3%
51.7%
Mambu
dynamic
44.0%
45.2%
67.3%
51.8%
28, 18.1%
51.9%
48.0%
370M
static
28. 28.9%
38.5%
58.7%
40.4%
25.9%
51.4%
40.6%
Mamba-PTO
10.3%
31.0%
58.8%
51.0%
Mamba-SmQ
44.3%
44.3%
66.3%
51.2%
27.7%
54.7%
48.1%
Mamba Had
53.2%
46.3%
68.6%
53.2%
27.1%
51.6%
50.0%
Quamba (Ours)
50.5%
46.2%
67.1%
51.6%
26.9%
51.9%
49.0%
Quamba- (Ours)
53.2%
46.2%
68.4%
53.5%
27.2%
53.5%
50.4%
64.9%
59.1%
74.2%
65.5%
32.8%
61.5%
59.7%
dynamic
52.2%
56.6%
70.4%
61.7%
31.7%
59.2%
1.4B
static
20.3%
46.7%
63.0%
44.4%
27.8%
50.5%
42.1%
Mamba-PTQ
55.4%
43.8%
70.2%
54.3%
Mamba-SmQ
50.8%
56.9%
71.9%
61.8%
31.4%
57.5%
55.1%
Mamba-Had
63.1%
58.7%
72.6%
64.1%
32.1%
58.1%
58.1%
Quamba (Ours)
62.6%
58.4%
72.7%
64.5%
33.4%
58.6%
58.4%
Quamba- O (Ours)
63.8%
58.8%
73.7%
64.3%
32.3%
57.6%
58.4%
FP16
69.1%
65.9%
75.6%
69.2%
35.8%
63.0%
63.1%
dynamic
59.2%
63.9%
72.7%
67.8%
34.4%
58.1%
59.4%
2.8B
static
20.0%
51.2%
64.8%
55.1%
31.0%
53.5%
45.9%
Mamba-PTQ PTQ
51.4%
47.6%
70.2%
57.6%
Mamba-SmQ
46.6%
62.8%
73.3%
66.5%
35.0%
59.8%
57.3%
Mamba-H.e Had
65.9%
65.6%
74.3%
68.6%
36.8%
63.3%
62.4%
Quamba (Ours)
65 9%
65.3%
73.9%
68.7%
36.5%
62.9%
62.2%
Quamba- (Ours)
66.9%
65.6%
74.6%
68.9%
35.9%
63.0%
62.5%
B PERPLEXITY EVALUATION
Table presents the perplexity results of Quamba and the baseline methods on the Mamba family of SSMs. Static quantization fails to maintain the precision in SSM quantization. resulting in significant performance degradation (i.e., increased perplexity, where lower better). Even with scaling factors calculated dynamically, which introduces significant computational overhead during inference, it still results a considerable increase in perplexity (+7.5) on Mamba-2.88 Although Mamba-SmQ mitigates the issue of outliers in the output of the SSM, a performance gap remains when compared to the non-quantized Mamba because fails to address the issue of quantizing sensitive x input tensors to SSMs. Quamba achieves similar perplexity to Mamba-Had but delivers better speedup both A5000 and Nano, as shown in Table and Figure (a). Since Mamba-H is not optimized for SSMs, requires extra transpose and Hadamard transforms to handle the SSM input activations.
16

---

Published as a conference paper at ICLR 2025
Table Perplexity results of different quantization methods applied on Mamba (Gu & Dao 2023 family of SSMs. We evaluate the quantized models on subset of Pile and Wikitext2 datasets. Quamba closes the performance gap and delivers a better trade-off between latency and accuracy (ref. Figure (a)). Bold is the best, and underline is the second best.
Wikitex12 Perplexity (1)
Pile Perplexity (U)
latency (.)
Model
Methods
130M
370M
1.4B
2.8B
130M
370M
1.4B
2.8B
2.8B
FP16
15.14
12.68
7.62
6.89
Pythia
SmQ
38.24
14.03
14.86
7.37
FP16
20.61
14.31
10.75
9.45
11.50
8.81
7.11
6.39
103.56
dynamic
57.18
24.58
19.32
16.92
28.74
15.28
12.37
10.56
Mamba
static
139.90
84.69
60.87
78.63
62.48
40.43
32.33
35.08
Mamba-SmQ
29.51
19.29
14.23
13.59
15.81
11.40
9.14
8.59
56,53
Mambu-Had
32.43
16.29
11.39
9.89
16.78
9.87
7.49
6.66
67.76
Quamba (Ours)
25.09
16.18
11.35
9.91
13.63
9.84
7.49
6.67
60.17
C FULL PARETO FRONT ANALYSIS FOR ACCURACY VS. LATENCY
Figure 6 iillustrates the average accuracy across six zero-shot tasks (y-axis) versus latency (x-axis, in log-scale) on the A5000 cloud platform and Nano 8G edge GPU. In panel (a), we profile TTFT fime-to-first-toke in milliseconds using 4K input tokens on A5000.. For comparison of end-to-end latency, we profile TTLT (tme-to-last-token in seconds, with 2K input tokens and 2K generated tokens A5000, as shown in panel (b). On Orin Nano 8G, we reduce the input length to IK and profile TTFT and TTLT, as shown in panel (c) and (d). For QuaRot (Ashkboos et al. 2024b), we use the official implementation and profile latencies for Llama-2 (Touvron et al. (023) and Nano. We note that latency is merely estimated for models that do not fit within the 24GB/8G memory of the A5000/Nano and is represented with dashed lines. Half-precisio Llama-2 models not included in panels (c) and (d) as they do not fit on the Nano. Our method is on the Pareto front and offers the best trade-off between average accuracy and latency, outperforming half-precision Mamba (Gu & Dao 2023) and Mamba-Har on both A5000 and Nano.
both A5000
WBAB
VHAA
OOM (Est.)
Hemory
Uama (FP16) [Tbuwrnn et
Liama (WADAKV4) [Ashiboos
Mamba (FP16) [Gu Dao)
Mamba-Had (WBAB)
Quamba (WBAB)
Acc. (%)
2.85
4
2.00
5.3G
1.48
15.4
2.58
2885
1448
1.40
Latency
Latency
(a) TTFT (L=4096) A5000
TTLT (2048 2048) A5000
Avg. Acc. (9))
7:45
G2-2
9.1
2.85
2.85
1.40
X01
Insect
284
Iseo
Latency
Latency
(c) TTFT (L=1024) on Nano
(d) TILT (512+ 512) on Nano
Figure 6: Pareto front analysis for accuracy vs. latency on A5000 and Nano. Quamba models on the Pareto front for average accuracy and latency when compared to other SSM and transformer-basee LLMs, while also featuring lower memory footprint as evidenced in the figure (size of the circle).
17

---

Published as a conference paper at ICLR 2025
D TTFT PROFILING ON NANO 8G: 4-BIT LLAMA-2-7B vs. 8-BIT QUAMBA
We use the official implementation from QuaRot (Ashkboos et al. 2024b) profile TTFT for Llama-2-7B (Touvron et al. 2023) (W4A4KV4) on Nano 8G for reference purposes. The quantized W4A4KV4 Llama-2-7B is roughly comparable in model size Quamba (8-bit Mamba), making a reasonably equivalent basis for comparison. We report the TTFT in milliseconds (msec.) for all latencies Nano 8G in T
Empirically, find that the 4-bit Llama-2-7h crashes with 8K input. We provide the theoretical memory cost for 4-bit Llama-2-77 in Figure [
Table 8: Lateney and accuracy comparison between Llama 2-7B (Tovroo al. 2023 (QuaRot
(Ashkboos et al., 2024b)) and Quamba-2.8B across different sequence lengths. OOM indicates out-of-memory.
Latency (mscc.)
Model
Bit-width
Model size
Acc.
1k
4k
8k
Llama-2-7B (QuaRot)
W4A4KV4
3.5G
2430.6
4529.1
9212.8
OOM
65.6%
Quamba-2.88
W8A8
2.7G
1181.1
2354.9
4706.2
10888.9
62.5%
E DETAILED LATENCY RESULTS ON A5000
We report the Mean Std in Table 9 on A5000 in milliseconds (msec.) for all latencies.
Table 9: Detailed Latency comparison of Mamba-2.88 and Quamba-2.8B under various sequence
lengths (L).
Model
Bit-width
Generate (I==1)
I.=512
1.=1024
1.=2048
Mamba-2.8B
FP16
9.86#0.038
61.19+2.79
102.29
184.16+1.89
Quamba-2.8B
W8A8
8.1200.011
48.24=1.53
84.84+0.46
165.13+0.8T
F REAL-TTME GENERATION ON EDGE GPUs
deploy both Quamba and Mamba on an Nvidia Nano 8G, comparing their speedups for real user experiences. We use the pre-trained weights of Mamba-Chal (Mattern & Hohr (023) and apply our quantization techniques. Figure shows a screen snapshot taken during the demo. We input the same prompt the model at the beginning (a) T 0. At the initial time point (a) T 0, the same prompt provided to both models. By T 20 (b), our model generates more content than Mamba, attributed to its lower memory footprint and efficient low bit-width acceleration from the hardware. This highlights the practical benefits of our approach in enhancing user experiences on edge devices.
Mamba
Quamba (Ours)
Mamba
Quamba (Ours)
(lay ng th orighal pwweratie On None
playing with or oo poreriiinn apord Darrs
(a) =
(b) T 20
Figure Demo on Orin Nano 8G.
18

---

Published as a conference paper at ICLR 2025
G GENERATION QUALITY
We also applied Quamba to a chat language model to evaluate its effectiveness in preserving generation quality. Specifically, we used the Mamba-2.8B-Chal model (Mattern & Hohr 2023), which is based on Mamba-2.80 and fine-tuned on Ultrahhat200k (Tunstall et al. 2023). We compared the sentences generated by the quantized and non-quantized models, as shown Table The results demonstrate that the compressed model is capable of generating complete and informative sentences.
Table 10: Generated Examples of Quamba (W8A8) and Mamba (FP16)
Model
Generated Sentences
Mamba (FP16)
What's great about the holiday season? do not have a personal opinion or feelings, However, some people may enjoy spending time with family and friends during this special period of year as brings joy to their lives while giving them an opportunity for reflection on important values such as love, gratitude, generosity etc.. also provides opportunities like gift-giving which can be used in various ways depending upon one's 's financial status/cireumstannes but ultimately serves humanity by spreading happiness around us through acts that are selfless rather than silfishly motivated towards materialistic gains only. Overall.the holidays bring together families from all over world under same roof creating bonds stronger then ever before making every moment count more precious because we know how short life is! So let me wish you happy new years!!! Happy Holidays!! May your days ahead filled up fullness & peacefulness!!!
Quamba (W8A8)
What's great about the holiday season? do not have a personal opinion or feelings, However, can provide you with some general information: The holidays are an exciting time of year filled with family and friends coming together to celebrate joyous occasions such as birthdays, wedkiingseenaggementt (ur anniversaries). new babies in our lives etc.. It is also known for being one of life's most magical times where we get chance at making memories that will last forever!- There may be challenges during this period like work commitments but they're usually worth because these moments make us feel so much more grateful towards each other.- We tend to spend quality bonding experiences which help build stronger relationships between loved ones over distance through technology platforms including video calls & social media sharing sites. This helps bring families closer than ever hefore even when physically separated by miles away from home due quarantine measures imposed everyone around world right now!!! So yes,the Holidays definitely brings out best sides within ourselves!! Happy Holiday Season Everyone!!!
H RE-IMPLEMENTATION OF QUAROT ON MAMBA
We re-implement QuaRot (Ashkboos et al. 2024b), a state-of-the-art low-bitwidth quantization method for Transformers, for the Mamba structure. Our re-implementation is denoted by Mamba- Had. Figure g (b) illustrates the details of our re-implementation. While Mamba-Had is on-par with Quamba in terms of perplexity and accuracy, it not optimized for the structure of SSMs and requires additional Hadamard transforms and transpositions to process the SSM input activations. The additional online Hadamard transforms are particularly costly because they require extra matrix transpositions and memory contiguous due to mismatched output and input shapes between the causal convolution and selective scan. Specifically, the process involves the following sequence: Causal Conv + [b, dim, seglen] + transpose & contiguous -> [b, seglen, dim] + forward Hadamard & quantize- + [6, seqlen, dim] + backward Hadamard + [b, seqlen, dim] + transpose & contiguous + 16, dim, seqlen] selective scan. In contrast, our method avoids these additional transpositions and memory contiguous steps. The streamlined process is as follows: Causal Conv & quantize - [b, dim, seglen] selective scan. Our approach is friendly to hardware and effectively improves the quantization precision of the x tensor, delivering real speedups on both cloud and edge devices (see [T)
19

---

Published as a conference paper at ICLR 2025
Int8 Path
FP16 Path
Channel-miking
Token-mixing
Normalization
Online Transform
2
(Out Proj) H*
Selective
Selective SSM
(Oot #t
yz
d
B
C
x
yz
de
B
C
HadQuant
HadQuant
Proj
y SSM
de Proj
y
(x Proj) H²
T T(*))
h
T(x")
In Proj
Proj
HadQuant
Causal Canv
h
T(x)
RMSNorm
Lth Layer
RMSNorm
Eth layer
Causal Conv
he
h2
(a) Quamba (Ours)
(b) Mamba-Had
Figure 8: The figure compares Quamba to Mamba-Had. We fuse the Hadamard matrices in the linear layers to achieve compute-invariance invariance. F represents the 8-bit x tensor with percentile clipping. Our method provides real speedups on cloud and edge devices, while QuaRot requires extra Hadamard transforms and transpositions for SSM input activations. Residual connections are simplified to reduce clutter in the figure.
1 SENSITIVITY ANALYSIS OF QUANTIZING SSMS
We perform sensitivity analysis of quantiz- ing the SSM input and output (SSM 1/O) ac- tivations. Figure illustrates that the model collapses when all activations and weights are quantized bits. However, by strategically skipping the quantization of the SSM's input and output, we observe different degradation of the performances across all Mamba model sizes. Notably, quantizing the SSM output results in severe performance degradation (orange, SSM I/O FP16/18) due to the skewed quantization resolution caused by large outliers, particularly extreme values (2 100). To address this, we apply Hadamard transform to the output acti- vations, transforming and quantizing them to an outlier-free space. Although the numerical val- ues in the tensor are small (typically < 10), we find that SSM output sensitive the quantiza- tion errors of input (i.e., the quantization errors of x leading to large deviations in output y) due to the causal relationship modeled in Equation I as shown in Figure31 (a). We employ clipping to improve the input's guantization precision thereby reducing the deviation at the SSM output. We show that applying clipping to enhance the quantization precision of the x tensor proves to be a more efficient strategy. This approach places Quamba ahead of Mamba-Had Had the Pareto front (ref. Section C. Additional quantization variants for the x tensor are provided in Appendix K
WSAS (55 WSAS (SSM/SPMMSPPMII
WIAA FPAG/1P15)
80
Zero-shot accuracy
60
20
2
120d
370M
2.88
Model size (M)
Figure 9: We analyzed the sensitivity of quantiz- ing SSM input and output (SSM 1/O), reporting zero-shot accuracy on the LAMBADA dataset. Ac- curacy highly sensitive input/output precision. Quamba quantizes SSM I/O to 8-bit, reducing the performance gap with FP16 (red).
J LAYER-WISER DISTRIBUTION OF SSMS
We analyzed the input and output distributions of SSM for Mamba family, as shown in Figure The box plots reveal the presence of outliers in both inputs and outputs.
20

---

Published as a conference paper at ICLR 2025
SSM input 2. In Figure 10ftet. skewed well asymmetric distribution is present in all layers of SSM inputs. Although the activation values are relatively small (< 10), however, due to the sensitivity of the linear recurrent dynamics, representing the input activations with low bit-width data causes significant performance drops for all sizes of Mamba-family. Therefore, clipping the distributions results in better quantization precision and avoids the accuracy degradation. Though asymmetric quantization performs slightly better than symmetric quantization as shown in Table [T] we choose symmetric quantization due the support of most frameworks, e.g., CUTLASS, We leave the asymmetric optimization in the future work.
SSM output y/- In Figure TO right, large outliers are observed in the SSM outputs. The outliers pose great challenge in quantizing SSMs, due to the skewed quantization resolution caused by the extreme values (2 100). This finding in SSMs echoes the outlier phenomenon observed in Transformers. We apply the Hadamard matrices to transform the output activations space, making quantization easier. Interestingly, the layers closer to the model output have larger outlier values, suggesting that different quantization schemes can be applied the earlier layers. We leave the study for future work.
outlier-free free
K EXPLORE QUANTIZATION ALGORITHMS FOR SSM INPUTS
In our investigation of 8-bit quantization strategies for the state space model (SSM), we observed that the SSM imput æ exhibits outliers. Although these outliers are not excessively large, their presence significantly impacts the quantization error, affecting the SSM output In this section, we present the results and analysis of several other 8-bit quantization options in Table though not necessarily hardware efficient.
Table II:
We explore different methods for quantizating SSM inputs into 8-bit. For the quantization of other tensors, we use the same settings as Quamba. We evaluate the quantized models and report the accuracy on the LAMBADA dataset. Our method is generalized to current mainstream freamworks and toolchains. (Sym., Asym., and Per. are short for symmetric, asymmetric, and percentile)
Framework Supports
LAMBADA Accuracy (1)
Methods for SSM Inputs
130m
370m
1.4b
2.8b
FP16
Yes
41.24%
51.81%
63.82%
67.51%
Dynamic
MinMax Sym.
Ye
40.38%
51.45%
62.55%
66.62%
Static
MinMax Sym.
Yes
34.10%
45.78%
44.89%
55.71%
MinMax Sym. Log2
No
40.31 %
51.80%
63.57%
67.51
MinMax Asym. Per.
No
40.73%
50.46%
63.09%
66.76%
MinMax Sym. Per. (Ours)
Yes
40.61%
50.37%
60.43%
65.67%
Dynamic quantization. One direct approach provide more accurate quantization mapping is through dynamic quantization. By dynamically capturing the activation range based on the current inputs, map the floating value into 8-bit with precise scaling factors. The approach boosts the accuracy and closes the performance gap between FP16. However, the dynamic approach will result in extra execution overhead on re-calculating the quantization scales, leading to sub-optimal computation efficiency.
Asymmetric quantization. We notice that the visualized tensor distribution of SSM inputs is asymmetric in Figure 10 To better utilize the bit-width, we could apply asymmetric quantization to the SSM inputs. As shown in Table asymmetric quantization yields better accuracy in the zero- shot tasks, particularly for Mamba-1
and Mamba-2.8b models. However, asymmetric quantization increases the computational load during inference and requires specific software framework and hardware supports. We leave the asymmetric optimization for future work.
Thhe dynamic approach will result in extra exccution overhead re-calculating the quantization scales.
21

---

Published as a conference paper at ICLR 2025
Mamba- I 30m (SSM Tuput x)
Mamba- 130m (SSM Output
10a
Activation Values
Activation Values
10a
10a
-109
10a
11011211144511177110222222
Layer Index
Layer Index
Mamba-370m 370m (SSM luput
Mamba-370m (SSM Output
10
10a
Activation Values
Activation Values
10a
109
100
100
-10
-107
2 4 8 101214161888 0002144612202222662833300
2 101214166 10121411662222222228888 3864444-2646
Index
Layer Index
Mamba-1 1.4b (SSM luput
Mamba-1 1.4mba-1.4b (SSM Output y)
10a
Activation Values
Activation Values
10a
109
-10
101
10a
2 4 11111111111100 122.211662222222623833
464244444
2 212.1300021111222222320222222210
Layer Index
Index
Mamba-2.88 (SSM Tuput x)
Mamba-2.88 2.8b (SSM Output y)
10a
Activation Values
Activation Values
101
10a
100
109
-10
12
20
28
36
52
60
52
Layer Index
Layer
Figure 10: Box plots of the SSM inputs and outputs distributions for Mamba family. We obtained the distributions using the Pile dataset, which serves as the calibration dataset for all our experiments.
22

---

Published as a conference paper at ICLR 2025
Log2 quantization. To avoid the quantization step skewed by the outliers, and to ensure that smaller values in a tensor are accurately mapped. we can quantify the tensor in log-scale. Here, we implement log2 quantization (Alsuhli et al. 2023; Miyashita et al. 2016), which maps the values to the nearest power of 2, achieving the desired non-uniform mapping. Our log2 version slightly outperforms Quamba. However, log2 matrix multiplication requires specifie optimization on both software and hardware levels. Our method is more generalizable to mainstream frameworks and toolchains (e., PyTorch, CUTLASS).
Other alternatives. Exploring the power of exponents with advanced low-bit floating point data types (uuzmin et al., 2024), such as ESM2 or E4M3 (Choquette, 2023), currently supported by NVIDIA Hopper GPUs, might also be an effective solution for quantizing SSM inputs. A more comprehensive study on quantizing SSMs using floating-point quantization is left for future work.
L Low BIT-WIDTH QUANTIZATION
We show that Transformer- quantization methods do not generalize well Mamba blocks. We re-implement state-of oftte-of-thearrr low bit- hit-width quantization methods for Transformers. Quip# (weight- only quantization, W2A16) (Tseng et al., 2024) and QuaRot (W4A4) (Ashkboos et al. 2024b), for the Mamba structure. These are denoted by Mamba-Quip# and Mamba-Had, respectively Our experiments show that they fail to effectively quantize Mamba in low bit-width settings, as shown in Tables [2] and [3 In Table [2] applying Quip# and QuaRot Mamba results in much higher perplexity. leading to worse performance compared to Transformers. For instance, Mamba-Quipi Quipt quantizes Llama-2-7b with W2A16, causing only 1.02 increase in perplexity, whereas our implementation on Mamba results in 1.34x increase. In Table [3] although models with different bit-widths cannot be directly compared, the results show that both Mamba-Quipf and Mamba-Han reduce the average accuracy across six zero-shot downstream tasks. Both Mamba-Had and Quamba fail at the 4-bit level, reducing the average accuracy from 63.1% 1% to 30.2% and 29.3%, respectively. Our study highlights that quantizing SSMs is particularly challenging, especially in low bit-width settings, as their input and output activations exhibit a causal relationship with varying levels of outliers, a phenomenon unique to SSMs (ref. Section M). We note that similar results were found recently for large transformer-baste based L.LMS which were shown to have best performance under memory constraints for bit-width of 6-8, with worse results for a bit-width of 4 (Kumar et al. 2024).
Table 12: Quantizing Llama-2-7b and Mamba 2.8B with low bit-width methods. The perplexity on Wiki2 is reported.
Table 13: Quantizing Mamba 2.8B with low bit-width methods. The average accuracy on six zero-shot tasks is reported.
Methods
Precision
Llama-2-7b
Mamba-2.8b
Methods
Precision
Mambu-2.8h
Mamha
FP16
5.47
9.45
Mamba
FP16
63.1%
Mamba-Quiph Mamba- Had
W2A16
5.56 (1.02 x) 6.10 x)
12.71
(1.34 failed
Mamba-Quupp
W2A16
58.5%
Mamba Had
W4A4
30.2%
W4A4
Quamba (Ours)
W4A4
29.3%
Quamba (Ours)
W8A8
9.91
Quamba (Ours)
W8A8
62.2%
M COMPARING SSMS WITH SELF- AATE-TTTENTOONN LAYERS
Comparing the quantization sensitivity. We explore the quantization sensitivity of input and output activation maps for both SSMs and self-attention layers, as shown in Figure [I] We conduct experiments on Mamba 2.8B, a same-sized Transformer Pythia 2.8B, and a recently published, comparably sized Transformer Llama 3.2 3B. Quantizing the input and output tensors leads to the most significant accuracy drop on the LAMBADA dataset for SSMs, compared to other tensors. In contrast, quantizing the tensors in self-attention layers (i.e. h, q. k, U, and output y) results in minimal accuracy loss. For transformers, using 8-bit ha tensors in feedforward layers significantly degrades model performance. Due to the differing quantization sensitivity patterns between Mamba and Transformer blocks, we highlight the smooth, outlier, and sensitive paths for both in Figure [2 This highlights that different quantization methods needed for SSM-based models.
23

---

Published as a conference paper at ICLR 2025
70.2%
70
70
64.8%
57.5%
55.9%
50-
50
X 50
50
25, .%
10
10
10
W8A16
Pc
de
W8A16
0u
Rg
W8A16
y/
Fry
Pg
B-bit Activation
B-bit Activation
8-bit Activation
(a) Mamba 2.8B
(b) Pythia 2.8B
(c) Llama3.2 3B
Figure 11: Quantizing the input æ and output y tensors causes the most significant accuracy drop on the LAMBADA dataset for SSMs. In contrast, quantizing the tensors in self-attention layers results minimal accuracy loss.
Smooth Path
Outtier Path
Sensitive Path
Norm Path
Channel-mixing
Token-miking
Normalization
Down Proj
Out Proj
Selective SSM
h
Out Proj
y
de
B
C
X
Gated Proj
y
SSM
de
h2
Salf-attention
htc
RMSNorm
K
x Proj
Proj
MHSA
Q
Proj
Proj
V Proj
X
4th layer
Causal Conv
layer
RMSNorm
RMSNorm
htc
h
(a) The Mamha block
(b) SSM
(c) The Transformer block
(d) MHSA
Figure 12: We annotate the different sensitivity patterns for SSMs and self-attention layers, which shows that they require different quantization techniques. The residual connections simplified to avoid cluttering the figure.
Auy: h
Auy: max 24.11
h
Auy: Z
X
mak
mak: 9:39
mas:: 10.45
14
Ko brldm
101 20 po
Molens
#tobms
#tobes
Aug. 2,70
B
Aug. 1.22
Avg. 2.02 dt
Avg: 0:35 y
mad 16.1
mask 9:38
mak: 74.17
mak:: 7915.0
123
t23
13
544
500
#takens
etolens
etoktos
200
#tokeos
Figure 13:
We visualize the activations of the SSM in the last layer of Mamba 2.8B. This figure shows that the values in the æ tensor are not significantly large. However, outliers are present in the tensor of the SSM output, making them difficult to represent in low bit-width data types.
Visualizing activations. We visualize these activation maps in Figures and 14 Figure 13 shows the SSM activation maps in the last layer of Mamba 2.8B, while Figure 14 shows those for the self-attention layer in Llama 3.2 3B. We use the same sample to visualize these activations.
24

---

Published as a conference paper at ICLR 2025
Notably, SSMs and self-attention layers exhibit distinct activation patterns. Outliers appear in the SSM output y, unlike in self-attention layers, where the outputs y remain smooth and do not present outliers. In Transformer blocks, outliers only occur in the h2 of the feedforward layers, making them difficult to quantize. In contrast, Mamba blocks have large outliers in the SSM output y tensor, which challenging represent with low-bit data types. While the tensor values not significantly large, they are highly sensitive to quantization errors. We address this issue in Mamba blocks and introduce Quamba manage the unique quantization patterns of SSMs.
Avg:
Avg:
Avg: K
Avg: 0,38
mas:: 5.67
200
max: 17.22
2000
max: 18.07
21.
max: 6:96
11
100
13.
200
2005
603
#tokons
100 di
#tobens
dim
#tokons
280
dim
#tokens
203
dm
Avg: 0.10 5.89
Avg: 0.30
hu
Avg: 0.16
10.42
130.13
12
inokens
#tolens
itokens
2000
Figure 14: We visualize the activations the self-attention layer in the last layer of Llama 3.2 3B. This figure shows that outliers present in the output y of the self-attention layers. In contrast to SSMs, the outlier issue arises in the hy tensor of the feedforward layers.
Comparing the precision mapping. Figure 15 compares precision mapping the standard
one used in Transformers (Zhao et al. 2023 Ashkboos et al. 2024b Lin et al. 2024 Xiao et al.
2023
Int8 Path
FP16 Path
Int8 OP
OP
Down Proj
(roj) ) n1
h
Selective SSM
Out Proj
Gate
HadQuant
d
y
y
Proj
Gated Proj
Self Gelf-attention
SSM
hg
Q
K
h
x Proj
RMSNorm
Q Proj
K Proj
Proj
Proj
gP
MHSA
Eth layer
Causal
h
RMSNorm
Zth layer
RMSNorm
hc
h
Precision Mapping of Quamba block (Ours)
(b) Precision Mapping of 8-bit Transformer block
Figure 15: We compare precision mapping the precision mapping commonly used in Trans- formers. The residual connections are simplified to avoid cluttering the figure.
25

---

Published as a conference paper at ICLR 2025
N QUANTIZATION ERROR ANALYSIS FOR SSMS
N.1 EMPIRICAL ANALYSIS OF A MATRICES
Given a discrete linear lime-nnvrrian (LTT) state space model: h(t) A(t)h(t -) + Bxx(t), we unroll the equation at time step t:
htt) A(t) + A(())(I - 1)Bx(t A(t) Bx(().
A(3)
We visualize the spectral norm of the cumulative production of matrices (i.e., (|A(t) ..... from pre-trained Mamba- 130m in Figure [6] The figure shows that SSM puts more weight on recent history by performing cumulative production of the A matrix. We leverage this property from SSMs to derive our quantization error bound and show the quantization error is bounded with respect to the input sequence length T'
A
Norm
A
AA
Spectral
AA---A--1
04
02
32
A3
Time Steps
Time Steps
(4) The first layer
(b) The last layer
Figure 16: The spectral norm of the cumulative production of A matrices from pre-trained Mamba- 130m, ie., [|A(t)A(t - 1) ..... 11...
N.2 THEORETICAL ERROR ANALYSIS
We consider discrete linear lmme-invariant (LTT) state space model at time step defined as h(t)
A(t)h(t- 1) Bx(t), where the state matrix A(t) E R N input matrix E XP. implicit
latent state htt) R x1, and input (x) E R Px1, Based on the behavior featured in Figure 16 we assume the norm of the state matrix can be bounded by an exponential function such that [IA(t)]l2 < a et
T--T, and the input matrix B is bounded by [ < b, where 0 -112). Furthermore, we assume the system input contains quantization error & (t) R Px1 such that 2x(1) x(t) + (t), where (t)l2 & and e 0. The system is initialized as h(0) = [0, 0, 0,...OTT.
1, >> 0, and
< < T. In our proof, we utilize the spectral norm ([
Theorem N.1. The quantization error of (tt) h(t) h(t) time step t for the given discrete linear time-invariant model is bounded such that: ||](t)]22 I cb (1-ae-r) Consequently. the global quantization error (i.e., t T) is bounded by | (T)
Proof. Given the quantization error [|52 (t)ll2 S c and the quantized input x(t) = x(t) - 6, (t) of each step, we have an original system h(t) and a quantized system h(t):
h(t) A(t)a(t 1) + Bx(t)
T((t) = A(t)T(( - 1) + ((x(t) + 5,
By subtracting two systems, have quantization error Att) Ti(t) h(t) given by: A(t)
A(1)(T(1 1) h(l 1)) + Bá_ A(I) A B6 (t). Let's consider the recurrence
26

---

Published as a conference paper at ICLR 2025
quantization error, assuming that A(0) 0:
t=,, ((1 = A(1) + (1x = (1)
2 ( (1)) (2) A1))
3,, (3) = A(3)A(2) + (62(3) = +(3) ((3) B3)
44 (4) A(4)A(3) (4) ((0448AAA2)8B(((+4)444444444
+ + (
At time step L, have
(t) = A(t)A(t - + + (t)
II 2 A (1) + (4) + II, A(M)86,)
(4)
+... + A(t)B8, 11
م ( (Lu) B8, - 11) + (t).
Let's consider quantization error (t) such that | = [[h(t) h()1l2:
||(()) = |ا [T A( (6) 1)) (61(
- - [1 [1 A
(u)B8,( 1)) ([)6
I _ [T
(6 1)) cb
I <b| E (TT
A(w)) + cb
I <b> +
A(u)l + +eb
[(4)] +eb
S = +
e" S <b( - "T + 1)
eb (b
1-5-7 +1)
eb(2
1=00 - 1)
ae
S <
+ +1)
T
ae' )
eb(
1
ac
< eb(
act-T
The time-step dependent bound becomes: [1A(t)|2 S eb( <b( Therefore, the global bound for given sequence length T (i.e., t is is AT))|
cb
I-a
O LIMITATIONS AND BROADER IMPACTS
We find that the accuracy degradation is not negligible in both accuracy and perplexity in Table 7 and Table Despite this, the performance trade-off is acceptable given the significant improvements in latency and resource efficiency. Our work enables large language models to be deployed on lesouree-limited devices. As positive feature, our method may push the development of privacy- centric on-device applications, where sensitive data can be processed locally without relying cloud services. However, our work may also present challenges such as increased device resource consumption and potential security vulnerabilities if the local devices compromised.
27


Reviews:
[{"review_id": "NxmpWR8SA1", "reviewer": "Reviewer_PfeA", "summary": "This work propose a PTQ solution for SSM model. Quamba notices the SSMs have highly sensitive feature maps within the selective scan mechanism (i.e., linear recurrence) and massive outliers in the output activations, and utilises two methods of outlier suppression - ①suppresses the maximum values of the input activations ②supply hadamard transform of the output activations, to solve the problem. Experiments show that show that this method can achieve 8-bit PTQ quantisation of SSM almost losslessly.", "questions": "- In line 228, you mentioned the **Theoretical error bound for SSM quantization**, and give a proof in appendix. However, this discussion seems to be isolated, so I do not particularly understand the connection between this discussion and the core approach of this paper.\n- In Table 1 and Table 2, you compared QuaRot and Quamba in W8A8 format. However, QuaRot mainly focus on **W4A4** solution for quantization. I would like to know if this method can be extended to W4A8 and W4A4 scenarios? If it can, how is his performance compared to QuaRot?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The approach is simple and effective, introducing little additional overhead\n- The experiment is solid: The author achieved real quantization - by handwriting fused operators and utilizing low-bit computational libraries (e.g. cutlass) to implement Quamba.", "weaknesses": "- The proposed method is effective, however, these methods are also widely used in Transformer, and authors may need to make their connection to SSM more explicit in their writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r9Qbtc1NHB", "reviewer": "Reviewer_TXWC", "summary": "This paper introduces a post-training quantization method tailored for Selective State Space Models (SSMs) to address challenges in model size reduction and computation efficiency, specifically in hardware-constrained environments. The authors propose an 8-bit static quantization approach that handles weight and activation quantization while mitigating the effects of outliers commonly found in SSM outputs.\n\nKey contributions of the paper include:\n1.  A method for input activation quantization that reduces the impact of extreme values by clipping based on the 99.999th percentile, improving quantization precision.\n2. Hadamard transforms to map output activations into a space with minimized outliers, allowing for accurate quantization without excessive memory and computational cost.\n3. Empirical analysis showing the effectiveness of this quantization approach in achieving a balance between model accuracy and reduced latency, making SSMs more feasible for edge and embedded applications.\n\nThe paper provides a comprehensive empirical evaluation, demonstrating how these techniques enable SSMs to perform efficiently on hardware with limited resources.", "questions": "1. As a researcher focused on large model quantization on edge devices, I have some concerns about the authors' claim regarding the lack of prominent outliers in transformers. Based on my own experiments and findings [1-2] from recent literature, larger language models (LLMs) actually tend to exhibit more significant outliers, especially in channel-wise activation. Can you clarify your claim about outliers in Transformers and provide evidence or citations supporting your view?\n\n2. Studies [3-4] on transformer quantization have shown that quantizing outliers can substantially impact model accuracy. This observation aligns with the authors' ablation study results. However, I am unclear on why certain techniques in this paper appear to alleviate the impact of outlier quantization. Additionally, shouldn’t each optimization step incrementally reduce accuracy? Surprisingly, the authors' ablation study shows an improvement in accuracy when all three techniques are applied together. Could you please provide a more detailed explanation of why your techniques, when combined, lead to improved accuracy rather than incremental reductions?\n\n3. In the introduction, the authors mention that this quantization method is applicable to both edge devices and cloud computing. However, the experiments only feature the high-compute edge device Orin Nano 8G (and the A5000, which would generally not be considered an edge device). There is no testing on cloud computing hardware, which would have been helpful.\n\n4. For the results in Table 1: Profiling Latency, I find some inconsistencies. As token length increases, both computation and memory should scale linearly, so I would expect a relatively stable speedup ratio. If anything, given the execution overhead of the nonlinear kernel within SSM, there might be a slight increase in latency with token length. However, the authors' data show an unusual spike in performance at $L=512$ and $L=1$. Could you please provide a more detailed explanation of the observed performance spikes, possibly including additional data points or experiments that could help clarify the relationship between token length and speedup ratio?\n\n5. Lastly, could the authors clarify how many hardware measurements were conducted to obtain the reported results? Given the inherent variability in hardware performance, multiple measurements are typically necessary to ensure stability and reliability. It would be helpful to understand the stability of the hardware results presented. Could you please provide details on the number of runs performed for each experiment, any measures taken to account for variability (e.g., averaging results, reporting standard deviations), and how you ensured the stability and reproducibility of your hardware measurements?\n\n[1] Shen, Xuan, et al. \"Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.\n[2] Xiao, Guangxuan, et al. \"Smoothquant: Accurate and efficient post-training quantization for large language models.\" International Conference on Machine Learning. PMLR, 2023.\n[3] Lin, Yang, et al. \"Fq-vit: Post-training quantization for fully quantized vision transformer.\" arXiv preprint arXiv:2111.13824 (2021).\n[4] Dong, Peiyan, et al. \"Packqvit: Faster sub-8-bit vision transformers via full and packed quantization on the mobile.\" Advances in Neural Information Processing Systems 36 (2024).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents a unique approach by applying quantization specifically to Selective SSMs, a topic with limited prior research in this area. The integration of percentile-based clipping and the Hadamard transform for managing outliers in SSMs represents an innovative combination of techniques. These methods are tailored specifically for the challenges of SSM quantization, marking a significant contribution to the field of model compression.\n2. The paper demonstrates a thorough experimental setup, with well-defined quantization methodologies for activations. The authors support their approach with empirical evidence, clearly detailing the implementation of 8-bit static quantization and its impact on latency and accuracy. The careful construction of quantization techniques for hardware constraints is grounded in a rigorous understanding of SSM behavior, lending credibility to the study’s findings.\n3. The proposed quantization approach has high potential significance, especially for applications involving SSMs in resource-limited environments. By addressing the specific challenges of quantizing SSMs, the paper extends the practical use of SSMs, making them feasible for deployment on low-power hardware without significant accuracy degradation.", "weaknesses": "The paper is innovative in its approach and thorough experimentation. However, there are several critical questions that I raised in the \"Question\" section, which I believe are essential for the clarity and robustness of the findings. I hope the authors can provide insights on these points, and I look forward to further discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xcxBvshcwv", "reviewer": "Reviewer_evxv", "summary": "The paper studies quantization of stat-space models. The propose Quamba which employs 8-bit quantization and Walsh-Hadamard transformations of activations to preserve accuracy.", "questions": "- Why is TTLT used? It makes more sense to evaluate latency (TTFT) and throughput (token-to-token).\n- Since extremely large outliers are identified as problematic for SSMs, have the authors considered using clipping to lower quantization noise. For example, OCTAV may be useful in this setup [1]\n- Empirical results with comparisons to transformers are focused on the W8A8 case. However, Transformers have been quantized to 4-bit (e.g., using Quarot).Is it therefore a fair comparison to use 8-bit transformers?\n\n[1] Sakr, Charbel, et al. \"Optimal clipping and magnitude-aware differentiation for improved quantization-aware training.\" International Conference on Machine Learning. PMLR, 2022.\n\n----\nGood responses, I am raising to 6.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- Studies quantization of SSMs, which is novel since SMMs themselves are new and quantization studies have not caught up with them.\n- The empirical results are not just scoped to accuracy studies using fake quantization, but actual profiling is done using INT8 and cutlass.", "weaknesses": "- Empirically, Quamba does not show advantages compared to Quarot.\n- MSE studies are used to compare difficulty of quantizing attention vs SSM (e.g., Fig 2). However, absolute numbers for MSE are not useful when inputs and models are different. A better way to visualize the quantization difficulty would be to employ a normalized MSE. In other words, the L2 norm of the delta divided by the L2 norm of the unquantized tensor, rather than just the L2 norm of the delta.\n- Theorem 4.1 is very badly written, I had to refer to the appendix to guess what the authors meant. Note that lowercase b and epsilon were never defined and they are the defining variables of the theorem! Please fix this and introduce b as being an element from the B matrix, and epsilon to be an upper bound of the quantization step. P.S., these are my guesses for what was meant. \n-While I appreciate Theorem 4.1 which states that quantization error compounds over time-steps (correct me if I am wrong). Where was this used? Which aspect of the proposed Quamba leverages Theorem 4.1? It seems like this was just added as decorative math.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JnLjYarPrj", "reviewer": "Reviewer_8GLX", "summary": "This paper proposes a post-training quantization method for state space models (SSMs), which enables the utilization of low-bit acceleration in modern GPUs and achieves notably lower generation latency and small accuracy penalty. The method is mainly comprised of two techniques: 1) static 8-bit per-tensor quantization for input activations, and 2) Hadamard transform for outlier-free output activation quantization.", "questions": "- Could the paper articulate more clearly what aspects of applying the two proposed techniques (clipping and Hadamard transform) to SSMs are novel or challenging compared to their use in other contexts?\n- Could the paper provide a better positioning of this paper, in terms of what the main difference is between the proposed method and the prior QuaRot? It seems that a trivial change on the original QuaRot (such as removing the extra Hadamard transform, which is a natural choice) can easily improve the performance.\n- In Figure 1 (b) and (c), could the paper provide the data for QuaRot-SSM for a more complete comparison?\n- In Figure 2 (b) and (c), it is unclear what y_t means. And why are there multiple (colors of) MSE curves? For example, for the curve of V_bar (red one in (b)), does it still represent the MSE of y_t and how should we understand this curve? Figure 3 (a) is much clearer in this sense.\n- Is it possible to support 4-bit quantization with the proposed method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Overall, the paper is well-structured, well-motivated, and well-written. The right amount of background introduction, motivating examples, and explanation of challenges make this paper easy to understand.\n- The paper did thorough experiments to show the generalizability and scalability of the proposed method on various SSM-based implementations. In the results, the proposed Quamba shows a clear latency improvement and makes it possible to run large-scale models on devices with a limited amount of memory.", "weaknesses": "- The novelty of the proposed method is not clear. As mentioned in the text, the paper acknowledges that clipping outliers in input activations with a percentile max is a widely applied technique, and smoothing out the outliers with the Hadamard transform is also not new and has been applied in prior quantization methods for transformer attention computation. The main novelty of this paper therefore may come from being the first to apply these two techniques to SSM.\n- There is no significant benefit over QuaRot-SSM, which is a re-implementation of QuaRot for Mamba. The positive side is that this does serve as a strong baseline, however, their performance is very similar in many experiments. Figure 7 in the appendix, to some extent, also shows that the differences in architecture are small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)