Quamba: A Post-Training Quantization Recipe for Selective State Space Models

## Abstract

Abstract State Space Models (SSMs) have emerged as an appealing alternative to Transformers for large language models, achieving state-of-the-art accuracy with constant memory complexity which allows for holding longer context lengths than attention-based networks.
The superior computational efficiency of SSMs in long sequence modeling positions them favorably over Transformers in many scenarios.
However, improving the efficiency of SSMs on request-intensive cloud-serving and resource-limited edge applications is still a formidable task.
SSM quantization is a possible solution to this problem, making SSMs more suitable for wide deployment, while still maintaining their accuracy.
Quantization is a common technique to reduce the model size and to utilize the low bit-width acceleration features on modern computing units, yet existing quantization techniques are poorly suited for SSMs.
Most notably, SSMs have highly sensitive feature maps within the selective scan mechanism ( i.e., linear recurrence) and massive outliers in the output activations which are not present in the output of token-mixing in the self-attention modules.
To address this issue, we propose a static 8-bit per-tensor SSM quantization method which suppresses the maximum values of the input activations to the selective SSM for finer quantization precision and quantizes the output activations in an outlier-free space with Hadamard transform.
Our 8-bit weight-activation quantized Mamba 2.8B SSM benefits from hardware acceleration and achieves a 1.72 × \times × lower generation latency on an Nvidia Orin Nano 8G, with only a 0.9% drop in average accuracy on zero-shot tasks.
When quantizing Jamba, a 52B parameter SSM-style language model, we observe only a 1 % percent 1 1\% 1 % drop in accuracy, demonstrating that our SSM quantization method is both effective and scalable for large language models, which require appropriate compression techniques for deployment.
The experiments demonstrate the effectiveness and practical applicability of our approach for deploying SSM-based models of all sizes on both cloud and edge platforms.
Code is released at https://github.com/enyac-group/Quamba .

## 1 Introduction

State Space Models (SSMs) \parencitegu2023mamba, lieber2024jamba
have attracted notable attention due to their efficiency in long sequence modeling and comparable performance to Transformers \parencitezhang2022opt, brown2020language.
Although Transformers have shown a strong ability to capture the causal relationships in long sequences, the self-attention module within Transformers incurs a quadratic computation complexity with respect to the context length in the prefilling stage as well as a linear memory complexity (*e.g.,* the K-V cache) in the generation stage.
In contrast, SSMs, an attractive substitute to Transformers, perform sequence modeling with a recurrent neural network-like (RNN-like) linear recurrence module (*i.e.,* selective scan) which has *linear* computation complexity in the prefilling stage and *constant* memory complexity in the generation stage.

Figure: Figure 1: We demonstrate that (a) our method achieves Pareto-optimality on the Nano 8G with 1K input tokens, (b) provides latency speedups for long input sequences on the A5000, and (c) maintains consistent memory usage across devices comparing to Pythia \parencitebiderman2023pythia, an iso-size Transformer. SmQ-SSM and QuaRot-SSM represent our re-implementations of SmoothQuant \parencitexiao2023smoothquant and QuaRot \parenciteashkboos2024quarot for the Mamba architecture \parencitegu2023mamba.
Refer to caption: extracted/5933363/figures/combined_latency_memory_requirements_side_by_side_2.png

However, despite their appealing computational attributes, deploying SSMs across a wide variety of hardware platforms is challenging due to their memory and latency constraints.
SSM quantization is a possible solution to this problem as it reduces model size and latency with low bit-width data types while preserving accuracy.
For example, 8-bit integer (INT8) quantization halves memory usage compared to FP16 and poised to benefit from hardware acceleration.
Quantizing SSMs is non-trivial as current post-training quantization (PTQ) techniques for Transformers \parencitexiao2023smoothquant, dettmers2022gpt3 fail to handle the sensitive activations of the selective scan (*i.e.*, linear recurrence) resulting in poor performance \parencitelieber2024jamba, zhao2024cobra.
Our study shows SSMs exhibit *distinct* outlier patterns compared to Transformers \parenciteashkboos2024quarot,dettmers2022gpt3, xiao2023smoothquant, zhao2023atom.
We show that outliers appear in the SSMs output (*i.e.* the $y$ tensor).
In contrast, the input and output of self-attention layers are relatively *smooth* and *do not* exhibit outlier issues.
More comparisons can be found in Section I.
This highlights that different quantization methods are needed for SSM-based models.
Due to the absence of an appropriate quantization technique for the selective scan mechanism, SSMs exhibit sub-optimal memory usage and latency on deployed hardware.

We first analyze the input and output activations of the selective scan module to reveal the quantization sensitivity and outliers in SSM activations (*ref.* Figure 2 and Figure 3).
Specifically, we show that quantizing SSMs is particularly challenging since SSM input and output activations present a causal relationship, making the input tensor (*i.e.,* $x$ in Eq. 1) sensitive to quantization errors.
Additionally, representing the large outliers using 8-bit precision in the output activations (*i.e.,* $y$ in Eq. 1), which are *not* present in the token-mixing output of self-attention modules, is difficult.
To address this, we propose Quamba, an 8-bit *static per-tensor* quantization method for selective SSMs that can leverage the low bit-width acceleration features on modern computing units with *minimal overhead*.
We suppress maximum values in input activations to SSMs, which is the most sensitive to the quantization error, for finer quantization precision.
For the extreme outliers in output activations from SSMs, we use the Hadamard transform to smooth out the activations.
Our quantized 8-bit 2.8B Mamba SSM benefits from the hardware acceleration and improves the generation latency by $1.72\times$ (*i.e.,* time-per-output-token, TPOT) on Nvidia Orin Nano 8G while only incurring a 0.9% accuracy drop in zero-shot tasks.
We demonstrate our method achieves Pareto-optimality on the Nano 8G and lower the latency by 1.2 $\times$ on A5000, as shown in Figure 1.
While quantizing Jamba, a 52B parameter SSM-style language model, we observe just a $\sim 1\%$ reduction in accuracy.
The effectiveness and scalability of our quantization technique for SSM-based models, as well as the practicality of our approach for deploying SSM-based models of various sizes on cloud and edge platforms.

Figure: Figure 2: We analyze the sensitivity of quantization errors for (b) self-attention layers and (c) SSMs input activations. Our study shows that the $x$ tensor causes huge errors at the output $y$ due to the causal relationship of the linear recurrence, which is unique to SSMs. Self-attention layers are more robust to quantization errors. Our method (d) reduces the quantization error for the input sample. In Figure 11, we highlight the smooth, outlier, and sensitive paths in SSMs and self-attention layers.
Refer to caption: extracted/5933363/figures/Sensitivity_2.png

## 2 Related Work

#### Model quantization.

Deploying large, over-parameterized neural networks on resource-constrained devices is challenging, and researchers have developed model quantization \parencitehan2015deep, jacob2018quantization, wang2019haq as a solution to this problem.
Quantization techniques reduce the data type precision (e.g. FP32 to INT4) to compress the model size and accelerate inference.
The quantization techniques are generally divided into two categories: Quantization-aware training (QAT) and post-training quantization (PTQ) \parencitezhu2023survey, gholami2022survey, zhou2024survey.
QAT \parenciteliu2023llm, dettmers2024qlora requires additional training efforts to adapt models to low bit-width, in exchange for better model performance.
Our work falls under PTQ, which does not require training and can therefore be plug-and-play.

#### LLM post-training quantization.

Post-training quantization (PTQ) techniques are generally broken down into two categories: weight-only quantization (*e.g.,* W4A16 and W2A16) and weight-activation quantization (*e.g.,* W8A8 and W4A4) \parencitezhu2023survey.
Weight-only quantization \parencitefrantar2022gptq, lin2023awq focuses on quantizing weight matrices (*e.g.,* 4-bit or 2-bit) while keeping the activations in half-precision floating point.
However, although weight-only quantization reduces memory usage, it still requires costly floating-point arithmetic for compute-intensive operations (*e.g.,* linear layers).
To utilize low bit-width operations, \textcitexiao2023smoothquant, zhao2023atom, ashkboos2024quarot, dettmers2022gpt3 study quantization for both weights and activations in Transformers.
They address outliers in activations by using mixed-precision \parencitedettmers2022gpt3, rescaling quantization factors \parencitexiao2023smoothquant, group quantization \parencitezhao2023atom, and quantizing activations in an outlier-free space \parenciteashkboos2024quarot.
Unfortunately, these techniques target Transformers, do not generalize well to SSMs, and either fail to handle the sensitive tensors in SSMs resulting in poor performance \parencitexiao2023smoothquant, dettmers2022gpt3 or introduce additional computational overhead to the input of the selective scan \parencitezhao2023atom, ashkboos2024quarot.
Our research addresses this gap by examining SSM weight-activation quantization, aiming to concurrently reduce memory and compute costs by harnessing hardware acceleration for integer operations.

#### State Space Models.

In recent times, a new wave of RNN-like models \parencitegu2021efficiently, smith2022simplified, peng2023rwkv, gu2023mamba, beck2024xlstm has emerged, noted for their efficacy in modeling long-range dependencies and achieving performance comparable to Transformers.
State Space Models (SSMs) \parencitegu2021efficiently, smith2022simplified, gu2023mamba are a promising class of architectures that have been successfully applied to various applications, such as text \parencitegu2023mamba, wang2024mambabyte, image \parencitezhu2024vision, nguyen2022s4nd, video \parenciteli2024videomamba, nguyen2022s4nd, and audio \parencitegoel2022s, saon2023diagonal.
Despite their successes, the challenge of deploying SSMs across resource-limited hardware platforms remains largely unresolved.
Our work addresses this challenge by proposing a quantization method specifically tailored for SSMs.

## 3 Background

### 3.1 Selective State Space Models

#### State Space Models.

Inspired by continuous systems, discrete linear time-invariant (LTI) SSMs \parencitegu2020hippo, gu2021efficiently, gu2022parameterization map input sequences $\mathbf{x}$ to output sequences $\mathbf{y}$.
Given a discrete input signal $x_{t}$ at time step $t$, the transformation $x_{t}\mapsto y_{t}$ through a hidden state $h$ is defined as

$$ $\displaystyle h_{t}=\dot{A}h_{t-1}+\dot{B}x_{t},\quad y_{t}=Ch_{t}+Dx_{t}$ (1) $$

where $\dot{A}$ and $\dot{B}$ are discrete parameters.
The discretization function for $\dot{A}$ and $\dot{B}$ with a given $\Delta$ is defined as $\dot{A}=\exp(\Delta A)$, $\dot{B}=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot\Delta B\approx\Delta B$.
This system uses $A$ as a state transition parameter and $B$ and $C$ as projection parameters.
$\Delta$ is the time-scale parameter that is used to discretize the continuous parameters $A$ and $B$.
$D$ is an optional residual parameter.
$(A,B,C,D,\Delta)$ are trainable parameters.
A residual branch $z_{t}$ is applied to the SSM output such that $y_{t}\cdot\text{SiLU}(z_{t})$ before the output projection.

#### SSMs with selection.

gu2023mamba improve SSMs with selection by letting their parameters $B$, $C$, and $\Delta$ be input-dependent, allowing the model to selectively remember or ignore inputs based on their content.
Specifically, the interaction with input $x_{t}$ is defined as $B_{t}=\text{F}_{B}(x_{t}),\quad C_{t}=\text{F}_{C}(x_{t}),\quad\Delta_{t}=
\text{softplus}(\text{F}_{\Delta}(x_{t}))$
where $\text{F}_{B}$ and $\text{F}_{C}$ are linear layers that map $x_{t}\mapsto B_{t},C_{t}$.
$\text{F}_{\Delta}$ use two consecutive projection layers, such that $\text{F}_{\Delta}=\text{Proj}(\text{Proj}(x))+\text{bias}$.
With the selection mechanism, the model has changed from time-invariant to time-varying.

### 3.2 Quantization

We focus on symmetric uniform quantization to approximate floating-point weights and activations with discrete 8-bit signed integers (*i.e.,* INT8) due to its hardware compatibility.
The general symmetric uniform quantization function is defined as

$$ $\overline{X}=\text{clamp}\Big{(}\left\lfloor\frac{X}{s}\right\rceil,-2^{N-1},2 ^{N-1}-1\Big{)},\quad s=\frac{\text{max}\big{(}\left\lvert X\right\rvert\big{) }}{2^{N-1}-1},$ (2) $$

where $\overline{X}$ represents the quantized weights or activations in INT8, $X$ is the input matrix in floating point, and $s$ is the scaling factor (*i.e.,* quantization step) that is determined by the target bit-width $N$ ($N=8$ in our setting).
The *static* scaling factor $s$ is pre-calibrated on a subset of data and is *fixed* during inference.
We use the notation $X$ to represent the floating-point matrices, and $\overline{X}$ to represent their quantized matrices with their floating-point scaling factors $s_{x}$.
For operators, we use $\overline{f}(\cdot)$ to represent the quantized version of the function $f(\cdot)$ (*i.e.,* the weights are quantized in the function $\overline{f}$).

### 3.3 Walsh–Hadamard Transform

#### Hadamard matrices.

A Hadamard matrix is an $n$-dimensional square matrix whose entries are either $+1$ or $-1$, and the rows and columns are mutually orthogonal with the computational property $\mathbf{H}_{n}\mathbf{H}_{n}^{\top}=n\mathbf{I}_{n}$.
Walsh-Hadamard matrix is a special category of Hadamard matrix, consisting of square matrices of size $2^{k}$ and can be constructed as follows:

$$ $\mathbf{H}_{2^{k}}=\begin{bmatrix}\mathbf{H}_{2^{k-1}}&\mathbf{H}_{2^{k-1}}\\ \mathbf{H}_{2^{k-1}}&-\mathbf{H}_{2^{k-1}}\end{bmatrix}=\mathbf{H}_{2}\otimes \mathbf{H}_{2^{k-1}}\quad\text{, where}\quad\mathbf{H}_{2}=\begin{bmatrix}1&1 \\ 1&-1\end{bmatrix}.$ $$

Figure: Figure 3: The primary difficulties in quantizing Mamba blocks lie in the precision of the activations input into and output from the selective SSM. Although inputs are numerically small, the quantization step is skewed by the maximum value, causing significant errors in the output SSMs after the linear recurrent system. In contrast, large outliers are observed in the outputs. We use Hadamard matrices to transform the outputs to an outlier-free space.
Refer to caption: extracted/5933363/figures/outliers.png

#### Walsh–Hadamard transform.

The Walsh–Hadamard transform (WHT), a generalized class of Fourier transforms, has been applied to many related areas, such as LLM quantization \parenciteashkboos2024quarot and efficient transfer learning \parenciteyang2024efficient, due to its favorable computational properties.
We perform WHT to remove outliers from the output of the selective SSM.
WHT projects a discrete input sequence signal onto a set of square waves (*i.e.,* Walsh functions).
Its forward and inverse transform can be expressed in matrix form as $\tilde{x}=\mathbf{H}_{n}x\quad\text{, and}\quad x=\mathbf{H}_{n}^{\top}\tilde{x}$.
$x$ is the input discrete sequence signal, and $\tilde{x}$ denotes the WHT coefficients (*i.e.,* sequence components) that describe the magnitude of the square waves.
WHT is efficient since the transform matrices consist only of real numbers, +1 or -1, so no multiplications are needed.
The fast Walsh–Hadamard transform (FWHT) can compute the transformation with a complexity of $n\text{log}n$ in a GPU-friendly (*i.e.,* parallelizable) fashion \parencitetridao_had.
For input dimension $n\neq 2^{k}$, we factorize $n=2^{p}m$, where $m$ is the size of a known Hadamard matrix \parencitesloane1999library.

## 4 Quamba: Quantizing Mamba Blocks

### 4.1 Preliminary Study

#### Theoretical error bound for SSM quantization.

For a comprehensive analysis of our proposed Mamba block quantization approach, we derive a theoretical error bound of a discrete 1D linear time-invariant (LTI) SSM. The details of the proof can be found in Section A in Appendix.

###### Theorem 4.1 .

The quantization error of $h[t]$ from each time step of the given discrete 1D linear time-invariant model is bounded by $b\epsilon\frac{e^{t-T}}{e-1}$ such that $|h[t]-\overline{h}[t]|\leq b\epsilon\frac{e^{t-T}}{e-1}$ .

Figure: Figure 4: The precision mapping and dataflow of Quamba. All scaling factors are fused in the quantized operations. Element-wise operations like non-linearity and residual addition are also fused into these operations.
Refer to caption: extracted/5933363/figures/quamba_overview.png

###### Theorem 4.1 .

#### Empirical analysis of SSM quantization.

As shown in Figure 2 and Figure 3, the main challenge in quantizing Mamba blocks is the precision of the SSM input activation $x$, which is sensitive to quantization errors, and the output activation $y$.
In Figure 1 (a), the naive 8-bit quantization introduces large quantization errors, resulting in model collapse for all model sizes.
We delve into the causal relationship between $(A,B,C,\Delta,x)$ and $y$ as modeled by the linear recurrent system in Equation 1.
As shown in Figure 2 (b) and Figure 3 (a), we find that $x$ is sensitive to quantization errors and it leads to the largest errors in the SSM output $y$, although the values in the $x$ tensor are numerically small (ref: Figure 12).
We conjecture the reason behind the phenomenon is that $(B,C,\Delta)$ are input-dependent (*i.e.,* $x$-dependent).
We note that this finding is specific to SSMs.
As shown in Figure 2 (a), self-attention layers are more resilient to quantization errors and do not experience the same issues.

#### SSM outliers.

Our study shows SSMs exhibit distinct outlier patterns compared to Transformers \parenciteashkboos2024quarot,dettmers2022gpt3, xiao2023smoothquant, zhao2023atom.
We show that outliers appear in the SSMs output (*i.e.* the $y$ tensor), which perform a similar token-mixing function to self-attention layers.
In contrast, the input and output of self-attention layers are relatively smooth and do not exhibit outlier issues.
More comparisons can be found in Section I.
This highlights that different quantization methods are needed for SSM-based models.
Therefore, we tailor two different quantization techniques: percentile-based quantization for the input activation $x$ and quantizing the output $y$ in an outlier-free space using the Hadamard transform.
Our method recovers performance by improving the quantization precision for the inputs and outputs of the SSM and achieves a better trade-off between accuracy and latency.

### 4.2 Quantization for Selective SSM

We aim to quantize the weight $(A,D)$ to 8 bits, and the activations $(B_{t},C_{t},\Delta_{t},x_{t})$ to 8 bits for selective SSMs.
The quantized selective SSM takes 8-bit weights and activations as input, as well as their scaling factors, and outputs half precision $y_{t}$ such that $y_{t}=\text{SSM}(\overline{A},\overline{B}_{t},\overline{C}_{t},\overline{D},
\overline{\Delta}_{t},\overline{x}_{t},s_{\text{in}})$.
$(\overline{A},\overline{B}_{t},\overline{C}_{t},\overline{D},\overline{\Delta}
_{t},\overline{x}_{t})$ are the quantized weights and activations, and their scaling factor $s_{\text{in}}$ in floating point.
$(\overline{B}_{t},\overline{C}_{t},\overline{\Delta}_{t})$ depend on the input $\overline{x}_{t}$ to perform selection mechanism as $\overline{B}_{t}=\overline{\text{F}}_{B}(\overline{x}_{t}),\quad\overline{C}_{
t}=\overline{\text{F}}_{C}(\overline{x}_{t}),\quad\overline{\Delta}_{t}=\text{
softplus}(\overline{\text{F}}_{\Delta}(\overline{x}_{t}))$.
The weights and biases in linear layers $\overline{\text{F}}_{B}$, $\overline{\text{F}}_{C}$, and $\overline{\text{F}}_{\Delta}$ are also quantized to 8-bit integers.
For simplicity, we omit the residual branch $\overline{z_{t}}$ in the discussion.

#### SSM inputs.

Our findings show that $x$ is highly sensitive to quantization errors, which results in the largest errors in the SSM output $y$.
Specifically, we found the quantization error of $x$ is dominated by outliers during the calibration.
Although they are numerically small ($\leq 10$), as shown in Figure 12, the small amounts of outliers ($\leq 0.1\%$) increase the quantization step (*i.e.,* scaling factors $s$ in Eq.2) and reduce the quantization precision for $x$.
Clipping the values with a percentile max \parencitezhao2022analysis, li2019fully is a simple solution to restrict the activation range and has no additional overhead during inference.
For example, using the 99th percentile to clip the top 1% of the highest activation values prevents the activation range from being skewed by extreme outliers.
We use percentile max to calculate the scaling factor for $x$: $s_{x}=(\text{max}^{p}\big{(}\left\lvert x\right\rvert\big{)})/(2^{N-1}-1)$,
where $p$ is a parameter for percentiles.
In our experiments, we found $p=99.999$ works well for Quamba.

#### SSM outputs.

We perform WHT to remove the outliers from the SSM output.
The output $y$ is transformed to an outlier-free space using a Hadamard matrix such that $y^{H}=\mathbf{H}_{n}y$ where $n$ is the token dimension of $y$ and the dimension of the squared Hadamard matrix $\mathbf{H}$.
We fuse the inverse Hadamard matrix into the output linear projection $\mathbf{W}_{\text{out}}^{H}=\mathbf{H}_{n}\mathbf{W}_{\text{out}}$ to avoid additional computation overhead and achieve compute-invariance \parenciteashkboos2024slicegpt, ashkboos2024quarot (*i.e.*, the same output) by
$\text{Mamba}_{\text{Output}}=\mathbf{W}_{\text{out}}^{\top}y=\mathbf{W}_{\text
{out}}^{\top}\mathbf{I}y=\mathbf{W}_{\text{out}}^{\top}(\frac{1}{n}\mathbf{H}_
{n}^{\top}\mathbf{H}_{n})y=\frac{1}{n}(\mathbf{W}_{\text{out}}^{H})^{\top}y^{H}.$
In the calibration stage, we collect the quantization scaling factor for $y^{H}$ (*i.e.,* transformed $y$).
Therefore, the fused Hadamard quantization layer can be expressed as

$$ $\overline{y}^{H}=\text{clamp}\Big{(}\left\lfloor\frac{y^{H}}{s_{y}}\right \rceil,-2^{N-1},2^{N-1}-1\Big{)},\quad s_{y}=\frac{\text{max}\big{(}\left \lvert y^{H}\right\rvert\big{)}}{2^{N-1}-1}\quad$ (3) $$

where $N$ represents the target bit-width.
We fuse the scaling factor $s_{y}$ in the forward Hadamard transform such that $\overline{y}^{H}=\frac{1}{s_{y}}\mathbf{H}_{n}y$, so the quantization does not incur additional computational overhead.
The fused Hadamard quantization layer is parallelizable and efficient on GPU with a complexity of $n\text{log}n$ \parencitetridao_had.

### 4.3 Other Operators

#### Projection layers.

Projection layers, which perform dense matrix multiplications, are the most over-parameterized and the most compute-intensive operators in the models.
With quantized activations and weights, projection layers benefit from hardware acceleration (*e.g.*, Tensor Cores) for 8-bit integers as well as reducing the memory costs by half.
We implement the 8-bit linear layers with commercial libraries, except for the output projection, which produces half-precision outputs for the subsequent normalization layer.

#### Fused causal convolution.

Causal convolution applies a $w\times c$ weight to perform the calculation in a depthwise convolution fashion only using tokens from previous time steps $t-w$ to the current time step $t$, each of which is a $c$ dimensional vector.
The operator is memory-bound, as typical depthwise convolution \parencitelu2021optimizing, zhang2020high, so applying quantization to the input and output activations and weights largely reduces memory pressure.
We quantize the inputs and weights as 8-bit integers and fuse the quantization operator before writing the result to memory.
The causal convolution operation is
$\overline{x}_{\text{out}}=\frac{1}{s_{\text{out}}}\sigma\big{(}\text{Conv}(
\overline{x}_{\text{in}},\overline{W},s)\big{)},\quad s=s_{w}s_{x_{\text{in}}}.$
The SiLU \parenciteelfwing2018sigmoid $\sigma$ is fused into the convolution, as described by \textcitegu2023mamba.

#### Fused RMSNorm.

We implement an operator that fuses the residual addition and *static* quantization with RMSNorm \parencitezhang2019root.
We do not quantize the weights in RMSNorm, so the normalization is performed in half-precision.
The fused operator takes as input a half-precision tuple $(x_{\text{out}},\;x_{\text{res}})$, where $x_{\text{out}}$ is the output from the Quamba block and $x_{\text{res}}$ is the residual.
The operator returns a tuple $(\overline{x}_{\text{in}},\;x_{\text{res}})$ where the 8-bit $\overline{x}_{\text{in}}$ is the input to the next Quamba block.
The operator can be expressed as
$(\overline{x}_{\text{in}}^{L+1},\;x_{\text{res}}^{L+1})=\Big{(}\frac{1}{s_{
\text{out}}}\text{RMSNorm}(x_{\text{out}}^{L}+x_{\text{res}}^{L}),\;x_{\text{
out}}^{L}+x_{\text{res}}^{L}\Big{)}$
where the $L$ is the layer number in the model, and the scaling factor $s_{\text{out}}$ is pre-calibrated.

## 5 Experiments

### 5.1 Experimental Setup

#### Model and datasets.

We evaluate Quamba on the open-sourced Mamba family of SSMs \parencitegu2023mamba and on Jamba \parenciteJamba, a hybrid architecture composed of self-attention, SSMs, and Mixture of Experts (MoE).
For zero-shot tasks, we use LM-EVAL \parenciteeval-harness, on which we evaluate baselines and Quamba on LAMBADA \parenciteLambdataDataset, HellaSwag \parenciteHellaSwag, PIQA \parencitePIQADataset, ARC \parencitearcDataset and WinoGrande \parenciteWinoGrandeDataset.
Model accuracy on each dataset and the average accuracy are reported.
We follow the evaluate protocol with Mamba [gu2023mamba], and report the accuracy for LAMBADA, WinoGrande, PIQA, and ARC-easy, and accuracy normalized by sequence length for HellaSwag and ARC-challenge.
For perplexity, we evaluate the models using the testing set of WikiText2 \parencitemerity2016pointer and a randomly sampled subset from validation set of Pile dataset \parencitePileDataset.

#### Quantization setup.

The calibration set is constructed by randomly sampling 512 sentences from the Pile dataset \parencitePileDataset.
We collect the *static* scaling factors for each operator based on the absolute maximum value observed from the calibration set and apply *symmetric per-tensor* quantization for weights and activations, except for the input to the SSM, where we use the $99.999$th percentile (*i.e.,* the $p$ described in Section 4.2) to clip the maximum values.
The same scaling factors are applied in all our experiments.
We note that our method does not require extra training efforts and can be plug-and-play.

#### Implementation.

We implement the INT8 linear layer using CUTLASS library \parenciteThakkar_CUTLASS_2023.
Quantization is integrated and adapted into the CUDA kernels of both the fast Hadamard transform \parencitetridao_had and causal convolution \parencitetridao_conv.
Additionally, the selective SSM CUDA kernel \parencitegu2023mamba is modified to accommodate inputs with quantized weights and activations, and their scaling factors.
We profile the latency and memory usage of baselines and Quamba on A5000 GPUs and Nvidia Orin Nano 8G.
For the latency, we perform a few warm-up iterations and then report the average latency of the following 100 iterations.

#### Baselines.

In our W8A8 setting, we compare Quamba with static quantization, dynamic quantization, and Mamba-PTQ \parencitepierro2024mamba.
We re-implement the state-of-the-art W8A8 SmoothQuant (SmQ) \parencitexiao2023smoothquant and W4A4 QuaRot \parenciteashkboos2024quarot in Transformers for 8-bit weight-activation SSM quantization as additional baselines.
These are denoted by SmQ-SSM and QuaRot-SSM, respectively.
Since QuaRot fails to quantize SSMs with W4A4 precision, we re-implement QuaRot to support W8A8 to fit our setting.
We include the details of our QuaRot re-implementation in Section C.
For the re-implemented SmoothQuant \parencitexiao2023smoothquant, we apply a smoothing factor $\alpha=0.5$ to all linear layers within Mamba SSM.
We find the low bit-width quantization methods for Transformers do not generalize well to SSMs and quantizing SSMs with low-bit-width remains unexplored, we include some of the results in Section E.

**Table 1: Profiling latency for Mamba and Quamba 2.8B on Nvidia A5000 GPU and Orin Nano 8G. The latency is measured milliseconds (*ms*). SmQ-SSM and QuaRot-SSM represent our re-implementations of SmoothQuant \parencitexiao2023smoothquant and QuaRot \parenciteashkboos2024quarot for the Mamba architecture.**
| Method | Precision | Size (G) | A5000 | Orin Nano 8G |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L=1 | L=512 | L=1024 | L=2048 | L=1 | L=512 | L=1024 | L=2048 |  |  |  |
| SmQ-SSM | W8A8 | 2.76 | 6.81 | 43.85 | 79.09 | 151.56 | 56.53 | 572.36 | 1123.91 | 2239.63 |
| Quarot-SSM | 10.46 | 56.87 | 103.98 | 199.83 | 67.76 | 746.31 | 1453.91 | 2892.95 |  |  |
| Quamba (Ours) | 8.12 | 48.24 | 84.84 | 165.13 | 60.17 | 607.25 | 1181.09 | 2354.85 |  |  |
| Mamba | FP16 | 5.29 | 9.86 | 61.19 | 102.29 | 184.16 | 103.56 | 730.16 | 1634.08 | 2756.28 |
| Quamba Reduction (Ours) | - | 1.91 $\times$ | 1.21 $\times$ | 1.27 $\times$ | 1.21 $\times$ | 1.12 $\times$ | 1.72 $\times$ | 1.20 $\times$ | 1.38 $\times$ | 1.17 $\times$ |

### 5.2 Model size and Latency

Quamba reduces the size of the 2.8B model nearly by half ($5.29$ GB *vs.* $2.76$ GB) by quantizing weights as 8-bit integers except for normalization layers, as shown in the Table 1.
We profile the latency on an A5000 GPU and an Orin Nano 8G for cloud and edge deployment scenarios.
Quamba enjoys the 8-bit acceleration and improves the latency by $1.27\times$ with a 512 input context (*i.e.,* time-to-first-token, TTFT) on the A5000 GPU and $1.21\times$ in the generation stage (*i.e.,* L=1, time-per-output-token, TPOT).
On the Orin Nano, Quamba improves the latency by $1.2\times$ with a 512 input context and $1.72\times$ in the generation stage.
Figure 9 shows the snapshot of real-time generation on Nano 8G.
Despite that having similar accuracy to the re-implemented QuaRot \parenciteashkboos2024quarot dubbed as QuaRot-SSM, Quamba delivers a better speedup on both A5000 and Nano, since QuaRot-SSM requires extra transpose and Hadamard transforms to handle the SSM input activations.
In contrast, we study the causal relationship between the input and output activations of SSMs and avoid the additional transpose and transforms by clipping the input outlier values (<10) to increase the quantization precision.
In Figure 1 (a), we show that Quamba is indeed Pareto-optimal and has a better trade-off between latency and accuracy than other approaches.
Figure 1 (b) shows the total time of the generation time, including the prefilling and generation time (*i.e.,* the time to last token, TTLT).
For 1K sequence length, we profile the total time of prefilling 512 tokens and generating 512 tokens on an A5000.
Quamba improves the TTLT by $1.2\times$ compared with Mamba.
Compared with Pythia \parencitebiderman2023pythia, more latency improvement is observed as the sequence length is increased, since SSMs do not require K-V cache for the generation.

**Table 2: Perplexity results of different quantization methods applied on Mamba \parencitegu2023mamba family of SSMs. We evaluate the quantized models on a subset of Pile and Wikitext2 datasets. We re-implement SmoothQuant (SmQ-SSM) \parencitexiao2023smoothquant and QuaRot (QuaRot-SSM) \parenciteashkboos2024quarot for 8-bit (W8A8) Mamba quantization. Quamba is a *static per-tensor* quantization method that closes the performance gap and delivers a better trade-off between latency and accuracy (ref. Figure 1 (a)). We show the generation latency (*ms*) of the 2.8B model on Orin Nano in this table, please find complete results in Table 1. Bold is the best, and underline is the second best.**
| Model | Methods | Wikitext2 Perplexity (↓) | Pile Perplexity (↓) | latency (↓) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 130M | 370M | 1.4B | 2.8B | 130M | 370M | 1.4B | 2.8B | 2.8B |  |  |
| Pythia | FP16 | – | – | 15.14 | 12.68 | – | – | 7.62 | 6.89 | - |
| SmQ | – | – | 38.24 | 14.03 | – | – | 14.86 | 7.37 | - |  |
| Mamba | FP16 | 20.61 | 14.31 | 10.75 | 9.45 | 11.50 | 8.81 | 7.11 | 6.39 | 103.56 |
| dynamic | 57.18 | 24.58 | 19.32 | 16.92 | 28.74 | 15.28 | 12.37 | 10.56 | - |  |
| static | 139.90 | 84.69 | 60.87 | 78.63 | 62.48 | 40.43 | 32.33 | 35.08 | - |  |
| SmQ-SSM | 29.51 | 19.29 | 14.23 | 13.59 | 15.81 | 11.40 | 9.14 | 8.59 | 56.53 |  |
| QuaRot-SSM | 32.43 | 16.29 | 11.39 | 9.89 | 16.78 | 9.87 | 7.49 | 6.66 | 67.76 |  |
|  | Quamba (Ours) | 25.09 | 16.18 | 11.35 | 9.91 | 13.63 | 9.84 | 7.49 | 6.67 | 60.17 |

### 5.3 Perplexity Evaluation

Table 2 presents the perplexity results of Quamba and the baseline methods on the Mamba family of SSMs.
Static quantization fails to maintain the precision in SSM quantization, resulting in significant performance degradation (*i.e.,* increased perplexity, where lower is better).
Even with scaling factors calculated dynamically, which introduces significant computational overhead during inference, it still results in a considerable increase in perplexity ($+7.5$) on Mamba-2.8B.
Although SmQ-SSM mitigates the issue of outliers in the output of the SSM, a performance gap remains when compared to the non-quantized Mamba because it fails to address the issue of quantizing sensitive $x$ input tensors to SSMs.
Quamba achieves similar perplexity to the re-implemented QuaRot-SSM \parenciteashkboos2024quarot but delivers a better speedup on both A5000 and Nano, as shown in Table 1 and Figure 1 (a).
Since QuaRot-SSM is not optimized for SSMs, it requires extra transpose and Hadamard transforms to handle the SSM input activations.

**Table 3: Zero-shot accuracy of W8A8 models on six common sense tasks. Quamba is a *static per-tensor* quantization method that closes the performance gap and outperforms the same-sized Transformers, Pythia \parencitebiderman2023pythia, in accuracy. We compare Quamba with Mamba-PTQ (W8A8) \parencitepierro2024mamba. We re-implement SmoothQuant (SmQ-SSM) \parencitexiao2023smoothquant and QuaRot (QuaRot-SSM) \parenciteashkboos2024quarot for 8-bit (W8A8) Mamba quantization as additional baselines. Bold is the best, and underline is the second best.**
| Model | Size | Methods | LAMBADA | HellaSwag | PIQA | Arc-E | Arc-C | WinoGrande | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pythia | 1.4B | FP16 | 62.0% | 41.8% | 72.0% | 61.7% | 27.4% | 56.5% | 53.6% |
| SmQ | 19.6% | 37.6% | 66.9% | 53.8% | 25.2% | 55.3% | 43.1% |  |  |
| 2.8B | FP16 | 65.2% | 59.4% | 74.1% | 63.5% | 30.0% | 58.5 % | 58.4% |  |
| SmQ | 59.7% | 58.4% | 73.2% | 62.1% | 28.4% | 57.1% | 56.5% |  |  |
| Mamba | 130M | FP16 | 44.2% | 35.3% | 64.5% | 48.0% | 24.3% | 51.9% | 44.7% |
| dynamic | 38.6% | 34.1% | 60.2% | 41.5% | 24.6% | 51.9% | 41.8% |  |  |
| static | 24.0% | 31.8% | 58.1% | 37.5% | 24.4% | 48.5% | 37.4% |  |  |
| Mamba-PTQ | 43.1% | 27.7% | 56.5% | - | - | 51.1% | - |  |  |
| SmQ-SSM | 41.1% | 34.4% | 63.0% | 43.6% | 23.6% | 51.9% | 42.9% |  |  |
|  | Quarot-SSM | 40.7% | 35.2% | 62.0% | 44.1% | 24.0% | 49.7% | 42.6% |  |
|  | Quamba (Ours) | 40.6% | 35.0% | 63.0% | 46.5% | 23.0% | 53.1% | 43.5% |  |
| 370M | FP16 | 55.6% | 46.5% | 69.5% | 55.1% | 28.0% | 55.3% | 51.7% |  |
| dynamic | 44.0% | 45.2% | 67.3% | 51.8% | 28.1% | 51.9% | 48.0% |  |  |
| static | 28.9% | 38.5% | 58.7% | 40.4% | 25.9% | 51.4% | 40.6% |  |  |
| Mamba-PTQ | 10.3% | 31.0% | 58.8% | - | - | 51.0% | - |  |  |
| SmQ-SSM | 44.3% | 44.3% | 66.3% | 51.2% | 27.7% | 54.7% | 48.1% |  |  |
|  | Quarot-SSM | 53.2% | 46.3% | 68.6% | 53.2% | 27.1% | 51.6% | 50.0% |  |
|  | Quamba (Ours) | 50.5% | 46.2% | 67.1% | 51.6% | 26.9% | 51.9% | 49.0% |  |
| 1.4B | FP16 | 64.9% | 59.1% | 74.2% | 65.5% | 32.8% | 61.5% | 59.7% |  |
| dynamic | 52.2% | 56.6% | 70.4% | 61.7% | 31.7% | 59.2% | 55.3% |  |  |
| static | 20.3% | 46.7% | 63.0% | 44.4% | 27.8% | 50.5% | 42.1% |  |  |
| Mamba-PTQ | 55.4% | 43.8% | 70.2% | - | - | 54.3% | - |  |  |
| SmQ-SSM | 50.8% | 56.9% | 71.9% | 61.8% | 31.4% | 57.5% | 55.1% |  |  |
|  | Quarot-SSM | 63.1% | 58.7% | 72.6% | 64.1% | 32.1% | 58.1% | 58.1% |  |
|  |  | Quamba (Ours) | 62.6% | 58.4% | 72.7% | 64.5% | 33.4% | 58.6% | 58.4% |
|  | 2.8B | FP16 | 69.1% | 65.9% | 75.6% | 69.2% | 35.8% | 63.0% | 63.1% |
|  | dynamic | 59.2% | 63.9% | 72.7% | 67.8% | 34.4% | 58.1% | 59.4% |  |
|  | static | 20.0% | 51.2% | 64.8% | 55.1% | 31.0% | 53.5% | 45.9% |  |
|  | Mamba-PTQ | 51.4% | 47.6% | 70.2% | - | - | 57.6% | - |  |
|  | SmQ-SSM | 46.6% | 62.8% | 73.3% | 66.5% | 35.0% | 59.8% | 57.3% |  |
|  |  | Quarot-SSM | 65.9% | 65.6% | 74.3% | 68.6% | 36.8% | 63.3% | 62.4% |
|  |  | Quamba (Ours) | 65.9% | 65.3% | 73.9% | 68.7% | 36.5% | 62.9% | 62.2% |

### 5.4 Zero-shot Evaluation

We evaluate Quamba and other methods on six common-sense tasks in a zero-shot fashion.
The accuracy of each task and the average accuracy across the tasks are reported in Table 3.
Quamba 2.8B has only a $0.9\%$ accuracy drop compared to floating-point Mamba 2.8B and outperforms Mamba-PTQ \parencitepierro2024mamba and SmQ-SSM \parencitexiao2023smoothquant in accuracy.
Quamba achieves similar accuracy to the re-implemented QuaRot-SSM \parenciteashkboos2024quarot but achieves a better trade-off between accuracy and latency, as shown in Figure 1 (a).

### 5.5 Quantizing Jamba: A Large-Scale Hybrid Mamba-Transformer LLM

Jamba \parencitelieber2024jamba is a hybrid transformer-mamba language model with 52B parameters, built with self-attention, Mixture of Experts (MoEs), and Mamba blocks, making it the first large-scale Mamba-style model with a number of parameters comparable to Mixtral \parencitejiang2024mixtral.
In Table 4, we compare the LAMBADA OpenAI accuracy of Jamba’s FP16 inference by combining off-the-shelf quantization methods with different quantization strategies.
Applying LLM.int8 \parencitedettmers2022gpt3 to self-attention and MoE preserves the model’s accuracy, whereas jointly quantizing Mamba with LLM.int8 \parencitedettmers2022gpt3 degrades the model and fails to produce meaningful accuracy.
In contrast, we combine Quamba with LLM.int8 \parencitedettmers2022gpt3 and as a result we achieve competitive accuracy (1.1% accuracy drop) with a lower model footprint than FP16.

**Table 4: Quantizing Jamba, a transformer-mamba hybrid model. We combine off-the-shelf quantization methods with our method. The zero-shot LAMBADA accuracy is reported. We apply SmoothQuant \parencitexiao2023smoothquant and LLM.int8\parencitedettmers2022gpt3 to self-attention and MoEs.**
| Self-attention | Mamba | MoE | Accuracy |
| --- | --- | --- | --- |
| FP16 | FP16 | FP16 | 74.0% |
| LLM.int8 | FP16 | LLM.int8 | 73.9% |
| SmQ | FP16 | LLM.int8 | 70.6% |
| LLM.int8 | LLM.int8 | LLM.int8 | fail |
| SmQ | Quamba (Ours) | LLM.int8 | 68.7% |
| LLM.int8 | Quamba (Ours) | LLM.int8 | 72.9% |

## 6 Ablation study

### 6.1 Quamba Ablation

In Table 6, We conduct a performance analysis on each component in Quamba and report the average accuracy across six zero-shot datasets.
Naive W8A8 quantization results in significant performance discrepancies across all sizes of models.
We improve the performance of quantized models by constraining the quantization range of the SSM input $x$ using percentile clipping ($+$ In Per.).
While addressing the large outlier in the SSM output using the Hadamard transform improves performance ($+$ Out Had.), the results remain unsatisfactory.
Quamba integrates two techniques, thereby closing the performance gaps across all model sizes.

**Table 5: Ablation study on Quamba. Avg. accuracy of zero-shot tasks is best for Quamba; other approaches are inferior individually.**
| Size | FP16 | W8A8 | + In Per. | + Out Had. | Quamba |
| --- | --- | --- | --- | --- | --- |
| 130M | 44.7% | 37.4% | 38.7% | 41.8% | 43.5% |
| 370M | 51.6% | 40.6% | 41.9% | 47.8% | 49.0% |
| 1.4B | 59.7% | 42.1% | 47.4% | 50.8% | 58.4% |
| 2.8B | 63.0% | 45.9% | 48.5% | 56.4% | 62.2% |

### 6.2 Percentile-based Activation Clamping

In Table 6, we conduct a sensitivity analysis on the percentile maximum clipping for the input $x$ to SSM.
We test different percentiles (*i.e.,* the $p$ described in 4.2) and report the accuracy on LAMBADA dataset.
The table shows more outliers in the larger models, while smaller amounts of outliers are clipped in the smaller models.
Therefore, clipping $0.001\%$ (*i.e., $p=99.999$*) of outliers in the model with 130m parameters produces best performance.
In contrast, for the model with 2.8b parameters, clipping $0.1\%$ (*i.e., $p=99.9$*) performs best on the LAMBADA dataset \parenciteLambdataDataset.

## 7 Conclusion

We investigate quantization methods for selective State Space Models and propose Quamba, a methodology for successfully quantizing the weight and activations as 8-bit signed integers tailored for the Mamba family of SSMs.
Our experiments show that Quamba maintains the original FP16 when accuracy compared with state-of-the-art counterparts, including current techniques for Transformers.
The profiling results on a wide variety of platforms show that the low bit-width representation of Quamba not only enables deployment to resource-constrained devices, such as edge GPUs, but also benefits from hardware acceleration with reduced latency.
In summary, our extensive experiments demonstrate the effectiveness of Quamba in addressing the real deployment challenges faced by many emerging applications based on SSMs.

## Acknowledgments

This work was supported in part by the ONR Minerva program, NSF CCF Grant No. 2107085, iMAGiNE - the Intelligent Machine Engineering Consortium at UT Austin, UT Cockrell School of Engineering Doctoral Fellowships, and Taiwan’s NSTC Grant No. 111-2221-E-A49-148-MY3.

## Appendix A Quantization Error Analysis for SSMs

We show the error bound of a discrete 1D linear time-invariant (LTI) SSM and the empirical experiments on the quantization errors of the discretized high-dimensional SSMs.

### A.1 Theoretical Error Analysis

We consider a discrete 1D linear time-invariant (LTI) state space model such that $h[t]=a(T,t)h[t-1]+bx[t]$, where $b$ is a constant, $a(T,t)$ is an exponential series with respect to the time step such that $a(T,t)=e^{t-T}$, T is a constant representing the total time step, and t is the current time step.
We assume that the system input has a quantization error $\delta_{x}[t]$ such that $\overline{x}[t]=x[t]+\delta_{x}[t]$ where $|\delta_{x}[t]|\leq\epsilon$.
The system is initialized to 0 0 such that $h[0]=\overline{h}[0]=0$.

###### Theorem A.1 .

The quantization error of $h[t]$ from each time step of the given discrete 1D linear time-invariant model is bounded by $b\epsilon\frac{e^{t-T}}{e-1}$ such that $|h[t]-\overline{h}[t]|\leq b\epsilon\frac{e^{t-T}}{e-1}$ .

###### Proof.

Given the quantization error $|\delta_{x}[t]|\leq\epsilon$ and the quantized input $\overline{x}[t]=x[t]+\delta_{x}[t]$ of each step, we have an original system $h[t]$ and a quantized system $\overline{h}[t]$

$$ $\displaystyle h[t]$ $\displaystyle=a(T,t)h[t-1]+b_{t}x[t]=e^{t-T}h[t-1]+bx[t]$ $\displaystyle\overline{h}[t]$ $\displaystyle=a(T,t)\overline{h}[t-1]+b_{t}\overline{x}[t]=e^{t-T}\overline{h} [t-1]+b\overline{x}[t]=e^{t-T}\overline{h}[t-1]+b(x[t]+\delta_{x}[t]).$ $$

By subtracting two systems, we have
$\overline{h}[t]-h[t]=e^{t-T}(\overline{h}[t-1]-h[t-1])+b\delta_{x}[t]$ .
Define the error term $\overline{h}[t]-h[t]=\omega[t]$ to simplify our notation, we have

$$ $\displaystyle\omega[t]$ $\displaystyle=e^{t-T}\omega[t-1]+b\delta_{x}[t]$ $\displaystyle|\omega[t]|$ $\displaystyle=|e^{t-T}\omega[t-1]+b\delta_{x}[t]|$ $\displaystyle|\omega[t]|$ $\displaystyle\leq|e^{t-T}\omega[t-1]|+|b\delta_{x}[t]|\quad\text{(using the triangle inequality)}$ $\displaystyle|\omega[t]|$ $\displaystyle\leq|e^{t-T}\omega[t-1]|+b\epsilon$ $\displaystyle|\omega[t]|$ $\displaystyle\leq e^{t-T}|\omega[t-1]|+b\epsilon.$ $$

We define $|\omega[t-1]|=\Omega[t-1]$ to simplify our notation, and the inequality becomes $\Omega[t]\leq e^{t-T}\Omega[t-1]+b\epsilon$.
Since $\Omega[0]=|\omega[0]|=|h[0]-\overline{h}[0]|=0$, by unrolling the recursive error function from $t=0$, we have

$$ $\displaystyle\text{t=0},\;$ $\displaystyle\Omega[1]\leq e^{1-T}\Omega[0]+b\epsilon=b\epsilon$ $\displaystyle\text{t=1},\;$ $\displaystyle\Omega[2]\leq e^{2-T}\Omega[1]+b\epsilon=e^{2-T}b\epsilon+b\epsilon$ $\displaystyle\text{...}\;.$ $$

Therefore, at time step $t$, the error is

$$ $\displaystyle\Omega[t]\leq e^{t-T}\Omega[t-1]+b\epsilon=b\epsilon\sum_{k=0}^{t -1}e^{k-T}.$ $$

$\sum_{k=0}^{t-1}e^{k-T}$ is a geometric series, therefore, we apply the sum of the first $t$ terms of the geometric series

$$ $\displaystyle\Omega[t]\leq b\epsilon\sum_{k=0}^{t-1}e^{k-T}=b\epsilon e^{-T} \sum_{k=0}^{t-1}e^{k}=b\epsilon e^{-T}\frac{e^{t}-1}{e-1}.$ $$

Then, we have the error bound

$$ $\displaystyle|h[t]-\overline{h}[t]|=\Omega[t]\leq b\epsilon e^{-T}\frac{e^{t}- 1}{e-1}=b\epsilon\frac{e^{t-T}-e^{-T}}{e-1}\leq b\epsilon\frac{e^{t-T}}{e-1}.$ $$

∎

The proof shows that for any time step $t$, the quantization error of $h[t]$ of the given discrete 1D linear time-invariant model is bounded.

Figure: (a) Materialized with HiPPO-LegT
Refer to caption: extracted/5933363/figures/HiPPO_LegT.png

###### Theorem A.1 .

###### Proof.

### A.2 Empirical Error Analysis

In Figure 5, we show the empirical results of the quantization errors for the discretized high-dimensional SSMs.
Given a discretized high-dimensional SSM

$$ $\displaystyle h[t]=\dot{A}h[t-1]+\dot{B}x[t],\quad y[t]=Ch[t]$ $$

where $x[t]\in\mathbb{R}^{p}$ is the input vector, $h[t]\in\mathbb{R}^{n}$ is the state vector, and $y[t]\in\mathbb{R}^{q}$ is the output vector, with the state matrix $A\in\mathbb{R}^{n\times n}$, input matrix $B\in\mathbb{R}^{n\times p}$, output matrix $C\in\mathbb{R}^{q\times n}$.
In the experiments, we set $n=p=q=4$, and the total time step $T$ to $100$.
We follow the official implementation in \textcitegu2020hippo.
HiPPO-LegT and HiPPO-LegS matrices \parencitegu2020hippo are used to materialize the $A$ and $B$ matrix and are discretized to $\dot{A}$ and $\dot{B}$.
We initialize $C$ matrix with a normal distribution $\mathcal{N}(0,1)$.
The input $x[t]$ of each time step is drawn from a normal distribution $\mathcal{N}(0,1)$, and is quantized into 8-bit representation $\overline{x}[t]$.
We initialize the $h[0]$ to a zero vector and update $h[t]$ with state space dynamics in each time step.
In Figure 5, we show the output errors $\texttt{Mean}(|y[t]-\overline{y}[t]|)$ induced by $\overline{x}[t]$. The errors are bounded as the time step $t$ increases.

## Appendix B Sensitivity Analysis of Quantizing SSMs

Figure: Figure 6: The sensitivity of quantizing SSM input and output (SSM I/O). We report zero-shot accuracy on the LAMBADA dataset. Accuracy is highly sensitive to input/output precision. Quamba quantizes SSM I/O to 8-bit, reducing the performance gap with FP16 (red, W8A8 SSM I/O FP16/FP16).
Refer to caption: extracted/5933363/figures/tensor_analysis.png

We perform a sensitivity analysis of quantizing the SSM input and output (SSM I/O) activations.
Figure 6 illustrates that the model collapses when all activations and weights are quantized to 8 bits.
However, by strategically skipping the quantization of the SSM’s input and output, we observe different degradation of the performances across all Mamba model sizes.
Notably, quantizing the SSM output results in severe performance degradation (orange, SSM I/O FP16/I8) due to the skewed quantization resolution caused by large outliers, particularly extreme values ($\geq$ 100).
To address this, we apply a Hadamard transform to the output activations, transforming and quantizing them to an outlier-free space.
Moreover, the small outlier values ($<$ 10) in the SSM input cause significant performance drops (green, SSM I/O I8/FP16), and the Hadamard transform does not resolve this issue.
To mitigate this, we clip the distribution results to preserve better quantization precision, thereby reducing the performance gap.

## Appendix C Re-implementation of QuaRot on Mamba

We re-implement QuaRot \parenciteashkboos2024quarot, a state-of-the-art low-bitwidth quantization method for Transformers, for the Mamba structure.
Our re-implementation is denoted by QuaRot-SSM.
Figure 7 (b) illustrates the details of our re-implementation.
While QuaRot-SSM is on-par with Quamba in terms of perplexity and accuracy, it is not optimized for the structure of SSMs and requires additional Hadamard transforms and transpositions to process the SSM input activations.
In contrast, our approach is friendly to hardware and effectively improves the quantization precision of the $x$ tensor, delivering real speedups on both cloud and edge devices (see Table 1).

Figure: Figure 7: The figure compares Quamba to QuaRot-SSM. We fuse the Hadamard matrices in the linear layers to achieve compute-invariance. $\overline{x}^{p}$ represents the 8-bit $x$ tensor with percentile clipping. Our method provides real speedups on cloud and edge devices, while QuaRot requires extra Hadamard transforms and transpositions for SSM input activations. Residual connections are simplified to reduce clutter in the figure.
Refer to caption: extracted/5933363/figures/mamba_quarot.png

## Appendix D Layer-wised Distribution of SSMs

Figure: Figure 8: Box plots of the SSM inputs and outputs distributions for Mamba family. We obtained the distributions using the Pile dataset, which serves as the calibration dataset for all our experiments.
Refer to caption: x1.png

We analyzed the input and output distributions of SSM for Mamba family, as shown in Figure 8.
The box plots reveal the presence of outliers in both inputs and outputs.

#### SSM input x 𝑥 x italic_x .

In Figure 8 left, a skewed as well as asymmetric distribution is present in all layers of SSM inputs.
Although the activation values are relatively small ($<$ 10), however, due to the sensitivity of the linear recurrent dynamics, representing the input activations with low bit-width data causes significant performance drops for all sizes of Mamba-family.
Therefore, clipping the distributions results in better quantization precision and avoids the accuracy degradation.
Though asymmetric quantization performs slightly better than symmetric quantization as shown in Table 9, we choose symmetric quantization due to the support of most frameworks, *e.g.,* CUTLASS.
We leave the asymmetric optimization in the future work.

#### SSM output y 𝑦 y italic_y .

In Figure 8 right, large outliers are observed in the SSM outputs.
The outliers pose a great challenge in quantizing SSMs, due to the skewed quantization resolution caused by the extreme values ($\geq$ 100).
This finding in SSMs echoes the outlier phenomenon observed in Transformers.
We apply the Hadamard matrices to transform the output activations to an outlier-free space, making quantization easier.
Interestingly, the layers closer to the model output have larger outlier values, suggesting that different quantization schemes can be applied to the earlier layers.
We leave the study for future work.

## Appendix E Low Bit-width Quantization

We show that Transformer-based quantization methods do not generalize well to Mamba blocks.
We re-implement state-of-the-art low bit-width quantization methods for Transformers, Quip# (weight-only quantization, W2A16) \parencitetseng2024quip and QuaRot (W4A4) \parenciteashkboos2024quarot, for the Mamba structure.
These are denoted by Quip#-SSM and QuaRot-SSM, respectively
Our experiments show that they fail to effectively quantize Mamba in low bit-width settings, as shown in Tables 8 and 8.
In Table 8, applying Quip# and QuaRot to Mamba results in much higher perplexity, leading to worse performance compared to Transformers.
For instance, Quip#-SSM quantizes Llama-2-7b with W2A16, causing only a 1.02$\times$ increase in perplexity, whereas our implementation on Mamba results in a 1.34$\times$ increase.
In Table 8, although models with different bit-widths cannot be directly compared, the results show that both Quip#-SSM and QuaRot-SSM reduce the average accuracy across six zero-shot downstream tasks.
Our study highlights that quantizing SSMs is particularly challenging, as their input and output activations exhibit a causal relationship with varying levels of outliers, a phenomenon unique to SSMs (*ref.* Section I).

**Table 7: Quantizing Llama-2-7b and Mamba 2.8B with low bit-width methods. The perplexity on Wiki2 is reported.**
| Methods | Precision | Llama-2-7b | Mamba-2.8b |
| --- | --- | --- | --- |
| Mamba | FP16 | 5.47 | 9.45 |
| Quip#-SSM | W2A16 | 5.56 (1.02 $\times$) | 12.71 (1.34 $\times$) |
| Quarot-SSM | W4A4 | 6.10 (1.11 $\times$) | failed |
| Quamba (Ours) | W8A8 | - | 9.91 |

## Appendix F Explore Quantization Algorithms for SSM Inputs

In our investigation of 8-bit quantization strategies for the state space model (SSM), we observed that the SSM input $x$ exhibits outliers.
Although these outliers are not excessively large, their presence significantly impacts the quantization error, affecting the SSM output $y$.
In this section, we present the results and analysis of several other 8-bit quantization options in Table 9, though *not* necessarily hardware efficient.

**Table 9: We explore different methods for quantizating SSM inputs into 8-bit. For the quantization of other tensors, we use the same settings as Quamba. We evaluate the quantized models and report the accuracy on the LAMBADA dataset. Our method is generalized to current mainstream freamworks and toolchains. (Sym., Asym., and Per. are short for symmetric, asymmetric, and percentile)**
| Methods for SSM Inputs | Framework<br>Supports | LAMBADA Accuracy ($\uparrow$) |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 130m | 370m | 1.4b | 2.8b |  |  |
| FP16 | Yes | 41.24% | 51.81% | 63.82% | 67.51% |
| Dynamic |  |  |  |  |  |
| MinMax Sym. | Yes(^3^33The dynamic approach will result in extra execution overhead on re-calculating the quantization scales.) | 40.38% | 51.45% | 62.55% | 66.62% |
| Static |  |  |  |  |  |
| MinMax Sym. | Yes | 34.10% | 45.78% | 44.89% | 55.71% |
| MinMax Sym. Log2 | No | 40.31% | 51.80% | 63.57% | 67.51% |
| MinMax Asym. Per. | No | 40.73% | 50.46% | 63.09% | 66.76% |
| MinMax Sym. Per. (Ours) | Yes | 40.61% | 50.37% | 60.43% | 65.67% |

#### Dynamic quantization.

One direct approach to provide a more accurate quantization mapping is through dynamic quantization.
By dynamically capturing the activation range based on the current inputs, we can map the floating value into 8-bit with precise scaling factors.
The approach boosts the accuracy and closes the performance gap between FP16.
However, the dynamic approach will result in extra execution overhead on re-calculating the quantization scales, leading to sub-optimal computation efficiency.

#### Asymmetric quantization.

We notice that the visualized tensor distribution of SSM inputs is asymmetric in Figure 8.
To better utilize the bit-width, we could apply asymmetric quantization to the SSM inputs.
As shown in Table 9, asymmetric quantization yields better accuracy in the zero-shot tasks, particularly for Mamba-1.4b and Mamba-2.8b models.
However, asymmetric quantization increases the computational load during inference and requires specific software framework and hardware supports.
We leave the asymmetric optimization for future work.

#### Log2 quantization.

To avoid the quantization step skewed by the outliers, and to ensure that smaller values in a tensor are accurately mapped, we can quantify the tensor in log-scale.
Here, we implement log2 quantization \parenciteNumberSystem, miyashita2016convolutional, which maps the values to the nearest power of 2, achieving the desired non-uniform mapping.
Our log2 version slightly outperforms Quamba.
However, log2 matrix multiplication requires specific optimization on both software and hardware levels.
Our method is more generalizable to mainstream frameworks and toolchains ( *i.e.,* PyTorch, CUTLASS ).

#### Other alternatives.

Exploring the power of exponents with advanced low-bit floating-point data types \parenciteFP8, such as E5M2 or E4M3 \parenciteh100, currently supported by NVIDIA Hopper GPUs, might also be an effective solution for quantizing SSM inputs.
A more comprehensive study on quantizing SSMs using floating-point quantization is left for future work.

## Appendix G Generation Quality

We also applied Quamba to a chat language model to evaluate its effectiveness in preserving generation quality.
Specifically, we used the Mamba-2.8B-Chat model \parencitehaven2023mambachat, which is based on Mamba-2.8B and fine-tuned on UltraChat200k \parencitetunstall2023zephyr.
We compared the sentences generated by the quantized and non-quantized models, as shown in Table 10.
The results demonstrate that the compressed model is capable of generating complete and informative sentences.

**Table 10: Generated Examples of Quamba (W8A8) and Mamba (FP16)**
| Model | Generated Sentences |
| --- | --- |
| Mamba<br>(FP16) | What’s great about the holiday season? I do not have a personal opinion or feelings. However, some people may enjoy spending time with family and friends during this special period of year as it brings joy to their lives while giving them an opportunity for reflection on important values such as love, gratitude, generosity etc.. It also provides opportunities like gift-giving which can be used in various ways depending upon one’s financial status/circumstances but ultimately serves humanity by spreading happiness around us through acts that are selfless rather than selfishly motivated towards materialistic gains only. Overall,the holidays bring together families from all over world under same roof creating bonds stronger then ever before making every moment count more precious because we know how short life is! So let me wish you happy new years!!! Happy Holidays!! May your days ahead filled up fullness & peacefulness!!! |
| Quamba<br>(W8A8) | What’s great about the holiday season? I do not have a personal opinion or feelings. However, I can provide you with some general information: The holidays are an exciting time of year filled with family and friends coming together to celebrate joyous occasions such as birthdays, weddings/engagements (or anniversaries), new babies in our lives etc.. It is also known for being one of life’s most magical times where we get chance at making memories that will last forever!- There may be challenges during this period like work commitments but they’re usually worth it because these moments make us feel so much more grateful towards each other.- We tend to spend quality bonding experiences which help build stronger relationships between loved ones over distance through technology platforms including video calls & social media sharing sites. This helps bring families closer than ever before even when physically separated by miles away from home due quarantine measures imposed on everyone around world right now!!! So yes,the Holidays definitely brings out best sides within ourselves!! Happy Holiday Season Everyone!!! |

## Appendix H Real-time Generation on Edge GPUs

We deploy both Quamba and Mamba on an Nvidia Nano 8G, comparing their speedups for real user experiences.
We use the pre-trained weights of Mamba-Chat \parencitehaven2023mambachat and apply our quantization techniques.
Figure 9 shows a screen snapshot taken during the demo.
We input the same prompt to the model at the beginning (a) $T=0$.
At the initial time point (a) $T=0$, the same prompt is provided to both models.
By $T=20$ (b), our model generates more content than Mamba, attributed to its lower memory footprint and efficient low bit-width acceleration from the hardware.
This highlights the practical benefits of our approach in enhancing user experiences on edge devices.

Figure: Figure 9: Demo on Orin Nano 8G.
Refer to caption: extracted/5933363/figures/demo.png

Figure: Figure 10: Quantizing the input $x$ and output $y$ tensors causes the most significant accuracy drop on the LAMBADA dataset for SSMs. In contrast, quantizing the tensors in self-attention layers results in minimal accuracy loss.
Refer to caption: extracted/5933363/figures/mamba_transformer_sensitivity.png

## Appendix I Comparing SSMs with self-attention layers

#### Comparing the quantization sensitivity.

We explore the quantization sensitivity of input and output activation maps for both SSMs and self-attention layers, as shown in Figure 10.
We conduct experiments on Mamba 2.8B, a same-sized Transformer Pythia 2.8B, and a recently published, comparably sized Transformer Llama 3.2 3B.
Quantizing the input $x$ and output $y$ tensors leads to the most significant accuracy drop on the LAMBADA dataset for SSMs, compared to other tensors.
In contrast, quantizing the tensors in self-attention layers (*i.e.* $h$, $q$, $k$, $v$, and output $y$) results in minimal accuracy loss.
For transformers, using 8-bit $h_{d}$ tensors in feedforward layers significantly degrades model performance.
Due to the differing quantization sensitivity patterns between Mamba and Transformer blocks, we highlight the *smooth*, *outlier*, and *sensitive* paths for both in Figure 11.
This highlights that different quantization methods are needed for SSM-based models.

Figure: Figure 11: We annotate the different sensitivity patterns for SSMs and self-attention layers, which shows that they require different quantization techniques. The residual connections are simplified to avoid cluttering the figure.
Refer to caption: extracted/5933363/figures/mamba_transformer.png

#### Visualizing activations.

We visualize these activation maps in Figures 12 and 13.
Figure 12 shows the SSM activation maps in the last layer of Mamba 2.8B, while Figure 13 shows those for the self-attention layer in Llama 3.2 3B.
We use the same test sample to visualize these activations.

Notably, SSMs and self-attention layers exhibit *distinct* activation patterns.
Outliers appear in the SSM output $y$, unlike in self-attention layers, where the outputs $y$ remain smooth and *do not* present outliers.
In Transformer blocks, outliers only occur in the $h_{d}$ of the feedforward layers, making them difficult to quantize.
In contrast, Mamba blocks have large outliers in the SSM output $y$ tensor, which are challenging to represent with low-bit data types.
While the $x$ tensor values are not significantly large, they are highly sensitive to quantization errors.
We address this issue in Mamba blocks and introduce Quamba to manage the unique quantization patterns of SSMs.

Figure: Figure 12: We visualize the activations of the SSM in the last layer of Mamba 2.8B. This figure shows that the values in the $x$ tensor are not significantly large. However, outliers are present in the $y$ tensor of the SSM output, making them difficult to represent in low bit-width data types.
Refer to caption: extracted/5933363/figures/mamba_tensors.png

Figure: Figure 13: We visualize the activations of the self-attention layer in the last layer of Llama 3.2 3B. This figure shows that outliers are *not* present in the output $y$ of the self-attention layers. In contrast to SSMs, the outlier issue arises in the $h_{d}$ tensor of the *feedforward layers*.
Refer to caption: extracted/5933363/figures/llama_tensors.png

#### Comparing the precision mapping.

Figure 14 compares our precision mapping to the standard one used in Transformers \parencitezhao2023atom, ashkboos2024quarot, lin2024qserve, xiao2023smoothquant.

Figure: Figure 14: We compare our precision mapping to the precision mapping commonly used in Transformers. The residual connections are simplified to avoid cluttering the figure.
Refer to caption: extracted/5933363/figures/mamba_transformer_precision.png

## Appendix J Limitations and Broader Impacts

We find that the accuracy degradation is not negligible in both accuracy and perplexity in Table 2 and Table 3.
Despite this, the performance trade-off is acceptable given the significant improvements in latency and resource efficiency.
Our work enables large language models to be deployed on resource-limited devices.
As a positive feature, our method may push the development of privacy-centric on-device applications, where sensitive data can be processed locally without relying on cloud services.
However, our work may also present challenges such as increased device resource consumption and potential security vulnerabilities if the local devices are compromised.

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: - In line 228, you mentioned the **Theoretical error bound for SSM quantization**, and give a proof in appendix. However, this discussion seems to be isolated, so I do not particularly understand the connection between this discussion and the core approach of this paper.
- In Table 1 and Table 2, you compared QuaRot and Quamba in W8A8 format. However, QuaRot mainly focus on **W4A4** solution for quantization. I would like to know if this method can be extended to W4A8 and W4A4 scenarios? If it can, how is his performance compared to QuaRot?
soundness: 3
strengths: - The approach is simple and effective, introducing little additional overhead
- The experiment is solid: The author achieved real quantization - by handwriting fused operators and utilizing low-bit computational libraries (e.g. cutlass) to implement Quamba.
summary: This work propose a PTQ solution for SSM model. Quamba notices the SSMs have highly sensitive feature maps within the selective scan mechanism (i.e., linear recurrence) and massive outliers in the output activations, and utilises two methods of outlier suppression - ①suppresses the maximum values of the input activations ②supply hadamard transform of the output activations, to solve the problem. Experiments show that show that this method can achieve 8-bit PTQ quantisation of SSM almost losslessly.
weaknesses: - The proposed method is effective, however, these methods are also widely used in Transformer, and authors may need to make their connection to SSM more explicit in their writing.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. As a researcher focused on large model quantization on edge devices, I have some concerns about the authors' claim regarding the lack of prominent outliers in transformers. Based on my own experiments and findings [1-2] from recent literature, larger language models (LLMs) actually tend to exhibit more significant outliers, especially in channel-wise activation. Can you clarify your claim about outliers in Transformers and provide evidence or citations supporting your view?

2. Studies [3-4] on transformer quantization have shown that quantizing outliers can substantially impact model accuracy. This observation aligns with the authors' ablation study results. However, I am unclear on why certain techniques in this paper appear to alleviate the impact of outlier quantization. Additionally, shouldn’t each optimization step incrementally reduce accuracy? Surprisingly, the authors' ablation study shows an improvement in accuracy when all three techniques are applied together. Could you please provide a more detailed explanation of why your techniques, when combined, lead to improved accuracy rather than incremental reductions?

3. In the introduction, the authors mention that this quantization method is applicable to both edge devices and cloud computing. However, the experiments only feature the high-compute edge device Orin Nano 8G (and the A5000, which would generally not be considered an edge device). There is no testing on cloud computing hardware, which would have been helpful.

4. For the results in Table 1: Profiling Latency, I find some inconsistencies. As token length increases, both computation and memory should scale linearly, so I would expect a relatively stable speedup ratio. If anything, given the execution overhead of the nonlinear kernel within SSM, there might be a slight increase in latency with token length. However, the authors' data show an unusual spike in performance at $L=512$ and $L=1$. Could you please provide a more detailed explanation of the observed performance spikes, possibly including additional data points or experiments that could help clarify the relationship between token length and speedup ratio?

5. Lastly, could the authors clarify how many hardware measurements were conducted to obtain the reported results? Given the inherent variability in hardware performance, multiple measurements are typically necessary to ensure stability and reliability. It would be helpful to understand the stability of the hardware results presented. Could you please provide details on the number of runs performed for each experiment, any measures taken to account for variability (e.g., averaging results, reporting standard deviations), and how you ensured the stability and reproducibility of your hardware measurements?

[1] Shen, Xuan, et al. "Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.
[2] Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." International Conference on Machine Learning. PMLR, 2023.
[3] Lin, Yang, et al. "Fq-vit: Post-training quantization for fully quantized vision transformer." arXiv preprint arXiv:2111.13824 (2021).
[4] Dong, Peiyan, et al. "Packqvit: Faster sub-8-bit vision transformers via full and packed quantization on the mobile." Advances in Neural Information Processing Systems 36 (2024).
soundness: 2
strengths: 1. The paper presents a unique approach by applying quantization specifically to Selective SSMs, a topic with limited prior research in this area. The integration of percentile-based clipping and the Hadamard transform for managing outliers in SSMs represents an innovative combination of techniques. These methods are tailored specifically for the challenges of SSM quantization, marking a significant contribution to the field of model compression.
2. The paper demonstrates a thorough experimental setup, with well-defined quantization methodologies for activations. The authors support their approach with empirical evidence, clearly detailing the implementation of 8-bit static quantization and its impact on latency and accuracy. The careful construction of quantization techniques for hardware constraints is grounded in a rigorous understanding of SSM behavior, lending credibility to the study’s findings.
3. The proposed quantization approach has high potential significance, especially for applications involving SSMs in resource-limited environments. By addressing the specific challenges of quantizing SSMs, the paper extends the practical use of SSMs, making them feasible for deployment on low-power hardware without significant accuracy degradation.
summary: This paper introduces a post-training quantization method tailored for Selective State Space Models (SSMs) to address challenges in model size reduction and computation efficiency, specifically in hardware-constrained environments. The authors propose an 8-bit static quantization approach that handles weight and activation quantization while mitigating the effects of outliers commonly found in SSM outputs.

Key contributions of the paper include:
1.  A method for input activation quantization that reduces the impact of extreme values by clipping based on the 99.999th percentile, improving quantization precision.
2. Hadamard transforms to map output activations into a space with minimized outliers, allowing for accurate quantization without excessive memory and computational cost.
3. Empirical analysis showing the effectiveness of this quantization approach in achieving a balance between model accuracy and reduced latency, making SSMs more feasible for edge and embedded applications.

The paper provides a comprehensive empirical evaluation, demonstrating how these techniques enable SSMs to perform efficiently on hardware with limited resources.
weaknesses: The paper is innovative in its approach and thorough experimentation. However, there are several critical questions that I raised in the "Question" section, which I believe are essential for the clarity and robustness of the findings. I hope the authors can provide insights on these points, and I look forward to further discussion.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 1
questions: - Why is TTLT used? It makes more sense to evaluate latency (TTFT) and throughput (token-to-token).
- Since extremely large outliers are identified as problematic for SSMs, have the authors considered using clipping to lower quantization noise. For example, OCTAV may be useful in this setup [1]
- Empirical results with comparisons to transformers are focused on the W8A8 case. However, Transformers have been quantized to 4-bit (e.g., using Quarot).Is it therefore a fair comparison to use 8-bit transformers?

[1] Sakr, Charbel, et al. "Optimal clipping and magnitude-aware differentiation for improved quantization-aware training." International Conference on Machine Learning. PMLR, 2022.

----
Good responses, I am raising to 6.
soundness: 2
strengths: - Studies quantization of SSMs, which is novel since SMMs themselves are new and quantization studies have not caught up with them.
- The empirical results are not just scoped to accuracy studies using fake quantization, but actual profiling is done using INT8 and cutlass.
summary: The paper studies quantization of stat-space models. The propose Quamba which employs 8-bit quantization and Walsh-Hadamard transformations of activations to preserve accuracy.
weaknesses: - Empirically, Quamba does not show advantages compared to Quarot.
- MSE studies are used to compare difficulty of quantizing attention vs SSM (e.g., Fig 2). However, absolute numbers for MSE are not useful when inputs and models are different. A better way to visualize the quantization difficulty would be to employ a normalized MSE. In other words, the L2 norm of the delta divided by the L2 norm of the unquantized tensor, rather than just the L2 norm of the delta.
- Theorem 4.1 is very badly written, I had to refer to the appendix to guess what the authors meant. Note that lowercase b and epsilon were never defined and they are the defining variables of the theorem! Please fix this and introduce b as being an element from the B matrix, and epsilon to be an upper bound of the quantization step. P.S., these are my guesses for what was meant. 
-While I appreciate Theorem 4.1 which states that quantization error compounds over time-steps (correct me if I am wrong). Where was this used? Which aspect of the proposed Quamba leverages Theorem 4.1? It seems like this was just added as decorative math.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Could the paper articulate more clearly what aspects of applying the two proposed techniques (clipping and Hadamard transform) to SSMs are novel or challenging compared to their use in other contexts?
- Could the paper provide a better positioning of this paper, in terms of what the main difference is between the proposed method and the prior QuaRot? It seems that a trivial change on the original QuaRot (such as removing the extra Hadamard transform, which is a natural choice) can easily improve the performance.
- In Figure 1 (b) and (c), could the paper provide the data for QuaRot-SSM for a more complete comparison?
- In Figure 2 (b) and (c), it is unclear what y_t means. And why are there multiple (colors of) MSE curves? For example, for the curve of V_bar (red one in (b)), does it still represent the MSE of y_t and how should we understand this curve? Figure 3 (a) is much clearer in this sense.
- Is it possible to support 4-bit quantization with the proposed method?
soundness: 3
strengths: - Overall, the paper is well-structured, well-motivated, and well-written. The right amount of background introduction, motivating examples, and explanation of challenges make this paper easy to understand.
- The paper did thorough experiments to show the generalizability and scalability of the proposed method on various SSM-based implementations. In the results, the proposed Quamba shows a clear latency improvement and makes it possible to run large-scale models on devices with a limited amount of memory.
summary: This paper proposes a post-training quantization method for state space models (SSMs), which enables the utilization of low-bit acceleration in modern GPUs and achieves notably lower generation latency and small accuracy penalty. The method is mainly comprised of two techniques: 1) static 8-bit per-tensor quantization for input activations, and 2) Hadamard transform for outlier-free output activation quantization.
weaknesses: - The novelty of the proposed method is not clear. As mentioned in the text, the paper acknowledges that clipping outliers in input activations with a percentile max is a widely applied technique, and smoothing out the outliers with the Hadamard transform is also not new and has been applied in prior quantization methods for transformer attention computation. The main novelty of this paper therefore may come from being the first to apply these two techniques to SSM.
- There is no significant benefit over QuaRot-SSM, which is a re-implementation of QuaRot for Mamba. The positive side is that this does serve as a strong baseline, however, their performance is very similar in many experiments. Figure 7 in the appendix, to some extent, also shows that the differences in architecture are small.

Meta-review: This paper presents an imporant contribution to SSM quantization, enabling faster inference with minimal accuracy loss. The authors carefully addressed initial reviewer concerns about novelty and theoretical clarity through additional experiments and improved explanations. Their framework, Quamba, shows strong latency reductions on real hardware without sacrificing performance. Overall, reviewers moved towards consensus after the rebuttal, recognizing the paper’s practical value and rigorous evaluation. While some minor points remain, they are not dealbreakers. The combined impression is that this work is both timely and impactful, making it deserving of acceptance.

Decision: Accept (Poster)