MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods
Authors: Zukang Xu
Yuxuan Yue
Xing Hu
Dawei Yang
Zhihang Yuan
Zixu Jiang
Zhixuan Chen
JiangyongYu
XUCHEN
Sifan Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Mamba
Quantization
Primary area: foundation or frontier models, including LLMs

Abstract: Mamba is an efficient sequence model that rivals Transformers and demonstrates significant potential as a foundational architecture for various tasks. Quantization is commonly used in neural networks to reduce model size and computational latency. However, applying quantization to Mamba remains underexplored, and existing quantization methods, which have been effective for CNN and Transformer models, appear inadequate for Mamba models (e.g., Quarot suffers a 21% accuracy drop on Vim-T$\dagger$ even under W8A8). We have pioneered the exploration of this issue and identified several key challenges. First, significant outliers arepresent in gate projections, output projections, and matrix multiplications. Second, Mamba’s unique parallel scan further amplifies these outliers, leading to uneven and heavy-tailed data distributions. Third, even with the application of the Hadamard transform, the variance across channels in weights and activations still remains inconsistent. To these ends, we propose MambaQuant, a post-training quantization (PTQ) framework consisting of: 1) Karhunen-Lo`eve Transformation (KLT) enhanced rotation, rendering the rotation matrix adaptable to diverse channel distributions. 2) Smooth-Fused rotation, which equalizes channel variances and can merge additional parameters into model weights. Experiments show that MambaQuant can quantize both weights and activations into 8-bit with less than 1% accuracy loss for Mamba-based vision and language tasks. To our knowledge, MambaQuant is the first comprehensive PTQ design for the Mamba family, paving the way for further advancements in its application.

Published as a conference paper at ICLR 2025
HAMBAQUANTI QUANTIZING THE MAMBA FAMILY WITH VARIANCE ALIGNED ROTATION METHODS
Zukang Xu Yuxuan Yue Xing Hu', Zhihang Yuan', Zixu Jiang! 131,
Zhixuan Chen', Jiangyong Yu', Chen Xu', Sifan Zhou Dawei Yangl
1 HHoumo AI
2 Harbin Institute of Technology (Shenzhen) Southeast University
3 NNanjing University
zukang. xuêhoumo.. ai
15s151633esuu wiit.edu.com
aajaaygmmail.... com
dawei yangghounggooooo ai
ABSTRACT
Mamba is an efficient sequence model that rivals Transformers and demonstrates significant potential as foundational architecture for various tasks. Quantization is commonly used in neural networks to reduce model size and computational latency. However, applying quantization to Mamba remains underexplored, and existing quantization methods, which have been effective for CNN and Trans- former models, appear inadequate for Mamba models (e.g.. Quarot suffers 21 % accuracy drop on Vim-T't even under W8A8). We have pioneered the exploration of this issue and identified several key challenges. First, significant outliers are present in gate projections, output projections, and matrix multiplications. Sec- ond, Mamba's unique parallel scan further amplifies these outliers, leading uneven and heavy-tailed data distributions. Third, even with the application of the Hadamard transform, the variance across channels in weights and activations still remains inconsistent. To these ends, we propose MambaQuant, post-training quantization (PTQ) framework consisting of: I) Karhunen-Loote Transformation (KLT) enhanced rotation, rendering the rotation matrix adaptable to diverse chan- nel distributions. 2) Smooth- -useed rotation, which equalizes channel variances and can merge additional parameters into model weights. Experiments show that MambaQuant can quantize both weights and activations into 8-bit with less than 1% accuracy loss for Mamba-base vision and language tasks. To the best of our knowledge, MambaQuant is the first comprehensive PTQ design for the Mamba family, paving the way for further advancements in its application.
INTRODUCTION
Mamba (Gu & Dao, 2023) is modern sequence model that competes with the Transformer (Vaswani al., 2017). particularly noted for its ability handle extremely long sequences. The model's design is inspired by the Structured State Space model (S4) (Gu et al., 2021) and integrates features from recurrent, convolutional, and continuous-tine models to effectively capture long-term periodic depen- dencies. Expanding upon the S4 paradigm, Mamba brings about several noteworthy improvements, especially in handling time-variant operations. These enhancements enable the effective and efficient processing of lengthy data sequences, positioning Mamba promising foundational architecture for vision (Zhu et al., 2024; Liu et al., 2024), language (Gu & Dao, 2023; Li et al., 2024), and multi-modality tasks (Zhao al., 2024).
Quantization is an essential technique for deploying deep neural networks (DNNs) in environments with limited computational resources and the demand for real-time processing. This process involves 32-bit converting weights and activation of neural networks from high precision 32-bit floating point numbers) to lower precision (e.g., 8-bit integers) reduce memory usage, computational burden, and energy consumption. Although quantization has been successfully utilized in convolutional
Corresponding author "Equal contribution
Thhss work conducted during
his intersship Houmo AT

---

Published as a conference paper at ICLR 2025
0.20
10
10
15
15
0.15
0.15
5
10
10
0.00
0.00
0
Out chanmel
200
In chantesi
100
Heddon
200
Token dm
Hotden diyy
200
Token dim
100
400
200
400
200
400
200
(a) Gate Projection
(b) Output Projection
(c) Matrix Multiplication
Figure I: Visualized distribution of hard layers for Mamba quantization. (a) denotes the weight of the gate projection, (b) denotes the input activations of the output projection. (c) represents the output of the parallel scan (PScan) operator, which is also one of the input to the matrix multiplication.
neural networks (Krishnamoorthi, 2018; Liu al., 2023) and Transformer-based large language models (T-LLMs) (Du et al., 2024; Yuan, 2024), its application within the Mamba family has not been systematically analyzed or studied.
To establish comprchensive quantization methodology for Mamba models, we first examine the potential constraints and challenges involved: 1 Significant outliers occur in both weights and activations of Mamba models. We observe the presence of outliers in the weights of linear layers, particularly in the gate projection layers (Figure (aa) of Mamba-LLM (Gu & Dao, 2023) for language tasks. We also find that certain inputs to linear layers exhibit significant variance in the channel dimension. This occurrence is particularly pronounced in the output projection layers (Figure 1(b)) of Vim (Xu et al., 2024) for vision tasks. @ Parallel Scan (PScan) further amplifies the outliers of activations. To obtain hidden states at each timestamp, the PScan operator (Smith et al., 2022) continuously performs self- seff-mutiplication of fixed parameter matrix. In this case, channels exhibiting higher values will be amplified, while those with comparatively lower values will be diminished. This obvious numerical difference across channels is directly expanded to activations (e.g., input variable to the matrix multiplication as shown in Figure I(c)).
Given that both Mamba and Transformer are sequence models with fully connected layers to be quantized, our initial solution involves exploring techniques that have been proven effective on Transformer-basee large language models (T-LLMs). Recently, Haadmmrrd-basee methods (Tseng et al., 2024), known for the capacity to uniform maximum values and the equivalent transformation property, have shown significant success in the quantization of T-L.MMs. For instance, quantizing LLAMA2-70B to 4 bits with QuaRot (Ashkboos et al., 2024b) maintains 99% of the zero-shot performance. However, directly applying this method to Mamba models leads to significant accuracy degradation (e.g., average more than 12% accuracy drop Vim (Xu et al., 2024) even 8 bits). Our analysis reveals that Hadamard transformation fails to achieve variance alignment across channels, shown in Figure 2(b)(e). The inconsistent variances inevitably result in uneven numerical distribution of the quantization data, thereby decreasing the accuracy.
To this end, propose Mamba framework tailored for Mamba models. The central concept of MambaQuant is to resolve the issue of inconsistent variances arising from the Hadamard transformation, thereby promoting the Mamba quantization. Specifically, MambaQuant considers two distinct situations depending on whether to integrate the rotation matrix into weights: the offline mode for exclusion and the online mode for inclusion. (1) We propose the Seahhunnn-LLève Transformation (KLT) enhanced rotation in the offline mode. This technique multiplies the Hadamard matrix with the KLT matrix, enabling the rotation matrix to accommodate various channel distributions. (2) We introduce the smooth-fused rotation in the online mode. This approach performs smoothing before the Hadamard transformation. The additional smoothing parameters are flexibly integrated into weights of Mamba blocks to avoid extra cost of memory space and inference step. Consequently, both the maximum values and the variances of the quantization data are sufficiently normalized in the channel dimension (i.e., they are consistent for the offline mode and closely aligned for the online mode as shown in Figure 2(c)(e)).
effective and efficient post-training quantization (PTQ)
Experiments show that MambaQuant outperforms existing methods across various tasks on different Mamba model families, including Vim (Zhu et al., 2024) and Mamba-ND (Li et al., 2024) for Mamba-hased vision tasks, well Mamba-LLIM (Gu & Dao, 2023) for Manba-based language
2

---

Published as a conference paper at ICLR 2025
tasks. MambaQuant quantizes both weights and activations into 8-bit with slight aceuracy drop (less than 1 1%) for all models. Additionally, it can quantize weights to 4-bit with minimal accuracy drop (about 1%) for vision tasks, and achieves significant accuracy improvements in language tasks compared to existing methods. Lastly, our contributions can be concluded as follows:
We identify that Mamba encounters quantization challenges primarily due to significant outliers, which are even amplified by PScan. Our analysis reveals that the Hadamard transformation is hindered by inconsistent channel variances to effectively solve these problems.
We propose MambaQuant. For offline mode, introduce the KLT-Enhanced rotation to equalize the channel variances. For online mode, we introduce smooth-fused rotation to normalize the channel variances. Both the offline and online transformation can achieve more uniform distributions prior to the quantization process.
To the best of our knowledge, MambaQuant is the first comprehensive PTQ framework for the Mamba family. It can efficiently quantize both weights and activations into 8-bit with less than 1 % accuracy loss for Mamba-based vision and language tasks.
As pioneering study on quantization within the Mamba family, we have published the code in the hope of promoting further research and facilitating advancements in this field.
2 RELATED WORK
Mamba Models Mamba (Gu & Dao, 2023) is selective structured state space model that sub- stantially improves the performance of state space models (SSM) handling sequential data. It transforms parameters in the structured state space model (S4) (Gu et al., 2021) into learnable func- tions and proposing a parallel scanning method. By overcoming the local perception limitations of convolutional neural networks (CNNs) (Krizhevsky et al., 2012; Yang al., 2024) and the quadratic computational complexity of Transformers (Vaswani et al., 2017), Mamba-bases networks (Xu et al., 2024) are widely applied in various tasks. For instance, the original Mamba (Gu & Dao, 2023) demonstrates comparable performance to Transformers in language modeling, audio generation, and DNA sequence prediction. Vision Mamba (Vim) (Zhu et al., 2024) marks the first introduction of Mamba to the field of computer vision, employing bidirectional SSM for global modeling and position embedding for position-aware understanding. Subsequently, VMamba (Liu al., 2024) proposes cross-scan module to address the direction-sensitivg challenges. LocalMamba (Huang et al., 2024) further improves performance by incorporating local inductive biases, while PlainMamba (Huang et al., 2024) is designed as a non-hierarchical structure for enhancing integration across the different
3.1
max
var
0.33
max
var
6.10
max
var
Max value
0.08
vananda
000
8003
0.0
0000
403
406
Hidden dim
Hiddon cim
Hiddan dim
(a) Gate Proj weight
(b) Hadamard rotation to (a)
(c) KLLTeenaaced rotation to (a)
max
var
var
max
var
Max value
varianco
1135
0325
Hidden dim
Hiden dim
Hidden
(d) MatMul input (PScan)
(e) Hadamard rotation to (d)
(f) Smooth-fused rolation (d)
Figure 2: Maximum values (blue color) and variances (red color) distribution across channels of: (a)
the original weight of the gate projection; (b) applying the standard offline Hadamard rotation to (a): (c) applying the proposed of the matrix multiplication; (c) applying the standard online Hadamard rotation to (d); (f) applying the proposed smooth-fuse fused rotation to (d).
rotation to (a); (d) the input activation (generated by PScan)
3

---

Published as a conference paper at ICLR 2025
scales. Mamba-ND (Li et al., 2024) simply alternates the order of sequence, effectively extending Mamba to multi-modal data including images and vidcos. Despite reduced computational demands and impressive performance, the large size of these models still limits their application edge devices.
Quantization Methods. Quantization is an effective model compression technique. Current meth- ods can be categorized into quantization aware training (QAT) and post training quantization (PTQ). While QAT typically necessitates full parameters training. which poses challenges for large models, PTQ (Zhou et al., 2024; Yue et al., 2024) has garnered more research attention. Quantizing full- precision variables of pre-trained models into low-bit integers, PTQ reduces the memory consumption and enhances the inference speed. For instance, in the field of Vision Transformer (Dosovitskiy, 2020), ViT (Lin et al., 2021) introduces a comprehensive quantization scheme for the first time, employing powers of two factors and Log2 quantizers for layer normalization and attention map- ping. RepQ- ViT (Li et al., 2023) further addresses the issue of extreme distribution in activations after layer normalization and SoftMax operations. In the field of Large Language Models (LLMs), GPTQ (Frantar al., 2022) introduces a layer-wise quantization technique based on approximate second-order information, quantizing weights to 3-4 bit with minimal accuracy loss. To suppress outliers in activations, SmoothQuant (Xiao et al., 2022) adopts a smoothing parameter that transfers the difficulty of quantizing activations to weights. Recently, QuaRot (Ashkboos et al., 2024b) adopts similar methodology. which combines the rotation in QulP (Chee et al., 2024) and the computational invariance SliceGPT (Ashkboos et al., 2024a), pushing PTQ to a new level. While these methods perform effectively for Transformer-based large language models, they do not work well with mamba models. Notably, to our knowledge, our method is the first PTQ solution specifically designed for Mamba models, applicable to both Mamba-bane based vision and language tasks.
3 PRELIMINARIES
3.1 STATE SPACE MODELS
The state space models (SSMs) are typically re- garded as contiguous linear time-invariant (LTI) systems (Kalman, 1960), which map input sig- nal s+) E R to its output (() E R through a hidden state h1) Rdx1:
SILU
Norm
LoRA
W B
W
W gate
W stale
proj)
[C
htt) = Ah(t 1) Bx((),
(1)
Gaad Proji
(Stali Proj)
Ysar (1) = Ch(()+ Dx((),
(2)
Convid
Mui
Softplus
where A E Rdxd B E Rdxi, C E IRd D g are weighting parameters, zt, and h(0)
E
SILU
SIUU
D Mui
is an initial hidden state.
SSM
Mui
EXP
Muil
3.2 MAMBA ARCHITECTURE
MalMul
ond
Since the usage of LTI system, the model parame- ters remain unchanged, decreasing the performance when representing changing inputs. To tackle this issue, Mamba (Gu & Dao, 2023) propose an implementation of selective SSM (Gu et al., 2021), which formulates parts of the parameters as functions a specific input sequence:
Figure 3: Mamba block architecture.
x' (DWConn
Projection(z))), A Sofplus(LoRA Module(æ'))
A = AoA B B Projecctioooz=' @ A, C C_Projection(z²),
where x' denotes the transformed input and represents the SiLU activation. Those input-dependent parameters and x' are used by the Parallel Scan (PScan) operator to generate y'amm The calculation process of PSean can be expressed as:
h(t) Ah(t - 1) Bx()), Vism (t) Ch(t),
(5)
This temporary output then element- -iineyt-wisely multiplied with a gated variable z to generate better outputs:
z = (Gate (rattePPrjection(())). Yout = Veom z.
(6)
4

---

Published as a conference paper at ICLR 2025
3.3 QUANTIZATION
Quantization is generally performed to obtain a low owwppecisiin representation (e.g., 4-bit integer) from a uniformly quantized to b-bits as follows (Jacob et al., 2018)) 8):
variable (e.g., 16-bit floating points). For tensor be quantized, can be
z 8
max (x) min(x)
min
σ = (clamp(
2, - 1) - 2).8. 8,
(7)
2
8
where 2 is the zero point, is the scale factor, [-] denotes the rounding-to-nearest operator, clamp is the clipping function.
METHOD
4.1 DIMINISHED EFFECTIVENESS OF HADAMARD TRANSFORMATION
Hadamard transformation is a promising quantization method for LLMs, recognized for its effective- ness in handling outliers and its computational simplicity and speed. It provides robust performance while efficiently managing data variability.
Hadamard matrices are square matrices with orthogonal rows and columns, where each element is either _ or _1_ (m is the order of the Hadamard matrix). By multiplying with such a uniformly given channel, thereby making the extreme values of the channels closer together (Tssng et al., 2024). Additionally, due to the orthogonal nature, the Hadamard matrix can be well integrated into model weights while ensuring the computational consistency.
vm ym distributed matrix, each row contributes relatively equally
We initially attempt to directly apply this method Mamba models, particularly to the gate projection, output projection, and the matmul layer. However, the Hadamard transformation is not sufficiently effective in normalizing the hard layers mentioned in Figure 1 of the Mamba architecture with significant outliers, as illustrated in Figure 2(b)(e).
To this end, we conduct a thorough analysis this issue and find that this method fails to align the channel variance of quantization variables, thereby overlooking the distribution consistency between different channels. In detail, given a centered data matrix (the columns of the matrix are zero-mean.) X (weights or activations) with dimensions (n, m) and the Hadamard transformation matrix H with dimensions (m, m.) the covariance matrix Cx of the transformed matrix X H can be expressed as:
CxZ = n-
XH (
1 H"KAK" H, 1 H' 1
(8)
n n
where X KAKT represents the eigenvalue decomposition. K is the eigenvectors matrix, and A is the diagonal eigenvalues matrix. Considering that H K and K" H are transposed matrices of each other, the 1-th diagonal elements of CxH can be expressed as:
m
m
(CxH)u n Σ I
HaKKy)
(9)
where X; the j-th eigenvalue of A. The complete derivation from Equation to Equation is provided in Appendix A.2. For given value of 7, Equation 9 represents the variance of the 1-th channel. As the vector H.j varies with the I, the channel variances cannot be proven to be numerically close in most cases refers to Appendix A.3. Further, considering that H is a fixed matrix while both K and À are input-dependent it is not feasible for the Hadamard transformation to uniformly adjust the channel variances across all scenarios. This property of Hadamard transformation inevitably formulates a distinct distribution for each channel, thus leading to sub-optimal quantization.
4.2 KLT-ENHANCED ROTATION FOR OFFLINE TRANSFORMATION
To overcome the constrain stated in Section 4.1, we introduce the Carhumen-Loöre Transformation (KLT) (Dony al., 2001) equalize channel variances. KLT identifies principal components in the data and projects onto these components, retaining the most critical information by focusing on directions of maximum variance. In practical, the mean value for each channel of the Mamba weights
5

---

Published as a conference paper at ICLR 2025
Offline Rotate
Offline Rotate Mamba Block By Block
fuse
Embed
Block1
Block
Head
de
fuse
K
fuse
Watata
fuse
fuse
Fatate
fuse
fuse
fuse
Wont
mut
H K
Wat
fuse
fuse
(a) LoRA module
(b) Inter-block connection
Figure 4: Offline transformation designs utilizing the KLT-Enhanced rotation.
and activations is typically close to zero, meeting the applicable conditions of KLT. Specifically, We apply KLT by performing eigenvalue decomposition the covariance matrix Cx of the centered matrix X derived from the calibration data.
Cx - -x=-_x²> n-
n- KAK".
(10)
Next, the KLT-Enhanced rotation matrix H can be obtained by applying the KLT to the Hadamard matrix H, as described Equation 11, and the Equation turns into Equation 12:
HK KH,
(11)
1 KAKK Hx H"K KH n
1 H'IAIH (12)
CXHR = n-1
where I denotes the identity matrix. Consequently, the Equation 9 thus turns to Equation 13:
(CxHR)u
Σ
m
m
Hale))
1
m Ag.
(13)
Imm
In this way, the variance of each channel becomes the same, making quantization much casier. This transformation serves a dual purpose: it not only equalizes the variance among different channels but also embodies the distinctive property Hadamard matrices, which is their ability to balance maximum values. We also provide detailed steps for the formula of performing KLT rotation followed by Hadamard rotation in Appendix A.4 to achieve variance balancing. practice the KLT is offline performed by using the calibration data to avoid extra computational costs. Still it can be rell-generalized to wider range inputs (detailed Appendix A.7).
To apply this Kl.T-Enhanced rotation matrix, we modify the offline transformation in QuaRot (Ashk- boos et al., 2024b) for the Mamba structure. As shown Figure 4, we employ this strategy for the LoRA module and the inter-block connection (where the output projection, gate projection and the state projection is transformed).
4.3 SMOOTH-FUSHH ROTATION FOR ONLINE TRANSFORMATION
To mitigate the shortcoming of the Hadamard rotation discussed in Section 4.1 where the online transformation is applied, we introduce the smoothing technique prior to its execution. The moti- vation of employing this method to uniform the channel variances through smoothing vector. Typically, the smoothing factors can be absorbed into the neighbored layers with the quantization of T-LLMS (Xiao et al., 2022: Shao et al., 2023). This operation effectively cireumvents the demand for additional memory allocation and computational overhead that would arise from the incorporation of extra parameters. However, this approach does not align with the Mamba modules due to non-linear SiLU operation and the complex loop structure of PScan. To this end, two distinct designs are proposed for output projection and matrix multiplication, respectively.
For the output projection layer: We improve the traditional SiLU activation function with Smooth SiLU (S-SiLU) (Hu et al., 2024) to meet the needs of smooth-fused quantization:
S-SiLU 8) 20(8 x)),
(14)

---

Published as a conference paper at ICLR 2025
fuse
W gate
W B
((I) (2)) (0)
fuse
gofee
WB
Ezcl
SILU
S-SILU
Mul
Mul
Mul
AddoMul
Mui
Mui
Mul
fuse
Mui
EXP
Mui
EXP
fuse
Wout
PScan
MatMul
Wc
PScan
MatMul
W
Origin
Ours
Origin
Ours
(a) Output Projection
(b) Matrix Multiplication
Figure Fusing smooth parameters into the Mamba structure.
where æ is an activation variable, a(.) represents the Sigmoid function, & denotes the introduced smoothing parameter, and "O" orrepresent element-wise multiplication. Depicted in Figure 5(a), the application of the S-SiLU function on the gate projection described by Equation 6 can be expressed as follows:
Yout
ssm C Sa, W,),, = [ysmm + S-SiLU((x, W S
(15)
where Ussm denotes the output activation of the SSM, W = W 2 outt and W Sout W. transformed weights of the gate projection (denoted with subscript 'g') and the output projection (denoted with subscript 'o'), "Ø' d'represents element-wise division, Sout is the absorbed smoothing factor, 2g is the input of the gate projection, and yout represents the final output of the Mamba block.
For the matrix multiplication layer: We also design a scheme to absorb the smoothing factor for the matrix multiplication operator within the Mamba block. One input stream of the multiplication the output of the C projection in Equation 4, which can directly fuse the smoothing factor Sym into the weight of C projection (Wc) as shown in Figure 5(b). Another input stream comes from the output of the parallel scan operator. As shown in Equation 5, the calculation PScan includes addition operator, and the smoothing factor 8mm will be transmitted along two routes on both sides of the addition operator. One route is transmitted through B and absorbed by the weight of B projection (W') B), and the other route is transmitted through A and absorbed by A, which defined in Equation 3. Because of the existence of exponential calculation in Equation 4, 1/5mm becomes - (n((5m)) when transmitted to A, and is absorbed by applying the addemul operator (PyTorch, 2023) to A(1) in Euation 16. is solely applied to the first token of A (A(1)).
addcmul( - In(s mm ), A (1), A) = (1) - In (m()m.).
(16)
After smoothing, the channel variances of activations for the output pro- jection and the matrix multiplication becomes relatively uniform. Subse- quently, we modify and apply the online Hadamard rotation (Ashkboos al., 2024b) for the Mamba structure as shown Figure 6. The Hadamard matrix H is dynamically applied to the input activation of the output projection and the matrix multiplication, while the transposed H can be absorbed into corresponding weights.
Online Hadamard Rotate
fuse
We
fuse
H
W
matmul
Figure 6: Online transfor- mation designs utilizing the smooth-fused rotation.
5 EXPERIMENTS
Models and datasets. We assess the general quantization capabilities of our proposed MambaQuant framework across three representative Mamba-based applications: Mamba (Gu & Dao, 2023), Vim (Zhu al., 2024), and Mamba-ND (Li et al., 2024). We evaluate the performance of the quantized Mamba model across vision and language tasks. For vision tasks, tested the model on the image classification dataset ImageNet (Russakovsky et al., 2015) and the video classification dataset UCF-101 (Soomro et al., 2012). the language domain, conducted evaluations five standard datasets: ARC-E (Boratko et al., 2018),
7

---

Published as a conference paper at ICLR 2025
Table I: Comparative results under different quantization settings for Vision Mamba models. The Vim models and Mamba-2d models are tested for accuracy on ImageNet, while the Mamba-3d model tested for accuracy on UCF-101. 1 indicates the fine-tuned model on Vim. denotes results based on official weights.
Vision Mamba
Mambu-ND
Bit Width
Methods
Vim-T
Vim-T
Vim-S
Vim-S!
Vim-B
mamba-2d
Mamba-2d
Mambu-3d
FP16
76.1
78.3
80.3
81.6
80.31
81.7
83.0
89.6
RTN
37.4
32.4
68.8
52.2
80.3
82.2
87.9
GPTQ+RIT
37.7
32.5
68.9
70.5
52.2
80.4
82.2
87.8
W8A8
SmoothQuant
37.7
32.3
68.7
72.9
52.1
80.3
82.2
87.9
QuaRot
59.3
57.4
73.8
75.5
73.8
80.8
82.3
88.0
Ours
75.6
77.8
80.3
81.4
80.1
81.2
82.8
89.0
RTN
26.3
25.0
66.1
70.0
46.2
40.6
78.8
86.
GPTQ+RTN
30.4
27.9
66.5
70.6
47.7
60.3
78.9
86.8
W4A8
SmoothQuant
27.0
26.0
66.4
70.2
46.7
59.7
80.2
86.9
QuaRot
52.7
48.5
72
74.0
72.8
80.1
82.0
86.9
Ours
72.1
73.7
79.4
80.4
79.8
80.4
81.9
88.4
ARC-C (Clark al., 2018), PIQA (Bisk et al., 2020), Winogrande (Sakaguchi al., 2021), and HellaSwag (Zellers et al., 2019), and reported the average performance across these datasets. The metric used for the evaluation of our test results these tasks is Accuracy (Acc).
Baselines and implementation details. For comparison, apply different quantization settings to the Mamba model and reported the performance under two configurations: W8A8 and W4A8 (weights and activations). Additionally, we compare with the different quantization methods, including the Round To Nearest (RTN) method, AmoothhQunt (Xiao et al., 2022), GPTQ (Frantar et al., 2022) for weights and RTN for activations (GPTQ+RTN), well as QuaRot (Ashkboos et al., 2024b). For the vision tasks, we utilize static quantization approach. The calibration data for image classification was randomly sampled from 128 images in the ImageNet (Russakovsky al., 2015) test set, while for video classification, we used samples from the UCF-101 (Soomro et al., 2012) test set for calibration. In contrast, for language tasks, we employ dynamic quantization better adapt the varying input structures during inference.
5.1 OVERALL RESULTS
Performance Comparison on Vision Model. Table I presents the results of various Mamba vision models under different quantization settings, including Vision Mamba and Mamba-ND, The quantization configurations evaluated are W8A8 (8-bit weights and activations) and W4A8 (4-bit weights and 8-bit activations). The table compares the performance of several quantization methods, including RTN, GPTQ+RTN, SmoothQuant, QuaRot, and our proposed method ("Ours")), across different Mamba model variants. Our proposed quantization method demonstrates significant improvement over baseline techniques. Under the W8A8 configuration, the performance of our method remains within points of the floating-poin bascline accuracy. the stricter W4A8 setting, our method substantially outperforms competing approaches, which experience more pronounced accuracy drops. These results indicate that our approach offers more robust solution for maintaining high accuracy in Vim and Mamba-No models under quantized settings. The findings suggest that our method is more resilient to the challenges of precision reduction and provides both practical and effective quantization solution for deploying Mamba models.
Performance Comparison on Language Model. Table 2 illustrates the quantization results for the Mamba model applied to language tasks. The models evaluated range from Mamba-370m to Mamba-2.8b. The table compares the accuracy of several quantization methods, includingRTN, GPTQ+RTN, SmoothQuant, QuaRot, and our method. Our approach delivers superior performance across different model sizes, particularly under the more challenging W4A8 configuration, where consistently outperforms baseline techniques by significant margin. These results demonstrate the robustness and efficiency of our quantization method for language model tasks.All experimental results for both vision and language models under our proposed method are provided Appendix A.6.
8

---

Published as a conference paper at ICLR 2025
Table 2: Comparative results under different quantization settings on language mamba models. Evalu- ations on the five standard datasets-ARC-E, ARC-C, PIQA, Winogrande and reesulted the reported average accuracy across these datasets.
Mamba-LL LLM
Width
Methods
Mamba-370m
Mamba-790m
Mamba-1
Mamba-2.88
FPI6
50.9
54.8
58.6
62.2
RTN
45.7
44.9
53.9
58.4
GPTQ+RTN
46.2
48.6
55.0
58.9
W8A8
SmoothQuant
45.2
41.7
54.2
58.7
QuaRot
48.8
51.6
56.9
59.3
Ours
50.0
53.8
58.3
62.1
RTN
36.2
35.4
51.6
54.8
GPTQ+RTN
36.7
36.0
51.1
53.6
W4A8
SmoothQuant
36.8
39.3
52.0
54.9
QuaRot
43.4
40.0
53.8
58.5
Ours
43.9
54.3
58.5
5.2
ABLATION STUDY
Table 3: Ablation Experiment For KL.T-Enhanced Rotation.
Width
Methods
Vim
Mamba-
Bit Width
Methods
Vim
Mamba-790m
FP16
78.3
54.8
FP16
78.3
54,8
Baseline(KTN)
32.4
44.2
(ascline(RTN)
25.0
35.4
WSAS
Hadamard Rotate
33.9(1 1.5) 47.701 15.3)
50,8(1 6.6)
W4A8
Hadamard Rotate
25.1(1 0.1)
40,2(1 4.8)
KLT E-T--nhnced Rotate
51.3(1 7.1)
KLT-Enhanced Rotate
38.9(1 3.9)
42,3(1 6.9)
1,99 Percenele
Percentile
1/09 Percentie Percentlee NinMax
25775 Percentile
2575 Percentle
Value
MixMax
MinMax
Hidden dimension index
Hidden dimension indixx
Hidden dimension index
20000
0.0051
loss: 0.0033
KLT-Ertanos5
L1 loss: 0.0016
Hodamontt 1003)
Count
0.015
0.15 -0015
0.015 0.015
0.015
Loss Value
Loss Value
Loss Value
(a) Gate Proj's input
(b) Hadamard rotation to (a)
(c) KLT-Enhanced rotation to (a)
Figure 7: Data of activatin of gate projection distribution and quantization of losses. (a) Original data (b) Hadamard rotateion to (a) (c) KLT-Enhanced rotation to (a). The first row of graphs depicts the
lineshowing the distribution of data values at different quantile points across various channels, while the second row illustrates the count bar graphs representing the different quantization losses.
Ablation Study KL.T-Enhanced Rotation. We conducted series of ablation studies the offline rotation layers descrihed Section 4.2, comparing the KLTTEEnhanced rotation and Hadamard rotation techniques proposed in our method. The ablation study comparisons from Table demonstrate the effectiveness of the KLT-enhanced rotation approach. Compared with the direct use of hadamard rotation, our method achieve a greater degree of progress improvement, which is about 14% higher than the direct hadamard method in the Vim-T quantization of W8A8.
In addition to the quantitative analysis of the ablation study results, Figure 7 visually conveys the optimization brought by the KLT-enhanced rotation over the Hadamard rotation. The upper part of Figure 7 shows the quantile distribution of data across different channels, while the lower part illustrates the quantization loss distribution using the 4-bit per-tensor method. In the bar graph, the horizontal axis represents quantization loss magnitude, and the vertical axis indicates the data point count. Subfigures (a), (b), and (c) display the quantization losses for the original data, Hadamard- rotated data, and KLT-enhanced rotation data, respectively. Comparing these, the KLTTenhancedd rotation clearly outperforms Hadamard rotation in smoothing quantization and reducing losses across channels. Moreover, our method reduces the LI loss from quantization by nearly half compared to using Hadamard rotation alone.

---

Published as a conference paper at ICLR 2025
Table 4: Ablation Experiment For Smoothed Rotation.
Bit Width
Methods
Vim-T
Mambb-790M
Bi Widhh
Methods
Vim-T
Mamba-700M
PP16
78.3
54.6
FPI6
78.3
55.6
Baseling KL Kototion)
47.7
51.3
Biselinct KL K-T-callned Kotssion)
SK9
42.3
WBAS
Hadumard Rotation
60 22.0)
0.5)
W4AS
Hadumard Rotation
23.1)
0.7)
Smooth Rosation
77.80 30.1)
53.301 2.00
Rotation
73.30 34.80
45.80 3.5)
0.15
1/50 Percentle
1/99 Parcentle
1/50 Percenble
25/75 Pecertilee
25/15 Percentile
25/75 Pecemtee
Value
Maxx
MinMax
MixMMa
-0.15
1500
3000
1500 Hidden dimension Indix
3000
1500
3000
dimension
Hidden dimension index
Origin
LT loss: 0.0037
Hodamard
L1 loss: 0.0030
Smooth Fused rojato
Count
Hadamand rutate
805
0.015
Loss Value
0.015 40015
Loss Value
0.015
0.015
Loss Value
(a) Output Proj's input
(b) Hadamard rotation to (a)
(c) Smooth-Fused rotation ro (a)
Figure 8: Data of weight of output projection distribution and quantization of losses. (a) Original data quantile and fake quantization loss, (b) Hadamard rotated data quantile and fake quantization loss, (c) Smooth based rotated data quantile and fake quantization loss.
Ablation Study on Smooth-fused Rotation. We applied the method described in Section 4.3, focusing on the sensitive matmul and output projection layers. The ablation results comparing the use of online Hadamard rotation and smooth-fused rotation are presented in Table 4. We found that the amooth-fused rotation can lead to significant improvements in quantization accuracy compared to directly using Hadamard rotation. Under the condition of W4A8 quantization configuration. in the Vim T model, the Smooth- Fused rotation method is 11.7% ahead of the direct online Hadamard rotation method in terms of accuracy.
Memory Occupancy and Computational Cost.
In method 4.3, we introduce a smooth scale parameter to optimize the quantization process, with minimal storage overhead since each quan- tization channel is represented by a single scalar, which has negligible impact overall model size. Additionally, the method introduces the application of the online Hadamard rotation technique described in the QuaRot (Ashkboos et al., 2024b), which ensures rapid transformations akin to FFT, minimizing computational impact on inference speed. Taking the Mamba-2.88 model as an example, the smooth scale adds only 329k parameters to the 2.8B model, while for a token sequence length of 1024, the computational increase is 25.6 GFLOPs over the bascline 2.8 TFLOPs. This translates to a 0.01% % increase in the parameter size and just a 0.91% increase in computational cost.
CONCLUSION
In this paper, we focus on introducing the quantization techniques into the realm of Mamba models. Firstly, we identify that significant outliers which challenge the quantization process are present in gate projection, output projection, and matrix multiplication, while the unique PScan operator further amplifies the numerical differences. Secondly, we find that Hadamard transformation method widely adopted in Transformer quantization performs less satisfying when quantizing these hard layers. Our analysis reveals that this method falls short of sufficiently aligning the channel variances, thus remaining an uneven distribution challenging the quantization process. To beyond this limitation, we propose MambaQuant, a comprehensive post training quantization framework especially designed for the Mamba models. The core idea of this strategy is to facilitating the Hadamard transformation with the ability to uniform the variance of each channel, thus enhancing the performance of Mamba quantization. Specifically, we introduce the Karhunen-Loëve Transformation to render the rotation matrix adaptable to diverse channel distributions. We also incorporate a smoothing methodology to uniform the channel variances, while additional parameters are fused into model weights to avoid extra overhead. Our proposed MambaQuant advances in accuracy for both Mamba-based vision and language tasks compared to existing methods, making Mamba models more practical for deployment in cssutrc-constrainee environments.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Dosovitskiy Alexey. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929, 2020. 19
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. International Conference on Learning Representations (ICLR), 2024a.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llss, arXiv preprint arXiv:244.00044 2024b. 2, 4, 7, 10
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Association for the Advancement of Artificial Intelligence, 2020. 8
Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew Pavan Kapanipathi, Nicholas Mattei, et al.
McCallum, Maria Chang, Achille systematic classification of knowledge, reasoning, and context within the are dataset. arXiv
preprint arXiv:1806.00:: 2018. 7
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024. 4
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. 2018. 8
R Dony et al. Karhunen-loeve transform. The transform and data compression handbook, 1 1(1-34):
29, 2001.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192) 2020. 4
Dayou Du Gu Gong, and Xiaowen Chu. Model quantization and hardware acceleration for vision
transformers: A comprehensive survey, arXiv preprint arXiv:2405.00314, 2024. 2
Fino and Algazi. Unified matrix treatment of the fast walsh-hadamard transform. IEEE Transactions
Computers, 100(11):1 (11111::11422-1 11:11144216666 1976. 17
Elias Frantar, Saleh Ashkboos, Torsten Hoetler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint UrXiv:2210.173233 2022, 4,
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint ar arX1v:2312.00752, 2023. 1, 2, 3, 7
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. 1,
Xing Hu, Yuan Chen, Dawei Yang. Sifan Zhou, Zhihang Yuan, Jiangyong Yu, and Chen Xu. I-llm: Efficient inleger-only inference for fully-quantized low-bit large language models. arXiv preprint arXiv:2405.17849 2024. 6
Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan. arXiv preprint 2033038403. 2024. 3
Benoit Jacob, Skinmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmeeic-only inference. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018. 5
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Journal of Basic Fngineering, 82(1):35 1960. 4
11

---

Published as a conference paper at ICLR 2025
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08.2 2018. 2
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012. 3
Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-nd: Selective state space modeling for multi-dimenssonal data. arXiv preprint arXiv:2402.05892, 2024. 1, 2, 4, 7
Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq-vit: Scale reparameterization for post-training quantization of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17227-17236, 2023.
Lin, Jiamiigg Tang, Haotian Tang, Shang Yang. Xingyu Dang, and Song Han. Awq: Activation- aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.0097 00978, 2023. 17
Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. Fq-vit: Post-training quantiza- tion for fully quantized vision transformer. arXiv preprint arXiv:2111.13824 2021. 4
Jiawei Liu, Lin Niu, Zhihang Yuan, Dawei Yang. Xinggang Wang. and Wenyu Liu. Pd-quant: Post-training quantization based on prediction difference metric. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24427-24437, 2023. 2
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint axXiv:2401.10166, 2024. 1,
PyTorch. toreh.addemul. https://pytorch. wrtg///cs/staale///eeeeeeeeeeeee http://////c//.aaaa//////er////e
html, 2023. 2024.09.24. 7
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal computer vision, 115:211-255 2015. 8
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 2021. 8
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Ommidirectionally calibrated quantization for large language models. CoRR, abs/2308. 13137, 2023. 6, 17
Jimmy TH Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.044: 049333, 2022. 2
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from vidcos in the wild. Center for Research in Computer Vision, 2(11):1-7, 2012. 7,
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipit: Even better I1m quantization with hadamard incoherence and lattice codebooks. Forty-first International Conference on Machine Learning, 2024. 2,5
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 2017. 3,
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint ar arXiv:2211.10438. 2022. 4, 6, 8, 17
Rui Xu, Shu Yang. Yihui Wang, Yu Cai, Bo Du. and Hao Chen. Visual mamba: A survey and new outlooks, 2024.7 2,3, 19
Dawei Yang, Yan Wang, Ran Wei, Jiapeng Guan, Xiaohua Huang, Wei Cai, and Zhe Jiang. An efficient multi-task learning enn for driver attention monitoring. Journal of Systems Architecture, 148:10300 103085, 2024.
12

---

Published as a conference paper at ICLR 2025
Zhihang Yuan. LLMViewer. https: httpsr//ghhh....cm///ayyaa////aa 2024. Accessed: Febru- ary 14, 2024.2
Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint vrXiv:2402.12065 2024. 4
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint
2019. 8
Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. arXiv preprint arXiv:2403. /4520, 2024. 1
Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey efficient inference for large language models. arXiv preprint arXiv:2404. 14294, 2024.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024. 1, 2, 3.
A APPENDIX
A.1 LIMITATIONS
Recently, Mamba models demonstrates superior accuracy across various vision and language tasks, indicating its strong capabilities in feature extraction and pattern recognition. However, its deploy- ment with the quantization methodology still remains largely under explored. We thus propose MambaQuant, an accurate and efficient post training quantization framework especially desined for the Mamba family.
While this approach can effectively quantize the weights and activations of Mamba models into 8-bit with less than a 1% drop in accuracy, it struggles to maintain such high level of accuracy when quantizing weights to 4-bit. In addition, we note that the proposed Karhunen-Loève Transformation (KLT) enhanced rotation is efficiently constrained if applied to the online Hadamard rotation. This is primarily due to the additional computation steps introduced by the eigenvalue decomposition (as stated in Equation 10) and the application to the Hadamard matrix (as stated in Equation 11). Despite the constrains, we hope that our work could inspire the research interest on Mamba quantization within the community. We are also committed to extending the KLT-Enhanced rotation method to online transformation in order to achieve better performance in low-bit quantization.
A.2 DERIVED THE INABILITY OF HADAMARD ROTATION TO ENSURE CONSISTENCY OF COLUMN VARIANCE.
Hadamard properties. An orthogonal matrix Q is square matrix such that QQ = I. In this work, we consider only real orthogonal matrices. A rotation matrix is an orthogonal matrix with [QI = 1. A Hadamard matrix is an orthogonal matrix with each element is either 1 or A 2x2 Hadamard matrix is defined as follows:
Vm
1 V2 (1 :1)
H
(17)
Covariance Calculation. Given matrix X with dimensions (n, m) with zero mean across each column. Alculate the covariance matrix Cx of X:
Cx = x n-1 KAKT
(18)
where K is the eigenvectors matrix, A is the diagonal eigenvalues matrix, n denotes the number of rows. We provide proof based the above properties of Hadamard that Hadamard cannot achieve
13

---

Published as a conference paper at ICLR 2025
column variance consistency. In detail, given Hadamard transformation matrix H with dimensions (m, mhe covariance matrix Cx the transformed matrix X can be expressed as:
Cxи (XH)* XH
(19)
Subsituate Equation 18 into Equation 19:
CxH=n-11'
Cxh = n
H X
H KAK" H,
(20)
Hadamard matrix expansion:
H1
H12
Him
H21
H2a
H2m
H
(21)
:
Hm
Hm2
Hmm
KLT matrix expansion:
K11
K12
Kim
K21
K22
K2m
K
(22)
:
Kmi
Km2
Kmm
We define the matrix P = H²K: K:
m
Hn
K
Ha Kim
Li=1
H K
Li=1
H2g
H Kim
P P=H*K=
(23)
1i=1
Kn
E Him K22
E Him Kim
Substitute Equation 23 into Equation 20:
Cxн PAP
p2
A
-=
Σ PE
m i=
p2 mi
A
m E 1=
Hap K,g))
1--1
E EE Him Kig)
(24)
m
(CxH)u n H
H
H HaKK, 23
(25)
Since the values of Psj are not equal, the variance for each column of matrix XH (which is the value on the diagonal of Cxн) is also not equal. Because the Hadamard transform is a fixed orthogonal transformation, fixed orthogonal transformation cannot uniformly adjust the variance in all directions, resulting in the variance after transformation still being uneven.
A.3 THE VARIANCE AFTER HADAMARD ROTATION IS STILL INCONSISTENT
We provide an example of simple random 4x4 matrix after it has undergone the Hadamard transform. Example of a simple random 4x4 matrix R and 4x4 Hadamard matrix H::
3
-1
0
-4
2
3
-3
1
(26)
1
33
-3
2
1
-1
6
14

---

Published as a conference paper at ICLR 2025
1
1
-1
1
-1
H
(27)
2
-1
Calculate the covariance matrix Cru of the matrix RH after the Hadamard transform:
1.83
-483
-2.83
2.00
4.83
30.50
2.17
-5.67
CRH =
(28)
22.83
2.17
8.75
-1.50
2.00
-5.67
-1.50
2.17
In the random example provided, it can be seen that after the Hadamard rotation, there is significant imbalance in the variance across the columns (which are the diagonal values of the covariance matrix CRur).
We observe the distribution within the actual Mamba network. In the Vim-T network, we compare the variances before and after the Hadamard rotation for the output projection inputs of the 1st, 10th, and 20th blocks, as shown in Fig. 9. Additionally, Hadamard rotation is applied to the gate projection inputs of the 1st, 10th, and 20th blocks in Fig. 10, as well to the gate projection weights of the 1st, 10th, and 20th blocks in Fig. 11.
A21
Origin
Vidsen drmession
Hddden disesicon Inden
iisden dimeniion indee
Hadamard Rotated
Veteene
Wine
Tiaddnn (b) block input of out projection in Vim-T
Teidee dimrrnsonn (c) the 20th block of the Input out projection Viin-T
(a) the 1st block of the Input of out projection Vim-T
Figure 9: Variances of input of output projection layer blocks in Vim-T: unequal across channels pre- and Hadamard rotation
2.40
1.5g
Origin
x20
Tradeen dimenioo
Hilcon on index
dknemiinneeiinnn iioe
coo
401
Hadamard Rotated
Hrddinn dimerisionn index
1odden dimematon
Hidden
(a) the 1st black of the Input of gate projection in Mamba-790m
(b) 10th block input
[c) the 20th block of the Input of gate projection Mambe-790m
gate projection in Mamba-790m
Figure 10: Variances of input of gate projection layer blocks in Mamba-790m: unequal across channels pre- and Hadamadd rotation
A.4 KLT-ENHANCED ROTATION TO ENSURE COLUMN VARIANCE BALANCE
KLT rotation before Hadamard rotation. We perform the KLT transformation on X to obtain a new matrix X', which is multiplying X by the matrix K the right:
X'= X
(29)
15

---

Published as a conference paper at ICLR 2025
Origin
coben dimena en modee
lddden dimensior index
13000
Hadamard
Rotated
50000
6.000
Hiddon
Hiddien Eiienssion
Hidden dinension kidex
(a) the 1st block of the weight of projection in Mamba-790m
weight
(c) the 20th block of the weight of gate projection
gate projection in Mamba-790m
Figure 11: Variances of weight of gate projection layer blocks in Mamba- 790m: unequal across channels pre- and post-Hadaman rotation
Equation 29 is the KLT transform. We calculate the covariance matrix of the data X' as Cx
Cx
X X'
(30)
n 1--1
We substitute Equation 29 into Eq 30:
Cx n- (XK) (XK) n
KT XK,
(31)
In Eq 19, we have x KAKT which can be substituted into Equation 31.
Cx
n- -ー
K"KKKKKKK
n-1 A,
(32)
We use the Hadamard matrix H to transform on X' to obtain X";
X" = X'H,
(33)
Calculate the covariance matrix Cx" of X":
Cx"
n1 1 (X")7T x"
n (X'H) X'H
n H x X'H
H Cx
n H AH.
(34)
n
Every element hij of the Hadamard matrix is 1/mm or -1/ V The composition of is
(A1,)2, An). Substitute Equation 21 into Equation 34.
m ===1 H A
Σ
H&N
Cx" - 1
(35)
E H \
Substitute the actual value of H3, 1/m, into Equation Furthrr calculate Cx and the result is:
E
A
Cx"
(36)
(n 1)m
The main diagonal elements of Equation 36 represent the variance of each column of the matrix X", which has been transformed first by KLT rotation and subsequently by Hadamard rotation. As evident from the equation, these variances are observed be perfectly uniform across all columns.
16

---

Published as a conference paper at ICLR 2025
A.5 THE CALCULATION PROCESS OF INCREASING PARAMETERS AND ADDING COMPUTATIONAL LOAD
In this section, we take the Mamba-2.88 model as an example to illustrate the increase in parameter count and computational load. The Mamba-2.8h model comprises 64 blocks, and we assume token quantity of 1024 for our calculations. We detail the computational overhead introduced by our smoothing process. The original parameter count of the Mamba-2.8b model is 2.8 billion. In our Method 4.3, we integrated a smoothing process by replacing the SiLU activation function with the SiLU activation function, which added 5,120 smoothing scale parameters. Additionally, modified the mul operation to an addemul operation, contributing an additional 16 parameters. Consequently, the percentage increase in parameter count is calculated as:
(5120 20 16) x 64) = 0.01%.
(37)
2.8 109
We estimate the original computational load of the Mamba-2.88 model to be 2.8 TFlops. In Method 4.2, we employed the online Hadamard technique, which has a complexity of (nlogg(n)) The complexity of the online Hadamard transformation is provided in Equation (1) of the paper by (Fino Algazi, 1976). Within the Mamba block, we inserted an online Hadamard transformation of size [16,16] for the matmul smoothing and similarly used a [5120,51200 120] Hadamard transformation for the output projection smoothing. Thus, the percentage increase in computational load calculated as:
(1024 x 5120 x 16 x (16) 1024 x 5120 x log, (5120)) x
R3 0.91%
(38)
2.8 x 1012
This analysis demonstrates that our proposed enhancements introduce minimal increases in both parameter count and computational load, making them practical for efficiency-criteee criticcl applications.
A.6 ADDITIONAL RESULTS
Table 5 shows the performance of increasing the KLT based rotate method and the smooth based rotate method on a variety of visual tasks.
Table 6 shows the performance of increasing the KLT based rotate method and the smooth based rotate method on a variety of language tasks.
Table 5: Add our methods sequentially vision tasks
Vision tasks
Vim
Mamba- ND)
Models
Vim-T
Vin-1
Vim-S
Vim-S²
Vim-BB B
Mamba-
Mamba 2d B
Mamba 3d
76.1
78.3
80.3
81.7
RIN
37.4
32.4
68.8
72.8
$2.2
79.9
82.2
88.9
W8A8
+ KLT EnEuunedd Rotation
48.4
47.7
73.4
77.2
72.9
80.5
82,2
89.2
-Smooth-Fused Rotation
75.6
77.8
80.3
81.4
80.1
81.5
82.8
89.2
RTN
26.3
25.0
46.2
40.6
78.8
87.1
W4A8
*KLT-Enhancee Rotation
41.7
38.9
71.3
75.7
71.2
79.2
81.7
88.0
+Smooth- Fused Rotation
72.1
73.7
79.4
80.4
79.6
80.4
82.5
88.4
A.7 GENERALIZATION EVALUTATION OF THE KLT-ENIANCED ROATION
To generate K without introducing online computational overhead, we use the calibration method, which is common practice of PTQ works (Lin et al., 2023; Xiao et al., 2022; Shao et al., 2023). In most cases, this method can effectively characterize the whole data distribution by a small subset to achieve rapid quantization optimization.
For instance, the experiments shown in Table are calibrated from the HellaSwag dataset. Zero-shot evaluation results, especially other datasets like ARC-E, ARC-C, PIQA, and Winogrande can effectively demonstrate the generalization ability.
Furthermore, for various inputs, visualize the activations distribution of the in projection layer of the first block in the Vim-T model. model. Then we respectively performs the classic Hadamard rotation and KI.T-enhanced rotation.
17

---

Published as a conference paper at ICLR 2025
Table 6: Add our methods sequentially on language tasks
Models
Width
Avg ACC
Arc-E
Arc-C
PIQA
WinoGrande
HellaSwag
FP16
44.7
48.0
24.3
RTN
41.5
41.0
25.4
56.3
51.7
32.9
W8A8
*KLT-Enhanced Rotation
42.2
42.3
25.1
59.4
50.4
33.9
Mamba-1300 130m
Smooth-Fused Rotation
43.9
24.2
62.5
34.7
RTN KLT--nhanced Rotation
35.2
27.6
22.4
49.8
51.0
25.1
W4A8
39.0
33.5
23.9
55.8
51.7
30.1
Smoott-Fused Rotation
39.8
36.2
24.6
56.9
50.9
30.5
FP16
55.1
28.0
46.5
RTN
45.7
43.8
27.5
59.6
53.5
44.0
W8A8
KLLTEnhanced Rotation
49.0
50.8
28.9
66.1
53.7
45.6
Mamba-370m
Smooth-Fusee Rotation
50.0
54.1
28.2
67.3
53.8
46.5
RTN -KIT--Fnannced Rotation
36.3
27.2
25.7
50.7
W4A8
43.1
37.0
27.0
60.1
51.6
39.7
Smmoot-FFuse Rotation
44.1
38.7
27.3
61.5
53.9
39.0
Fp16
54.8
61.2
29.5
72.1
56.0
55.1
RTN
44.2
43.7
26.1
60.7
52.7
W8A8
KLT-Gnhanced Rotation
51.3
52.4
32.3
65.1
55.0
51.9
Mamba-790m
Smooth-Fused Rotation
53.8
59.5
30.6
68.9
55.8
54.6
KIN *KLT-Enhanced Rotation
35.4
28.0
25.0
50.7
49.6
23.5
W4A8
42.3
33.6
27.1
53.5
Semooth-Fuseh Rotation
45.8
39.8
30.2
62.5
52.3
44.3
FP16
58.6
65.5
32.8
74.2
614
59.1
RTN KKLT-Enhanced Rotation
53.9
54.9
31.4
69.4
56.8
57.0
W8A8
56.1
60.8
32.7
71.3
Mamba-1
Smooth-Fused Rotation
58.3
64.9
32.8
73.8
60.9
59.1
RTN *KL.T-nhhanced Rotation
51.6
51.7
30.9
64.9
57.4
53.0
W4A8
54.3
57.8
30.6
70.4
57.0
55.9
Smootth-Fuse Rotation
FP16
62.2
69.7
36.3
75.2
63.5
66.1
RIN KLLT-Enanced Rotation
57.5
60.4
33.8
71.0
58.5
63.9
W8A8
60.5
68.3
35.3
73.9
60.2
64.7
Mamha-245
+Smooth Fused Rotation
RTN
53.7
54.5
32.6
66.2
56.7
58.6
W4A8
+KLT
+KLT-Enhanced Enhanced Rotation
$8.0
60.7
35.8
71.6
60.7
61.1
+Smooth-Fucc Rotation
58.5
62.2
35.5
72.0
60.9
62.1
(a)
(b)
Maye
devverates
(d)
data rotabed by Ait- Alt-enhanced matriis.
percentile
by
Figure 12: Analysis of Activation Values in the in projection of the block 0 of the Vim-T model. Display using 384 images from ImageNet for calibration. (b), (c) and (d) Display of other data outside the calibration set.
Figure 12(a) shows the calibration data and its distribution after KLT-enhanced rotation, while Figure 12(b), (c), and (d) display the situation for non-calibration data. Figure 12(b) and (c) clearly demonstrate that the KL.T-enhanced rotation also effectively maintain the uniformity of the
18

---

Published as a conference paper at ICLR 2025
maximum values and variances of data with similar distribution calibration. Its effect is significantly better than using only Hadamard rotation. In most cases, the calibration of PTQ can sufficiently characterize various data distributions. For extreme cireumstances where the distribution of the imputs are dissimilar to that the calibration (as shown Figure 12(d)), the effect of the KI.T-enhanced matrix is still not worse than that of Hadamard rotation.
A.8 COMPARISON OF NUMERICAL DISTRIBUTIONS BETWEEN VIT AND VIM MODELS
The distributions of Transformers and Mamba are significantly different. For instance, we randomly sample 96 images from ImageNet, then feed them into the classic Vit (Alexey, 2020) model and Vim (Xu et al., 2024) model. Next, we calculate the top-3 channel maximums and top-3 channel variances of the input activations of all quantized modules in the last block. can be clearly seen that Vim has more uneven distribution and more outliers than Vit, leading to great challenges for quantization.
Table 7: Statistics of the top 3 maximum values and maximum variances among activation channels. We choose the modules be quantized of the last block of Vim and Vit. Data are randomly sampled from the ImageNet dataset.
Model
Module attention.qky proj attention.gk matmul.q attention.ag qk matmul.k attention.o proj attention.pp matmul.p
Top3 Channel Maximums
Top3 Channel Variances
4.7/3.873.7
0.4/0.470.4
7.1/7.1/7.1
2.1/16/1.
10.8 10.6 10.5
4.5/4.444.
11.6/ 8,6
4.1/3.4/3.2
vit-base-patch16-22
4.8 4.2
0.1/0.1 0.1
attention.pp matmul.y
14.9 12.5
6.5/ 4.5
mlp.fel mlp.fe2
11.4 7.8
2.0 1.6
17.26 13.15 13.01
12.10 9.80/9984/// 7733
in in_proo convld ld
49.3 /48.8/ 38.9
230.8 95.5 62.8
48.2 36.2 36.2
41.3 27.8 27.7
x x_proj
15.1 12,0/ 11.1
8.6 3.9
vin-base-patch 16-224
dt_ dr_proj
14.9 / 13.6 4.1
11.0/ 10.9 10.6
matmul.in inn
75.2 59.0 58.2
20.8 19.84 18.4
matmul.in? inn
8.5 6.7 5,4
1.8 /1.6 1.6
out_p
1371.6 / 1064.3 930.3
8854.9 2513.8/ 1377.8
A.9 VISUALIZED AMPLIFICATION EFFECT OF THE PSCAN OPERATOR
In Section 1, it is stated that the Parallel Scan (PScan) further amplifies the outliers of activations. Despite the corresponding explanation, viewpoint. Specifically, we focus on the activation statistics of the Vim-T network on the ImageNet dataset and visually present the distribution differences between the inputs and outputs of Pscan in different blocks.
here provide the visualized data to further support this
Figure 13(a) presents the distribution of activation values input Pscan, which corresponds to Bx(t) in Equation 5. Meanwhile, Figure 13(b) illustrates the distribution of activation values subsequent to the application the Pscan operator. has been observed that following the application of the pscan operator, the distribution differences of activation values in the hidden dimension become more pronounced. This phenomenon can attributed to the multiple consecutive multiplications involved in the internal implementation of the pscan operator.
19

---

Published as a conference paper at ICLR 2025
1,0990 Percentre Parcertll
1/5093 Percentie
1/29 Percentle
25/55 Percentle
25775 Percentle VirUe3a
HinMax
layer
3000
1/9099 Percentlee
1/9900 Percenblee Percantiee
1 Percentiee
Percentic MindMox
23/73 Porcentiee HissMax
Adteston vale
layer
3000
Percentile MirMMax
25/75 Percentlee
HixHMax
Actooution vatue
layer 23
-20-
-20
1000
2000
3000
4000
5000
5000
1002
2000
3000
4000
5000
5000
Hilddinn dimension
Hidden dimennsion Indle:
Activation Before Pscan
(b) Activation After Pscan
Figure 13: The amplification phenomenon of activations by the Pscan operator in different blocks of the Vim-T model.
20


Reviews:
[{"review_id": "ddROtNn1Xj", "reviewer": "Reviewer_KhMz", "summary": "This paper proposes a quantization scheme for a family of under-explored models, mamba based models. The authors observe that the outliers exists in certain type of projections in mamba models, and the parallel scan further make the issue worse. And the Hadamard transform, commonly used in quantization literature for Transformer models, is insufficient to address the outlier issue. The authors design a Karhunen-Loeve Transformation enhanced rotation to control the variance for offline weight quantization and Smooth-Fused rotation to control the variance for online activation quantization.", "questions": "1. I think it is possible (also easier) to control the variance of Hadamard transformed weight by simply applying a diagonal scaling matrix so that the variance of each row is the same. This is similar to per channel quantization, where different scalings are applying to different rows of weight matrix before doing rounding. I am wondering how does this scheme compare to the proposed method. \n2. The visualized value distribution in Figure 1 looks very similar to the visualization for Transformer models. More discussion on why quantization for Transformer models failed for mamba would be interesting. \n3. 8 bit quantization for Transformer models is quite easy without noticeable performance drop, but it looks like that quantization for mamba is hard based on the experiments. Can the authors comment on this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The KLT enhanced rotation for controlling the variance is interesting. \n2. The experiments show significant improvement compared to quantization methods proposed for Transformer models.", "weaknesses": "See question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GQ83apF49x", "reviewer": "Reviewer_5nSC", "summary": "This paper find out current PTQ methods on Transformer-based LLMs failed to quantize Mamba. Then it proposes MambaQuant with offline KLT-enhanced rotation and online smooth-fused rotation to improve PTQ performance. Experiments show MambaQuant achieves 1\\% accuracy loss on W8A8, better than QuaRot and SmoothQuant.", "questions": "KLT-enhanced rotation $KH$ seems to be also fixed, like Hadmard matrix $H$, for different input. To perform offline transformation, eigenvectors $K$ also need to be fixed for all input.  Thus it may not uniformly transform each X's channel variance?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies the PTQ problems for Mamba,  finding out that current PTQ methods on Transformer-based LLMs failed to quantize Mamba.\n2. MambaQuant applies SmoothQuant's smoothing vector to Mamba's non-linear SiLU and PScan.\n3. Experiments show MambaQuant achieves 1\\% accuracy loss on W8A8.", "weaknesses": "1. This paper does not detail how to get $K$ for different input $X$.\n2.  KLT-enhanced ratation seems unable to solve all input's channel in-variance and outliers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mg89f0qazR", "reviewer": "Reviewer_AGfn", "summary": "The paper explores post-training quantization for Mamba family of state-space models and identifies several key challenges, namely the presence of outliers in gate projections, output projections and the fact that Mamba’s PScan operation further amplifies these outliers.\n\nAuthors proposed MambaQuant, a post-training quantization framework for Mamba model family, consisting of two extensions over Hadamard-based methods: 1) Karhunen-Loève Transformation (KLT) enhanced rotation; and 2) Smooth-fused rotation, which equalizes channel variances and can be merged into model weights.\n\nExperiments show that MambaQuant can quantize both weights and activations into 8-bit with less than 1% accuracy loss for Mamba-based vision and language tasks.", "questions": "* Would be great if authors could include detailed per-task results for LM-eval benchmark in the appendix, to improve reproducibility and make it easier to compare to.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* Authors demonstrate the effectiveness of MambaQuant on both vision and language tasks and different model sizes. Solid performance against the recent PTQ methods including GPTQ, SmoothQuant, QuaRot.\n* Ablation studies and figures to demonstrate that how the proposed method improves the distribution of problematic activations and hence improves their quantizability.\n* Authors intend to release the code.", "weaknesses": "* Paper claims to be the first comprehensive analysis of quantizability of Mamba, however many of the insights including the existence of outliers in gate and output projections were already discussed in “Mamba-PTQ” [1]. Could you elaborate on any additional differences between these two approaches?\n\n* It seems that KLT enhanced rotation is very straightforward and boils down to simply applying an SVD decomposition on weights/activations (or equivalently, eigenvalue decomposition on the covariance matrix), and fusing the emerging matrix K into H. Is there is any additional insight to it?\n\n[1] Pierro et al., \"Mamba-PTQ: Outlier Channels in Recurrent Large Language Models\". ArXiV:2407.12397", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4QQwJooLEo", "reviewer": "Reviewer_kvr1", "summary": "The authors propose a post-training quantization framework, called MambaQuant, which applies to Mamba models. MambaQuant relies on the observation that there are weights and outliers in Mamba models which are significantly larger than the rest, and that this disparity is increased by the parallel scan operation. The authors first study the effectiveness of the Hadamard transformation, which incorporates pre- and post- multiplies the features by Hadamard matrices so as to spread out the largest values, and show that it is ineffective at equalizing variance across channels, which makes quantization performance suffer. Then they discuss the MambaQuant framework which consists of two operations:\n- An offline whitening step, which they call the Karhunen-Loeve transform: first diagonalize the feature covariance matrix at the given layer, then multiply the appropriate Hadamard matrix by the matrix of eigenvectors of the feature covariance. Multiplying the features by this product matrix instead of the Hadamard matrix will spread out the massive values and also ensure that the variance across each channel is balanced.\n- An online smoothing step: do a transformation on the SILU, e.g., $x \\sigma(W x) \\mapsto x \\sigma(s \\odot [(W \\oslash s)x])$ where $\\odot, \\oslash$ are elementwise multiplication/division respectively and $s$ is a smoothing vector, and there exists an algorithm to compute $(W \\oslash s)x$ accurately and efficiently via fusing with multiple operations.", "questions": "- See weaknesses section; more empirical validation would be great in order to demonstrate the generalizability of the MambaQuant operations.\n- Is there any aspect of runtime which is significantly improved or made worse by these changes? E.g. prefill time or average runtime? Same with the memory constraints. It's acknowledged that the appendix contains some estimates for time and space complexity but it would be interesting to see how these bear out in practice.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed toolkit MambaQuant is clearly motivated, and the motivation follows from both theoretical and empirical perspectives; the weaknesses of the prior methodology (that is, smoothing large values via Hadamard transformation) are clearly demonstrated in the context of Mamba.\n- Accordingly, the solutions proposed by MambaQuant are shown to resolve these issues effectively using the same methodology.\n- The shown empirical results demonstrate minimal performance degradation compared to Mamba on both language and vision tasks.\n- The paper conducts several ablations to show that in their experimental setup, all changes significantly improve a property of the model to make quantization easier.", "weaknesses": "- The whitening transformation (KLT) seems to be computationally prohibitive to run. It is true that the paper thus classifies it as an offline transformation. However, it is still not clear whether the eigenvectors of the feature matrix are similar across batches (so that the transform truly whitens the data and equalizes variance across all channels). That would be an interesting insight, if true, but it's not confirmed here, and so the KLT needs more empirical evidence on more diverse data to be validated in real applications.\n- The methods proposed are only studied in the context of, and only apply to, the Mamba family of architectures. This is an explicit limitation of the paper. It would be great if the methods here transferred to different architectures without significant performance downgrades (i.e., they still sort of work).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)