SSDi8: Accurate and Efficient 8-bit Quantization for State Space Duality
Authors: Hyunwoo Kim
Byoungchan Ko
Minseok Kang
Minwoo Kim
Dongjin Lee
Jaehoon Lee
Sungroh Yoon
Dahuin Jung
Venue: ICLR 2026 Poster (2026)
Keywords: Mamba-2
State Space Duality (SSD)
Quantization
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Recent advances in sequence modeling have highlighted Mamba as a state space architecture offering efficient long-range dependency modeling and providing a viable alternative to Transformers. Building upon this, Mamba-2 introduces the Structured State Space Duality (SSD), which integrates recurrent and attention modes to achieve efficiency and scalability. However, this architectural expansion substantially increases memory and latency overhead, underscoring the need for efficient compression strategies tailored to SSD. In this work, we present SSDi8, the first post-training quantization framework specifically designed for SSD to maintain a persistent INT8 path. SSDi8 introduces a reformulation that decouples element-wise multiplications from matrix multiplications, enabling reuse of quantized activations across modules. Moreover, SSDi8 adaptively quantizes channel-varying activations at cost-effective points, further reducing latency. On the accuracy side, SSDi8 explicitly leverages the intrinsic dimensional decomposition of SSD, exploiting distinct outlier distributions across axes, and incorporates an error correction term based on per-channel error statistics. Comprehensive experiments demonstrate that SSDi8 achieves accuracy comparable to FP16 while delivering up to 1.4X speedup in W4A8 and W8A8 settings. We further validate its robustness in resource-constrained environments by deploying it on the Orin NX device. Code is available at https://github.com/cau-hai-lab/SSDi8.

Published as a conference paper at ICLR 2026
SSDI8: ACCURATE AND EFFICIENT 8IBIT QUANTIZATION FOR STATE SPACE DUALITY
Hyunwoo Kim
Byoungchan Koa- Minseok Kang²
Minwoo Kim²
Dongjin Lee
Jaehoon Lee" Sungroh 3.4.5
Dahuin Jung"
DDpaarmment of Artificial Intelligence, Chung-Ang University
School of Computer Science and Engineering, Soongsil University
3 DDpaarment of Electrical and Computer Engineering, Seoul National University
*Interdisciplinary Program in Artificial Intelligence, Scoul National University
AIIS, ASRI, INMC, and ISRC, Seoul National University
ABSTRACT
Recent advances in sequence modeling have highlighted Mamba as a state space architecture offering efficient long-range dependency modeling and providing a viable alternative to Transformers. Building upon this, Mamba-2 introduces the Structured State Space Duality (SSD), which integrates recurrent and attention modes achieve efficiency and scalability. However, this architectural expansion substantially increases memory and latency overhead, underscoring the need for efficient compression strategies tailored SSD. In this work, present SSDi8, the first post-training quantization framework specifically designed for SSD to maintain persistent INT8 path. SSDi8 introduces reformulation that decouples olement-wise multiplications from matrix multiplications, enabling reuse quan- tized activations across modules. Moreover, SSDi8 adaptively quantizes channel- varying activations at cost-effective points, further reducing latency. On the accu- racy side, SSDi8 explicitly leverages the intrinsic dimensional decomposition of SSD, exploiting distinct outlier distributions across axes, and incorporates an error correction term based on per-channel error statisties. Comprehensive experiments demonstrate that SSDi8 achieves accuracy comparable to FP16 while delivering up to 1.4x speedup in W4A8 and W8A8 settings. We further validate its robust- ness in resource-constrainee environments by deploying on the Orin NX device. Code is available at hwwp://twwwoooocccacaaaaaaaaaaaa
INTRODUCTION
Mamba (Gu Dao 2024 recent state space sequence model that builds upon the Structured State Space Model (SSM) (Gu et al. 2020: 2022) to provide efficient long-range -aage dependency model- ing with constant computation and memory usage. While global attention in Transformers (Vaswani et al.) 2017) can enhance performance as model size increases, it also incurs quadratic growth in computation and memory with respect to sequence length, which poses substantial challenges for large-scale training and deployment. In contrast, Mamba achieves performance comparable to ceeding state-of-the-art architectures across billion-scale language models, positioning it as a strong candidate for next cext-generation sequence modeling.
Despite its algorithmic efficiency, Mamba faces practical limitations: its specialized state space re- currence is difficult to parallelize modern accelerators, making it less handware-friendee than optimized Transformer kernels, and it shows relatively diminishing efficiency when scaled to larger parameter sizes. To overcome these issues, Mamba-2 (Dao & Gu 2024) introduces the Structured State Space Duality (SSD), a hybrid design that integrates recurrent mode with attention mode. Mamba-2 adds head dimension analogous multi-head attention to enhance scalability and em- ploys a dual representation that improves general matrix multiplication (GEMM) utilization,, yield- ing higher throughput on GPUs and TPUs. While the original Mamba exhibited limited efficiency
"Equal Contribution
1 "orresponning Authors

---

Published as a conference paper at ICLR 2026
beyond 2.7B parameters, Mamba-2 scales effectively to over 8B parameters and achieves competi- tive performance across language, audio (Lee et al. 2025), vision (Shi et al.) 2024), and multimodal tasks (Huang al. 2024). Yet this expansion also intensifies memory and latency overhead, high- lighting the need for efficient compression and optimization.
The recurrent mode of SSD is computationally efficient but system-inefficient,, while the attention mode is relatively computationally demanding. During its operation, SSD repeatedly invokes acti- vations across modules and performs sequential updates. In this process, activations reuse across modules necessitates frequent DRAM accesses, and the intrinsically higher latency of DRAM intro- duces considerable overhead.
As shown in Tab. I directly applying quantization methods as Hadamard
Table I: Accuracy under major layer quantization of Mamba-2. Significant degradation arises when SSD is quan- tized per-tensor.
originally designed for Transformers- rotation or GPTQ-to SSD layers leads to substantial ac- curacy degradation. This stems from the distinctive com- putational organization of SSD. First, the model dimen- sion is partitioned into the number of heads and the per- head dimension, each following markedly different statis- tical distributions; failure to account for this property re- sults in significant performance loss. Second, SSD contains dimension-varyl varying activations whose shapes differ between memory storage and computation, and these activations are repeatedly invoked across multiple modules. Third, element-wise multiplications are extensively in- tertwined with matrix multiplications, further complicating quantization. In this work, we conduct the first comprehensive analysis of SSD to maintain a persistent INT8 path, providing observations that reveal the internal factors contributing to its quantization sensitivity.
Quantized Layer(s)
Model
Bitwidth
ACC
FP16
63.8%
2.7B
In Proj
63.6%
W448
+ SSD
58.4%
+ Out Pruj
54.6%
Accordingly, we propose SSDi8, an accurate and efficient post-training quantization framework that reduces both inference latency and performance degradation within SSD. For latency reduc- tion, SSDi8 quantizes -hannel-variant and recurrent activations at optimal points and reuses them, ensuring an uninterrupted INT8 execution path from input to output. Furthermore, we address element-wise operations that disrupt this path by introducing a sparse-aware reformulation, with the guarantee formally established through mathematical analysis. This design keeps the execution in INT8 while substantially alleviating memory bottlenecks and computational overhead. For accuracy, SSDi8 leverages the intrinsic dimensional structure and properties of SSD. Specifically, external di- mensions entering SSD decomposed into two axes, each exhibiting distinet outlier distributions, which are explicitly exploited to reduce quantization error. Furthermore, we introduce an error cor- rection term based per-channel error means, yielding consistent gains in accuracy. Through these mechanisms, SSDi8 achieves a balanced optimization of both efficiency and performance.
SSDi8 achieves accuracy comparable FP16 while enabling up to 1.4x inference speedup under both W4A8 and W8A8 configurations, while excluding W4A4 due to hardware-inducce slowdowns as discussed in Lin al. Notably. in the context of SSD where error sensitivity often causes se- vere degradation our method incurs negligible accuracy loss while delivering substantial latency reductions, with single-inference speedups reaching 1 1.5x. To the best of our knowledge, this repre- sents the first successful application of persistent INT8 path within the Mamba-2 SSD) architecture. Furthermore, we demonstrate that SSDi8 maintains efficiency in resource-constrained environments through deployment on the Orin NX device.
2 RELATED WORK
Mamba Architecture. Mamba is a sequence modeling architecture built on SSMs, which has been explored as an alternative to Transformers in order to circumvent the quadratic complexity of self- attention (Gu & Dao 2024). Unlike conventional linear SSMs (Gu et al. 2022 Smith et al.) (0023), Mamba incorporates a selective state space mechanism that adaptively gates input-dependent state transitions and output projections, enabling more expressive sequence modeling. Mamba-2 extends this framework by introducing structured SSDs (Dao & Gul 2024), which establishes a formal equivalence between SSMs and linear attention and enables optimized GEMM-baseh implementa- tions. This design substantially improves hardware utilization on modern accelerators. Furthermore, Mamba-2 allows the state
constrained N 16 in Mamba-1 1- to scale sta-
2

---

Published as a conference paper at ICLR 2026
bly to N = 64-128 and beyond. In addition, Mamba-2 integrates multi-head structure analogous to multi-head attention, further enhancing scalability. These advances make large-scale parameter expansion feasible, but they also intensify memory and latency overhead, motivating the need for compression and deployment strategies.
Quantization for Mamba Models. Recently, several studies have begun explore quantization for the Mamba models (Tang et al.] 2024 Yu et al.] 2025). MambaQuant (Xu et al.) 2025) and Quambal (Chiang al. 2025b) introduced Post-Training Quantization (PTQ) methods targeting the original Mamba-1 architecture, but their approaches are not directly applicable to SSD-based Mamba-2. Quamba2 (Chiang al. 2025a) extended quantization and W8A8 settings that include SSD blocks. However, its method is limited to the inputs of SSD layers and does not adequately address precision issues within internal SSD computations, leaving the INT8 execution path incomplete and constraining latency optimization.
Mamba-2, applying W4A8
3 BACKGROUND
3.1 QUANTIZATION
Quantization discretizes continuous values into finite set of integer levels. particular, uniform quantization divides the value range into equal intervals, mapping each element of a tensor X to its nearest quantized level as follows:
X ax
max (I
x round
2 b-1
where X is the quantized tensor, ox is the scaling factor that defines the step size based on the maximum absolute value of X, and the bit-width.
3.2 MAMBA-1
Mamba an architecture built upon State Space Models (SSMs), composed solely of activation operations, where hidden state variable is employed efficiently compress and propagate mem- ory (Gu & Dao 2024). The fundamental state update and output equations are defined as follows:
11(()= Ah(() + Bx((), y(1) Ch((t.
Eq. 2 builds on the theoretical foundations of HiPPO (Gu et al., (020)) and S4 (Gu et al., 2022), which substantially improve both performance and efficiency. However, since SSMs are defined in continuous time, applying them to discrete inputs requires discretization. In practice, Zero-Order Hold is used to preserve previous values, and a time-step activation A introduced to discretize matrices A and B. These operations performed independently along the channel dimension of the input æ, so that each channel independently follows its own SSM formulation:
Sn(x) rWB, B, Sc(x) zWc, Sa(x) xWa-
(3)
Through input-dependent activations, Mamba highlights important information while suppressing noise, improving long-range dependency modeling.
METHODOLOGY
The overall workflow of SSDI8 is illustrated in Fig. [ A substantial portion of SSD modules is executed along the persistent INT8 representation path, reusing quantized activations and applying sparse-aware reformulation to element-wise operations that disrupt this path. The output tensor dAcs from ChunkCums um is negligible size compared to other tensors, yet its recovery after quantization is challenging due to the element-wise multiplication; hence. it is retained in FP16. In the same vein, ChunkScan2 remains in FP16 for analogous reasons. These choices are further elaborated within this section.
3

---

Published as a conference paper at ICLR 2026
Persistent INT8 Representation
Pro
Reformulation
Chunk Scate
State
State Passing
State
Chunk
Scan(1)
RMiSNoom
SSD
A
X
18 C
Quantize
Chunk Cumsum
Chunk Scan(2)
Quantizee
CORV
Correction
CB
Chunk
RMMS
BMM
(a): :amba-2
(b): SSDi8
Figure I: (a) Mamba-2 block architecture. (b) SSD pipeline in SSDi8. SSDi8 enables the persistent INT8 representation path through reformulation and quantized activation reuse, while mitigating performance degradation via channel-award quantization and mean correction.
4.1 PRELIMINARY STUDY: MAMBA-2'S STRUCTURED STATE SPACE DUALITY
The Structured State Space Duality (SSD) in Mamba-2 consists solely of activation operations and unifies the recurrent and attention modes, thereby reducing computational cost and improving effi- ciency over the recurrence- -omeennnneeddinteeddddeee operations of conventional SSMs. Concretely, the SSM com- putation can be expressed as a lower-triangee structured matrix: the diagonal block, which directly influences the output, is computed via the attention formulation using matrix multiplications, while the off-diagonal blocks, which require recurrence, are computed by leveraging the semiseparable property, which admits low-rank factorizations.
key distinction from Mamba is that Mamba-2 introduces a number of heads H, analogous to the multi-head structure in Transformers. As shown in Fig. 2 the value of H is formally defined by D HOP, where D denotes the model dimension and P the head dimension. Notably, H and P remain independent axes, with chosen to be much larger than For efficiency, the input-dependent B and C are parameterized with an auxiliary dimension G, and broadcast to H when required.
Formally, the input activations of SSD and its dimension before discretization are given as follows: R(D)
A
A E (B,
X
E (.B.L,
B E R(BLGG,,)
C E (R Y E R(B.L,
where B denotes the batch size, L the sequence length, H the number of heads, G the number of groups, the head dimension, the state dimension, and the final output of SSD. To shorten the effective recurrent path and enable parallelism, the sequence is partitioned as L = c C 1, where the number of chunks and the chunk size. The computation then proceeds through five mod- ules Coss-ChuuCCuu.sui ChunkState, StatePass ing. Chunk3MM, and ChunkScan which together yield the SSD output Y. Additional details are provided in Appendix
ChunkCumsum (Input (A, A) Output (A, d). ChunkCumsum applies a softplus trans- formation A, a time-step dependent scaling factor introduced for discretization, and discretizes the decay activation A that governs recurrent dynamics. It then prepares the cumulative decay term d.Acs, which is subsequently consumed by downstream modules for state updates.
ChunkState (Input (dAcs, A, B, X) Output (State)). The ChunkState module dis- cretizes the projection matrix B, applies the decay factor, and multiplies with the input X to gener- ate the hidden state. The cumulative decay is computed Decay simplicity, we denote @ Decay state by LUT where @ denotes element-wise multiplication, in the following modules. The resulting state update is formulated
exp /dA.fimall -ocaaldda ddccs ). For
State = X x C LUT
StatePassing (Input (State, dA_) + Output (State)). This module integrates the states com- puted from independent chunks into the actual recurrent state through decay. The decay term is given
4

---

Published as a conference paper at ICLR 2026
Activation X outside SSD
Activation X inside SSD
8000
127
Separable patterns
XL
6000
0.25
appear alonng H-dim
/ 64
D 4000
0.20
0.25
2000
0.15
2000 4000 6000 8000
0:3
0
0.15
0,20
0.00
D
0.00
0
500
1000
1500
2000
20 40
60
P
Figure 2: Visualization of activation X the 16th block of Mamba-2 8B before and after the SSD in- put transformation. The pre-SSD dimension (B, I,, D) exhibits no clear token-wise pattern, whereas the transformed dimension (B, L, H, P) within SSD reveals distinct patterns along the H-dim.
by
Docay pass = cxp(dA fiaa)
(5)
and the recurrent update is performed over the chunk as
State, E States;1 <- State;+1 +
Decay @ State;,
i == 0......., 2. (6)
ChunkBMM (Input (B.C) + Output (CB)). ChunkBMM performs a batched matrix multiplica- tion between C and B. This operation extracts the diagonal blocks of the product, yielding C.B, which is used in the output computation within SSD.
Output (oufur ChunkScan1 computes the off-diagonal interaction term by performing a matrix multiplication between the recurrent state State and the projection matrix C. The decay contribution is modeled Decay, scant exp(dAcs). and combined with A to form LUT (= A C Decayscau1) The final off- off-diagonal output is obtained as ouforr-ding (State x C) @ scanl
ChunkScan1 (Input (State, C, d, A)
ChunkScan2 (Input (X, CB, dAes: A Output (outdigg) )). ChunkScan2 computes the diag- onal contribution by projecting the input representation X with the combined activation C.B, while modulating the result using the decay and discretization terms (dAes, A). This module complements the off-diagonal pathway from ChunkScanl, and together they form the complete output of SSD: outoff-diag + oufdiag-
4.2 SSD18
Quantization of B,C. Within SSDi8, the handling of the channel-dependent activations B and C constitutes one of the strategies, since they are repeatedly invoked across three SSD submodules. Rather than quantizing them separately within each module, SSDi8 quantizes once and reuses the resulting INT8 tensors, thereby reducing memory traffic and enabling a consistent low-precision execution path. challenge arises because B and C are defined along the group dimension but broadcast to the head dimension H during computation, with typically an order of magnitude larger than G. Naively applying quantization after broadcasting induces significant overhead (up to 4x), which SSDi8 addresses by optimizing the placement of quantization operations.
To minimize redundant overhead, SSDi8 performs an early quantization of the channel-varying ac- tivations B and C' once along the group axis G at the beginning of cach SSD layer. The resulting INT8 tensors are then reused across all downstream modules, maintaining a consistent bow-bitwidth representation without repeated quantization. Since [G] H., quantization along G is considerably more efficient, adding only about 3% to the total SSD latency. Moreover, as shown in Figs. 2 and the head dimension exhibits highly heterogeneous value distributions across heads up to 5x variation-making direct per-head quantization unstable, Similarly, the group dimension G shows distinct characteristics and must be considered in quantization. While the state dimension exhibits relatively consistent statistics, it directly participates in subsequent matrix multiplications, where quantization errors cannot restored. Thus, excluded from the quantization axes.
Sparse-aware Reformulation. As defined in Eq. 4 the ChunkState computation applies B O LUTamee prior to the matrix multiplication with X E (R, Here, LUT E a c, is multiplied element-wise with B H,, H, 1, to impose a decay pattern across the steps within
each H, and The resulting B LUT state is then multiplied with X along the 1-axis project
5

---

Published as a conference paper at ICLR 2026
X
X Scooel
State
10th head
0000
63
0,33
33
20
10th head chank
High Sparsity
State
Sealed
LUT
130
uaa
1
20th head
1931
15th chunk
(a)X with Sparse - Aware Reformulation
(b) State per head
Figure 3: (a) Distribution plots of head-wise X and LUTT in the 27th block of the ChunkState module, and their element- wise product after reformulation Xscled- The channel-wise (P-dim) dis- tribution of x is highly sparse. (b) Head-wise distribution plots of State.
the 1 sequence steps into N. The operations are executed independently and in parallel across B, H, and c. However, this ordering introduces three critical limitations: (i) although is quantized to INT8, the presence of LUT in FP16 enforces floating-pols pootin execution path, undermining the efficiency of INT8 GEMM; (ii) because LUT state exhibits exponential variation along the chunk axis 1. any quantization scheme other than per-/ quantization introduces substantial error, while even per- quantization is infeasible due to quantization eπrоr accumulation after the /-axis matrix multiplication; (iii) attempting A Q(B @ sUT requires quantization after the G H expansion, which incurs significant overhead. To enable a fully INT8 execution path, SSDi8 reformulates the computation
StateInT32 (Xaacalo) x Q(B), X LU'T X, StateInT32 (B (7)
where Q(-) denotes quantization. This reformulation is valid because LUT applies its multi- plication along the I-dimension shared by both X and B, while all other dimensions operate in- dependently. This property ensures that moving the scaling operation from B to X preserves the computational result, and quantizing the resulting Xscaled mitigates the limitations. In this case, ((Xeaale is quantized along the (P. H) axes because LU'T'stae broadcast along the P axis while X preserves consistency across and per-(H) ) heterogeneity as shown in Fig. Fig.3 3aa and Fig. 2 Quan- tization simulations show that X exhibits pronounced outliers along the channel axis, which makes accurate quantization challenging. However, the actual quantization error of ((X does not significantly increase despite the presence of such outliers. From a distributional perspective, this robustness can be attributed to the high sparsity of X as shown in Fig. 3 (a), which leads reduced quantization errors overall. To formally validate this property, we prove in Appendix A that, under mild conditions, the quantization error of X is smaller than that of (XX C LUT, state- This sparsity- aware proof justifies the proposed reformulation, and empirical results further confirm that the resulting performance degradation remains negligible.
Persistent INT8 Representation Recurrent States. StatcıN132 obtained from the proposed re- formulation is accumulated in INT32. Since INT32 consumes twice the memory of FP16, SSDi8 reduces memory traffic by directly converting INT32 to INT8 in registers with quantization scales:
S2
StateiNTs
Round StateInT32
Amax
85
where 8z, Sb; 8, denote the quantization scales of X, B, and State, respectively. The resulting INT8 tensor then stored in DRAM, avoiding intermediate FP16 representations and thereby reducing memory bandwidth usage. State also exhibits variation across heads Н. As shown in Fig. 3 (b), consistency observed along both the P and N, since N participates subsequent multiplications within ChunkScan1, quantization along N is not adopted. StatcINts is thus quantized per-(H, P).
In the StatePassing module, independently computed chunkwise states are recurrently accu- mulated with decay to form the actual state, shown in Eq. 6 Since State is already in INT8, maintaining the INTS execution path requires quantizing the FP16 Decay. The computation pro- ceeds independently along 3, H and recurrently along c, where each Decay is scalar. This enables

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation of Mamba-2 (1.3B, 2.7B, 8B) with three quantization methods (Quamba, Quamba2, and SSDi8) on six zero-shot tasks (LA, HS, PIQA, Arc-E, Arc-C, WG).
Model
Size
Methoos
Bitwidth
LA
HS
PIQA
Arc-E
Arc-C
WG
Avg.
FP16
65.6%
59.9%
73.3%
64.1%
33.3%
60.8%
59.5%
Quamba
W8A8
49.8%
58.5%
71.2%
61.9%
32.1%
58.1%
55.2%
W8A8
62.0%
59.2%
72.5%
63.4%
32.7%
60.0%
58.3%
1.3B
Quamba2
W4A8
61.0%
58.8%
72.4%
62.7%
32.6%
59.1%
57.7%
W8A8
64.7%
59.7 %
72.7 %
64.0%
32.8%
60.9%
59.1 %
SSDi8 (Ours)
W4A8
63.6%
59.2%
72.7% %
63.5% %
33.5 %
60.4%
58.8%
FP16
69.5%
66.6%
76.4%
69.5%
36,4%
64.2%
63.8%
Quamba
W8A8
52.4%
60.4%
71.6%
62.9%
33.7%
58.0%
56.5%
Mamba-2
2.7B
Quamba2
W8A8
W4A8
66.1%
65.6%
65.1%
65.5%
74.7%
74.4%
68.1%
68.45%
37.1 %
36.1%
63.7%
62.8%
62.5%
62.1%
W8A8
68.3%
66.2%
75.6%
69.0%
36.8%
63.4%
63.2%
SSDi8 (Ours)
W4A8
67.4%
65.3%
75.6 %
68.9%
35.2%
63.55 %
62.6%
FP16
70.9%
77.7%
79.7%
76.0%
48.0%
72.0%
70.7%
Quamba
W8A8
54.0%
74.6%
77.1%
73.5%
44.2%
65.5%
64.8%
W8A8
69.8%
77.8%
79.1%
75.9%
46.9%
69.0%
69.8%
8B
Quamba2
W4A8
68.8%
77.1%
79.1%
75.0%
46.0%
68.7%
69.1%
SSDi8 (Ours)
W8A8
70.4%
77.2%
79.6%
75.5%
47.2%
71.2%
70.2%
W4A8
69.9 %
76.5%
79.1 %
75.4%
46.2 %
70.6%
69.6 %
element-wise fixed-point quantization of Decay. Formally,
Q(Stater+1) + Q(State:+1) + Q(Decay:+1) C Q(State;),
1= 1........,--- 2,
where S is a gating constant chosen as 2 to enable bit-shift operations for minimal latency (with k 7 in experiments). Per-H, P quantization ensures that all StateINTs across с share common scale. This allows recurrent updates to be performed by simple bit-shift operations. As result, StateiNTs can be persisted through ChunkScan1, enabling INT8 Tensor Core multiplications with CINTS. Here, Decay E H,, aligns with the output outotf-diag IR (R so element-wise multiplication is applied directly after the matrix multiplication.
Quantization of ChunkBMM and ChunkScan2. As shown in Fig. I the quantized activations BINTS and CINTS are reused in the ChunkEMM module. Because both are defined on the group dimension G. the multiplication proceeds without conversion to the head dimension H. producing CBINT322- The tensor CB ((,, is larger than X, so its quantization yields substantial memory savings. Similar to ChunkState, single INT32 + INT8 step is applied to minimize memory traffic. In ChunkScan2, (LUT scan2 @ (CB)) x X involves X in FP16, enforeing a floating-poin path. Due to its shape, LUT is element wise multiplied with CB, making post- quantization recovery difficult and rendering reformulation infeasible due to a shape mismatch with X. The dequantization scale of CB is fused into LUT reducing overhead while allowing partial FP16 execution. Experiments demonstrate that this process alone yields substantial latency gains.
Leveraging the persistent INT8 representation of recurrent states together with the sparse-aware re- formulation and reuse of activation, SSDi8 achieves up to 1.38x speedup overall, with gains reach- ing 1.6x in the ChunkScan module compared to FP16 execution.
Mean Correction for SSD Quantization Error. To further mitigate the accumulation of quanti- zation errors across SSD layers, introduce per-channel mean correction strategy. Given full- precision and dequantized results XW = Y e R N." and X'W' = Y' E N.P." minimizing the error between Y and Y' can be formulated as a least-squares problem, and the optimal correction vector e* is given in closed form as the channel-wise mean of the quantization error:
P N
N
Eg = Y Y (Y' + +c ΣΣ (y---Y') -p)) E =
Σ YY-Y')
(10)
p=1==
To ensure accurate estimation, we adopt a layer-wise sequential update strategy, enabling subse- quent layers to reflect the applied corrections and, thereby, capture activation shifts induced by
7

---

Published as a conference paper at ICLR 2026
FP
Quambe2
SSD/8
Lateney (ma)
20
15
X1.06
Latency (mne)
16
Latency (1m4)
20
ChunkBMM
ChumkCCumeom
Guanlizese
10
10
10
KLL
ChunkSoan
Stute
PIR
0
ChunkState
Batch
8
16
Length
256
512
1024
2048
FP Quasmbe22 S5Ds8
(a) Latency vs. Batch size
(b) Latency Sequence length
(c) (c) wise latency breakdown to SSD
Figure 4: SSD latency of quantization methods on Mamba-2 2.7B: (a) varying batch (L. 2048), (b)
varying length (B = 32), and (c) module-wise latency (B = 32, L 2048). PIR denotes Persistent
INT8 Representation. SSDi8 achieves up 1.47x overall speedup and 1.77x in the State path.
earlier updates. For a detailed description of the sequential update algorithm, please refer to Al- gorithm Appendix B To minimize overhead, e applied only to the output projection layer, whose dimensionality is half that of the input projection layer and where quantization error is most pronounced. This design achieves consistent accuracy gains while incurring only marginal latency overhead (<= 1-2%).
EXPERIMENTS
Experimental Setup. We conduct PTQ experiments on Mamba-2 (Dao & Gu 2024) models with 1.3B, 2.7B, and 8B parameters. Experiments are primarily conducted on NVIDIA A5000 GPUs. We evaluate zero-shot performance on LAMBADA (Paperno et al. 2016), WinoGrande (Sakaguchi al. 2020), PIQA (Bisk et al., 2020), HellaSwag (Zellers et al. 2019), ARC-Easy, and ARC- Challenge (Clark et al.) (018)) benchmarks, and additionally assess language modeling capability via WikiText2 perplexity. Results are compared against the FP16 baseline, Quamba (Chiang et al.) 2025b) and Quamba2 (Chiang et al. 2025a). and the HAD (HadMamba2)) bascline, where HAD ap- plies the Hadamard rotation to the Mamba-2 projection layers (Chiang et al. 2025a). GPTQ weight quantization and RTN quantization of SSD inputs.
Quantization Setup. We use symmetric, static quantization on both W8A8 and W4A8 configu- rations. For 4-bit weight quantization, employ GPTQ (Frantar et al. 2023), combined with Hadamard--ranssormeed (Ashkboos et al., 2024) projection layers. To handle RMSNorm-induced outliers, migrate the 7 parameter (Wei al. 2022), and apply mean correction with factor of 0.15 to prevent estimation overfitting.
5.1 EVALUATION OF ZERO-SHIOT AND GENERALIZATION PERFORMANCE
Tab. reports zero-shot task performance of Mamba-2 models (1.3B, 2.7B, 8B) un- der FP16, Quamba, Quamba2, and SSDi8 quantization. Average accuracy is computed over six benchmarks. Across all bit-widths (W8A8, W4A8) and model scales, SSDi8 consistently outperforms Quamba2. For example, model with W4A8, SSDi8 improves over Quamba2 (62.7% vs. 62.1%). and on the 8B model with W8A8, achieves 70.2% compared to 69.8%. These results underscore the robustness of SSDi8 across diverse configurations. Full comparisons, including HadMamba-2 and Quamba2 with W4A16, are provided in Appendix
Table 3: Wikitext2 perplexity with L = 2048.
Wikitext2 Perplexity ())
Methods
Bitwidth
1.3B
2.7B
8B
FP16
10.42
9.06
7.25
HAD
W4A8
11.63
11.85
8.79
the 2.7B
Quambu2
W8A8
10.80
9.32
7.79
W4A8
11.08
9.54
7.94
W3A8
SSDi8 (Ours)
W4A8
10.92
9.43
7.62
Perplexity Results. To assess linguistic fluency and generalization. we report WikiText2 perplexity in Tab. 3 Across all model scales and bit-widths, SSDi8 consistently achieves lower perplexity than Quamba2 while narrowing the gap to FP16. In particular, for the 8B model. SSDi8 yields reductions of 3.9% (7.49 vs. 7.79) under W8A8 and 4.0% (7.62 vs. 7.94) under W4A8. These results demonstrate that SSDi8 preserves linguistic fluency and generalization under quantization.
8

---

Published as a conference paper at ICLR 2026
5.2 LATENCY AND MODEL SIZE
In Fig. 4 (a) and (b), compare SSDi8 with FP16 and Quamba2
on NVIDIA A5000 (24GB) across varying batch sizes (B & 32) and sequence lengths (T. < 2048). Lateney is measured in milliseconds as the average of 100 runs after warm-up. On Mamba-2 2.7B with B 32, L 2048, SSDi8 achieves 1.47x speedup over FP16 and 1.38x x improvement over Quamba2. The benefit increases with larger batch sizes and longer sequences, where greater chunk-level parallelism amplifies throughput, while short sequences (e.g., L 256) may show higher FP16 efficiency due to lower computational intensity. Fig. (c) reports module-level latency breakdown for 2.7B at B 32, I. 2048. With persistent INT8 representation, ChunkScan achieves up to 1.77x speedup over FPI6 and 1.50x over Quamba2, while StatePassing yields 2.25x and 2.17x improvements, respectively. As demonstrated in Fig. similar gains are observed under W8A8, and results Mamba-2 8B provided in Appendix
PP
Quandba
SSDI
90
Latency (me)
60
Bulch 16
32
Figure 5: Forward la-
tency of W8A8 (I.
2048) on 2.7B.
To further assess deployability under resource- constrained conditions, evaluate SSDi8 the NVIDIA Orin NX 16G, as shown Tab. 4 Using the Mamba-2 2.7B model, measure SSI) latency across varying sequence lengths with batch size of 16, comparing W4A8 and W8A8 quantization against Quamba2. Across all configurations, SSDi8 consistently outperforms Quamba2, demonstrating its robustness beyond high-scale accelerators.
Table 4: SSD latency (ms) of SSDi8 vs. Quamba2 Orin NX 16G.
GPU
Orin
Bitwidth
W4A8
W8A8
Method
Quumba2
SSDi8
Quambu2
SSDi8
256
55,30
44.71
51.03
41.30
512
76.10
68.00
70.95
60.49
L 1024
134.40
127.51
139.10
114.36
L 2048
262.90
240.54
249.29
217.69
Additional results evaluating longer sequence
lengths and larger batch sizes are reported in Appendix H and Appendix K respectively.
5.3 ABLATION STUDIES
present ablation results on Mamba-2 2.7B. The baseline retains FP16 only within SSD while applying W4A8 elsewhere. Compar- ing Q(X) with the pro- posed reformulated Q(X o LUTame) shows negligible quantization error, consis- tent with theoretical proof. Avoiding clement-wise multiplications after head expansion of B yields measurable latency gains. Without reformulation, quantizing X alone prevents the use of the persistent INT8 path, and the final latency improvement from quantizing B, C, and CB limited to 1.07 x. By contrast, our reformulation enables INTS execution in ChunkScanl, improving latency by 1.08 x, and further quantization of ChunkBMM achieves a 1.32x speedup. Perplexity degradation remains below 0.1, indicating that our channel-aware quantization preserves accuracy. Further results provided in Appendix
Table 5: Ablation results for internal SSD quantization (Q(SSD)).
ChunkState Q(X)
Sparse Reform.
Persistent INT8
Quant. ChunkBMM
width
of B,C
Latency
PPL
8.63
9.34
8.58
9.35
8.05
9.37
W4A8
7.60
9.39
8.35
9.36
8.00
9.42
6.53
9.43
We perform an ablation study on SSD quantization and mean cor- rection using the Lambada dataset, which exhibits minimal perfor- mance variance, and report Tab. 6 On Mamba-2 2.7B under the W4A8 setting, HadMamba quantization yields only 51.2% accu- racy, whereas applying SSD quantization substantially boosts per- formance to 67.2%. Incorporating mean correction provides an ad- ditional improvement to 67. 17.4% achieving consistent accuracy gains with only 1-22% overhead. These results demonstrate that SSDi8 achieves both accuracy and efficiency, while mean correction offers effective correction with negligible additional latency.
Table 6: Ablation results of SSDi8: Q(SSD) and correc- tion c.
Bitwidth
QSSSD) Correct
SSDi8
Acc.
FP16
69.5%
$1.2%
W4A8
67.2%
67.4%

---

Published as a conference paper at ICLR 2026
Table Hybrid Mamba-Transforme results on Nemotron-H-SBP ----rroon-------eaonnnnnnnn SSDi8 applied only to the SSD path, while other modules remain in FP16.
Method
Wino
PIQA
ARC-C
ARC-E
Hclla
Lamb
Avg.
PPL
SSD Latency (ms)
Fwd. Latency (ms)
FP16
73.8
80.9
55.8
81.4
80.6
66.2
73.1
8.42
19.834 9.156
109.873
INT8
73.5
80.7
55.9
81.5
80.6
66.3
8.65
98.904
5.4
RESULTS ON HYBRID MODEL:: NEMOTRON-H--B-RRANE REASONNNG
We report results for applying SSDi8 to the Nemotron-H-Be H-BB ----r-nsn---B- Reasonins model in Tab. 7 which adopts Tranf-Trrnsformer hybrid architecture. In this setting, INT8 quantization is applied ex- clusively to the SSD path, while all other components are kept in FP16, allowing us to isolate the effect of SSD-path quantization within the hybrid architecture. Tab. Taa..innoe Iincludes zero-shot benchmark accuracies and average accuracy, perplexity, as well as latency, enabling a joint assessment of modeling performance and computational efficiency.
latency and end-to-end forward
Applying INT8 quantization only the SSD path results in minimal changes in zero-shot perfor- mance across Wino, PIQA, ARC-C/E, Hella, and Lamb compared to FP16. The average accuracy decreases slightly from 73.1% (FP16) to 73.0% (INT8), and perplexity exhibits a limited increase from 8.42 to 8.65. In contrast, latency-related metries show clearer differences. The average SSD- module latency reduced from 19.834 ms to 9.156 ms, corresponding to approximately 2x re- duction, and the overall forward latency decreases from 109.873 ms to 98,904 ms. These results quantitatively illustrate the contribution of the SSD path overall inference time and the efficiency gains obtained by INT8 quantization of this component. In this experiment, quantization is restricted to the SSD path, while MI.P and attention modules remain in FP16; extending quantization beyond the SSD path is left for future investigation.
CONCLUSION
In this work, we presented SSDi8, an INT8 quantization framework developed in the context of the SSD of Mamba-2. Unlike prior approaches limited to projections or partial SSD operations, SSDi8 establishes persistent INT8 representations through activation reuse and a sparse-enhanced refor- mulation. It further explores optimal quantization strategies by analyzing internal activations and incorporates mean correction to compensate for accumulated errors, enabling accurate and efficient inference for large-scale Mamba-2 models. SSDi8 achieves FP16-level accuracy while delivering up to 1.47x speedup over FP16 and 1.38 x over Quamba2, and further demonstrates superior efficiency on edge devices such as NVIDIA Orin NX, as well as across diverse batch-sequence settings. SSDi8 provides mathematical intuition for sparse-tensor quantization and offers guidance for quantization in environments where element-wise and recurrent operations are prevalent.
ACKNOWLEDGMENTS
This work was supported by the Institute of Information & Communications Technology Plan- ning & Evaluation (IIP)) grants funded by the Korea government (MSIT) [RS-2021 11221-111211411111 Artificial Intelligence Graduate School Program (Chung-Ang University): RSS-222-II211 1134333,, Ar- tificial Intelligence Graduate School Program (Seoul National University)], the National Research Foundation of Korea (NRF) grants funded by the Korea government (MSIT) 20550555-0055944: 2022R 1 A3B 1077700), the Al Computing Infrastructure Enhancement (GPU Rental Support) User Support Program funded by the Ministry of Science and ICT (MSIT), Republic of Korea (RQT-25- 090040), and the BK21 FOUR program (Education and Research Program for Future ICT Pioneers) Seoul National University in 2025.
REFERENCES
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alis- tarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems (NeurIPS), 2024.
10

---

Published as a conference paper at ICLR 2026
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourty AAAI Conference on Artificial Intelligence (AAAI), 2020.
Hung- Yueh Chiang, Chi-Chih Chang, Natalia Frumkin, Kai-Chiang Wu, Mohamed Abdelfattah,, and Diana Marculescu. Quamba2: A robust and scalable post-training quantization framework for selective state space models. In Forty-second International Conference on Machine Learning, 2025a.
Hung- Yuch Chiang, Chi-Chih Chang, Natalia Frumkin, Kai-Chiang Wu, and Diana Marculescu. Quamba: A post-training quantization recipe for selective state space models. In International Conference on Learning Representations (ICLR), 2025b.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and
Oyvind Tafjord. Think you have solved question answering? try are, the A12 reasoning challenge. CoRR, 2018.
Tri Dao. Causal depthwise convld in cuda with a pytorch interface, 2024a.
URL https:/ //
gi github. github.ccm/Deateeee catp:///aao-AAaaaccaaaaaaaaaaaaa
Tri Dao. Fast hadamard transform cuda, with a pytorch interface, 2024b.
URL https:
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In International Conference Machine Learning, pp. 10041- 10071. PMLR, 2024.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: quantization for generative pre-trained transformers. International Conference on Learning Rep- resentations (ICLR), 2023.
Accurate post-training
Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-power computer vision, pp. 291-326. Chapman and Hall/CRC, 2022.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First net formmm?
Conference on Language Modeling, 2024. URL
VVY2
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations, 2022. URL https:
//openzeview.nee net.e/ewweetteeeeeee www.:nnee/ooommmmmmmmmme attp://ooormmmmaaaaaaoaollll.
Wenjun Huang, Jiakai Pan, Jiahao Tang, Yanyu Ding, Yifei Xing, Yuhe Wang, Zhengzhuo Wang, and Jianguo Hu. Ml-mamba: Efficient multi-moda large language model utilizing mamba-2 arXiv preprini arXiv:2407. 19832, 2024.
Taehan Lee, Jaehan Jung, and Hyukjun Lee. Mamba-2 audio captioning: design space exploration and analysis, 2025. URL http::htttsttttttttl wttpxvv..rr//bb//555.... 15680
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awan aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems (MLSYS), 2024.
Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient Ilm serving. Eighth Conference Machine Learning and Systems.

---

Published as a conference paper at ICLR 2026
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Lim quantization with learned rotations. In The Thirteenth International Conference on Learning Representations.
Bruce Lee LY. Cuda hgemm, 2024a. URL httpps//gtthbbb.mm////eeeeeeeeee hgemm
Bruce Lee LY. Cuda hgemv, 2024b. hgemv
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernard,, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, (ACL). The Association for Computer Linguis- tics, 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. In The Thirty-Fourd AAAI Conference on Artificial Intelligence (AAAI), 2020.
Yuheng Shi, Minjing Dong. Mingjia Li, and Chang Xu. Vssd: Vision mamba with non-casual state space duality. CoRR, 2024.
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In International Conference on Learning Representations (ICI.), 2023.
Shengkun Tang, Liqun Ma, Haonan Li, Mingjie Sun, and Zhiqiang Shen. Bi-mamba: Towards accurate b-bit state space models. arXiv preprint 1rXiv:2411.11844 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 1397,,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Feng- wei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer lan- guage models. Advances in Neural Information Processing Systems, 35:1 7402-1744, 2022.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning (ICML), 2023.
Zukang Xu, Yuxuan Yue, Xing Hu, Zhihang Yuan, Zixu Jiang, Zhixuan Chen, Jiangyong Yu, Chen Xu, Sifan Zhou, and Dawei Yang. Mambaquant: Quantizing the mamba family with variance aligned rotation methods. In International Conference on Learning Representations (ICLR), 2025.
Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo, and Yusuke Iwasawa. Slender-mamba: Fully quan- tized mamba in 1.58 bits from head to toe. In Proceedings of the 31st International Conference on Computational Linguistics (COLING), 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Pro- ceedings of the 57th Conference of the Association for Computational Linguistics (ACL), 2019.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
CONTENTS
Proof of Proposed Quantization Error Reduction via Reformulation
14
Algorithm
16
Additional Related Works
18
Additional Experimental Setting
18
C Additional Accuracy Results
19
F Additional Ablation Studies
21
G Additional Latency and Model Size Results
22
Longer Context Results
23
Calibration Sensitivity Analysis
24
J
Latency breakdown of the entire Mamba-2 block
25
Batch-Size Sensitivity Analysis
26
I. Distributions of SSD Tensors
27
LLM Usage
29
13

---

Published as a conference paper at ICLR 2026
A PROOF OF PROPOSED QUANTIZATION ERROR REDUCTION VIA REFORMULATION
Proposition
Suppose that
P
P
A
R.P
p.P
P(yp #
1|lul
12
x.p
12
12
z.P
p=1
Then holds that
MSE
MSE
Notation.
(1) We denote the Hadamard product by O. The quantization step size is A
Range
2
(2) The dequantized input is x' deq(g(x)). The output is Wt.p Fl,p ① luti-
(3) Let Pp P(yp + O) and Vi.p {Np # 0}.
(4) Vectors are denoted by Zp = (2O.p. LL.)) and V/p (yo VL,p), with error vector
Cg.p (eo.z.p...., eL..p.
(5) The L-vector lut = (luto.... lutz) is fixed and deterministic.
Assumptions.
(1) min(yp) < 0 and max(y;) 0.
(2) Quantization errors satisfy Cx.lpp U( Aapp Ane), Cdd.p U(- Aye. Av
2
(3) Outliers are not considered in Up.
(4) 0 < Pp < 1.
(5) lut is not random variable.
Proof. Step 1. Step size relation. In symmetric quantization, the step size A is determined by the min/max values.
By Assumption (1), we have A = A..p. Let Sp = Av.p/Az, so that Ay.p = SpAr. and hence v.P >A..p
Step Case y' = (x @ Iul)..
The reconstructed output is
1í. = Vī + t.pp with prob. Pp:
00
with prob. Pp-
Thus
MSE Pp E(( 31.
Since the error "y,.p = Vi.p Vi has zero mean, we have
B((CtA,))] Var(e
Therefore,
MSE Pp E((C Pg: Var(e
Under the standard quantization noise model,
(A m.p)
Var(e
12
so that
(A g.p) y2
MSE Bemka.p = Pp
12
14

---

Published as a conference paper at ICLR 2026
Averaging over p gives
P Σ Pp 12
(A )
MSE spaled
p
pml
Step Case y' x' lut.
We expand
MSE., = B[lbb/ [ol1š] = [E[((a,) - -xp) ④ Lue
By component,
[|x=,, xp) o lutli
Σ
(ond lutt)²
1=1
Taking expectation,
Elllez,p 2 /utll Σ luff Ble
Since Endyy is uniform, Bie A p 12. Therefore,
/12.
A
MSEz.p = ||ut
12
Averaging gives
P E
x.
MSE
||rut|13
12
p=1
Step 4. Comparison. Substituting Ag,p 8q Sppp.
P
A
MSE
Fooled
P
Pp
Sp
2
12
2.P
p=1
Thus, if
P
A2
P
A2
Pps
P.P
<
lut 1
12
12
p=1
p=1
then
MSF
MSSE
MSE
Mildness of the sufficient condition. This condition is mild. First, scaling typically reduces the dynamic range so that Ay.p S Az.p. i.e., 8p < 1. Second, due to the sparsity of Xsealed the activation probability is small (pp & 1), which diminishes the left-hand side. Third, the lul vector carries negligible energy across dimensions, so ||/ut|l2 is not small. Consequently, in these typical regimes,
P
Pps
Ilut
a.
12
12
and thus MSE Suyted < MSE follows naturally. For a detailed discussion of the empirical character- istics of the distributions of seaaled and lut, please refer to Fig. 3 and Appendix
15

---

Published as a conference paper at ICLR 2026
B ALGORITHM
Algorithm Sequential Mean Correction Update
Require: Quantized Blocks Bu.,, fp16 means [op[1:]], number of samples S, sequence length T decaying factor 7), target-layer set Ligs
Fix initial inputs
1: for <- to S do
2:
[[s) < Embedding D(s), T))
3: end for
4:
for 1--1 1 to L do
5:
if & Lugi then
6:
for s <- 1 to S do
7:
Y 4- (x)+) X|s]
9:
end for continue end if
10:
11:
12:
0; N ++1 My
13:
for
8 + too S do
14:
Y < (((((s)
15:
mı Y.mean(( 1)
16:
ns Y.shape [0] Y shspppe[1]
nh. N N + mei ws N
17:
18:
14g - + uis (m 14)
19:
end for
20:
8
pmp
Pq
21:
<[t]
< v 8 8
22:
for 8 < 1 to S do
23:
Y Yoomp t [x] apply e(()
24:
X|s} Ycomp
25:
end for
26:
end for
27:
return model with corrections applicd
Algorithm Sanrhhhmmmmmennnnn Timplements a layer-wise sequential mean correction strategy. Mean correction estimates a channel-ve -ise correction term from the quantized output mean Ma, and the accuracy of this estimate depends the input distribution under which it is measured. If correction terms are estimated inde- pendently using a fixed initial input distribution, later layers will observe statistics that differ from those encountered during actual inference, leading to biased correction terms. Instead, update layers sequentially and propagate corrected activations forward, so that each layer estimates un- der the distribution produced by upstream corrections. This procedure aligns correction estimation with the inference-time signal flow and stabilizes cumulative quantization error across layers.
16

---

Published as a conference paper at ICLR 2026
Algorithm 2 SSD Layer
Require: X E L, H. AER(B, L, H). decay activation A E R(H).
I: B E R(B,L,G,N) C' CE, L
Module 1: ChunkCumsum (Input (A, A) Output (A, d.
3: A <- softplus(2) A)
4: A discretize(A) A)
dA CunnSunDecayAA' (A+)
D R(B,
Module 2: ChunkState (Input (dAcs, A, B.X) - Output State) dAcs)
6: Decay state exp(dAcal by 1,1111111
sUTsttate + Decay atate
DER H, HE
8: State X (B @ LUT
Module 3: StatePassing (Input (State, d.A.)) Output State)
9: Decay pass exp(dAcs [-:;11111
D DE BE, HI,)
for to c-2
11: State[i+1]] + State + Decay pass [i+1] O State[i]
12:
end for
Module 4: ChunkBMM (Input (B,C) Output CB)
13: CB t- Cx
DE
Module 5: ChunkScan1 (Input (Stale, C, C, A) - out.r)
14: Decay scant exp(dAcs)
15: LUTscann + A Decay scan1
16: out <- (Stale x C "") LUT
DE ((,,
Module ChunkSean2 (Input (X, CB, dAes, A) + outaigg)
Let ( E ((B.H.c...) dA(s) E ((B.H,c...)) be broadcasted forms dAcs. 18: scanc t- A exp(dA(() -dA())
H,,, P (Bc,H.,
outding f (CBC LU'ssmm))
Final Output
20: Y outoff outdiag
21: return Y
RRB H,c, H,, P.I)
SSD layer computes the state-space dynamics of Mamba-2 using parallel chunked formulation. Given input activations X, the layer first discretizes the step size A and decay activation A, and constructs per-chunk cumulative decay factors through ChunkCumsum.
ChunkState performs the input-to-state projection within each chunk in parallel, while
StatePassing propagates recurrent information across chunks to restore the global sequence depen- dency.
ChunkBMM computes the block-diagonal interaction matrix CB,
which is exclusively used in the diagonal path.
ChunkScanl generates the off-diagonal contribution from the recurrent state, and
ChunkScan2 produces the diagonal contribution from the input representation with CB.
The final SSD output is obtained by summing these two terms.
17

---

Published as a conference paper at ICLR 2026
C ADDITIONAL RELATED WORKS
Post- post-Training Quantization and LLM Quantization. Quantization approaches are generally di- vided into Quantization-Awan Training (QAT) (Gholami al.) 2022). which integrates quantiza-
tion into the training process, and Post-Training Quantization (PTQ) (Frnnaar et al. 2023 Xiao
al., 2023 Lin al., 2024). which applies quantization models after pretraining. QAT often considered strong in preserving accuracy. but for large-scale models the associated retraining cost can become prohibitively high. As result, many recent studies have shifted attention toward PTQ, particularly in the context of large language models (LLMs) (Touvron et al. 2023).
Among representative PTQ approaches, GPTQ (Frantar et al. 2023) proposes weight- compensation PTQ method by leveraging approximate second-order information via the Hessian. SmoothQuant (Xiao et al.) 2023) shifts the difficulty of activation quantization into weights, en- abling stable WSA8 and W4A8 performance. QuaRot (Ashkboos et al., 2024) and SpinQuant (Liu et al.) achieve precise 4-bit quantization by applying random or learned rotation matrices to mitigate outliers. QServe (Lin et al.) highlights the practicality of W4A8 quantization in real environments, demonstrating its effectiveness in reducing inference latency for L.MMs. However, these methods inherently optimized for the structural properties of Transformers- such as self-attention and KV caching-ane thus are not directly applicable to architectures like selective state space models, where continuous state updates and activation reuse play a central role.
D ADDITIONAL EXPERIMENTAL SETTING
Implementation. For quantization, we use a calibration of 512 samples drawn from the Pile dataset. We apply 4-bit weight quantization to the in projection and out projection layers using GPTQ. To improve efficiency, the scaling parameter 7 of RMSNorm fused into the in projection layer (Wei et al. 2022). Except for the SSD module, activations are quantized to 8-bit with per- tensor quantization, while the fast Hadamard transform (Ashkboos et al. 2024) is fused into the corresponding layers. Inside the SSD, we adopt the same Triton (Dao) 2024b a) as used in Mamba- 2, but modified to fit the SSDi8 method. CUDA LY 2024a b) based causal Conv 1d operator is used without modification.
18

---

Published as a conference paper at ICLR 2026
E ADDITIONAL ACCURACY RESULTS
Evaluations are conducted on the same datasets, where HAD denotes applying Hadamard and 4-bit GPTQ quantization to Mamba-2. SSDi8 achieves performance comparable to Quamba2 under W4A16 quantization, even with W4A8 quantization.
Tab. presents an extended version of the accuracy results in Tab. 2
Table Evaluation of Mamba-2 models 1.3B, 2.7B, and 8B scales using four quantization meth- HAD, Quamba, Quamba2, and SSDi8 ASDis-across across six zero-shot tasks: LA, HS, PIQA, Arc-E, Are-C, and WG.
Model
Size
Methods
Bitwidth
LA
HS
PIQA
Arc-E
Arc-C
WG
Avg.
FP16
65.6%
59.9%
73.3%
64.1%
33.3%
60.8%
59.5%
W8A8
55.3%
59.4%
73.2%
64.0%
33.5%
58.25
57.3%
HAD
W4A8
53.9%
58.9%
72.3%
63.6%
33.9%
1%
57.0%
Quamba
W8A8
49.8%
58.5%
71.2%
61.9%
32.1 1%
58.1
55.2%
W4A16
64.3%
59.2%
72.6%
63.8%
33.1 %
60.3%
58.9%
Quamba2
W8A8
62.0%
59.2%
72.5%
63.4%
32.7%
60.0%
58.3%
W4A8
61.0%
58.8%
72.4%
62.7%
32.6%
59.1%
57.7%
W8A8
64.7%
59.7%
72.7%
64.0%
32.8%
60.9%
SSDi8 (Ours)
W4A8
63.6%
59.2%
72.7%
63.5%
33.5%
60.4%
58.8%
FP16
69.5%
66.6%
76.4%
69.5%
36.4%
64.2%
63.8%
W8A8
53.8%
60.8%
73.8%
64.8%
35.8%
62.2%
58.5%
HAD
W4A8
51.2%
59.7%
73.0%
64.9%
34.6%
60.2%
57.3%
Mamba-2
Quamba
W8A8
52.4%
71.6%
62.9%
33.7%
58.0%
56.5%
2.7B
W4A16
68.8%
65.6%
75.5%
68.66 68.6%
36.6%
64.9%
63.3%
Quamba2
W8A8
66.1%
65.5%
74.4%
68.4%
37.1%
63.7% 7%
62.5%
W4A8
65.6%
65.1%
74.7%
68.1%
36.1%
62.8%
62.1%
W8A8
68.3%
66.2%
75.6%
69.0%
36.8%
63.4%
63.2%
SSDi8 (Ours)
W4A8
67.6%
65.3%
75.6%
68.9%
35.2%
63.5%
62.7%
FP16
70.9%
77.7%
79.7%
76.0%
48.0%
72.0%
70.7%
W8A8
56.7%
75.3%
78. 11%
74.1%
45.0%
65 65.6%
65.8%
HAD
W4A8
56.1%
74.6%
77.3%
73.8%
44.5%
66.0%
65.4%
Quamba
W8A8
54.0%
74.6%
77. %
73.5%
44.2%
65.5%
64.8%
SB
W4A16
71.2%
76.8%
79.1% 19.1%
75.2%
45.9%
70.8%
69.8%
Quamha2
W8AS
69.8%
77.8%
79.1%
75.9%
46.9%
69.0%
69.8%
W4A8
68.8%
77.1%
79.1
75.0%
46.0%
68.7%
69.1
W8A8
70.4%
77.2%
79.6%
75.5%
47.2%
71.2%
70.2%
SSDi8 (Ours)
W4A8
69.9%
76.5%
79.1
75.4%%
46.2%
70.6%
69.6%
19

---

Published as a conference paper at ICLR 2026
We evaluate perplexity on the Pile benchmark for the 1.3B and 2.7B models. Across both model scales, SSDi8 surpasses Quamba2 and approaches FP16-level performance under W8A8 quantiza- tion, shown in Tab. 9
Table 9: Pile perplexity with 7. = 2048
Pile Perplexity (1)
Model
Methods
Bitwidth
1.3B
2.7B
FP16
6.99
6.27
W8A8
7.46
7.77
HAD
W4A8
7.87
8.17
Mamba-2
W8A8
7.20
6.44
Quamba2
W4A8
7.55
6.68
W8A8
7.08
6.34
SSDi8 (Ours)
W4A8
7.41
6.57
20

---

Published as a conference paper at ICLR 2026
F ADDITIONAL ABLATION STUDIES
Tab. 10 shows ablation results on the quantization axis of activations within SSD, evaluated on Wikitex(2 perplexity. For activations B, C. per-G, yields the best performance, though the differ- ence from per-G is negligible (0.02). In contrast, X and State are highly sensitive to the choice of quantization axis, showing substantial degradation when either the or axis is not considered.
Table 10: Ablation study for quantization axis.
Model
Bitwidth
Activation
per-T
per-P(N)
per H()
Wikitex12 Perplexity
7.42
7.59
B,C
V
7.43
W4A8 SSD-FPI6
V
7.44
8B
V
V
7.42
11.97
X,State
V
8.59
V
8.15
7.42
We also analyze latency and accuracy variations with respect to the placement of mean correction. The highest accuracy gain is observed when mean correction is applied immediately after SSD layers, indicating the out-projection layer, yielding the lowest latency when mean correction is applied. Considering the trade-off between latency and accuracy, we therefore apply mean correction only the out- projection layer, as summarized in Tab. II
accumulation within SSD). In Mamba-2, the model dimension is halved after
Table 11: Accuracy and speedup for W4A8. Experiments are conducted on the LAMBADA dataset, using SSDiS without mean correction as the baseline.
Bitwidth
Project
Speedup
Acc.
None x1.00
67.2%
In x0.945
67.4%
W4A8
SSD 00.975
67.5%
Out
0..987
67.4%
21

---

Published as a conference paper at ICLR 2026
G ADDITIONAL LATENCY AND MODEL SIZE RESULTS
We further analyze the memory footprint and SSD latency of SSDi8, using FP16 and Quamba2 as reference baselines.
Table 12: Memory usage comparison
Model
Size
Method
W8A8
W4A8
FP16
5.154 GB
2.7B
Quamba2
2.948GB
1.766GB
SSDi8 (Ours)
2.953GB
1.774GB
Mamba2
FP16
15.710 GB
8B
Quamba2
9.860GB
7.028GB
SSDiS (Ours)
9.867GB
7.038GB
As shown in Table 12 the memory usage of SSDi8 is nearly identical to Quamba2. For the 2.7B model under W8A8, SSDi8 requires 2.953GB compared to 2.948GB for Quamba2, resulting only a ++.117 17% increase due additional static quantization scales. A similar trend is observed for the 8B model (9.867GB vs. 9.860GB, +0.07%).
FP
Quamba2
SSDi8
30
Latency (ms)
15
TAS
0
Batch
8
16
32
Figure 6: Comparison with Quamba2 under W4A8 quantization on the 8B model is also reported. OOM denotes Oft-Or--emory.
Figure Segure@repone reports SSD-module latency across batch sizes. Compared to FP16, SSDi8 consistently re- duces latency at moderate batch sizes. At batch 16, SSDi8 reduces latency from 14.745ms (FP16) to 10.646ms, corresponding to 27.8% reduction. At batch 32, FP16 encounters OOM, while Quamba2 requires 25.797ms. Under the same setting, SSDi8 achieves 19.000ms, yielding 26.3% latency duction compared to Quamba2.
22

---

Published as a conference paper at ICLR 2026
H LONGER CONTEXT RESULTS
comparing FP16, Quamba2, and SSDi8 under identical experimental settings. All experiments are conducted on an A5000 GPU using the 2.7B Mamba-2 model with a batch size of 8 and sequence lengths ranging from 2k to 14k. For each context length, we measure SSD-module latency, end-to-end throughput (tokens/s), and perplexity (PPL) to assess both computational efficiency and modeling behavior in the long-soqueee sequence regime.
We report additional results on long-context evaluation beyond 2k tokens(Tab.. 13
Table 13: Long-context results (B=8): SSD latency, end-to-end throughput (tokens/s), and perplexity (PPL).
Length
Ranch
QuambaZ
Throughput (3)s)
Throughpu: (D5)
(ma)
Throughpu: (1))
4,990
11981
B-8
9,299
10716
13.345
8,938
12333
11.502
7.231
12818
9.021
6k
B==8
13.564
10922
113-407
13.363
12648
100.438
10.367
13038
8.943
8t
B-88
17.80 867
11175
444 46-
15 15.932
13079
309 182
12,635
13418
9.040
B-8
22,06]
11167
1577.003
19. 19.805
13121
1253.093
15.827
13804
9.122
27.058
26504220
13079
18.168
13743
9.239
3881,000
13291
Across all evaluated sequence lengths, SSDi8 consistently achieves higher throughput and lower SSD latency than both FP16 and Quamba2, with the performance gap widening as the context length increases. At 14k tokens, SSDi8 improves throughput from 11,362 tokens/s (FP16) to 14,200 tokens/s, corresponding to gain of +2,838 tokens/s (+25.0%). Over the same setting, SSD la- tency is reduced from 30.870 ms to 21.138 ms, yielding a 9.73 ms reduction (-31.5%). In terms of perplexity, FP16 Mamba-2 exhibits rapid degradation beyond context length of 4k, with PPL increasing from 13.845 (4k) to 113,407 (6k) and further escalating at longer contexts. Quamba2 shows similar instability trend. In contrast, SSDi8 maintains stable perplexity across the entire evaluated range (2k-14k), with PPL. remaining within narrow band of 8.94-9.36. These results indicate that SSDi8 preserves numerical stability the long-context regime while simultaneously improving computational efficiency. The stability of SSDi8 at extended sequence lengths suggests that careful quantization of the SSD path mitigates the oong-sequence numerical issues observed in full-precision and prior quantized baselines.
23

---

Published as a conference paper at ICLR 2026
I CALIBRATION SENSITIVITY ANALYSIS
We analyze the sensitivity SSDi8 to calibration settings. Specifically, we consider both the number of calibration samples and the choice of calibration dataset. First, we report zero-shot performance while varying the calibration sample size from 128 to 2048 under an identical evaluation proto- col, in order examine how the performance changes as the amount calibration data varies(see Tab. [14.. We then compare performance across different calibration datasets, including commonly used corpora such as C4, PTD, and Pile, study the effect of the calibration data source (Tab. 15
Table 14: Sensitivity to the number of calibration samples. Accuracy (%) on six zero-shot tasks.
Calib Size
Wino
PiQA
ARC-C
ARC-E
Hella
Lamb
Avg.
128
64.4
75.5
35.1
68.2
65.0
68.0
62.7
256
64.5
75.3
36.4
67.8
64.8
67.0
62.6
512
64.2
75.5
35.8
67.9
65.1
67.7
62.7
1024
64.3
75.7
36.4
67.6
64.8
66.3
62.5
2048
64.8
75.1
36.2
68.1
65.2
67.1
62.7
SSDi8 shows only small variations with respect to both the number of calibration samples and the choice of calibration dataset. When increasing the calibration sample size from 128 2048, the av- erage accuracy across six zero-shot tasks remains largely stable, with only minor differences across individual benchmarks. Similarly, changing the calibration dataset leads comparable performance, and no single calibration corpus consistently results in lower accuracy. Overall, these results suggest that the performance of SSDi8 remains stable across range of calibration configurations.
Table 15: Sensitivity to the choice of calibration dataset. Accuracy (%) on six zero-shot tasks.
Calib Data
Wino
PiQA
ARC-C
ARC-E
Hella
Lamb
Avg.
C4
65.4
75.2
36.1
68.5
65.3
67.7
63.0
PTD
65.1
75.1
36.5
68.6
65.5
67.6
63.0
Pile
64.5
75.3
36.4
67.8
64.8
67.0
62.6
24

---

Published as a conference paper at ICLR 2026
J LATENCY BREAKDOWN OF THE ENTIRE MAMBA-2 BLOCK
Tab. T6 presents a module-wise latency breakdown of a single Mamba-2 block, decomposing the block into major computational components: input projection (in-proj), convolution (conv), SSD, normalization (norm), output projection (out-proj), and the correction term. The results are reported for the 55th block of the 2.7B Mamba-2 model under batch sizes 8, 16, and 32, and compare FP16 and SSDi8 under identical experimental settings. This breakdown is intended to quantify where latency reductions are achieved by SSDi8 and clarify the contribution of each submodule the overall block latency.
Table 16: Latency breakdown (ms) of the Mamba-2 block (2.7B, I. 2048) under different batch sizes.
Method
Batch
In-Proj
Conv
SSD
Norm
Out-Proj
Correction
FP16
B=8
9.829
0.781
4.795
1.196
4.919
FP16
B=16
20.759
1.357
9.426
1.640
10.272
FP16
B=32
41.297
2.526
17.640
3.086
20.474
SSDi8
B=8
8.124
0.587
4.481
1.089
3.266
0.426
SSDi8
B=16
16.758
1.011
7.293
1.442
6.677
0.698
SSDi8
B=32
33.893
1.799
13.199
2.291
12.914
1.252
under FP16 execution, the projection layers constitute the dominant latency bottleneck, followed by the SSD computation, while normalization and other components contribute relatively little. In SSDi8, Hadamard-based activation quantization and GPTQ weight quantization are applied to the projection layers, while the primary optimization focus placed on the SSD module, which represents the second-largest bottleneck in FP16. Across batch sizes, SSD latency increases ap- proximately linearly, exhibiting a scaling trend similar to that of the projection layers, whereas the remaining submodules remain negligible. We further observe that the output projection reduces the dimensionality the hidden representation, allowing the mean-correction term introduced by SSDi8 to be applied at minimal cost. As result, the correction latency remains lower than that of the con- volution layer, which the least expensive FP16 component. Overall, this breakdown illustrates how SSDi8 reduces latency primarily by optimizing the most time-consuming components within Mamba-2 block, while preserving favorable scaling behavior across batch sizes.
25

---

Published as a conference paper at ICLR 2026
K BATCH-SIZE SENSITIVITY ANALYSIS
To further examine the computational behavior of SSDi8, we evaluate SSD-module latency and throughput across varying batch sizes, ranging from edge-oriented small batches large-batch cloud-serving scenarios. This analysis allows us to assess how INT8 quantization of the SSD path scales under different levels computational intensity (see Tab, 17
Table 17: SSD latency and throughput comparison 2048)
Value
Batch Size
Method
Latency (ms)
Throughput (T/s)
FP16 SSDi8
2.757
9918
16
2.671
10299
FP16
4.898
10653
32
SSDi8
4.510
11976
FP16
9.271
10405
64
SSDi8
7.173
12114
FP16
18,003
11222
128
SSDi8
13.406
13407
FP16
35. 194
11333
256
SSDi8
23.730
13798
SSDi8 consistently reduces SSD-module latency and improves throughput across all batch sizes. At batch size 16. SSD latency decreases from 2.757 ms to 2.671 ms (3.1 % reduction), while through- put improves by 3.8%. As batch size increases, the performance gap widens. At batch size 128, SSD latency is reduced by 25.5% and throughput increases by 19.5%. In the large-batch regime (256), SSDi8 achieves a 32.6% reduction in SSD latency (35.194 mss + 23.730 ms) and a 21.8% improvement throughput.
These results indicate that the benefits of quantizing Ctunk and ChunkBMM become increas- ingly pronounced under higher computational intensity. While small-batch settings already show consistent gains, larger batch sizes amplify the arithmetic and memory-efficiency advantages of INT8 execution. Overall, SSDi8 demonstrates favorable scaling behavior across both edge and cloud-serving deployment regimes.
26

---

Published as a conference paper at ICLR 2026
L DISTRIBUTIONS OF SSD TENSORS
Visualization of Activations. Figure 7 represents that visualization of B, C. and CB by group in the first, middle, and last blocks of the Mamba-2 8B model. As argued in Sec. the distributions differ across groups. CB is masked as it is used for computing ouldiag.
CB
11 0
2.0
oth Group
0.0
04
oth Block
1000
2000
N
1003
3043
200
3rd Group
0.4
1.0
0.0
01
1000
1090
1090
7.
3303
B
C
CB
055
10
0.1
0.2
oth
2000
N
1000 1/
1000
2000
27th Block
2.
3rd Group
1.5
100
121
3000
3000
1000
2001
2000
CB
0.5
oth Group
1000
1003
2000
55th Block
1.2
0.3
0.6 2
grd Group
0.5
3.0
N
C
2002
2003
Lub
2000
Figure 7: Visualization of the distributions of activations B, C. and CB in in the first, middle, and last block of Mamba-2 8B.
27

---

Published as a conference paper at ICLR 2026
Figure shows shows the visualization of X, LUT state, and Xscaled in the last block of Mamba-2 8B. The first row illustrates the full sequence length, while the second row depicts its partition into nehunks with the corresponding chunk size. Both LUT and X ssoatd exhibit exponential growth as the chunksize index increases.
X
LUT state
X scaled
6.0
0.2
3.0
HXL
0.0
0.0
60
1000
30
P
1000
30
P
2000
20000
0.2
1001
ex1
0.0
0.00
30
P
00
120
1
120
Figure 8: Visualization of the distributions of activations X, LUT states and X seated in the last block of Mamba-2 8B.
28

---

Published as a conference paper at ICLR 2026
M LLM USAGE
During the manuscript preparation, we used OpenAl's GPTS (https://chttgt..omm)) a Large Lan- guage Model, to proofread our work. Our interaction with the LLM was iterative and focused exclu- sively on improving the quality of the writing. We affirm that the LLM served as an assistive tool and did not contribute core research ideas, experimental design, analysis, and results presented in this paper. The final scientific content and all claims made in this paper are the sole responsibility of the authors.
29


Reviews:
[{"review_id": "8HG8G8jcvV", "reviewer": "Reviewer_Kz2V", "summary": "The paper introduces SSDi8, a post-training quantization (PTQ) framework specifically designed for Mamba-2's Structured State Space Duality (SSD) architecture. The authors first identify key challenges that make naive PTQ methods unsuitable for SSDs, such as heterogeneous activation scales and the mix of element-wise and GEMM operations. To address this, SSDi8 introduces three core contributions: (1) a sparse-aware reformulation that pre-scales activations to maintain a persistent INT8 data path within the SSD block; (2) a \"quantize-once, reuse\" strategy for channel-varying activations \\(B,C\\) along the group axis to minimize DRAM traffic; and (3) a lightweight per-channel mean correction technique to mitigate quantization bias.\n\nEmpirically, SSDi8 demonstrates near-FP16 accuracy on Mamba-2 models up to 8B parameters. It achieves up to a 1.47x speedup for the SSD block compared to FP16 (1.38x over the Quamba2 baseline) and shows practical latency gains on edge hardware like the Orin Nano. The framework strategically keeps two submodules, ChunkCumsum and ChunkScan2, in FP16 to preserve stability. A theoretical proposition in the appendix further supports the sparse-aware reformulation, showing it can reduce quantization MSE under certain conditions.", "questions": "1.  The evaluation is performed on context lengths up to 4k. I believe this is limited by the base Mamba-2 models. However, a key advantage of quantizing SSDs may become apparent when scaling the sequence length to a longer one. It might be good and interesting to see how the e2e latency gains are there at longer context (e.g., 8k, 16k or longer)\n\n2. Can SSDi8 apply on mabma2-transformer hybrid models [1,2]?  \n\n[1] https://huggingface.co/nvidia/mamba2-hybrid-8b-3t-128k\n[2] https://huggingface.co/nvidia/Nemotron-H-8B-Reasoning-128K", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- **Principled, SSD-Specific Design:** The work is well-motivated, starting from a clear analysis of why standard PTQ fails for Mamba-2. The proposed sparse-aware reformulation and persistent INT8 state are theoretically grounded (as shown in the Appendix) and effectively target the unique properties of the SSD architecture.\n- **Consistent Accuracy Gains:** The method's effectiveness is demonstrated across a range of model scales (1.3B, 2.7B, 8B) with perplexity and accuracy metrics that are nearly on par with the FP16 baseline and competitive with or better than Quamba2. The inclusion of latency measurements on edge hardware (Orin Nano) highlights the practical deployability of the framework.\n- **Low-Overhead Accuracy Recovery:** The per-channel mean correction method effectively improves accuracy with a negligible latency overhead (1-2%), offering a practical and efficient solution for mitigating quantization bias.", "weaknesses": "- **Marginal End-to-End Speedup Over Prior Work:** While the SSD block itself shows significant speedups (up to 1.47x), the reported end-to-end latency gains over the most relevant baseline, Quamba2, appear modest. This raises questions about the overall practical impact, as other parts of the model may become bottlenecks. A more detailed breakdown of end-to-end latency would help clarify the real-world benefits.\n- **Limited Evaluation on Recent Mamba-2 Models:** The evaluation is confined to the base Mamba-2 models. The paper misses an opportunity to demonstrate the framework's relevance and scalability by applying it to recent, influential foundation models that incorporate Mamba-2, such as Nemotron-4-340B-Base, which use hybrid SSM-Attention architectures and operate on much longer contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tND7Fd2ZeR", "reviewer": "Reviewer_akaA", "summary": "This paper presents SSDi8, a post-training quantization method built specifically for Mamba-2’s Structured State Space Duality (SSD) blocks. Instead of applying generic Transformer quantization, SSDi8 restructures SSD computations so they can run almost entirely in INT8. It quantizes the group-wise activations B and C once and reuses them across modules, reformulates ChunkState to keep the matrix multiplications in low precision, and adds a small per-channel mean correction to offset quantization drift. The result is near-FP16 accuracy with up to 1.47× faster inference over FP16 and 1.38× over Quamba2, marking the first fully INT8 data path within the SSD architecture.", "questions": "1. Why is maintaining a fully continuous INT8 execution path so essential for SSD efficiency?\n\n2. Do the authors have profiling data showing where FP16 operations become actual latency bottlenecks in SSD?\n\n3. How much runtime is really lost when LUT_state state remains in FP16—can this be quantified?\n\n4. Could a mixed-precision (INT8 + FP16) path achieve similar performance without full reformulation?\n\n5. The paper claims that FP16 activations “eliminate” INT8 GEMM efficiency—what hardware-level evidence supports that?\n\n6. In what specific ways does SSDi8 differ from Quamba2, which already quantizes SSD blocks in Mamba-2?\n\n7. Is the main novelty the sparse-aware reformulation, the activation reuse strategy, or the persistent INT8 recurrence path?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "A key strength of the paper is its architecture-aware approach to quantization. Rather than applying generic Transformer-style methods, SSDi8 directly models the structure of Mamba-2’s SSD blocks and exploits their specific activation patterns.\n\nThe design choices are well motivated by empirical observations, such as head-wise activation variance and per-axis sparsity.\n\nThe experiments are comprehensive, covering multiple model scales, bit-width regimes, and hardware platforms, with consistent accuracy close to FP16 and up to 1.47× latency improvement", "weaknesses": "The paper overemphasizes the need for a fully continuous INT8 execution path through SSD without clearly explaining why this is critical for Mamba-2’s efficiency. While the idea of keeping all operations in INT8 sounds appealing, the authors do not show evidence that occasional FP16 operations meaningfully degrade throughput. The claim that the presence of an FP16 activation such as LUTstate “eliminates the efficiency of INT8 GEMM” is not well supported. A mixed-precision path would not inherently destroy INT8 performance; it would simply require limited casting or dequantization overhead. More quantitative profiling is needed to show where and how FP16 operations bottleneck the pipeline.\n\nThe novelty claim that SSDi8 is “the first post-training quantization framework specifically designed for SSD” is also overstated. Quamba2 already targets the SSD modules in Mamba-2 and applies structured activation quantization within them. The paper should clarify what makes SSDi8 distinct from existing SSD-aware methods, such as whether the main innovation is the sparse-aware reformulation, activation reuse, or the persistent INT8 recurrence path, rather than relying on the claim of being the first.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XaoH9BWzwd", "reviewer": "Reviewer_mHsN", "summary": "This paper presents a post-training quantization framework specifically designed for the Structured State Space Duality (SSD) layers in Mamba-2 models. They enable INT-8 quantization inside SSD blocks through a combination of: a sparse-aware reformulation that decouples element-wise multiplications from matrix multiplications, strategic placement of quantization operations to minimize overhead, and a mean correction mechanism. The authors achieve comparable accuracy to FP16 while delivering up to 1.4x speedup in inference. The work addresses the challenge that existing quantization methods designed for Transformers fail when applied to SSD layers due to their unique computational structure involving recurrent states and channel-varying activations.", "questions": "1. Do you observe any mismatch between step-by-step vs. chunked vs. parallel forward passes, given that quantization introduces errors in the recurrence?\n2. In Section 1, you claim to show results for Hadamard rotation and GPTQ but these are not in Table 1 or other results tables. Could you clarify where these comparisons are?\n3. Why is the mean correction applied only to the output projection layer? What happens if applied elsewhere?\n4. How sensitive is the method to the calibration dataset size and choice?\n5. Could you provide more details on how the sparse-aware reformulation maintains numerical stability?\n6. How does the method perform on longer sequences beyond $L=2048$?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- first work to successfully apply INT-8 quantization within Mamba-2's SSD architecture\n- Thorough mathematical analysis\n- comprehensive experimental validation across multiple model sizes up to 8B\n- Deployment validation on resource-constrained devices (Orin Nano)\n- clear identification and analysis of why existing methods fail on SSD layers\n- well-structured paper with detailed ablation studies", "weaknesses": "- unclear notation and presentation around the reformulation (mixing step-by-step vs chunked vs parallel forward passes)\n- limited theoretical justification for why the mean correction works\n-some implementation details are vague (e.g., specific quantization schemes for different components)\n- the paper could benefit from more intuitive explanations alongside the mathematical formulations\n- limited discussion of potential limitations or failure cases\n- not immediately clear what the overhead is of computing and storing the quantization scales for the various per-channel schemes", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sUlivSK1a8", "reviewer": "Reviewer_9vyW", "summary": "The paper introduces SSDi8, a framework designed to accelerate Structured State Space Duality (SSD) computation using INT8 representation. Unlike prior methods (e.g., Quamba2) that quantize only the SSD inputs while performing state updates in FP16, SSDi8 establishes a persistent INT8 execution path by quantizing internal SSD computations, reusing quantized activations, and reformulating element-wise operations to ensure low-precision consistency. The framework further incorporates a sparse-aware reformulation to mitigate quantization errors and a mean-correction mechanism to compensate for layer-wise activation drift. Experiments on Mamba-2 models (1.3B, 2.7B, and 8B) demonstrate that SSDi8 achieves up to 1.47× speedup over FP16 while maintaining accuracy comparable to full precision across six zero-shot benchmarks. Moreover, SSDi8 exhibits robust performance on edge devices (e.g., NVIDIA Orin Nano), highlighting its practical deployability under resource constraints.", "questions": "- Could the authors provide a deeper comparison with Quamba2 [1]? In particular, it would be helpful to include Quamba2 results in Table 5 for a more direct and quantitative comparison. If my understanding is correct, Quamba2 already supports ChunkState quantization Q(X) and quantization of matrices B and C. A side-by-side evaluation would better clarify the incremental benefits of the proposed framework over this closely related baseline.\n- Would the authors also provide a more detailed latency breakdown of the entire SSD block? Specifically, analyzing the latency contributions of the input projection, SSD computation, normalization, and output projection layers under different input batch sizes would offer valuable insight into the scalability and bottlenecks of the proposed quantization method. \n\nI am willing to reconsider and potentially raise my overall score if the above concerns are adequately addressed in the authors’ rebuttal or follow-up discussion.\n\n\n### Minor Suggestions\n- I recommend that the authors simplify the shape conversion and tensor dimension descriptions in the SSD computation steps. The current presentation is dense and difficult to follow, and a clearer explanation (potentially with a schematic or pseudo-code) would improve readability.\n- I suggest that the authors include a discussion on large-batch serving scenarios, as quantization behavior and latency characteristics can differ significantly between small-batch (edge inference) and large-batch (cloud serving) settings. Highlighting how SSDi8 performs under these conditions would enhance the paper’s practical relevance and deployment perspective.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "SSDi8 provides a approach to quantizing Mamba-2’s SSD modules with 8-bit, addressing quantization sensitivity and memory inefficiency not handled by prior work. Its sparse-aware reformulation and activation reuse are both theoretically grounded and empirically validated, enabling a full INT8 inference pipeline without significant accuracy loss. The paper includes comprehensive experiments across multiple model scales, showing consistent improvements over baselines like Quamba2 in both accuracy and latency. Furthermore, the mean-correction mechanism is elegantly simple yet effective, improving quantized results with minimal overhead. Finally, the inclusion of both theoretical proofs and deployment evaluations (e.g., Orin Nano) enhances the paper’s completeness and real-world relevance.", "weaknesses": "### Major Concerns\n- The paper is somewhat difficult to follow, particularly in the presentation of the Sparse-aware Reformulation, which appears to be one of the main contributions. I recommend the authors derive Equation (7) explicitly from the original ChunkState formulation to establish a clearer connection between the reformulated SSD equations and the preceding context.\n- The source of the reported accuracy and latency improvements is unclear. To the best of my understanding, Quamba2 [1] already quantizes inputs (x), B, and C to INT8, then dequantizes within the SSM module to perform state computation in FP16. The authors should clarify how SSDi8 maintains state propagation and accumulation in INT8 precision, and why this results in both higher accuracy and lower latency compared to Quamba2. A detailed explanation or ablation isolating the impact of full INT8 state computation would greatly strengthen the technical claims.\n- Some related methods, such as HadMamba, are mentioned without citation or discussion. Properly citing these works and clarifying their relationship to SSDi8 would improve the completeness of the related-work section.\n\n\n\n\n\n[1] Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)