STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization
Authors: Marco Federici
Riccardo Del Chiaro
Boris van Breugel
Paul N. Whatmough
Markus Nagel
Venue: ICLR 2026 Poster (2026)
Keywords: Quantization
Efficient Deep Learning
Activation Quantization
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: STaMP applies sequence transforms with mixed-precision quantization to exploit token activation correlations, enabling accurate low bit width inference for LLMs and LVMs while complementing existing feature transforms and weight quantization methods.

Abstract: Quantization is the key method for reducing inference latency, power and memory footprint of generative AI models.
However, accuracy often degrades sharply when activations are at low bit widths. 
Recent work suggests that invertible linear transformations (e.g. rotations) can aid quantization, by reparameterizing feature channels and weights. 
In this paper, we propose Sequence Transformation and Mixed Precision (STaMP) quantization, a novel strategy that applies linear transformations along the sequence dimension to exploit the strong local correlation in language and visual data. 
By keeping a small number of tokens in each intermediate activation at higher precision, we can maintain model accuracy at lower (average) activations bit-widths.
We evaluate STaMP on recent LVM and LLM architectures, demonstrating that it significantly improves low bit width activation quantization and complements established activation and weight quantization methods including recent feature transformations.

Published as a conference paper at ICLR 2026
STAMP: SEQUENCE TRANSFORMATION AND MIXED PRECISION FOR LOW-PRECISION ACTIVATION QUANTIZATION
Marco Federici
Riccardo Del Chiaro Qualcomm Al Research {rdelchia}
Boris van Breugel
Qualcomm AI Research
Qualcomm AI Research (bvanbreu}
(mfederic) qua comm com
Paul Whatmough
Markus Nagel
Qualcomm AI Research
Qualcomm AI Research
(pwhatmou)
{markusn)
ABSTRACT
Quantization is the key method for reducing inference latency, power and mem- ory footprint generative AI models. However, accuracy often degrades sharply when activations are quantized below eight bits. Recent work suggests that in- vertible linear transformations (e.g. rotations) can aid quantization, by reparame- terizing feature channels and weights. In this paper, we propose Sequence Trans- formation and Mixed Precision (STaMP) quantization, novel strategy that ap- plies linear transformations along the sequence dimension exploit the strong local correlation in language and visual data. By keeping a small number of to- kens in each intermediate activation at higher precision, we can maintain model accuracy at lower (average) activations bit-widths. We evaluate STaMP on re- cent LVM and LLM architectures. demonstrating that it significantly improves low bit width activation quantization and complements established activation and weight quantization methods including recent feature transformations. The code is available at https: /ttt:////hhhb. com/ Qual comm AI research/ stamp stamp-quantian
A4
+TTMMP
INTRODUCTION
SQNR: 0.64
SQNR: 5.00
Modern generative models, such as large lan- guage models (LLMs) and large vision models (LVMs), achieve state-of-the-art performance in text and image generation but at the cost of massive computational and memory require- ments. As the demand for scalable and efficient deployment of these models grows, both in the cloud and on edge devices where computational resources are scarce, achieving inference effi- ciency has become a critical area of research.
SQNR: 7.07
SQNR: 11.47
#Feature Transform
Post-Training Quantization (PTQ) of weights and activations fundamental to enhancing inference efficiency, especially for demand- ing operations such as large matrix multiplica- tions in linear layers, which dominate power consumption. However, PTQ faces substan- tial challenges when pushing the quantization bitwidth down to 4-bits, often due to the pres- ence of outliers in weights and activations.
Figure 1: STaMP and feature transformations ap- plied to PixArt-E with 4-bit activations. The ben- efit of STaMP is orthogonal (Hadamard) feature transformation, drastically reducing artifacts.
Qualcomm AI Research is initiative of Qualcomm Technologies, Inc. All emails @qua comm. com

---

Published as a conference paper at ICLR 2026
To mitigate outliers, prior work applies function-presserving transformations (van Breugel et al. 2023) to weights and activations. For example, [iao et al. (2023) scales down outlier activations and compensates by scaling up subsequent weights, preserving the model output. Similarly, Hadamard-
based feature mixing (Ashkboos et al. 2024 Liu et al. 2024 Ma et al. 2024 Zhao et al. 2025)
reduces activation variance and spreads outliers across dimensions.
However, these methods operate primarily along the feature dimension and ignore correlations across the sequence dimension. Drawing inspiration from traditional media compression methods, in this paper we propose Sequence Transformation and Mixed Precision (STaMP) quantization, a complementary approach that leverages sequence structure to improve activation quantization.
Our main contributions are summarized below:
We introduce a new class of activation transformations operating along the sequence di- mension, complementary existing feature transformations (Figure I
We characterize the quantization error for sequence transforms and design novel mixed- precision quantization scheme to exploit local activation correlation.
We demonstrate that STaMP consistently improves the model accuracy when combined with feature transformations and weight quantization both LLM and LVM.
2 BACKGROUND
2.1 ACTIVATION QUANTIZATION
Consider X E Raxd as an activation matrix of shape sequence length s X feature size (batch size omitted for the sake of clarity). Integer quantization refers the operation of converting the activations to integer values with lower bit width. The activation quantization operation Raxd -> Nexd defined as:
def Biy Q()); = clamp
([
Fij Sij
+ zıj, 0, 2, -
(1)
in which Σij and 8ij can be interpreted as an offset and scaling parameter, while bi refers to the number bits used to quantize the entry 2ij. In order to make quantization efficient, offsets, scales and bit widths are shared across all feature channels: Siz
8;. Zij 24. and bij b;.
The de-quantization function Q-1 Nsxd + Raxd maps the integer-quantized activation into the (arl) 25y)88 We will refer to the combination of the quan-
original domain: Q Q
tization and de-quantization operation with Q (X) def QQ()), omitting the quantization parameters for brevity. The expected activation quantization error introduced by the quantization and de-quantization operations is commonly defined as:
((x,)
((X) def E [[Q(X)) - x -
1 (x.) x;
(2)
which the expectation computed with respect to the activation distribution p( X) and 11-122 refers to the squared Frobenius norm. Nagel et al. (202T) identified two main causes of quantization enror: clipping error (due to the clamp operator) and rounding error (introduced by the rounding). As is common practice in the literature,
will focus
setting in which the scale s, and offset 2s
for the i-th token are set based on the range (x:) def max, æij min, mij to prevent any clipping error: 3, def 21
rango(z.). 2 def min; 25, In this setting, the quantization error for each token æ; is determined by its quantization scale 5;;
range (ax;)*
L ((x)) <
(3)
22b. 1)2
As the activation quantization error increases, so does the deterioration in performance for the model outputs since the network output diverge from the original (unquantized) model.
2

---

Published as a conference paper at ICLR 2026
2.2 FEATURE TRANSFORMATIONS
In order to reduce the activation quantization enror, recent literature has introduced linear function- preserving transformations the form of (right) invertible matrices R. which are applied prior to the quantization operation (van Breugel et al. 2025). Clearly, the additional flexibility introduced by Feature Transformations can aid in reducing the activation quantization error:
C (X) 2 min
(4)
2(X;R)
In particular, rotation matrices can effectively reduce the activation range, effectively spreading outliers across multiple channels and consequently reducing the subsequent quantization error:
E range Rx)
E range (ax)))
S (X)
(5)
Hadamard matrices have proven very effective in reducing the number of outliers with limited ad- ditional overhead: matrix multiplications with Hadamard matrices can be performed efficiency in Ossdlog d) thanks the butterfly algorithm (Fino & Algazi 1976). and the inverse Hadamard matrix can be fused into linear layer weights (Ashkboos et al. 2024).
Existing feature transformation techniques reduce activation quantization error by redistributing ac- tivation ranges across features. However, they operate exclusively along the feature dimension and ignore correlations across the sequence dimension. Visual and textual data exhibit strong local cor- relations relations-neighboriine
pixels in images and adjacent tokens in text are strongly dependent. This suggests that a similar structure could exist in the model intermediate activations and could be lever- aged improve quantization efficiency, which is described in the next section.
METHOD
We define a Sequence Transform as linear transformation of X across sequence dimension de- fined by (left) invertible matrix L. Similarly to feature transformations R, sequence transforma- tions can reduce the quantization error, and the two can be easily combined to further the error:
(x) min E
| 2 X12]
min E
Q
R X
(6)
LRR
((X:L)
L(X;L,R)
Sequence transformations are linear, hence they commute with other linear operations. For linear layer:
((X)) w + 13 = -- (Q(()) +1 = [((X)) + (L1) 8T
in which represents a vector of ones of size s, indicating that the same bias 3 is applied to all the tokens. This implies that in a linear layer, or (ii) postpone this operation and use sequence-transformee the scale & may differ across different tokens function of L. The algorithm for sequence transformed linear layer is reported in Figure 2a
invert sequence transformation (i) right before applying the bias
bias f in which
We emphasize that, contrary feature transformations, sequence transforms do affect weights, and hence they are orthogonal to more advanced weight quantization methods such as vector quan- tization, GPTQ (Guo al. 2024). and SVDQuant (Li et al. 2025).
3.1 SEQUENCE TRANSFORM AND MIXED PRECISION (STAMP)
To understand how sequence transformations affect quantization error, we first formalize the rela- tionship between quantization error and sequence-transformee tokens (proof in Appendix A.T].
3

---

Published as a conference paper at ICLR 2026
1:
function STAMP LINEAR(X)
Uniform
STaMP
Quant. Error= 7.26x00
Quant. Error=5.56x10
2:
6- LX
10a
5 bits
8 bits
bits
3:
for = 1..8 do
10
4:
2j < Q(x;; bi)
101
end for
6:
Y - XWT
10a
Y Y
10a
8:
Y Y
10a
Upper-bound bound
Quantization Error
return Y
10a
10: end function
0
500
1000
1500 2000
0
500
1000
1500 2000
Token Index
Token Index
(a) STaMP Linear Layer Pscudocode
(b) Comparison of Upper-Bound and Quantization
Figure 2: Summary of the STaMP Procedure. The sequence Transform L aims to concentrate the energy in the initial tokens, which are quantized at higher precision. This reduces the value of the upper-bound in Equation 8 (blue) and, consequently, the overall activation quantization error (purple). For fixed average bit width
bits, combining energy concentration with two precision levels 2h right) achieves lower error than uniform quantization scheme without sequence trans- formations 2h left). Activations are collected from the input to Layer 20 of LLaMA v3 8B.
Theorem 1. The expected quantization error for activations X transformed by an orthogonal se- quence transformation L and quantized using min-max scale for each token is upper-bounded by the weighted sum of the expected norm of the transformed tokens:
E
X
LX;;
2
(8)
1)2
Note that sequence transformations L do not alter the total energy E' = Ei= Ei, any improve- ment to the bound in Equation [ must come from redistributing the bit width. This observation motivates mixed precision strategy: instead of keeping energy and bit width uniform, ately concentrate most of the energy into a few tokens and allocate more bits to them. Because the denominator in Equation 8 grows exponentially with b, allocating extra bits tokens with large energy yields a disproportionately large reduction in their contribution to the error. In other words, redistributing bit from a low-energy token high-energy token reduces the total error more than keeping bit widths and energy uniform. This property is illustrated in Figure 2b and further elaborated in Appendix A..3
deliber-
Therefore, to improve activation quantization performance, we propose Sequence Transform and Mixed Precision (STaMP) a simple strategy that concentrates activation energy into small set of tokens and assigns them higher precision. In the next section, we describe how to design transformation that achieves this efficiently and how to determine the bit width allocation.
3.2 EFFICIENT ENERGY CONCENTRATION
The energy of the i-th sequence--rannffrmed token e; can be also seen the projection of the autocorrelation matrix S E along the direction L;
e = | - SL.
he direction that maximizes this energy is the cigenvector u₁ associated with the largest eigen- value A of S. Similarly, the second largest energy corresponds u2 and so on. Therefore, given the eigendecomposition S = UAUT, the optimal orthogonal transformation L for concentrating the token norm U In this case, the energy e the transformed tokens LX aligns with the squared eigenvalues X This linear transformation is also known as the Karhunen-Loève Trans- form (KLT), which requires a representative calibration set to estimate U. Despite its optimality, KLT has the same computational complexity of full-rank matrix multiplication, which is imprac- tical since each transform needs to be applied twice for each linear layer. Estimating for each activation would further adds costly calibration step.
4

---

Published as a conference paper at ICLR 2026
KLT
10a
Transtorm L Identity
PixArt-2
Token Energy:
10c
KLT
DCT
DCT
150
10a
DWT
10a
10a
10a
10a
Token Index
10"
Transform
1
Identity
Llama 3 8B
KLT
Token Energy:
10*
DCT
12
2.
DWT
30
14
30
10a
10
10a
10a
Token Index
Component Index:
(a) Autocorrelation
(b) Energy Distribution
(c) Transform Components
Figure 3: Visualization of 3a a portion of the autocorrelation matrix, 3b transformed token en- ergy distribution, and (3c) transformation components for the input to the cross-attention layer of PixArt-E and the attention layer 20 of L.LMAAv3-8B, computed on COCO and Wikitext, respec- tively. The block structure in the LVM activations arises from flattening 2D data into a ID sequence. Both matrices exhibit Toeplitz-like diagonal structure, allowing their KLT eigenbases to be ef- ficiently approximated by DCT (Figure Bc), which concentrates the token energy close to optimal distribution (Figure 3bb. The DWT closely approximates the optimal energy with discrete levels.
Fortunately, the autocorrelation matrix of common LVM and LLM activations S exhibits a strong structure induced by the properties of natural images and text. Figure Ba shows that tokens corre- sponding to spatially or sequentially adjacent activations are highly correlated, while distant tokens weakly correlated. As result S is approximately (block) Toeplitz, whose eigenvectors can be well-approximated by a Fourier basi In particular, since the autocorrelation is real and symmetric, we can use a Discrete Cosine Transform (DCT) instead of complex Fourier basis.
The complexity of DCT (ds log s) is lower than full matrix multiplication since the transfor- mation requires only log steps of the Fast Fourier Transform algorithm. Further simplification is possible by retaining only the sign of Fourier coefficients, yielding the Walsh-Hadamard Trans- form (WHT), which approximates DCT while enabling more efficient hardware implementations.
Finally, the Discrete Wavelet Transform (DWT further reduces the computational complexity to (ds) while effectively concentrating the energy of the activations at each intermediate step. Each DWT step pushes the energy in the first half (one quarter for 2D signal) of the tokens, requiring log steps to fully concentrate the energy. Figure 3 compares energy distributions (3b) and basis compo- nents 3c for KLT, DCT, and DWT on intermediate activations of LVM and LLM architectures.
'This follows from Szegö's thcorem.
We use the Haar wavelet for simplicity and minimal padding requirements.
5

---

Published as a conference paper at ICLR 2026
Uniform
STaMP(4-8)
STaMP
Average bit width: 4.0 bits Average bit width: 4.0625 bits Average bit width: 4.0 bits
10
Distribution
Optimal:
Selected:
SQNA
10
Method
Unioomm
STaMP(0-0) STaMP*
109
10a
10a
10
10a
10
10a
107
10:
10a
10a
102
4.00
4.25
4.50
4.75
5.00
Takan index
Token Index
Taken index
Average bit width
Bit width allocation and transformation strategies
(b) width SQNR tradeoff
Figure 4: Comparison of energy and bit width allocation strategies. Figure 4a compares uni- form allocation without transformation against two STaMP strategies based on the DWT transform. Restricting to only 4- and 8-bits precision (in yellow) introduces minimal overhead while signifi- cantly improving output quality. The results in Figure 4b are obtained by varying the number of high-precision tokens and adjusting the uniform quantization bit width for per token-activation- quantized PixArt-E model with QuaRot feature transformations.
3.3 OPTIMAL BIT ALLOCATION
Given vector of energies and total maximum allocation of B bits, the optimal bit distribution follows the logarithm of the squared root of the token energy: b, = loga vei C, with C (B Ei= logg /ea) /s. In practice, we are restricted to integer bit widths [b*]. Furthermore, due to practical limitations, it is beneficial to use only small number of different bit precisions which are supported on hardware, such as 4 or 8 bits.
For this reason, although the DWT energy concentration sub-optimal, its property of creating discrete number of energy levels makes it more suitable to our use case. As illustrated in Figure 4a in yellow, we propose simple allocation scheme that uses only two bit widths: the first 64 tokens are kept at 8 bits, while the rest uses 4 bits, resulting only in minor bit width overhead (4.0625 on the PixArt-E), while significantly improving the model accuracy (Figure 4b).. For this reason, STaMP with DWT and 2 precision level will be our main focus in the experimental section.
4 RELATED WORK
Quantization is a fundamental technique for reducing the computational and memory footprint of deep neural networks, enabling efficient inference with minimal accuracy loss. The rapid growth of LLMs and LVMs has intensified interest in post training quantization (PTQ) (Nagel et al. [202]] Gholami al. 2022). retraining these models is often impractical. Recent PTQ approaches focus on removing the outliers and reducing the dynamic range of weights and activations, to improve quantization robustness.
SmoothQuant [iioo et al.) 2023) reduces activations outliers by scaling the feature channels, shifting quantization difficulty from activations to weights. QuaRot and FlatQuant apply invertible feature transforms over weights and activations to spread outliers across channels, employing randomized Hadamard (Ashkboos et al. 2024) or learning lightweight affine transforms (Sun al., 2025). van Breugel et al. (2025) develop transforms that commute with transformer operations, such that they can merged into linear weights. Zhao et al.] (2025) develops Static-Dynamie Channel Bal- ancing (SDCB) procedure based on channel scaling and mixing on Diffusion Transformer (DIT) architectures and retaining certain ientiiiaationseessstti layers to higher bitwidth to achieve W8A8. Li al. 22025) absorbs activation outliers into high-precision low-rank branch via singular value decomposition, while quantizing the residuals to b-tit with a per-- terr-ttkke/eeerreeeeeeeee ularity to achieve W4A4 mixed-precision quantization dynamic range of DiT activations by subtracting the sequence average from each token and apply- ing Hadamard feature rotations, the cost of processing extra token.
quantization gran- DIT. Federici et al. (2025) reduces the

---

Published as a conference paper at ICLR 2026
hidden state
text emmbedinng
Norm
Norm
Norm
FIGTA
WITH
Linear Layer
Quant. Lincar
10-9
10-0
up proj
Other Op.
Seq. Trans.
SDPA
SDPA
Act
Seq. Inv. Trans,
FILI
Quantization
FP Act.
pooj
Quantized Act.
1-1
attl
attm2
hidden
Figure 5: Diagram of LVM transformer block based the PixArt-E architecture. The quantization and sequence transformations operations are indicated explicitly. LLM architectures use the same quantization scheme for the alin/ and ffn blocks. Note that no transform is applied to the alIn2. Louut activations since the sequence autocorrelation does not present the block diagonal structure because of its dependency the pooled textual embedding.
While these approaches have advanced PTQ for large models, they operate exclusively along the feature dimension and ignore correlations across the sequence dimension. In contrast, STaMP intro- duces sequence-awe awaeee transformations that exploit local token correlations by using mixed preci- sion allocation strategy under fixed average budget. Our approach is strongly relates to classical media compression techniques that leverage frequency-domain transforms concentrate energy and enable adaptive quantization. JPEG (Leger et al. [991] and JPEG2000 (Christopoulos et al. 2002) rely on the Discrete Cosine Transform and Discrete Wavelet Transform, respectively, to decorrelate spatial data and allocate bits based on perceptual importance. Similar principles underlie modern video coding standards such AVC (Wiegand audio codecs like MP3 and AAC, which use the Modified Discrete Cosine Transform (Princen et al.) 1987.. STaMP demonstrates that possible to apply similar principles to the activation space of generative models, where local correlation in the sequence dimension allows energy compaction and selective precision allocation, improving quantization efficiency without retraining.
2003) and HEVC (Sullivan et al., 2012), well as
5 EXPERIMENTAL RESULTS
We demonstrate the effectiveness of STaMP on both LVM and LLM models, comparing against uniform activation quantization, and showing its effectiveness in combination with popular feature transformation and weight quantization approaches. Throughout this section, we focus on STaMP with DWT due to its computational efficiency. We refer to Figure f for visualization of how sequence transformations are applied the architectures, and to Appendix C for additional results.
5.1 VISION LANGUAGE MODELS
Set-up Our LVM experiments focus the DiT architectures of PixArt-E CCen al. 2024b) and SANA (Xie et al. 2025). We quantize activation before each linear layer in transformer blocks using asymmetric quantization with minmax scaling. As common practice, unless otherwise specified,

---

Published as a conference paper at ICLR 2026
Table STaMP consistently improves LVM quantization. Image SQNR and Image Reward (IR) for W4A4 per block quantization with block size 64. For all the STaMP results we keep 64 tokens 8-bits. STaMP consistently improves baselines.
COCO
MJHQ
SQNR
IR
SQNR
IR
STaMP
X
X
x
X
FP
+00
0.90
+00
0.96
PixArt-E
RTN
5,88
6.16
0.38
0.80
5.75
6.23
0.38
0.76
ViDiT-Q
7.82
6.37
0.83
0.84
7.55
8.53
0.76
0.86
SVDQuant
8.78
9.72
0.90
0.91
8.83
9.75
0.86
0.89
+00
0.87
+00
0.97
SANA
RTN
8.63
9.32
0.89
0.91
8.56
9.40
0.95
0.99
ViDiT-Q
10.03
10.74
0.89
0.86
10.04
10.69
0.96
0.97
SVDQuant
9.99
10.69
0.87
0.90
9.88
10.51
0.93
0.98
RTN
ViDiT-Q
SVDQuant
FP32
RTN + STaMP
ViDiT-Q + STaMP
SVDQuant + STaMP
"a cute little dog nugget looking up at the stars in the sky, filled with hope and determination." "
Figure 6: Visualization of PixArt-S, sample generation for the results reported in Table 1
separate scale s; for each token and weight output channel. Following common procedure in literature, in LVMs we do not quantize activations and weights corresponding to the cross-attentee key and value since their effect accounts for less than 5% of the runtime (Li al. 2025).
We evaluate the fidelity of the quantized LVMs by computing the Signal to Quantized Noise Ratio
SQNR(O, 10 1og
| / o 0 both in the diffusion latent space and image space.
We further compute CLIP Score (Hessel et al. 2021) to assess alignment with the textual prompt, CLIP IQA (Wang et al. 2023). and Image Reward (Xu et al. 2023) to evaluate the generated image quality. Following standard procedures, we compute the metrics using 1000 prompts and images from the COCO (Lin al. 2014) and MJHQ (Li al. 2024) datasets.
We demonstrate the effectiveness STaMP alone and by combining it with other quantization meth- ods developed recent literature. We combine STaMP with feature transform methods such as SmoothQuant (Xiao et al. 2023) and QuaRot (Ashkboos al.) (024), and Static-Dynamic Channel Balancing (SDCB) described in ViDiT-Q (Zhao al. (025). We further evaluate STaMP with the mixed precision low-rank weight quantization method described in SVDQuant (Li et al. 2025). We apply STaMP before each linear layer in the Transformer blocks, inverting it right after each linear layer. We use 2-dimensional DWT with 64 8-bit (high precision) tokens unless otherwise specified.
8

---

Published as a conference paper at ICLR 2026
Table STaMP always improves LLM quantization. Perplexity (PPL) for W4A4KV4 quantiza- tion, using the same setting as (Ashkboos et al. 2024 Sun et al. 2025). We use 64 8-bit tokens for activations and KV-cache for all methods and baselines, even if we do not apply the sequence trans- form (effectively W4A4. 125KV4.125) The STaMP sequence transform improves all baselines.
Llama 3 8B X
Llama 3.2 IB it
Llama 3.2 3B it
Qwen 2.5 3B it
STaMP
x /
X
/
X
FP
6.14
13.
11.27
8,56
RTN
668
95.3
1795
700
483
159
99723
18767
Smooth Guant Quant
531
93.8
883
407
177
88.5
66929
29063
Quarot
9.05
8.66
25.78
23.72
18.43
17.57
94.86
71.13
FlatQuant
6.89
6.77
15.72
15.16
12.71
12.40
9.29
9.19
Results Table I reports the results obtained by combining STaMP with recent LVM quantization method using the same quantization setting for all the baselines. Both activation and weights quantized at bits with blocks size 64, following the setup described in (Li et al. (025). We observe that STaMP consistently improves upon all the reported metrics different models and architectures resulting in visually more accurate generations, as shown in Figure
5.2 NUMBER OF HIGH PRECISION TOKENS
(beure44ddemonssee demonsttrsss the trade-off between bit width and SQNR observed by changing the number of high-precision tokens in STaMP, while fixing the high and low precision bit widths to 8 and 4 bits respectively. Only activations are quantized to focus the analysis solely the activation quantization error. We observe a sharp increase in SQNR whenever even few high precision tokens are introduced. Even in the bits regime, STaMP achieves better performance uniform quantization. Additional comparison with per- blo-klock activation quantization can be found in Appendix C
5.3
LARGE LANGUAGE MODELS
Set-up We evaluate on language models of different sizes and model classes, including Llama 8B (Grattafiori et al. 2024), Llama 3.2 IB and 3B instruct, and Qwen 2.5 3B instruct (Qwen et al.
2025). We take popular feature transforms from literature, including SmoothQuant (Xiao et al. 2023). QuaRot (Ashkboos et al.) 2024), and the state-of-the-art FlatQuant (Sun et al.) 2025). and evaluate whether STaMP (DWT) brings additional gains these baselines. We use the exact same quantization set-up used in (Ashkboos et al. 2024: Sun et al. 2025), W4A4KV4 per token acti- vation quantization, and like them, evaluate Wikitext-2 (Merity et al. 2017) perplexity at sequence length 2048. We use round-to-neares, (RTN) for weight quantization (vs. GPTQ (Frantar et al.) p. as weight quantization keep the first 64 tokens in 8 bits for all baselines, which means all methods use effective activation/KV bit width of 4.125 bits. Note that, on L.Mss, STaMP can be effectively applied only for the prompt-processing phase, since during token generation, only activation available the time. Despite this limitation, STaMP remains useful for prompt processing, which is typically compute-bound (Agrawal et al. 2023 Chen al. 2024a Kamath et al.) 2025), reducing the
completely perpendicular to sequence transforms. In this setup,
activation size lowers total compute and latency. See Appendix B.2 for additional details.
Results We observe (Table 2) that all baselines improve consistently when STaMP is added. This is especially apparent for scenarios where baselines are far from FP performance- e.g. for the small Llama 3.2 IB and 3B instruct models, which are evidently hard to quantize at bit. This demon- strates that STaMP is not competitor of other quantization techniques (e.g. feature transforms), but an additional tool for achieving extremely low bit width guantization which can be added without any manual tuning training to existing quantization methods. Appendix C includes additional evaluations on few-shot reasoning tasks, which show the same trend as the perplexity evaluation, with STaMP consistently increasing accuracy.

---

Published as a conference paper at ICLR 2026
SQNR (image)
Image Reward
Wiki-2 Perplexity
Feature Transform
12.50
SDCB
0:83
0.96
0.95
0.97
FlatQuuant
0.33
QuaRot
7.07
11.87
11.08
11.47
QuaRot
0.71
0.94
0.96
0.92
QuaRcc
6.90
6.68
6.68
6.09
SmocthhQuart
5.54
6.40
SmoothQuant
-1.87
Smoothiquart
Identity
0.36
5.65
5.11
5.00
Ideetitty
2.20
0.12
-0.99
0.99
Idenkity
111.00
13.53
13. 46
22.53
Identity DCT
WHT
DWT
Identity
DCT
DWT
Ideniityy
DWT Sequence Transform
DCT
WHT
Sequence Transform
Sequence Transform
(a) A4 P SQNR and Image Reward
(b) A4 Llama v3 SB Perplexity
Figure 7: Effect of combining Feature Transforms (rows) and STaMP (columns) with A4 on the PixArt-E and LLama v3 8B models.
5.4 COMBINING FRATURE AND SEQUENCE TRANSFORMATIONS
In Figure we assess the effectivenes of STaMP for three different sequence transformations in combination with popular feature transforms. Overall, we observe that improvements are largely complementary, especially for LVMs. Furthermore, results show that DCT, WHT and DWT perform similarly to each other, demonstrating that there is little price to pay for switching from the more accurate and computationally expensive DCT to the cheaper approximate DWT.
5.5 OVERHEAD ESTIMATES
Table 3 reports that the theoretical compute overhead and latency of CUDA for latent de- noising step with STaMP (DWT) is compara- ble to Hadamard transforms on the features, ac- counting for less than 5% of the total runtime. The DWT transform is applied in three levels, consistently with the results reported in Table and Figure 4a using a specialized CUDA ker- nel that considers the memory layout of the ac- tivation tensors. Appendix B.3 includes fur- ther details on the benchmarking procedure. The minimal overhead in the number of float- ing point operations (flops) suggests that the tency overhead can be further reduced with bet- ter optimized kernels or specialized hardware.
Table 3: Overhead in terms of latency on CUDA and extra flops for single PixArt-E denoising step. STaMP with DWT has little impact on the latency and number of floating point operations.
Transformation
Overhead [%] flops
Feature
Sequence
CUDA
Hadamard
0.24
3.0
Hadamard
0.69
56.8
DWT
0.21
4.8
Hadamard
DWT
0.44
7.8
6 CONCLUSION
This work explores uncharted territories in the recent advancements of LLM and LVM quantization, applying invertible transformations over the sequence dimension further reduce quantization error. Drawing inspiration from traditional signal processing, introduce a novel method that exploits the autocorrelation structure that naturally present in the intermediate activations of generative models to enable more efficient mixed-precisio activation quantization scheme.
We demonstrate that STaMP complements existing PTQ techniques such as SmoothQuant and QuaRot, and plays critical role in advancing oo-precision activation quantization, pushing the boundaries of W4A4 quantization for both LLMs and LVMs and offering training solution for deploying hiff-peeformance models in resource-constrained environments.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Images were generated using the PixArt-E model licensed under Apache 2.0 and SANA model licensed under Apache 2.0

---

Published as a conference paper at ICLR 2026
REFERENCES
Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ra- machandran Ramjee. Sarathi: Efficient Ilm inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308. 16369, 2023.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. In Advances in Neural Information Processing Systems 38: Annual Conference Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http: httpt:::/ppaar.... . nips cc paper_ www.efilllspppeeeee httpleeeppppp////// htm.
Hao Mark Chen, Fuwen Tan, Alexandros Kouris, Royson Lee, Hongxiang Fan, and Stylianos I Venieris. Progressive mixed-precision decoding for efficient Ilm inference. arXiv preprint vrXiv:2410.13461, 2024a.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-E: Weak Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In Computer Vision ECCV 2024 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXXII, volume 15090 Lecture Notes in Computer Science, pp. 74-91. Springer, 2024b. doi: 10. 10.1007/978-3-031-7755 URL wwtphhtps:/eee /ttp://doi.ie org http:///oo..ggg//...o 1007 1087/978777711 3- 001117777---0011111133---3
Charilaos Christopoulos, Athanassios Skodras, and Touradj Ebrahimi. The jpeg2000 still image coding system: overview. IEEE transactions consumer electronics, 46(4): 103 127,, 2002.
Marco Federici, Riccardo Del Chiaro. Boris van Breugel, Paul Whatmough, and Markus Nagel. Hadanorm: Diffusion transformer quantization through mean-centered transformations. arXiv preprint (rXXi:2500609932, 2025.
Fino and Algazi. Unified matrix treatment of the fast halsm-d transform. IEEE Transactions on Computers, 100(11):11) 1146, 1976.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17322 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL htttttttp//eeeeddee wrw.:/oordd.........oooo...o.... 12608602]
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-power computer vision, pp. 291-326. Chapman and Hall/CRC, 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprini 24Xi:22407.217888 2024.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When Attention Sink Emerges in Language Models: An Empirical View. In International Conference on Learning Representationss arXiv. March 2025. doi: 10,48550Varxiv.4410.rr
URL
abs/ 10781 arXiv:2410.10781 [cs].
Yipin Guo, Yilin Lang. and Qinyuan Ren. GPTQT: quantize large language models twice to push the efficiency. In IEEE International Conference on Cybernetics and Intelligent Sys- tems, CIS 2024, and IEEE International Conference on Robotics, Automation and Mechatron- ics. RAM 2024, Hangzhou, China, August 8-11, 2024, pp. 368 373. IEEE, 2024. doi: 10. 109/ CIS 2...RAA619999.2221667777777 2024 10672819
URL https: doi org / 1109/C19 SAA.9//ISSRRAAM61911 RAM619366999.
12

---

Published as a conference paper at ICLR 2026
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A fference-ffree evaluation metric for image captioning. In Proceedings of the 2021 Confer-
Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp- 7514-7528. Association for Com- URL https:
putational Linguistics, 2021. doi:
FBGGS////22011EMMMPPPPOO emnip mimann.995
Aditya K Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. Pod-attention: Unlocking full prefill-decode overlap for faster Ilm inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Program- ming Languages and Operating Systems, Volume 2, pp. 897-912, 2025.
Alain M Leger, Takao Omachi, and Gregory Wallace. Jpeg still picture compression algorithm. Optical Engineering, 30(7))947-955 1991.
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation. CoRR, abs/2402. 17245, 2024. doi: 10445500AAXXIV22422.1222 URL httth::t///det http:::////ii..//1.... org/10.
48550/arX 2402 2402..2245
17245
Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun Yan Zhu, and Song Han. Svdquant: Absorbing outliers by low-rank component for 4-bit diffusion models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https:ttp///.mm..... /ttpttts//ppeereriiiee /wwp//veenreee.eeeeoee net/forum? d=vwR333U Qui
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014, pp. 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3- 319-10602-1.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: LLM quanti- zation with learned rotations. CoRR, abs/2405. 16406, 2024, doi: 10.48550/ARXIV:2405. 16406. 48550/8504000 arwiv. 2405 .12.4.1...466 16406
URL
Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang. Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, net form/forrmmdd
May 7-11, 2024. OpenReview.net, 2024. URL of2rhA1q81
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. CoRR, abs/2106.08295,, 2021. URL/ttpps: htttttss///xxiv.orpgp//oo http:v://arxvvvvv wwtp/////////////o///.///e/eeeea aaaaaabss.... /wt...21.... 08295
J. Princen, A. Johnson, and A. Bradley. Subband/transform coding using filter bank designs based on time domain aliasing cancellation. In ICASSP '87. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 12, pp. 2161-2164 1987. doi: 10. 10.1109/ICASSPP11997 1169405.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu. Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia. Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL htlpstt::: / xx v. orgg abs/ 2412. 15115
13

---

Published as a conference paper at ICLR 2026
Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. IEEE Transactions on Circuits and Systems for Video Technology, 22(12):1649-15) 1668, 2012. doi: 10.1 10.1199/TCSVT222122222111
Mingjie Sun, Xinlei Chen, Zico Kolter, and Zhuang Liu. Massive Activations in Large Language Models. In First Conference on Language Modeling (COML), arXiv, August 2024. doi: 10.48550/ arXiv.2402.177 17762. URL http: aawiiiiiiivv...... org/ abs/ http//bb///222///// 17762 arXiv:440.117662 [cs] version: 2.
Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, JiaxinHu, Xianzhi Yu, Lu Hou, Chun Yuan, Xin Jiang, Wulong Liu, and Jun Yao. Flatquant: Flatness matters for LLM quantization. In Ferty- International Conference Machine Learning, 2025. URL htttps://opezzeview net/ forum? 2iu
Boris van Breugel, Yelysei Bondarenko, Paul Whatmough, and Markus Nagel. Fptquant: Function- preserving transforms for IIm quantization. arXiv preprini urXiv:2506.04985 2025.
Jianyi Wang, Kelvin C. K. Chan, and Chen Change Loy. Exploring CLIP for assessing the look and feel of images. In Thirty-Sevente AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty- Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Sym- posium Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pp. 2565-2563, AAAI Press, 2023. doi: 10. 10.1609/AAALV331122555.
URL https: / /tt::://doo.oooo org http:///i..oggg...../ www.::11.1100aa...... aaai. ialaai.v37 v3712 23312.223333255
T. Wiegand, G.J. Sullivan, G. Bjontegaard, and A. Luthra. Overview of the h.264/ave video coding standard. IEEE Transactions on Circuits and Systems for Video Technology. 13(7):560-576, 2003. doi: 10. 10.1100/CSVT..200.1555..
Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, vol- ume 202 of Proceedings of Machine Learning Research, pp. 38087-38099, PMLR, 2023. URL hkttps:/prroceedingg www.mmrrrr..ep...mmmmmmmmmmmel.e /ttpp///e/////e//////////222ee2e v202/ xiao23c......................e.. 2htmml1
mlr
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient Streaming Language Models with Attention Sinks. In International Conference of Learning Representa- fions, April 2024. doi: 10.48350/arxiv.2309. 17453. URL httt::///axxiiii wttpl///rggbbbb/////b////// 2309.
17453 13X4v:2309.17553 [cs].
Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. SANA: efficient high-resolution text-to-image syn- thesis with linear diffusion transformers. ing Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL //tpr////peeeeeiee net/ forum? wwouumidd-eee id N80 N80
The Thirteenth International Conference on Learn-
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yux- iao Dong. Imagereward: Learning and evaluating human preferences for text-to-image genera- tion. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023. URL htL:: http:://aappss.ee n nips. wttpa///wtt////ap.............o.
httpee/e//11/pppp//////////// html]
Tianchen Zhao, Tongcheng Fang, Haofeng Huang, Rui Wan, Widyadewi Soedarmadji, Enshu Liu, Shiyao Li, Zinan Lin, Guohao Dai. Shengen Yan. Huazhong Yang. Xuefei Ning, and Yu Wang. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video gen- eration. In The Thirteenth International Conference on Learning Representations, 2025. URL https::/cpperevll reteeeret/eee.....e htt.://eettoutttttt/eeeem forum?
14

---

Published as a conference paper at ICLR 2026
A PROOFS
A.1 PROOF FOR THEOREM
1
We prove the result reported in Theorem I in three steps. First, demonstrate that the quantization eπror for an orthogonal sequence transformation is the same as the quantization eπrоr on the sequence transformed inputs:
C (X; (LX)
(10)
Proof. This simply follows from the invariance of the Frobenious norm to orthogonal transforma- tions.
(X: (Lx) x12
| L LX
(Q(L)) - -XX)
-IX(LX)- -
= -XXX)
Secondly, for completeness, we prove the upper-bound reported in Equation scheme with minmax scales shared across all feature channels Stz = 3/;
For quantization
range (ar:)
(xx))= I
(1I)
4
(2 1)²
Proof.
C (a,)
E Q (a::)
+
2/
3
E
24
24
E
E
3,
E
range (ax;) 27
EE
(2b:
1)
Lastly, we bound the range using the norm 2:
range (zx)) < 2 2
(12)
Equality is attained whenever à; consists of two non-zero entries - and v, for which range (a) 2v and ||æ]|| = 22..
=
Using this three steps write the proof for Theorem
15

---

Published as a conference paper at ICLR 2026
Proof.
(x; L)
10
(LX)
(1,x)
range (l 2
(2% 1
E
||,XX 1
12
d
2
(2
1)²
A.2 OPTIMAL BIT WIDTH ALLOCATION
We determine the optimal bit width allocation b" for energy vector by determining the bit width
for which the ratio eg/220; = (2 1)2 is constant for all tokens:
k
(13)
2i=1 bj B
log2 C₁ logg
6
(14)
2
log2 Gi
loog k
B
(15)
1 Σ
2B
loga
k
log2 C₁
(16)
1=1
log2 C2
Li=1 log2 es + 2B
bf
(17)
2
B Σ i= log2 Vei
bj logz Vei-
+
(18)
A.3 EFFECTIVENESS OF ENERGY CONCENTRATION
We demonstrate the effectiveness of the proposed Sequence Transform and Mixed precision scheme by considering the upper bound in Equation 8 in two different settings. In order to simplify compu- tation, we will consider 2 negligible for practical values of b,.
et_ instead of 12y since the difference between the two quantities is
1. Uniform Energy:
In this scenario we have e = E/s and b, = B/s. We note that the energy for each token is equal to the average of the squared eigenvalues X3 of S:
1
e = E/s Trace(S)/:
x2
(19)
8
Therefore:
d 5
e
X²
ds
210g,
2B/s//s
(20)
2
226
22B/#
2
i=1
2. Maximum Energy Concentration:
We consider scenario in which the energy corresponds with the squared eigenvalues e;
16

---

Published as a conference paper at ICLR 2026
X The optimal bit allocation is given by Equation 18
B
2 loge A
bg llog A; +
(21)
ToRa
Therefore:
d
Σ i=1
C 22b.
Σ 1=1
A
ds 210g;
22-2
(22)
2
21og x,+ 2log,
2
Comparing the two results is equivalent to comparing log, X2 (uniform) and log2 X (max concen- tration). Using Jensen's 's inequality and the convexity of log2. we have:
loga X Ellog2 X?1
log B[A
log2 A/
(23)
Therefore, the Maximum Energy concentration strategies achieves lower value than the uniform scheme.
17

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL DETAILS
B.1 LVMs
Baselines We implement SmoothQuant, QuaRot, ViDiT-Q and SVDQuant closely following the details reported in the paper and the respective codebases. Specifically, for ViDiT-Q (Zhao et 2025) we use = 0.01, as reported in their PixArt-E setup. For SVDQuant and SuoothQuant we use a default value of = 0.5.
al.
Quantization In order to promote a fair comparison with the other models, for the results reported in T instead of using the 8-bits weight quatization scheme described in ViDiTT-Q (8 bits for the FFN blocks, first and last transformer blocks), we use per-block weight and activation quantization, as described in SVDQuant (Li al., 2025), which improves the ViDiT-Q results. In line with the SVDQuant methodology, we retain the depth-wise convolutions within the feed-forward layers of the SANA transformer blocks in full precision. Meanwhile, the two point-wise convolutions are quantized by treating them as linear layers. Notably, unlike the original SVDQuant implementation, our experiments do not incorporate GPTQ at any stage. The diivation-quantization only results reported in Figure44ban and Figure Juse bits per token quantization.
STaMP For the STaMP results, excluding the specific ablation study, 64 high-precision tokens, which results in an effective activation bit-width of 4.0625 bits on the PixAnt-E model and 4.125 bits the SANA model. Non-STaMP result do not use any mixed precision tokens.
SQNR Our experimental results report the average value of SQNR in decibels (dB) for each im- age. In other words, for N 2-dimensional signals O.,..., ON and their quantized counterparts O ÖN consisting of tokens (pixels) and f features, we compute:
.......,
E E of
SQN
10
Σ log10
10 N
log10
|0 10,
E
E ous)²
1
The SQNR of each image is averaged in log space to mitigate the effect of extreme outliers.
B.2 LLMS
Quantization We use dynamic quantization for the KV cache and activations. Quantization scales and offsets are determined per token, sequence, and (for KV) head. We follow (Ashkboos al.) 2024 Sun et al. 2025)) and only quantize weights, KV cache, and inputs to linear layers within the transformer block. We use coundto-nneares weight quantization, as weights are unaffected by "TaMP-mort advanced weight quantization schemes could improve results further, but this is per- pendicular our contribution. we range set the weights by computing the weight quantization squared error for grid of candidate ranges and selecting the candidate with lowest error. For fair- ness, for all experiments (inel. baselines) we keep the first 64 tokens in bits, which gives an effective bit width of 4.5 for both STaMP and Non-STaMP results.
Baselines For Smoothhuant we calibrate the scales based the Wikitext-2 training dataset acti- vations. For QuaRot, we follow the original paper and reduce the minmax activation range by 10%. For FlatQuant, we use their recommended settings for all models (e.g. training for 15 epochs on 128 Wikitext-2 training sequences).
The first token in most LLMs acts an attention sink Xiao al. (2024): Gu et al. 20255) and typically contains massive outliers Sun et al. (004)). Keeping the first token in 8 bits helps to accurately represent these massive outhiers. To ensure the massive outlier stays in the first token, STaMP is not applied to the first token. Note that the baselines that do not use STaMP's transform, do benefit from keeping the first token (and next 63) in high-precision in experiments.
STaMP
18

---

Published as a conference paper at ICLR 2026
M=4096, k=1152, N=1152 to wttg///o/ktoo.. s/too_ttooooooooee out
M=4096, K=1152, N=3456
to_q/to k/to (Fused)
0.00035
(tarch)
0.0012
(torch)
(torch) HP (cuBbos)
(tarch)
HP rcutless)
0.00030
ir44 [cutlass)
114 (cuclass)
0.0008
0.00020
Timing
Timing
0.0006
0.00015
0.00010
0.0002
0.00005
Batch Size
Batch Size
M=4096, K=4608, N=1152 down_proj
M=4096, K=1152, N=4608 up_proj
0,0016
0.00175
(cutlass)
0.0014
HP (cutlass)
0.00150
[ruelees]
(cuelees)
0.0012
0.00125
0.0010
0.00100
Timing
Timing
3.0008
0.00075
0.00050
0.00025
0,0002
0.00000
Batch Size
Batch Size
Figure 8: Visualization of runtime estimation in terms of latency on CUDA for running matrix multiplication kernels with different data types, shapes, and batch sizes averaged over 1000 runs. The proposed mixed precision approach has a minimal overhead that does not depend on the batch size.
B.3 NOTES ON THE RUNTIME ESTIMATION
B.3.1 DWT OVERHEAD
To compute the overhead associated with the sequence and feature transforms, we measured the time required to run single denoising step with the original PixArt- model, and compared it with the modified architectures (having the extra transform operations). Measurements are performed on an A100-SXM4-80GB GPU using python 3.10.12 12, PyTorch 2.5.1+cu122) and CUDA drivers 12.5. Hadamard transforms are based on the CUDA accellracee kernels from the built specialized CUDA kernel, optimized for applying DWT over the sequence dimension. The increased latency measured with the Hadamard applied on the sequence dimension can be mainly attributed to the memory reshaping operations required to use the fast- hadamard modammard-transfom kernel.
sonddamard-transffonn
P package For Haar DWT operations,
B.3.2 MIXED-PRECISION MATRIX MULTIPLICATION
When using STaMP, the first tokens of the activation tensors are stored at a higher bit width with respect to the subsequent ones and bits in our experimental settings). This implies that the subsequent linear operation should perform part of the computation with higher precision. The number extra operations for the bit part is minimal (about 1.56% on PixArt-Sigma).
It is important to note that STaMP 4-8 bits mixed precision scheme drastically reduces the number of binary operations when compared to in18 matrix multiplications. However, the latency measure-
hhaps///gttuu..co//aooo/4 htw.a//o--lllaaaaaaaaaaaaaaaaraa
19

---

Published as a conference paper at ICLR 2026
ments are heavily influenced by the support for specific data types and operations in the hardware and software stack of the target device. Demonstrating the effectiveness of im4 quantization is an important step towards developing support for more power-efficient solutions in commercial-graen -grade hardware.
We assessed the effectiveness of the proposed mixed precision design by measuring the time to perform a mixed precision matrix multiplication on CUDA, and we compared to performing the same operation with fpl6, int8 in14 data types. When using Nvidia hardware, one can take advantage of Tensor Cores to accelerate low-precision operations and allow higher computational throughput. PyTorch uses Tensor Cores kernels for most of the low-precision data types (such as rf32, fp/6, bfl6, int8), but the Py PyTorch stack lacks an int4 accelerated kemnel. Following the procedure from Sun et al. (2025) we used Cutlass-basse germm kernel with minor modifications to the python bindings to enable inplace operations.
In Figure we report the outcome of our benchmakrs for different batch sizes and for the tensor shapes used in the Pixart-E architecture. The measured overhead for the mixed-precision matrix multiplication over the in14 one for batch-size (required for generating a single output image) vary from about 30% (for smaller shapes) to approximately 10% (for fused Q/K/V computation and up projection). Increasing the batch size to 8 reduces the relative overhead to under 10% on all layers.
One should consider that Mixed- Precision implementation essentially runs two matmuls: one using the custom cutlass kernel over the int4 data souble-packed into an int8 tensor, the other using the standard torch iut8 kernel the first 64 tokens and writing the results the first 64 tokens of the output tensor. For batch size 2, we also perform an additional inr8 operation over the first 64 tokens of the second sample. For batch sizes 4 and larger we slice the tensors retrieve the first 64 tokens of each batch element and pack into contiguous memory before running a single int8 matmul. Additional memory transfer optimization and kernel fusion could further decrease the overhead introduced by the STaMP operations.
C ADDITIONAL RESULTS
C.I LVMS
We report additional results for the setups described in Section including ablation of the effect of STaMP on different activations (Table bit width vs SQNR comparison against per-block acti- vation quantization (Figure TO). and additional metrics (Table 5 and visualizations (Figure for the results reported in the main text.
Table 4: Effect of A4 activation quantization on single activations of the Pixart-E model on the Image SQNR. Fach entry corresponds model for which only the corresponding input activation is quantized. Note that STaMP has little to no effect on att22.to since the structure is determined by pooled textual embedding, which does not present the same correlation structure as the other activations in the network.
Transform atml attll.to.om attn2.to-g attn2.to_out ffn.up-proj ffn.down.pro)
Identity 6.01
10.09
0.40
15.48
1.15
4.92
QuaRot
13.80
16.01
16.29
23.20
11.36
8.42
STaMP
7.68
13.89
11.60
15,49
8.75
9.61
QuaRot-STaMP
15.28
16.78
18.31
23.23
13.84
14.91
C.2 LLMS
We report additional LLM evaluation involving common few-shots reasoning tasks in Table The evaluation is performed using the an-eval-harness toolkit (Gao et al. 2024).
20

---

Published as a conference paper at ICLR 2026
"a cat that has shirt on its back"
Sequence: Identity
Sequence: DCT
Sequence: WHT
Sequence: DWT
Feature: Identity
Feature: SmoothQuant
Feature: QuaRot
Feature: SDCB
Figure 9: Visualization of the Images generated with A4 activation quantization and several com- bination of Feature and Sequence transforms for the Pixart-Y architecture. The images refer to the same setup described in Figure 7
D USE OF LLMS
Large Language Models have been used only for sentence rephrasing and grammar check, which resulted in minor alterations the paper text.
21

---

Published as a conference paper at ICLR 2026
Quantization Scheme Comparison
Per-Token Bits SONR (image): 7.28
Per-Token Bits SQNR (image): 11.37
13
SQNR (Image)
11
10
Method
9
pt
8
pb
pt+ STaMP(48)
7
4.00
4.25
4.50
4.75
5.00
Average bit width
Per-Block: 1024
Per-Block: 256
Per-Block: 64 4.25 Bits
Per-Block: 16 5.0 Bits SQNR (image): 12.0
4.015625 Bits SQNR (image): 8.72
4.0625 Bits
SQNR (image): 10.27
SQNR (image): 10.9
High Prec. Tokens 4.0009765625 Bits SONR (image): 9.43
64 High Prec. Tokens 4.0625 Bits SQNR (image): 11.8
High Prec. Tokens 4.25 Bits SQNR (image): 12.51
1024 High Prec. Tokens 5.0 Bits SONR (image): 13.43
Figure 10: Tradeoff between SQNR and bit width for per-token activation quantization (pt) per-block activation quantization (pb) at different block size (from 16 to 1024), and per-token with STaMP (pb+STaMP) varying number high precision tokens on the PixArt-E model. We consider 16 bits scales for each scale parameter. The visualization correspond to the Prompt 'A guy with backpack looking at the ground to his left."
22

---

Published as a conference paper at ICLR 2026
FP32
RTN
ViDiT-Q
SVDQuant
RTN +STaMP
ViDiT-Q +STMMP
SVDQuant +STaMP
"portrait of an army of wild mountain hures are rouming the streets of central Stockholm, summertine, warm ans sunny, sunset life style stock image popular no text prompt trend. pinterest contest winner"
Figure 11: Visualization of sample generation for the SANA for the results reported in Table 1
23

---

Published as a conference paper at ICLR 2026
Table 5: Additional metrics for the experiments reported in Table
Dataset
LVM
Method
STaMP
CLIP
CLIP IQA
SQNR (latent)
FP
31.54
0.91
too
X
31.23
0.79
0.19
RTN
31.88
0.91
0.49
PixArt-
X
31.68
0.91
0.96
SVDQuant
/
31.76
0.91
1.47
X
31.63
0.87
0.73
ViDIT-Q
V
31.92
0.87
1.03
COCO
FP
31.85
0.89
+oo
X
31.85
0.89
3.39
RTN
31.94
0.89
4.09
SANA
X
31.83
0.90
4.61
SVDQuant
/
31.88
0.90
5.30
X
31.79
0.89
4.63
ViDiT-Q
V
31.82
0.89
5.42
FP
31.46
0.85
+00
X
29.90
0.75
0.28
RTN
30.96
0.87
0.71
PixArt-
X
31.13
0.86
1.27
SVDQuant
/
31.28
0.87
1.86
X
30.97
0.81
0.86
ViDiT-Q
V
31.20
0.83
1.31
MJHQ
FP
31.57
0.83
+o0
X
31.51
0.84
4.45
RTN
/
31.60
0.84
5.39
SANA
X
31.52
0.83
5.69
SVDQuant
/
31.55
0.83
6.46
X
31.60
0.83
5.76
ViDIT-Q
31.60
0.83
6.65
24

---

Published as a conference paper at ICLR 2026
Table 6: L.M evaluation Common Reasoning tasks. The experimental setup is equivalent the one described in Table
Llama
Llama 3.2 1B it
Llama 3.2 3B it
Qwen 2.5
Task
STaMP
x
X
X
X
V
FP
54,4
36.1
46.4
56.0
RTN
43.9
27.9
28.4
35.7
35.9
36.6
37.0
are challenge
SmoothQuant
40.3
40.2
30.5
30.5
34.5
34.6
39.2
39.3
QuaRot
39.0
38.7
30.5
30.5
37.4
37.1
42.8
43.2
FlatQuant
52.0
52.5
33.9
34.2
44.4
45.5
52.0
54.0
FP
84.2
70.7
79.7
83.5
RTN
75.2
75.8
56.2
56.3
63.6
63.8
66.7
66.8
are easy
SmoothQuant
71.4
72.1
61.7
62.4
64.5
65.1
69.0
69.3
QuaRot
71.5
71.3
58.2
58.3
66.5
66.5
73.5
73.6
FlatQuant
81.3
82.4
67.6
67.8
77.5
77.7
82.2
82.4
FP
61.3
44.3
52.8
56.2
RTN
39.2
50.2
31.0
34,0
37.2
41.9
32.4
33.2
hellaswag
SmoothQuant
39.1
47.4
33.3
36.5
40.1
45.3
33.3
33.8
QuaRot
49.5
51.2
37.9
38.9
45.9
46.9
33.9
FlatQuant
58.7
59.7
41.2
41.9
50.4
50.8
54.5
FP
63.3
RIN
0.2
1.2
0.3
0.9
0.7
3.6
0.0
0.0
lambada
SmoothQuant
0.5
2.7
0.6
1.6
2.4
9.0
0.0
0.0
QuaRot
15.1
24.6
16.4
20.6
37.6
40.6
0.0
0.0
FlatQuant
67.1
67.1
44.1
48.3
58.1
58.9
55.0
57.5
FP
81.1
74.0
77.3
78.0
RTN
75.4
76.5
66.2
66.5
69.7
70.3
71.5
71.6
piqa
SmoothQuant
74.1
74.3
68.7
68.8
71.6
72.0
72.1
72.3
QuaRot
74.2
73.9
68.9
68.4
70.9
70.7
73.1
72.9
FlatQuant
79.4
80.2
72.1
73.0
76.1
75.7
77.5
77.6
FP
77.5
62.0
70.6
70.6
RTN
71.0
71.0
55.2
55.2
63.1
63.1
62.0
62.0
winogrande
SmoothQuant
70.8
70.8
54.9
54.9
64.8
64.8
65.6
65.6
QuaRot
70.8
70.8
57.6
57.6
62.6
62.6
65.9
65.9
FlatQuant
75.2
74,9
58.6
60.1
66.4
68.8
66.6
68.1
FP
69.7
55.7
63.8
65.5
RTN
50.8
53.1
39.4
40.2
45.0
46.5
44.9
45.1
Average
SmoothQuant
49.4
51.2
41.6
42.5
46.3
48.5
46.5
46.7
QuaRot
53.3
55.1
44.9
45.7
53.5
54.1
48.2
48.2
FlatQuant
67.4
67.9
50.9
52.3
60.7
61.5
62.7
63.9
25


Reviews:
[{"review_id": "o9jjZagCK8", "reviewer": "Reviewer_F9G6", "summary": "1. The paper proposes STaMP quantization to mitigate accuracy loss in low-precision activation quantization for LLMs and LVMs.\n2. Unlike existing feature-wise methods, STaMP applies linear transformations along the sequence dimension to exploit local correlations in language and visual data.\n3. It uses a mixed-precision scheme: a small number of high-energy tokens are kept at higher precision to maintain accuracy with lower average bit-widths.\n4. STaMP complements established feature transformations and weight quantization methods without conflicting with them.\n5. Experiments on models like Llama 3 and PixArt-Σ demonstrate consistent accuracy improvements in 4-bit activation quantization.\n6. For efficiency, STaMP adopts practical transforms instead of complex KLT, balancing energy concentration and computational cost.", "questions": "See Weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is really well-written in both figures, tabs, equations, and texts. \n2. The paper contains experiments on both SD and LLM. \n3. The method design features strong complementarity and is compatible with the existing technical system.", "weaknesses": "1. I a m confused  with the argument in QuaRot that rotation only occurs in the feature dimension, because some weights (such as up/gate projection) are left-multiplied by the rotation matrix.\n2. The experiments on Large Language Models is insufficient: (1) The model size is relatively small, and it has not been scaled up to the magnitude greater than 8B; (2) There are only PPL experiments, with the lack of mainstream zero-shot reasoning experiments, which are more crucial for verifying the effectiveness of PTQ.\n3. The paper introduces left-invertible matrices, but it does not provide a detailed and clear explanation of how such matrices are integrated into the weight parameters of the original architecture (like QuaRot and SpinQuant).\n4. As a work focusing on quantization, the paper does not report experimental results related to accelerated inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oQdn7LFvxa", "reviewer": "Reviewer_jfMb", "summary": "This paper proposes Sequence Transformation and Mixed Precision(STaMP), a new PTQ method that operates along the sequence dimension, different from most recent methods that focus on utilizing rotation to modify along the feature dimension. The key idea is to exploit local correlation between sequential tokens by applying a sequence-wise linear transform (e.g., DCT, DWT) before quantization, followed by a mixed-precision strategy that assigns higher bit-widths to a subset of tokens carrying higher \"energy.\"", "questions": "See Above", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. New Perspective: The paper introduces quantization transformations along the sequence dimension—a novel and orthogonal approach compared to existing methods. This direction effectively exploits temporal or spatial token correlations often overlooked by prior research.\n\n2. Solid Theoretical Analysis: The work offers a well-defined mathematical treatment of quantization error and establishes a theoretical upper bound (Theorem 1) that connects token energy, bit allocation, and resulting error.\n\n3. Synergistic with Existing Methods: The proposed approach integrates seamlessly with leading quantization strategies (e.g., QuaRot, SmoothQuant, FlatQuant, SVDQuant), delivering complementary and cumulative performance improvements.", "weaknesses": "1. Missing Real-Time and Latency Evaluation: The paper does not report inference speed, latency, or hardware efficiency metrics—critical aspects for assessing quantization performance. Given that the multiplication of XL occurs online, it would be valuable to analyze its impact during inference.\n\n2. Absence of Calibration Set Ablation: There is no investigation into how the size of the calibration dataset influences performance, which is an essential factor for post-training quantization (PTQ) methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c3GYQ9TTCk", "reviewer": "Reviewer_sLp3", "summary": "This manuscript introduces STaMP (Sequence Transformation and Mixed Precision), a novel post-training quantization (PTQ) framework aimed at addressing low-bit activation quantization in large generative models (LLMs and LVMs). The core innovation, which distinguishes this work from prior art focused on feature-dimension transforms, is the application of linear transformations (e.g., DWT) along the sequence dimension. This method seeks to exploit local token correlations to compact activation energy into a minority of transformed tokens. Subsequently, the framework employs a mixed-precision strategy, allocating higher bit-widths to these high-energy tokens while quantizing the remainder more aggressively. The authors demonstrate through experiments that STaMP can serve as an orthogonal module that  consistently improves the performance of various existing PTQ methods under a W4A4 setting, without requiring any retraining.", "questions": "1. Regarding practical implementation on current mainstream GPUs (e.g., NVIDIA A100/H100), could the authors elaborate on a viable strategy for the proposed mixed-precision scheme? What is the anticipated latency overhead introduced by dequantization or conditional logic compared to a uniform W4A4 implementation? \n2. To contextualize the cost of the proposed transform, could the authors provide a brief computational complexity analysis comparing the $O(ds)$ cost of DWT to the $O(s \\cdot d \\cdot d_{ffn})$ cost of a typical Transformer FFN layer, thereby quantifying the relative overhead introduced by the transform itself? \n3. In the LLM experiments, a fixed strategy of keeping the first 64 tokens in high precision was applied to all methods. Have the authors considered whether this is an optimal strategy for the baseline methods, which do not employ sequence transforms? An ablation study that implements and compares against a dynamic, magnitude-based token selection strategy for the baselines could further strengthen the fairness of the experimental comparison. \n4. To strengthen the paper's motivation, particularly the analogy to media compression, further insight is requested. Technologies like JPEG use energy compaction to more aggressively compress high-frequency information, which is less perceptible to the human eye. What is the conceptual equivalent in the context of STaMP? What kind of information corresponds to the low-energy components after the DWT/DCT transform, which are subsequently assigned lower precision? A more intuitive explanation of which aspects of the token sequence are deemed \"less critical\" by the transform would significantly enhance the persuasiveness of the paper's motivation.\n$\\textbf{If these issues can be resolved, I will consider giving a higher score.}$", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. High Novelty: The primary strength of this work lies in its high degree of originality. By shifting the focus of quantization transforms from the feature dimension to the sequence dimension, the authors introduce a new research paradigm.\n 2. Solid Experimental Results: The paper's claims are supported by consistent and significant performance improvements across a range of models (LLMs and LVMs), datasets, and strong baselines. \n3. Excellent Complementarity: The experiments clearly demonstrate that STaMP is an orthogonal improvement that can be combined with state-of-the-art methods like QuaRot to achieve further gains. \n4. Clear Motivation and Exposition: The analogy to classical media compression provides a strong theoretical basis and an intuitive understanding of the proposed method.", "weaknesses": "1. Unaddressed Implementation Overhead: The most significant weakness is the failure to discuss or quantify the runtime overhead of the per-token mixed-precision scheme on current hardware. The lack of native hardware support may necessitate inefficient dequantization operations, which would severely impact the method's practical speedup. \n2. Unquantified Transform Cost: While the computational cost of the DWT/DCT transform is theoretically small, the paper fails to provide a simple quantitative analysis to substantiate this claim (e.g., by comparing it to the computational load of an FFN layer in a Transformer block), leaving its relative overhead ambiguous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "er2fTHmTKs", "reviewer": "Reviewer_jGRZ", "summary": "This paper proposes STaMP (Sequence Transformation and Mixed Precision), a novel post-training activation quantization method for LLMs and LVMs. Unlike prior work that applies invertible feature transformations across the channel dimension, STaMP introduces linear sequence transformations to exploit correlations between neighboring tokens. By concentrating activation energy into a few tokens and assigning them higher precision under a fixed average bit budget, STaMP effectively reduces quantization error. The authors approximate the optimal Karhunen–Loève Transform using efficient DCT or DWT implementations and derive theoretical bounds on quantization loss. Extensive experiments on PixArt-Σ, SANA, and LLaMA/Qwen models show consistent improvements over existing methods such as SmoothQuant, QuaRot, and SVDQuant, both in SQNR and downstream performance, with minimal computational overhead.", "questions": "1. How would STaMP operate during autoregressive decoding, where tokens are generated sequentially? Since the sequence transform matrix L require access to the full sequence, does it mean that we need to recompute L in every decoding step?\n2. Could you provide quantitative measurements of the runtime and memory overhead introduced by applying and inverting the sequence transform in large transformer models? How does this cost compare to the savings from reduced activation bitwidth?\n3. How is the mix-precision processed in STaMP? Specifically, since there are 64 tokens kept at 8 bits, and the rest uses 4 bits, how do you run the mix-precision model efficiently?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces sequence-domain transformations for activation quantization, a direction largely unexplored compared to prior channel-wise or feature-domain approaches. It can be combined with existing quantization methods and feature transformations without retraining or altering model weights.\n2. Extensive results on both LVMs (PixArt-Σ, SANA) and LLMs (LLaMA 3, Qwen 2.5) show clear improvements in SQNR, perplexity, and visual quality under low-bit settings (W4A4), confirming robustness and generality.\n3. The paper provides clear figures (e.g., Figs. 2–6) illustrating how energy concentration improves quantization performance and complements feature transforms.", "weaknesses": "1. The paper does not quantify the additional latency or memory cost of applying and inverting sequence transforms in large-scale models. Without such analysis, it is difficult to assess whether the efficiency gains from lower-bit activations outweigh the added transform cost in practical deployment scenarios.\n2. For the decoding stage of LLMs, STaMP may not be directly applicable, since each newly generated token would require recomputing the sequence transform matrix L.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)