STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization

## Abstract

Abstract Quantization is the key method for reducing inference latency, power and memory footprint of generative AI models.
However, accuracy often degrades sharply when activations are quantized below eight bits.
Recent work suggests that invertible linear transformations (e.g. rotations) can aid quantization, by reparameterizing feature channels and weights.
In this paper, we propose Sequence Transformation and Mixed Precision (STaMP) quantization, a novel strategy that applies linear transformations along the sequence dimension to exploit the strong local correlation in language and visual data.
By keeping a small number of tokens in each intermediate activation at higher precision, we can maintain model accuracy at lower (average) activations bit-widths.
We evaluate STaMP on recent LVM and LLM architectures, demonstrating that it significantly improves low bit width activation quantization and complements established activation and weight quantization methods including recent feature transformations.

## 1 Introduction

Figure: Figure 1: STaMP and feature transformations applied to PixArt-$\Sigma$ with 4-bit activations. The benefit of STaMP is orthogonal to (Hadamard) feature transformation, drastically reducing artifacts.
Refer to caption: figures/Id_Id.png

Modern generative models, such as large language models (LLMs) and large vision models (LVMs), achieve state-of-the-art performance in text and image generation but at the cost of massive computational and memory requirements.
As the demand for scalable and efficient deployment of these models grows, both in
the cloud and on edge devices where computational resources are scarce,
achieving inference efficiency has become a critical area of research.

Post-Training Quantization (PTQ) of weights and activations is fundamental to enhancing
inference efficiency, especially for
demanding operations such as large matrix multiplications in linear layers, which dominate power consumption.
However,
PTQ faces substantial challenges when pushing the quantization bitwidth down to 4-bits, often due to the presence of outliers
in weights and activations.

To mitigate outliers, prior work applies function-preserving transformations (van Breugel et al., 2025) to weights and activations.
For example, Xiao et al. (2023) scales down outlier activations and compensates by scaling up subsequent weights, preserving the model output. Similarly, Hadamard-based feature mixing (Ashkboos et al., 2024; Liu et al., 2024; Ma et al., 2024; Zhao et al., 2025) reduces activation variance and spreads outliers across dimensions.

However, these methods operate primarily along the feature dimension and ignore correlations across the sequence dimension. Drawing inspiration from traditional media compression methods, in this paper we propose *Sequence Transformation and Mixed Precision*
(STaMP) quantization, a complementary approach that leverages sequence structure to improve activation quantization.

Our main contributions are summarized below:

- •
We introduce a new class of activation transformations operating along the sequence dimension, complementary to existing feature transformations (Figure [1](https://arxiv.org/html/2510.26771v1#S1.F1)).
- •
We characterize the quantization error for sequence transforms and design a novel mixed-precision quantization scheme to exploit local activation correlation.
- •
We demonstrate that STaMP consistently improves the model accuracy when combined with feature transformations and weight quantization on both LLM and LVM.

## 2 Background

### 2.1 Activation Quantization

Consider ${\bm{X}}\in\mathbb{R}^{s\times d}$ as an activation matrix of shape sequence length $s$ $\times$ feature size $d$ (batch size is omitted for the sake of clarity). Integer quantization refers to the operation of converting the activations to integer values with lower bit width. The activation quantization operation $\mathbf{Q}:\mathbb{R}^{s\times d}\to\mathbb{N}^{s\times d}$ is defined as:

$$ $\displaystyle x^{\text{int}}_{ij}\stackrel{{\scriptstyle def}}{{=}}\mathbf{Q}({\bm{X}})_{ij}=\text{clamp}\left(\left\lfloor\frac{x_{ij}}{s_{ij}}\right\rceil+z_{ij},0,2^{b_{ij}}-1\right),$ (1) $$

in which $z_{ij}$ and $s_{ij}$ can be interpreted as an offset and scaling parameter, while $b_{ij}$ refers to the number of bits used to quantize the entry $x_{ij}$. In order to make quantization efficient, offsets, scales and bit widths are shared across all feature channels: $s_{ij}=s_{i}$, $z_{ij}=z_{i}$, and $b_{ij}=b_{i}$.

The de-quantization function $\mathbf{Q}^{-1}:\mathbb{N}^{s\times d}\to\mathbb{R}^{s\times d}$ maps the integer-quantized activation into the original domain:
$\mathbf{Q}^{-1}({\bm{X}}^{\text{int}})_{ij}=(x_{ij}^{\text{int}}-z_{ij})s_{ij}$.
We will refer to the combination of the quantization and de-quantization operation with $\mathcal{Q}\left({\bm{X}}\right)\stackrel{{\scriptstyle def}}{{=}}\mathbf{Q}^{-1}(\mathbf{Q}({\bm{X}}))$, omitting the quantization parameters for brevity.
The expected activation quantization error introduced by the quantization and de-quantization operations is commonly defined as:

$$ $\displaystyle\mathcal{L}\left({\bm{X}}\right)$ $\displaystyle\stackrel{{\scriptstyle def}}{{=}}\mathbb{E}\left[\left\|\mathcal{Q}\left({\bm{X}}\right)-{\bm{X}}\right\|_{2}^{2}\right]=\sum_{i=1}^{s}\overbrace{\mathbb{E}\left[\left\|\mathcal{Q}\left({\bm{x}}_{i}\right)-{\bm{x}}_{i}\right\|_{2}^{2}\right]}^{\mathcal{L}\left({\bm{x}}_{i}\right)},$ (2) $$

in which the expectation is computed with respect to the activation distribution $p({\bm{X}})$ and $\left\|\cdot\right\|_{2}^{2}$ refers to the squared Frobenius norm.
Nagel et al. (2021) identified two main causes of quantization error: clipping error (due to the clamp operator) and rounding error (introduced by the rounding). As is common practice in the literature, we will focus on a setting in which the scale $s_{i}$ and offset $z_{i}$ for the i-th token are set based on the $\text{range}\left({\bm{x}}_{i}\right)\stackrel{{\scriptstyle def}}{{=}}\max_{j}{\bm{x}}_{ij}-\min_{j}{\bm{x}}_{ij}$ to prevent any clipping error: $\overline{s_{i}}\stackrel{{\scriptstyle def}}{{=}}\frac{2^{b_{i}}-1}{\text{range}\left(x_{i}\right)}$, $\overline{z_{i}}\stackrel{{\scriptstyle def}}{{=}}-\frac{\min_{j}x_{ij}}{\overline{s_{i}}}$.
In this setting, the quantization error for each token ${\bm{x}}_{i}$ is determined by its quantization scale $\overline{s}_{i}$:

$$ $\displaystyle\mathcal{L}\left({\bm{x}}_{i}\right)\leq\frac{d}{4}\mathbb{E}\left[\overline{s_{i}}^{2}\right]=\frac{d}{4}\frac{\mathbb{E}\left[\text{range}\left({\bm{x}}_{i}\right)^{2}\right]}{(2^{b_{i}}-1)^{2}}.$ (3) $$

As the activation quantization error increases, so does the deterioration in performance for the model outputs since the network output diverge from the original (unquantized) model.

### 2.2 Feature Transformations

In order to reduce the activation quantization error, recent literature has introduced linear function-preserving transformations in the form of (right) invertible matrices ${\bm{R}}$, which are applied prior to the quantization operation (van Breugel et al., 2025). Clearly, the additional flexibility introduced by Feature Transformations can aid in reducing the activation quantization error:

$$ $\displaystyle\mathcal{L}\left({\bm{X}}\right)\geq\min_{{\bm{R}}}\underbrace{\mathbb{E}\left[\left\|\mathcal{Q}\left({\bm{X}}{\bm{R}}\right){\bm{R}}^{-1}-{\bm{X}}\right\|_{2}^{2}\right]}_{\mathcal{L}\left({\bm{X}};{\bm{R}}\right)}.$ (4) $$

In particular, rotation matrices can effectively reduce the activation range, effectively spreading outliers across multiple channels and consequently reducing the subsequent quantization error:

$$ $\displaystyle\sum_{i=1}^{s}\mathbb{E}\left[\text{range}\left({\bm{x}}_{i}{\bm{R}}\right)^{2}\right]\leq\sum_{i=1}^{s}\mathbb{E}\left[\text{range}\left({\bm{x}}_{i}\right)^{2}\right]\implies\mathcal{L}\left({\bm{X}};{\bm{R}}\right)\leq\mathcal{L}\left({\bm{X}}\right)$ (5) $$

Hadamard matrices have proven very effective in reducing the number of outliers with limited additional overhead: matrix multiplications with Hadamard matrices can be performed efficiency in $O(sd\log d)$ thanks to the butterfly algorithm (Fino and Algazi, 1976) , and the inverse Hadamard matrix can be fused into linear layer weights (Ashkboos et al., 2024).

Existing feature transformation techniques reduce activation quantization error by redistributing activation ranges across features. However, they operate exclusively along the feature dimension and ignore correlations across the sequence dimension. Visual and textual data exhibit strong local correlations—neighboring pixels in images and adjacent tokens in text are strongly dependent. This suggests that a similar structure could exist in the model intermediate activations and could be leveraged to improve quantization efficiency, which is described in the next section.

## 3 Method

We define a Sequence Transform as a linear transformation of ${\bm{X}}$ across sequence dimension defined by a (left) invertible matrix ${\bm{L}}$.
Similarly to feature transformations ${\bm{R}}$, sequence transformations can reduce the quantization error, and the two can be easily combined to further the error:

$$ $\displaystyle\mathcal{L}\left({\bm{X}}\right)\geq\min_{{\bm{L}}}\underbrace{\mathbb{E}\left[\left\|{\bm{L}}^{-1}\mathcal{Q}\left({\bm{L}}{\bm{X}}\right)-{\bm{X}}\right\|_{2}^{2}\right]}_{\mathcal{L}\left({\bm{X}};{\bm{L}}\right)}\geq\min_{{\bm{L}},{\bm{R}}}\underbrace{\mathbb{E}\left[\left\|{\bm{L}}^{-1}\mathcal{Q}\left({\bm{L}}{\bm{X}}{\bm{R}}\right){\bm{R}}^{-1}-{\bm{X}}\right\|_{2}^{2}\right]}_{\mathcal{L}\left({\bm{X}};{\bm{L}},{\bm{R}}\right)}.$ (6) $$

Sequence transformations are linear, hence they commute with other linear operations. For a linear layer:

$$ $\displaystyle\left({\bm{L}}^{-1}\mathcal{Q}\left({\bm{L}}{\bm{X}}\right)\right){\bm{W}}+{\bm{1}}\bm{\beta}^{T}={\bm{L}}^{-1}\left(\mathcal{Q}\left({\bm{L}}{\bm{X}}\right){\bm{W}}\right)+{\bm{1}}\bm{\beta}^{T}={\bm{L}}^{-1}\left(\mathcal{Q}\left({\bm{L}}{\bm{X}}\right){\bm{W}}+\underbrace{\left({\bm{L}}{\bm{1}}\right)}_{\bm{\ell}}\bm{\beta}^{T}\right),$ (7) $$

in which ${\bm{1}}$ represents a vector of ones of size $s$, indicating that the same bias $\bm{\beta}$ is applied to all the tokens. This implies that we can invert sequence transformation (i) right before applying the bias in a linear layer, or (ii) postpone this operation and use a sequence-transformed bias $\bm{\ell}\bm{\beta}^{T}$ in which the scale $\ell_{i}$ may differ across different tokens as a function of ${\bm{L}}$. The algorithm for a sequence transformed linear layer is reported in Figure [2](https://arxiv.org/html/2510.26771v1#S3.F2).

We emphasize that, contrary to feature transformations, sequence transforms do not affect weights, and hence they are orthogonal to more advanced weight quantization methods such as vector quantization, GPTQ (Guo et al., 2024), and SVDQuant (Li et al., 2025).

### 3.1 Sequence Transform and Mixed Precision (STaMP)

Figure: (a) STaMP Linear Layer Pseudocode
Refer to caption: figures/bound_and_error.png

To understand how sequence transformations affect quantization error, we first formalize the relationship between quantization error and sequence-transformed tokens (proof in Appendix [A.1](https://arxiv.org/html/2510.26771v1#A1.SS1)):

###### Theorem 1 .

The expected quantization error for activations ${\bm{X}}$ transformed by an orthogonal sequence transformation ${\bm{L}}$ and quantized using a min-max scale for each token is upper-bounded by the weighted sum of
the expected norm of the transformed tokens:

$$ $\displaystyle\mathcal{L}\left({\bm{X}};{\bm{L}}\right)\leq\frac{d}{2}\sum_{i=1}^{s}\frac{\overbrace{\mathbb{E}\left[\left\|{\bm{l}}_{i}^{T}{\bm{X}}\right\|_{2}^{2}\right]}^{e_{i}}}{(2^{b_{i}}-1)^{2}}.$ (8) $$

Note that sequence transformations ${\bm{L}}$ do not alter the total energy $E=\sum_{i=1}^{s}e_{i}$, so any improvement to the bound in Equation [8](https://arxiv.org/html/2510.26771v1#S3.E8) must come from redistributing the bit width.
This observation motivates a mixed precision strategy: instead of keeping energy and bit width uniform, we deliberately concentrate most of the energy into a few tokens and allocate more bits to them. Because the denominator in Equation [8](https://arxiv.org/html/2510.26771v1#S3.E8) grows exponentially with $b_{i}$, allocating extra bits to tokens with large energy yields a disproportionately large reduction in their contribution to the error. In other words, redistributing a bit from a low-energy token to a high-energy token reduces the total error more than keeping bit widths and energy uniform.
This property is illustrated in Figure [2](https://arxiv.org/html/2510.26771v1#S3.F2) and further elaborated in Appendix [A.3](https://arxiv.org/html/2510.26771v1#A1.SS3).

Therefore, to improve activation quantization performance, we propose Sequence Transform and Mixed Precision (STaMP) a simple yet effective strategy that concentrates activation energy into a small set of tokens and assigns them higher precision. In the next section, we describe how to design a transformation ${\bm{L}}$ that achieves this efficiently and how to determine the corresponding bit width allocation.

Figure: (a) Autocorrelation
Refer to caption: figures/autocorrelation_pixart-sigma.png

###### Theorem 1 .

### 3.2 Efficient Energy Concentration

The energy of the i-th sequence-transformed token $e_{i}$ can be also seen as the projection of the autocorrelation matrix ${\bm{S}}=\mathbb{E}[{\bm{X}}{\bm{X}}^{T}]$ along the direction ${\bm{l}}_{i}$:

$$ $\displaystyle e_{i}=\mathbb{E}\left[\left\|{\bm{l}}_{i}{\bm{X}}\right\|_{2}^{2}\right]={\bm{l}}_{i}^{T}{\bm{S}}{\bm{l}}_{i}.$ (9) $$

he direction that maximizes this energy is the eigenvector ${\bm{u}}_{1}$ associated with the largest eigenvalue $\lambda_{1}$ of ${\bm{S}}$. Similarly, the second largest energy corresponds to ${\bm{u}}_{2}$ and so on.
Therefore, given the eigendecomposition ${\bm{S}}={\bm{U}}{\bm{\Lambda}}{\bm{U}}^{T}$, the optimal orthogonal transformation ${\bm{L}}$ for concentrating the token norm is ${\bm{U}}^{T}$. In this case, the energy ${\bm{e}}$ of the transformed tokens ${\bm{L}}{\bm{X}}$ aligns with the squared eigenvalues $\bm{\lambda}^{2}$.
This linear transformation is also known as the Karhunen-Loève Transform (KLT), which requires a representative calibration set to estimate ${\bm{U}}$.
Despite its optimality, KLT
has the same computational complexity of a full-rank matrix multiplication, which is impractical since each transform needs to be applied twice for each linear layer. Estimating ${\bm{S}}$ for each activation would further adds a costly calibration step.

Fortunately, the autocorrelation matrix of common LVM and LLM activations ${\bm{S}}$ exhibits a strong structure induced by the properties of natural images and text. Figure [3](https://arxiv.org/html/2510.26771v1#S3.F3) shows that tokens corresponding to spatially or sequentially adjacent activations are highly correlated, while distant tokens are weakly correlated.
As a result ${\bm{S}}$ is approximately (block) Toeplitz, whose eigenvectors can be well-approximated by a Fourier basis(^1^11This follows from Szegő’s theorem.).
In particular, since the autocorrelation is real and symmetric, we can use a Discrete Cosine Transform (DCT) instead of a complex Fourier basis.

The complexity of DCT $\mathcal{O}(ds\log s)$ is lower than a full matrix multiplication since the transformation requires only $\log s$ steps of the Fast Fourier Transform algorithm.
Further simplification is possible by retaining only the sign of Fourier coefficients, yielding the Walsh-Hadamard Transform (WHT), which approximates DCT while enabling more efficient hardware implementations.

Finally, the Discrete Wavelet Transform (DWT)(^2^22We use the Haar wavelet for its simplicity and minimal padding requirements.) further reduces the computational complexity to $\mathcal{O}(ds)$ while effectively concentrating the energy of the activations at each intermediate step.
Each DWT step pushes the energy in the first half (one quarter for 2D signal) of the tokens, requiring $\log s$ steps to fully concentrate the energy.

Figure [3](https://arxiv.org/html/2510.26771v1#S3.F3) compares energy distributions ([3](https://arxiv.org/html/2510.26771v1#S3.F3)) and basis components ([3](https://arxiv.org/html/2510.26771v1#S3.F3)) for KLT, DCT, and DWT on intermediate activations of LVM and LLM architectures.

### 3.3 Optimal bit allocation

Given a vector of energies ${\bm{e}}$, the optimal bit distribution ${\bm{b}}$ for a total maximum allocation of $B$ bits follows the logarithm of the squared root of the token energy: $b_{i}^{*}=\log_{2}\sqrt{e_{i}}-C$,
with $C=\left(B-\sum_{i=1}^{s}\log_{2}\sqrt{e_{i}}\right)/s$. ${\bm{b}}^{*}$ gives us an indication of the optimal bit width for each token, however, in practice we are restricted to integer bit widths $\left\lfloor{\bm{b}}^{*}\right\rceil$. Furthermore, due to practical limitations, it is beneficial to use only a small number of different bit precisions which are supported on hardware, such as 4 or 8 bits.

Figure: (a) Bit width allocation and transformation strategies
Refer to caption: figures/optimal_bitwidth.png

For this reason, although the DWT energy concentration is sub-optimal, its property of creating a discrete number of energy levels makes it more suitable to our use case. As illustrated in Figure [4](https://arxiv.org/html/2510.26771v1#S3.F4) in yellow, we propose a simple allocation scheme that uses only two bit widths: the first 64 tokens are kept at 8 bits, while the rest uses 4 bits, resulting only in a minor bit width overhead (4.0625 on the PixArt-$\Sigma$), while significantly improving the model accuracy (Figure [4](https://arxiv.org/html/2510.26771v1#S3.F4)). For this reason, STaMP with DWT and 2 precision level will be our main focus in the experimental section.

## 4 Related Work

Quantization is a fundamental technique for reducing the computational and memory footprint of deep neural networks, enabling efficient inference with minimal accuracy loss.
The rapid growth of LLMs and LVMs has intensified interest in
post training quantization (PTQ) Nagel et al. (2021); Gholami et al. (2022),
as retraining these models is often impractical.
Recent PTQ approaches focus on removing the outliers and reducing the dynamic range
of weights and activations, to improve quantization robustness.

SmoothQuant (Xiao et al., 2023) reduces activations outliers by scaling the feature channels, shifting quantization difficulty from activations to weights.
*QuaRot* and *FlatQuant* apply invertible feature transforms over weights and activations to spread outliers across channels, employing randomized Hadamard (Ashkboos et al., 2024) or learning lightweight affine transforms (Sun et al., 2025). van Breugel et al. (2025) develop transforms that commute with transformer operations, such that they can be merged into linear weights.
Zhao et al. (2025)
develops a Static-Dynamic Channel Balancing (SDCB) procedure based on channel scaling and mixing on Diffusion Transformer (DiT) architectures and retaining certain quantization-sensitive layers to higher bitwidth to achieve W8A8.
Li et al. (2025)
absorbs activation outliers into a high-precision low-rank branch via singular value decomposition,
while quantizing the residuals to 4-bit with a per-token/per-group quantization granularity to achieve W4A4 mixed-precision quantization on DiT.
Federici et al. (2025) reduces the dynamic range of DiT activations by subtracting the sequence average from each token and applying Hadamard feature rotations, at the cost
of processing an extra token.

While these approaches have advanced PTQ for large models, they operate exclusively along the feature dimension and ignore correlations across the sequence dimension. In contrast, STaMP introduces sequence-aware transformations that exploit local token correlations by using a mixed precision allocation strategy under a fixed average budget.
Our approach is strongly relates to classical media compression techniques that leverage frequency-domain transforms to concentrate energy and enable adaptive quantization. JPEG (Leger et al., 1991) and JPEG2000 (Christopoulos et al., 2002) rely on the Discrete Cosine Transform and Discrete Wavelet Transform, respectively, to decorrelate spatial data and allocate bits based on perceptual importance.
Similar principles underlie modern video coding standards such AVC Wiegand et al. (2003) and HEVC (Sullivan et al., 2012), as well as audio codecs like MP3 and AAC, which use the Modified Discrete Cosine Transform (Princen et al., 1987). STaMP demonstrates that it is possible to apply similar principles to the activation space of generative models, where local correlation in the sequence dimension allows energy compaction and selective precision allocation, improving quantization efficiency without retraining.

## 5 Experimental Results

We demonstrate the effectiveness of STaMP on both LVM and LLM models, comparing against uniform activation quantization, and showing its effectiveness in combination with popular feature transformation and weight quantization approaches. Throughout this section, we focus on STaMP with DWT due to its computational efficiency. We refer to Figure [5](https://arxiv.org/html/2510.26771v1#S5.F5)
for a visualization of how the sequence transformations are applied to the architectures, and to Appendix [C](https://arxiv.org/html/2510.26771v1#A3) for additional results.

Figure: Figure 5: Diagram of a LVM transformer block based on the PixArt-$\Sigma$ architecture. The quantization and sequence transformations operations are indicated explicitly. LLM architectures use the same quantization scheme for the attn1 and ffn blocks. Note that no transform is applied to the attn2.to_out activations since the sequence autocorrelation does not present the block diagonal structure because of its dependency on the pooled textual embedding.

### 5.1 Vision Language Models

**Table 1: STaMP consistently improves LVM quantization. Image SQNR and Image Reward (IR) for W4A4 per block quantization with block size 64. For all the STaMP results we keep 64 tokens at 8-bits. STaMP consistently improves baselines.**
|  |  | COCO | MJHQ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | SQNR | IR | SQNR | IR |  |  |  |  |
|  | STaMP $\rightarrow$ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ |
| PixArt-$\Sigma$ | FP | $+\infty$ | 0.90 | $+\infty$ | 0.96 |  |  |  |  |
| RTN | 5.88 | 6.16 | 0.38 | 0.80 | 5.75 | 6.23 | 0.38 | 0.76 |  |
| ViDiT-Q | 7.82 | 6.37 | 0.83 | 0.84 | 7.55 | 8.53 | 0.76 | 0.86 |  |
| SVDQuant | 8.78 | 9.72 | 0.90 | 0.91 | 8.83 | 9.75 | 0.86 | 0.89 |  |
| SANA | FP | $+\infty$ | 0.87 | $+\infty$ | 0.97 |  |  |  |  |
| RTN | 8.63 | 9.32 | 0.89 | 0.91 | 8.56 | 9.40 | 0.95 | 0.99 |  |
| ViDiT-Q | 10.03 | 10.74 | 0.89 | 0.86 | 10.04 | 10.69 | 0.96 | 0.97 |  |
| SVDQuant | 9.99 | 10.69 | 0.87 | 0.90 | 9.88 | 10.51 | 0.93 | 0.98 |  |

Figure: Figure 6: Visualization of PixArt-$\Sigma$ sample generation for the results reported in Table [1](https://arxiv.org/html/2510.26771v1#S5.T1).
Refer to caption: figures/visual_results/FP.png

#### Set-up

Our LVM experiments focus on the DiT architectures of PixArt-$\Sigma$ (Chen et al., 2024) and SANA (Xie et al., 2025).
We quantize activation before each linear layer in transformer blocks using asymmetric quantization with minmax scaling. As common practice, unless otherwise specified, we use a separate scale $s_{i}$ for each token and weight output channel. Following common procedure in literature, in LVMs we do not quantize activations and weights corresponding to the cross-attention key and value since their effect accounts for less than 5% of the runtime (Li et al., 2025).

We evaluate the fidelity of the quantized LVMs by computing the Signal to Quantized Noise Ratio $\text{SQNR}({\bm{O}}_{\text{orig}},{\bm{O}}_{\text{quant}})=10\log_{10}\left(\left\|{\bm{O}}_{\text{orig}}\right\|_{2}^{2}/\left\|{\bm{O}}_{\text{orig}}-{\bm{O}}_{\text{quant}}\right\|_{2}^{2}\right)$ both in the diffusion latent space and image space. We further compute CLIP Score (Hessel et al., 2021) to assess alignment with the textual prompt, CLIP IQA (Wang et al., 2023), and Image Reward (Xu et al., 2023)
to evaluate the generated image quality. Following standard procedures, we compute the metrics using 1000 prompts and images from the COCO (Lin et al., 2014) and MJHQ (Li et al., 2024) datasets.

We demonstrate the effectiveness STaMP alone and by combining it with other quantization methods developed in recent literature. We combine STaMP with feature transform methods such as SmoothQuant (Xiao et al., 2023) and QuaRot (Ashkboos et al., 2024), and Static-Dynamic Channel Balancing (SDCB) described in ViDiT-Q (Zhao et al., 2025). We further demonstrate the effectiveness of STaMP with the mixed precision low-rank weight quantization method described in SVDQuant (Li et al., 2025). We apply STaMP before each linear layer in the Transformer blocks, inverting it right after each linear layer. We use 2-dimensional DWT with 64 8-bit (high precision) tokens unless otherwise specified.

#### Results

Table [1](https://arxiv.org/html/2510.26771v1#S5.T1) reports the results obtained by combining STaMP with recent LVM quantization method using the same quantization setting for all the baselines. Both activation and weights are quantized at 4 bits with blocks of size 64, following the setup described in (Li et al., 2025). We observe that STaMP consistently improves upon all the reported metrics on different models and architectures resulting in visually more accurate generations, as shown in Figure [6](https://arxiv.org/html/2510.26771v1#S5.F6).

### 5.2 Number of high precision tokens

Figure [4](https://arxiv.org/html/2510.26771v1#S3.F4) demonstrates the trade-off between bit width and SQNR observed by changing the number of high-precision tokens in STaMP, while fixing the high and low precision bit widths to 8 and 4 bits respectively. Only activations are quantized to focus the analysis solely on the activation quantization error. We observe a sharp increase in SQNR whenever even a few high precision tokens are introduced. Even in the 5 bits regime, STaMP achieves better performance uniform quantization.
Additional comparison with per-block activation quantization can be found in Appendix [C](https://arxiv.org/html/2510.26771v1#A3).

### 5.3 Large Language Models

**Table 2: STaMP always improves LLM quantization. Perplexity (PPL) for W4A4KV4 quantization, using the same setting as (Ashkboos et al., 2024; Sun et al., 2025). We use 64 8-bit tokens for activations and KV-cache for all methods and baselines, even if we do not apply the sequence transform (effectively W4A4.125KV4.125). The STaMP sequence transform improves all baselines.**
|  | Llama 3 8B | Llama 3.2 1B it | Llama 3.2 3B it | Qwen 2.5 3B it |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| STaMP $\rightarrow$ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ |
| FP | 6.14 | 13.16 | 11.27 | 8.56 |  |  |  |  |
| RTN | 668 | 95.3 | 1795 | 700 | 483 | 159 | 99723 | 18767 |
| SmoothQuant | 531 | 93.8 | 883 | 407 | 177 | 88.5 | 66929 | 29063 |
| Quarot | 9.05 | 8.66 | 25.78 | 23.72 | 18.43 | 17.57 | 94.86 | 71.13 |
| FlatQuant | 6.89 | 6.77 | 15.72 | 15.16 | 12.71 | 12.40 | 9.29 | 9.19 |

#### Set-up

We evaluate on language models of different sizes and model classes, including Llama 3 8B (Grattafiori et al., 2024), Llama 3.2 1B and 3B instruct, and Qwen 2.5 3B instruct (Qwen et al., 2025). We take popular *feature* transforms from literature, including SmoothQuant (Xiao et al., 2023), QuaRot (Ashkboos et al., 2024), and the state-of-the-art FlatQuant (Sun et al., 2025), and evaluate whether STaMP (DWT) brings additional gains to these baselines.
We use the exact same quantization set-up as used in (Ashkboos et al., 2024; Sun et al., 2025), W4A4KV4 per token activation quantization, and like them, evaluate Wikitext-2 (Merity et al., 2017) perplexity at sequence length 2048. We use round-to-nearest (RTN) for weight quantization (vs. GPTQ (Frantar et al., 2022)), as weight quantization is completely perpendicular to sequence transforms. In this setup, we keep the first 64 tokens in 8 bits for all baselines, which means all methods use an effective activation/KV bit width of 4.125 bits. See Appendix [B.2](https://arxiv.org/html/2510.26771v1#A2.SS2) for additional details.

#### Results

We observe (Table [2](https://arxiv.org/html/2510.26771v1#S5.T2)) that all baselines improve consistently when STaMP is added. This is especially apparent for scenarios where baselines are far from FP performance—e.g. for the small Llama 3.2 1B and 3B instruct models, which are evidently hard to quantize at 4 bit. This demonstrates that STaMP is not a competitor of other quantization techniques (e.g. feature transforms), but an additional tool for achieving extremely low bit width quantization—which can be added without any manual tuning or training to existing quantization methods.

### 5.4 Combining Feature and Sequence Transformations

Figure: (a) A4 PixArt-$\bm{\Sigma}$ SQNR and Image Reward
Refer to caption: x5.png

In Figure [7](https://arxiv.org/html/2510.26771v1#S5.F7) we assess the effectivenes of STaMP for three different sequence transformations in combination with popular feature transforms. Overall, we observe that improvements are largely complementary, especially for LVMs. Furthermore, results show that DCT, WHT and DWT perform similarly to each other, demonstrating that there is little price to pay for switching from the more accurate and computationally expensive DCT to the cheaper approximate DWT.

### 5.5 Overhead Estimates

**Table 3: Overhead in terms of latency on CUDA and extra FLOPS for a single PixArt-$\Sigma$ denoising step. STaMP with DWT has little impact on the latency and number of floating point operations.**
| Transformation | Overhead [%] |  |  |
| --- | --- | --- | --- |
| Feature | Sequence | FLOPS | CUDA |
| Hadamard | - | 0.24 | 3.0 |
| - | Hadamard | 0.74 | 56.8 |
| - | DWT | 0.21 | 4.8 |
| Hadamard | DWT | 0.44 | 7.8 |

Table [3](https://arxiv.org/html/2510.26771v1#S5.T3) reports that the theoretical compute overhead and latency of CUDA for a latent denoising step with STaMP (DWT) is comparable to Hadamard transforms on the features, accounting for less than 5% of the total runtime.
The DWT transform is applied in three levels,
consistently with the results reported in Table [1](https://arxiv.org/html/2510.26771v1#S5.T1)
and Figure [4](https://arxiv.org/html/2510.26771v1#S3.F4), using a specialized CUDA kernel that considers
the memory layout of the activation tensors.
Appendix [B.3](https://arxiv.org/html/2510.26771v1#A2.SS3) includes further details on the benchmarking procedure. The minimal FLOPS overhead suggests that the latency overhead can be further reduced with better optimized kernels or specialized hardware.

## 6 Conclusion

This work explores uncharted territories in the recent advancements of LLM and LVM quantization,
applying invertible transformations over the sequence dimension to further reduce quantization error.
Drawing inspiration from traditional signal processing, we introduce a novel method that exploits the autocorrelation structure that is naturally present in the intermediate activations of generative models to enable a more efficient mixed-precision activation quantization scheme.

We demonstrate that STaMP complements existing PTQ techniques such as *SmoothQuant* and *QuaRot*,
and plays a critical role in advancing low-precision activation quantization, pushing the boundaries of
W4A4 quantization for both LLMs and LVMs and offering training-free solution for
deploying high-performance models in resource-constrained environments.

## Acknowledgments and Disclosure of Funding

Images were generated using the PixArt-$\Sigma$ model licensed under [Apache 2.0](https://github.com/PixArt-alpha/PixArt-sigma/blob/master/LICENSE) and SANA model licensed under [Apache 2.0](https://github.com/NVlabs/Sana/blob/main/LICENSE).

## References

- van Breugel et al. (2025)
Boris van Breugel, Yelysei Bondarenko, Paul Whatmough, and Markus Nagel.
Fptquant: Function-preserving transforms for llm quantization.
*arXiv preprint arXiv:2506.04985*, 2025.
- Xiao et al. (2023)
Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han.
Smoothquant: Accurate and efficient post-training quantization for large language models.
In *International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA*, volume 202 of *Proceedings of Machine Learning Research*, pages 38087–38099. PMLR, 2023.
URL [https://proceedings.mlr.press/v202/xiao23c.html](https://proceedings.mlr.press/v202/xiao23c.html).
- Ashkboos et al. (2024)
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman.
Quarot: Outlier-free 4-bit inference in rotated llms.
In *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024*, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/b5b939436789f76f08b9d0da5e81af7c-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2024/hash/b5b939436789f76f08b9d0da5e81af7c-Abstract-Conference.html).
- Liu et al. (2024)
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort.
Spinquant: LLM quantization with learned rotations.
*CoRR*, abs/2405.16406, 2024.
doi: 10.48550/ARXIV.2405.16406.
URL [https://doi.org/10.48550/arXiv.2405.16406](https://doi.org/10.48550/arXiv.2405.16406).
- Ma et al. (2024)
Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji.
Affinequant: Affine transformation quantization for large language models.
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=of2rhALq8l](https://openreview.net/forum?id=of2rhALq8l).
- Zhao et al. (2025)
Tianchen Zhao, Tongcheng Fang, Haofeng Huang, Rui Wan, Widyadewi Soedarmadji, Enshu Liu, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang.
Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=E1N1oxd63b](https://openreview.net/forum?id=E1N1oxd63b).
- Nagel et al. (2021)
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort.
A white paper on neural network quantization.
*CoRR*, abs/2106.08295, 2021.
URL [https://arxiv.org/abs/2106.08295](https://arxiv.org/abs/2106.08295).
- Fino and Algazi (1976)
Fino and Algazi.
Unified matrix treatment of the fast walsh-hadamard transform.
*IEEE Transactions on Computers*, 100(11):1142–1146, 1976.
- Guo et al. (2024)
Yipin Guo, Yilin Lang, and Qinyuan Ren.
GPTQT: quantize large language models twice to push the efficiency.
In *IEEE International Conference on Cybernetics and Intelligent Systems, CIS 2024, and IEEE International Conference on Robotics, Automation and Mechatronics, RAM 2024, Hangzhou, China, August 8-11, 2024*, pages 368–373. IEEE, 2024.
doi: 10.1109/CIS-RAM61939.2024.10672819.
URL [https://doi.org/10.1109/CIS-RAM61939.2024.10672819](https://doi.org/10.1109/CIS-RAM61939.2024.10672819).
- Li et al. (2025)
Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Zhu, and Song Han.
Svdquant: Absorbing outliers by low-rank component for 4-bit diffusion models.
In *The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025*. OpenReview.net, 2025.
URL [https://openreview.net/forum?id=vWR3KuiQur](https://openreview.net/forum?id=vWR3KuiQur).
- Gholami et al. (2022)
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer.
A survey of quantization methods for efficient neural network inference.
In *Low-power computer vision*, pages 291–326. Chapman and Hall/CRC, 2022.
- Sun et al. (2025)
Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, JiaxinHu, Xianzhi Yu, Lu Hou, Chun Yuan, Xin Jiang, Wulong Liu, and Jun Yao.
Flatquant: Flatness matters for LLM quantization.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=uTz2Utym5n](https://openreview.net/forum?id=uTz2Utym5n).
- Federici et al. (2025)
Marco Federici, Riccardo Del Chiaro, Boris van Breugel, Paul Whatmough, and Markus Nagel.
Hadanorm: Diffusion transformer quantization through mean-centered transformations.
*arXiv preprint arXiv:2506.09932*, 2025.
- Leger et al. (1991)
Alain M Leger, Takao Omachi, and Gregory K Wallace.
Jpeg still picture compression algorithm.
*Optical Engineering*, 30(7):947–954, 1991.
- Christopoulos et al. (2002)
Charilaos Christopoulos, Athanassios Skodras, and Touradj Ebrahimi.
The jpeg2000 still image coding system: an overview.
*IEEE transactions on consumer electronics*, 46(4):1103–1127, 2002.
- Wiegand et al. (2003)
T. Wiegand, G.J. Sullivan, G. Bjontegaard, and A. Luthra.
Overview of the h.264/avc video coding standard.
*IEEE Transactions on Circuits and Systems for Video Technology*, 13(7):560–576, 2003.
doi: 10.1109/TCSVT.2003.815165.
- Sullivan et al. (2012)
Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand.
Overview of the high efficiency video coding (hevc) standard.
*IEEE Transactions on Circuits and Systems for Video Technology*, 22(12):1649–1668, 2012.
doi: 10.1109/TCSVT.2012.2221191.
- Princen et al. (1987)
J. Princen, A. Johnson, and A. Bradley.
Subband/transform coding using filter bank designs based on time domain aliasing cancellation.
In *ICASSP ’87. IEEE International Conference on Acoustics, Speech, and Signal Processing*, volume 12, pages 2161–2164, 1987.
doi: 10.1109/ICASSP.1987.1169405.
- Chen et al. (2024)
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li.
Pixart-$\Sigma$: Weak-to-strong training of diffusion transformer for 4k text-to-image generation.
In *Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXXII*, volume 15090 of *Lecture Notes in Computer Science*, pages 74–91. Springer, 2024.
doi: 10.1007/978-3-031-73411-3“˙5.
URL [https://doi.org/10.1007/978-3-031-73411-3_5](https://doi.org/10.1007/978-3-031-73411-3_5).
- Xie et al. (2025)
Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han.
SANA: efficient high-resolution text-to-image synthesis with linear diffusion transformers.
In *The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025*. OpenReview.net, 2025.
URL [https://openreview.net/forum?id=N8Oj1XhtYZ](https://openreview.net/forum?id=N8Oj1XhtYZ).
- Hessel et al. (2021)
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.
Clipscore: A reference-free evaluation metric for image captioning.
In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021*, pages 7514–7528. Association for Computational Linguistics, 2021.
doi: 10.18653/V1/2021.EMNLP-MAIN.595.
URL [https://doi.org/10.18653/v1/2021.emnlp-main.595](https://doi.org/10.18653/v1/2021.emnlp-main.595).
- Wang et al. (2023)
Jianyi Wang, Kelvin C. K. Chan, and Chen Change Loy.
Exploring CLIP for assessing the look and feel of images.
In *Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023*, pages 2555–2563. AAAI Press, 2023.
doi: 10.1609/AAAI.V37I2.25353.
URL [https://doi.org/10.1609/aaai.v37i2.25353](https://doi.org/10.1609/aaai.v37i2.25353).
- Xu et al. (2023)
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong.
Imagereward: Learning and evaluating human preferences for text-to-image generation.
In *Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023*, 2023.
URL [http://papers.nips.cc/paper_files/paper/2023/hash/33646ef0ed554145eab65f6250fab0c9-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2023/hash/33646ef0ed554145eab65f6250fab0c9-Abstract-Conference.html).
- Lin et al. (2014)
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.
Microsoft coco: Common objects in context.
In *Computer Vision – ECCV 2014*, pages 740–755, Cham, 2014. Springer International Publishing.
ISBN 978-3-319-10602-1.
- Li et al. (2024)
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi.
Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation.
*CoRR*, abs/2402.17245, 2024.
doi: 10.48550/ARXIV.2402.17245.
URL [https://doi.org/10.48550/arXiv.2402.17245](https://doi.org/10.48550/arXiv.2402.17245).
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Qwen et al. (2025)
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report, 2025.
URL [https://arxiv.org/abs/2412.15115](https://arxiv.org/abs/2412.15115).
- Merity et al. (2017)
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.
Pointer sentinel mixture models.
In *International Conference on Learning Representations*, 2017.
- Frantar et al. (2022)
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.
Gptq: Accurate post-training quantization for generative pre-trained transformers.
*arXiv preprint arXiv:2210.17323*, 2022.
- Xiao et al. (2024)
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.
Efficient Streaming Language Models with Attention Sinks.
In *International Conference of Learning Representations*, April 2024.
doi: 10.48550/arXiv.2309.17453.
URL [http://arxiv.org/abs/2309.17453](http://arxiv.org/abs/2309.17453).
arXiv:2309.17453 [cs].
- Gu et al. (2025)
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin.
When Attention Sink Emerges in Language Models: An Empirical View.
In *International Conference on Learning Representations*. arXiv, March 2025.
doi: 10.48550/arXiv.2410.10781.
URL [http://arxiv.org/abs/2410.10781](http://arxiv.org/abs/2410.10781).
arXiv:2410.10781 [cs].
- Sun et al. (2024)
Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu.
Massive Activations in Large Language Models.
In *First Conference on Language Modeling (COML)*. arXiv, August 2024.
doi: 10.48550/arXiv.2402.17762.
URL [http://arxiv.org/abs/2402.17762](http://arxiv.org/abs/2402.17762).
arXiv:2402.17762 [cs] version: 2.

## Appendix A Proofs

### A.1 Proof for Theorem 1

We prove the result reported in Theorem [1](https://arxiv.org/html/2510.26771v1#Thmtheorem1) in three steps.
First, we demonstrate that the quantization error for an orthogonal sequence transformation is the same as the quantization error on the sequence transformed inputs:

$$ $\displaystyle\mathcal{L}\left({\bm{X}};{\bm{L}}\right)=\mathcal{L}\left({\bm{L}}{\bm{X}}\right)$ (10) $$

###### Proof.

This simply follows from the invariance of the Frobenious norm to orthogonal transformations.

$$ $\displaystyle\mathcal{L}\left({\bm{X}};{\bm{L}}\right)$ $\displaystyle=\left\|{\bm{L}}^{-1}\mathcal{Q}\left({\bm{L}}{\bm{X}}\right)-{\bm{X}}\right\|_{2}^{2}$ $\displaystyle=\left\|{\bm{L}}^{-1}\mathcal{Q}\left({\bm{L}}{\bm{X}}\right)-{\bm{L}}^{-1}{\bm{L}}{\bm{X}}\right\|_{2}^{2}$ $\displaystyle=\left\|{\bm{L}}^{-1}\left(\mathcal{Q}\left({\bm{L}}{\bm{X}}\right)-{\bm{L}}{\bm{X}}\right)\right\|_{2}^{2}$ $\displaystyle=\left\|\mathcal{Q}\left({\bm{L}}{\bm{X}}\right)-{\bm{L}}{\bm{X}}\right\|_{2}^{2}$ $\displaystyle=\mathcal{L}\left({\bm{L}}{\bm{X}}\right)$ $$

∎

Secondly, for completeness, we prove the upper-bound reported in Equation [3](https://arxiv.org/html/2510.26771v1#S2.E3). For quantization scheme with minmax scales shared across all feature channels $s_{ij}=\overline{s}_{i}$:

$$ $\displaystyle\mathcal{L}\left({\bm{x}}_{i}\right)\leq\frac{d}{4}\frac{\mathbb{E}\left[\text{range}\left({\bm{x}}_{i}\right)^{2}\right]}{(2^{b_{i}}-1)^{2}}$ (11) $$

###### Proof.

$$ $\displaystyle\mathcal{L}\left({\bm{x}}_{i}\right)$ $\displaystyle=\mathbb{E}\left[\left\|\mathcal{Q}\left({\bm{x}}_{i}\right)-{\bm{x}}_{i}\right\|_{2}^{2}\right]$ $\displaystyle=\mathbb{E}\left[\left\|\mathbf{Q}^{-1}\left(\left\lfloor\frac{x_{i}}{\overline{s}_{i}}\right\rceil+z_{i}\right)-{\bm{x}}_{i}\right\|_{2}^{2}\right]$ $\displaystyle=\mathbb{E}\left[\left\|\left(\left\lfloor\frac{{\bm{x}}_{i}}{\overline{s}_{i}}\right\rceil+z_{i}-z_{i}\right)\overline{s}_{i}-{\bm{x}}_{i}\right\|_{2}^{2}\right]$ $\displaystyle=\mathbb{E}\left[\left\|\left(\left\lfloor\frac{{\bm{x}}_{i}}{\overline{s}_{i}}\right\rceil-\frac{{\bm{x}}_{i}}{\overline{s}_{i}}\right)\overline{s}_{i}\right\|_{2}^{2}\right]$ $\displaystyle=\mathbb{E}\left[\left\|\left(\left\lfloor\frac{{\bm{x}}_{i}}{\overline{s}_{i}}\right\rceil-\frac{{\bm{x}}_{i}}{\overline{s}_{i}}\right)\right\|_{2}^{2}\overline{s}_{i}^{2}\right]$ $\displaystyle\leq\mathbb{E}\left[\left\|\frac{{\bm{1}}}{2}\right\|_{2}^{2}\overline{s}_{i}^{2}\right]$ $\displaystyle=\frac{d}{4}\frac{\mathbb{E}\left[\text{range}\left({\bm{x}}_{i}\right)^{2}\right]}{(2^{b_{i}}-1)^{2}}.$ $$

∎

Lastly, we bound the range using the norm 2:

$$ $\displaystyle\text{range}\left({\bm{x}}_{i}\right)^{2}\leq 2\left\|{\bm{x}}_{i}\right\|_{2}^{2}.$ (12) $$

Equality is attained whenever $\hat{{\bm{x}}}_{i}$ consists of two non-zero entries $-v$ and $v$, for which $\text{range}\left({\bm{x}}\right)=2v$ and $\left\|{\bm{x}}\right\|_{2}=\sqrt{2}v$.

Using this three steps we can write the proof for Theorem [1](https://arxiv.org/html/2510.26771v1#Thmtheorem1)

###### Proof.

$$ $\displaystyle\mathcal{L}\left({\bm{X}};{\bm{L}}\right)$ $\displaystyle\stackrel{{\scriptstyle\ref{eq:ortho_invariance}}}{{=}}\mathcal{L}\left({\bm{L}}{\bm{X}}\right)$ $\displaystyle=\sum_{i=1}^{s}\mathcal{L}\left({\bm{l}}_{i}{\bm{X}}\right)$ $\displaystyle\stackrel{{\scriptstyle\ref{eq:range_ubound_proof}}}{{\leq}}\frac{d}{4}\sum_{i=1}^{s}\frac{\mathbb{E}\left[\text{range}\left({\bm{l}}_{i}{\bm{X}}\right)^{2}\right]}{(2^{b_{i}}-1)^{2}}$ $\displaystyle\stackrel{{\scriptstyle\ref{eq:range_norm}}}{{\leq}}\frac{d}{2}\sum_{i=1}^{s}\frac{\mathbb{E}\left[\left\|{\bm{l}}_{i}{\bm{X}}\right\|_{2}^{2}\right]}{(2^{b_{i}}-1)^{2}}$ $$

∎

###### Proof.

###### Proof.

###### Proof.

### A.2 Optimal bit width allocation

We determine the optimal bit width allocation ${\bm{b}}^{*}$ for an energy vector ${\bm{e}}$ by
determining the bit width for which the ratio $e_{i}/2^{2b_{i}^{*}}\approx e_{i}/(2^{b_{i}}-1)^{2}$ is constant for all tokens:

$$ $\displaystyle\begin{cases}\frac{e_{i}}{2^{2b_{i}^{*}}}=k\\ \sum_{i=1}^{s}b_{i}^{*}=B\end{cases}$ (13) $\displaystyle\implies$ $\displaystyle b_{i}^{*}=\frac{\log_{2}e_{i}-\log_{2}k}{2}$ (14) $\displaystyle\implies$ $\displaystyle\sum_{i=1}^{s}\frac{\log_{2}e_{i}-\log_{2}k}{2}=B$ (15) $\displaystyle\implies$ $\displaystyle\log_{2}k=\frac{1}{s}\sum_{i=1}^{s}\log_{2}e_{i}-\frac{2B}{s}$ (16) $\displaystyle\implies$ $\displaystyle b_{i}^{*}=\frac{\log_{2}e_{i}-\frac{1}{s}\sum_{i=1}^{s}\log_{2}e_{i}+\frac{2B}{s}}{2}$ (17) $\displaystyle\implies$ $\displaystyle b_{i}^{*}=\log_{2}\sqrt{e_{i}}+\underbrace{\frac{B-\sum_{i=1}^{s}\log_{2}\sqrt{e_{i}}}{s}}_{C}$ (18) $$

### A.3 Effectiveness of energy concentration

We demonstrate the effectiveness of the proposed Sequence Transform and Mixed precision scheme by considering the upper bound in Equation [8](https://arxiv.org/html/2510.26771v1#S3.E8) in two different settings.
In order to simplify computation, we will consider $\frac{e_{i}}{2^{2b_{i}}}$ instead of $\frac{e_{i}}{(2^{b_{i}}-1)^{2}}$ since the difference between the two quantities is negligible for practical values of $b_{i}$.

- 1.
Uniform Energy:
In this scenario we have $e_{i}=E/s$ and $b_{i}=B/s$.
We note that the energy for each token is equal to the average of the squared eigenvalues $\lambda_{i}^{2}$ of ${\bm{S}}$:
$\displaystyle e_{i}=E/s=\text{Trace}({\bm{S}})/s=\frac{1}{s}\sum_{i=1}^{s}\lambda_{i}^{2}\stackrel{{\scriptstyle def}}{{=}}\overline{\lambda^{2}}.$
(19)
Therefore:
$\displaystyle\frac{d}{2}\sum_{i=1}^{s}\frac{e_{i}}{2^{2b_{i}}}=\frac{d}{2}\sum_{i=1}^{s}\frac{\overline{\lambda^{2}}}{2^{2B/s}}=\frac{ds}{2}2^{\log_{2}\overline{\lambda^{2}}-2B/s}$
(20)
- 2.
Maximum Energy Concentration:
We consider a scenario in which the energy corresponds with the squared eigenvalues $e_{i}=\lambda_{i}^{2}$. The optimal bit allocation is given by Equation [18](https://arxiv.org/html/2510.26771v1#A1.E18):
$\displaystyle b_{i}^{*}=\log\lambda_{i}+\frac{B}{s}-\underbrace{\frac{\sum_{i=1}^{s}\log_{2}\lambda_{i}}{s}}_{\overline{\log_{2}\lambda}}$
(21)
Therefore:
$\displaystyle\frac{d}{2}\sum_{i=1}^{s}\frac{e_{i}}{2^{2b_{i}}}=\frac{d}{2}\sum_{i=1}^{s}\frac{\lambda_{i}^{2}}{2^{\log\lambda_{i}^{2}+\frac{2B}{s}-2\overline{\log_{2}\lambda}}}=\frac{ds}{2}2^{\overline{\log_{2}\lambda^{2}}-2B/s}$
(22)

Comparing the two results is equivalent to comparing $\log_{2}\overline{\lambda^{2}}$ (uniform) and $\overline{\log_{2}\lambda^{2}}$ (max concentration).
Using Jensen’s inequality and the convexity of $log_{2}$, we have:

$$ $\displaystyle\overline{\log_{2}\lambda^{2}}=\mathbb{E}[\log_{2}\lambda_{i}^{2}]\leq\log\mathbb{E}[\lambda_{i}^{2}]=\log_{2}\overline{\lambda_{i}^{2}}.$ (23) $$

Therefore, the Maximum Energy concentration strategies achieves a lower value than the uniform scheme.

## Appendix B Experimental details

### B.1 LVMs

#### Baselines

We implement SmoothQuant, QuaRot, ViDiT-Q and SVDQuant closely following the details reported in the paper and the respective codebases. Specifically, for ViDiT-Q [Zhao et al., 2025] we use $\alpha=0.01$, as reported in their PixArt-$\Sigma$ setup. For SVDQuant and SmoothQuant we use a default value of $\alpha=0.5$.

#### Quantization

In order to promote a fair comparison with the other models, for the results reported in Table [1](https://arxiv.org/html/2510.26771v1#S5.T1), instead of using the 8-bits weight quatization scheme described in ViDiT-Q (8 bits for the FFN blocks, first and last transformer blocks), we use per-block weight and activation quantization, as described in SVDQuant [Li et al., 2025], which improves the ViDiT-Q results.
In line with the SVDQuant methodology, we retain the depth-wise convolutions within the feed-forward layers
of the SANA transformer blocks in full precision. Meanwhile, the two point-wise convolutions are quantized
by treating them as linear layers.
Notably, unlike the original SVDQuant implementation, our experiments do not incorporate GPTQ at any stage.
The activation-quantization only results reported in Figure [4](https://arxiv.org/html/2510.26771v1#S3.F4) and Figure [7](https://arxiv.org/html/2510.26771v1#S5.F7) use 4 bits per token quantization.

#### STaMP

For the STaMP results, excluding the specific ablation study, we use 64 high-precision tokens, which results in an effective activation bit-width of 4.0625 bits on the PixArt-$\Sigma$ model and 4.125 bits on the SANA model. Non-STaMP result do not use any mixed precision tokens.

### B.2 LLMs

#### Quantization

We use dynamic quantization for the KV cache and activations. Quantization scales and offsets are determined per token, sequence, and (for KV) head. We follow [Ashkboos et al., 2024, Sun et al., 2025] and only quantize weights, KV cache, and inputs to linear layers within the transformer block. We use round-to-nearest weight quantization, as weights are unaffected by STaMP—more advanced weight quantization schemes could improve results further, but this is perpendicular to our contribution. we range set the weights by computing the weight quantization squared error for a grid of candidate ranges and selecting the candidate with lowest error. For fairness, for all experiments (incl. baselines) we keep the first 64 tokens in 8 bits, which gives an effective bit width of 4.125 for both STaMP and non-STaMP results.

#### Baselines

For SmoothQuant, we calibrate the scales based on the Wikitext-2 training dataset activations.
For QuaRot, we follow the original paper and reduce the minmax activation range by 10%. For FlatQuant, we use their recommended settings for all models (e.g. training for 15 epochs on 128 Wikitext-2 training sequences).

#### STaMP

The first token in most LLMs acts as an attention sink Xiao et al. [2024], Gu et al. [2025], and typically contains massive outliers Sun et al. [2024]. Keeping the first token in 8 bits helps to accurately represent these massive outliers. To ensure the massive outlier stays in the first token, STaMP is not applied to the first token. Note that the baselines that do not use STaMP’s transform, do benefit from keeping the first token (and next 63) in high-precision in our experiments.

### B.3 Notes on the runtime estimation

To compute the overhead associated with the sequence and feature transforms, we measured the time
required to run a single denoising step with the original PixArt-$\Sigma$ model, and compared it
with the modified architectures (having the extra transform operations).
Measurements are performed on an A100 GPU using PyTorch.
Hadamard transforms are based on the CUDA-accelerated kernels from the fast-hadamard-transform package (^3^33https://github.com/Dao-AILab/fast-hadamard-transform).
For Haar DWT operations, we built a specialized CUDA kernel, optimized for applying DWT over the sequence dimension.
The increased latency measured with the Hadamard applied on the sequence dimension can be mainly attributed to the memory reshaping operations required to use the fast-hadamard-transform kernel.

## Appendix C Additional Results

We report additional results for the setups described in Section [5](https://arxiv.org/html/2510.26771v1#S5), including an ablation of the effect of STaMP on different activations (Table [4](https://arxiv.org/html/2510.26771v1#A3.T4)), bit width vs SQNR comparison against per-block activation quantization (Figure [9](https://arxiv.org/html/2510.26771v1#A3.F9)), and additional metrics (Table [5](https://arxiv.org/html/2510.26771v1#A3.T5)) and visualizations (Figure [10](https://arxiv.org/html/2510.26771v1#A3.F10)) for the results reported in the main text.

**Table 4: Effect of A4 activation quantization on single activations of the Pixart-$\Sigma$ model on the Image SQNR. Each entry corresponds to a model for which only the corresponding input activation is quantized. Note that STaMP has little to no effect on attn2.to_out since the structure is determined by pooled textual embedding, which does not present the same correlation structure as the other activations in the network.**
| Transform | attn1 | attn1.to_out | attn2.to_q | attn2.to_out | ffn.up_proj | ffn.down_proj |
| --- | --- | --- | --- | --- | --- | --- |
| Identity | 6.01 | 10.09 | 0.40 | 15.48 | 1.15 | 4.92 |
| QuaRot | 13.80 | 16.01 | 16.29 | 23.20 | 11.36 | 8.42 |
| STaMP | 7.68 | 13.89 | 11.60 | 15.49 | 8.75 | 9.61 |
| QuaRot+STaMP | 15.28 | 16.78 | 18.31 | 23.23 | 13.84 | 14.91 |

Figure: Figure 8: Visualization of the Images generated with A4 activation quantization and several combination of Feature and Sequence transforms for the Pixart-$\Sigma$ architecture. The images refer to the same setup described in Figure [7](https://arxiv.org/html/2510.26771v1#S5.F7).
Refer to caption: figures/heatmap_pixart-sigma_images.png

Figure: Figure 9: Tradeoff between SQNR and bit width for per-token activation quantization (pt) per-block activation quantization (pb) at different block size (from 16 to 1024), and per-token with STaMP (pb+STaMP) at varying number of high precision tokens on the PixArt-$\Sigma$ model. We consider 16 bits scales for each scale parameter. The visualization correspond to the Prompt ’A guy with a backpack looking at the ground to his left.’.
Refer to caption: x8.png

Figure: Figure 10: Visualization of a sample generation for the SANA for the results reported in Table [1](https://arxiv.org/html/2510.26771v1#S5.T1).
Refer to caption: figures/visual_results/FP_2.png

**Table 5: Additional metrics for the experiments reported in Table [1](https://arxiv.org/html/2510.26771v1#S5.T1).**
| Dataset | LVM | Method | STaMP | CLIP | CLIP IQA | SQNR (latent) |
| --- | --- | --- | --- | --- | --- | --- |
| COCO | PixArt-$\Sigma$ | FP |  | 31.54 | 0.91 | $+\infty$ |
| RTN | ✗ | 31.23 | 0.79 | 0.19 |  |  |
| ✓ | 31.88 | 0.91 | 0.49 |  |  |  |
| SVDQuant | ✗ | 31.68 | 0.91 | 0.96 |  |  |
| ✓ | 31.76 | 0.91 | 1.47 |  |  |  |
| ViDiT-Q | ✗ | 31.63 | 0.87 | 0.73 |  |  |
| ✓ | 31.92 | 0.87 | 1.03 |  |  |  |
| SANA | FP |  | 31.85 | 0.89 | $+\infty$ |  |
| RTN | ✗ | 31.85 | 0.89 | 3.39 |  |  |
| ✓ | 31.94 | 0.89 | 4.09 |  |  |  |
| SVDQuant | ✗ | 31.83 | 0.90 | 4.61 |  |  |
| ✓ | 31.88 | 0.90 | 5.30 |  |  |  |
| ViDiT-Q | ✗ | 31.79 | 0.89 | 4.63 |  |  |
| ✓ | 31.82 | 0.89 | 5.42 |  |  |  |
| MJHQ | PixArt-$\Sigma$ | FP |  | 31.46 | 0.85 | $+\infty$ |
| RTN | ✗ | 29.90 | 0.75 | 0.28 |  |  |
| ✓ | 30.96 | 0.87 | 0.71 |  |  |  |
| SVDQuant | ✗ | 31.13 | 0.86 | 1.27 |  |  |
| ✓ | 31.28 | 0.87 | 1.86 |  |  |  |
| ViDiT-Q | ✗ | 30.97 | 0.81 | 0.86 |  |  |
| ✓ | 31.20 | 0.83 | 1.31 |  |  |  |
| SANA | FP |  | 31.57 | 0.83 | $+\infty$ |  |
| RTN | ✗ | 31.51 | 0.84 | 4.45 |  |  |
| ✓ | 31.60 | 0.84 | 5.39 |  |  |  |
| SVDQuant | ✗ | 31.52 | 0.83 | 5.69 |  |  |
| ✓ | 31.55 | 0.83 | 6.46 |  |  |  |
| ViDiT-Q | ✗ | 31.60 | 0.83 | 5.76 |  |  |
| ✓ | 31.60 | 0.83 | 6.65 |  |  |  |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 4
questions: See Weakness.
soundness: 3
strengths: 1. The paper is really well-written in both figures, tabs, equations, and texts. 
2. The paper contains experiments on both SD and LLM. 
3. The method design features strong complementarity and is compatible with the existing technical system.
summary: 1. The paper proposes STaMP quantization to mitigate accuracy loss in low-precision activation quantization for LLMs and LVMs.
2. Unlike existing feature-wise methods, STaMP applies linear transformations along the sequence dimension to exploit local correlations in language and visual data.
3. It uses a mixed-precision scheme: a small number of high-energy tokens are kept at higher precision to maintain accuracy with lower average bit-widths.
4. STaMP complements established feature transformations and weight quantization methods without conflicting with them.
5. Experiments on models like Llama 3 and PixArt-Σ demonstrate consistent accuracy improvements in 4-bit activation quantization.
6. For efficiency, STaMP adopts practical transforms instead of complex KLT, balancing energy concentration and computational cost.
weaknesses: 1. I a m confused  with the argument in QuaRot that rotation only occurs in the feature dimension, because some weights (such as up/gate projection) are left-multiplied by the rotation matrix.
2. The experiments on Large Language Models is insufficient: (1) The model size is relatively small, and it has not been scaled up to the magnitude greater than 8B; (2) There are only PPL experiments, with the lack of mainstream zero-shot reasoning experiments, which are more crucial for verifying the effectiveness of PTQ.
3. The paper introduces left-invertible matrices, but it does not provide a detailed and clear explanation of how such matrices are integrated into the weight parameters of the original architecture (like QuaRot and SpinQuant).
4. As a work focusing on quantization, the paper does not report experimental results related to accelerated inference.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See Above
soundness: 4
strengths: 1. New Perspective: The paper introduces quantization transformations along the sequence dimension—a novel and orthogonal approach compared to existing methods. This direction effectively exploits temporal or spatial token correlations often overlooked by prior research.

2. Solid Theoretical Analysis: The work offers a well-defined mathematical treatment of quantization error and establishes a theoretical upper bound (Theorem 1) that connects token energy, bit allocation, and resulting error.

3. Synergistic with Existing Methods: The proposed approach integrates seamlessly with leading quantization strategies (e.g., QuaRot, SmoothQuant, FlatQuant, SVDQuant), delivering complementary and cumulative performance improvements.
summary: This paper proposes Sequence Transformation and Mixed Precision(STaMP), a new PTQ method that operates along the sequence dimension, different from most recent methods that focus on utilizing rotation to modify along the feature dimension. The key idea is to exploit local correlation between sequential tokens by applying a sequence-wise linear transform (e.g., DCT, DWT) before quantization, followed by a mixed-precision strategy that assigns higher bit-widths to a subset of tokens carrying higher "energy."
weaknesses: 1. Missing Real-Time and Latency Evaluation: The paper does not report inference speed, latency, or hardware efficiency metrics—critical aspects for assessing quantization performance. Given that the multiplication of XL occurs online, it would be valuable to analyze its impact during inference.

2. Absence of Calibration Set Ablation: There is no investigation into how the size of the calibration dataset influences performance, which is an essential factor for post-training quantization (PTQ) methods.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: 1. Regarding practical implementation on current mainstream GPUs (e.g., NVIDIA A100/H100), could the authors elaborate on a viable strategy for the proposed mixed-precision scheme? What is the anticipated latency overhead introduced by dequantization or conditional logic compared to a uniform W4A4 implementation? 
2. To contextualize the cost of the proposed transform, could the authors provide a brief computational complexity analysis comparing the $O(ds)$ cost of DWT to the $O(s \cdot d \cdot d_{ffn})$ cost of a typical Transformer FFN layer, thereby quantifying the relative overhead introduced by the transform itself? 
3. In the LLM experiments, a fixed strategy of keeping the first 64 tokens in high precision was applied to all methods. Have the authors considered whether this is an optimal strategy for the baseline methods, which do not employ sequence transforms? An ablation study that implements and compares against a dynamic, magnitude-based token selection strategy for the baselines could further strengthen the fairness of the experimental comparison. 
4. To strengthen the paper's motivation, particularly the analogy to media compression, further insight is requested. Technologies like JPEG use energy compaction to more aggressively compress high-frequency information, which is less perceptible to the human eye. What is the conceptual equivalent in the context of STaMP? What kind of information corresponds to the low-energy components after the DWT/DCT transform, which are subsequently assigned lower precision? A more intuitive explanation of which aspects of the token sequence are deemed "less critical" by the transform would significantly enhance the persuasiveness of the paper's motivation.
$\textbf{If these issues can be resolved, I will consider giving a higher score.}$
soundness: 3
strengths: 1. High Novelty: The primary strength of this work lies in its high degree of originality. By shifting the focus of quantization transforms from the feature dimension to the sequence dimension, the authors introduce a new research paradigm.
 2. Solid Experimental Results: The paper's claims are supported by consistent and significant performance improvements across a range of models (LLMs and LVMs), datasets, and strong baselines. 
3. Excellent Complementarity: The experiments clearly demonstrate that STaMP is an orthogonal improvement that can be combined with state-of-the-art methods like QuaRot to achieve further gains. 
4. Clear Motivation and Exposition: The analogy to classical media compression provides a strong theoretical basis and an intuitive understanding of the proposed method.
summary: This manuscript introduces STaMP (Sequence Transformation and Mixed Precision), a novel post-training quantization (PTQ) framework aimed at addressing low-bit activation quantization in large generative models (LLMs and LVMs). The core innovation, which distinguishes this work from prior art focused on feature-dimension transforms, is the application of linear transformations (e.g., DWT) along the sequence dimension. This method seeks to exploit local token correlations to compact activation energy into a minority of transformed tokens. Subsequently, the framework employs a mixed-precision strategy, allocating higher bit-widths to these high-energy tokens while quantizing the remainder more aggressively. The authors demonstrate through experiments that STaMP can serve as an orthogonal module that  consistently improves the performance of various existing PTQ methods under a W4A4 setting, without requiring any retraining.
weaknesses: 1. Unaddressed Implementation Overhead: The most significant weakness is the failure to discuss or quantify the runtime overhead of the per-token mixed-precision scheme on current hardware. The lack of native hardware support may necessitate inefficient dequantization operations, which would severely impact the method's practical speedup. 
2. Unquantified Transform Cost: While the computational cost of the DWT/DCT transform is theoretically small, the paper fails to provide a simple quantitative analysis to substantiate this claim (e.g., by comparing it to the computational load of an FFN layer in a Transformer block), leaving its relative overhead ambiguous.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. How would STaMP operate during autoregressive decoding, where tokens are generated sequentially? Since the sequence transform matrix L require access to the full sequence, does it mean that we need to recompute L in every decoding step?
2. Could you provide quantitative measurements of the runtime and memory overhead introduced by applying and inverting the sequence transform in large transformer models? How does this cost compare to the savings from reduced activation bitwidth?
3. How is the mix-precision processed in STaMP? Specifically, since there are 64 tokens kept at 8 bits, and the rest uses 4 bits, how do you run the mix-precision model efficiently?
soundness: 3
strengths: 1. The paper introduces sequence-domain transformations for activation quantization, a direction largely unexplored compared to prior channel-wise or feature-domain approaches. It can be combined with existing quantization methods and feature transformations without retraining or altering model weights.
2. Extensive results on both LVMs (PixArt-Σ, SANA) and LLMs (LLaMA 3, Qwen 2.5) show clear improvements in SQNR, perplexity, and visual quality under low-bit settings (W4A4), confirming robustness and generality.
3. The paper provides clear figures (e.g., Figs. 2–6) illustrating how energy concentration improves quantization performance and complements feature transforms.
summary: This paper proposes STaMP (Sequence Transformation and Mixed Precision), a novel post-training activation quantization method for LLMs and LVMs. Unlike prior work that applies invertible feature transformations across the channel dimension, STaMP introduces linear sequence transformations to exploit correlations between neighboring tokens. By concentrating activation energy into a few tokens and assigning them higher precision under a fixed average bit budget, STaMP effectively reduces quantization error. The authors approximate the optimal Karhunen–Loève Transform using efficient DCT or DWT implementations and derive theoretical bounds on quantization loss. Extensive experiments on PixArt-Σ, SANA, and LLaMA/Qwen models show consistent improvements over existing methods such as SmoothQuant, QuaRot, and SVDQuant, both in SQNR and downstream performance, with minimal computational overhead.
weaknesses: 1. The paper does not quantify the additional latency or memory cost of applying and inverting sequence transforms in large-scale models. Without such analysis, it is difficult to assess whether the efficiency gains from lower-bit activations outweigh the added transform cost in practical deployment scenarios.
2. For the decoding stage of LLMs, STaMP may not be directly applicable, since each newly generated token would require recomputing the sequence transform matrix L.


Decision: Accept (Poster)