Shift-and-Sum Quantization for Visual Autoregressive Models
Authors: Jaehyeon Moon
Bumsub Ham
Venue: ICLR 2026 Poster (2026)
Keywords: VAR
network quantization
Primary area: generative models

Abstract: Post-training quantization (PTQ) enables efficient deployment of deep networks using a small set of data. Its application to visual autoregressive models (VAR), however, remains relatively unexplored. We identify two key challenges for applying PTQ to VAR: (i) large reconstruction errors in attention–value products, especially at coarse scales where high attention scores occur more frequently; and (ii) a discrepancy between the sampling frequencies of codebook entries and their predicted probabilities due to limited calibration data. To address these challenges, we propose a PTQ framework tailored for VAR. First, we introduce a shift-and-sum quantization method that reduces reconstruction errors by aggregating quantized results from symmetrically shifted duplicates of value tokens. Second, we present a resampling strategy for calibration data that aligns sampling frequencies of codebook entries with their predicted probabilities. Experiments on class-conditional image generation, in-painting, out-painting, and class-conditional editing show consistent improvements across VAR architectures, establishing a new state of the art in PTQ for VAR.

Published as a conference paper at ICLR 2026
SHIFT-AND-SUM QUANTIZATION FOR VISUAL AU- TOREGRESSIVE MODELS
Jaehyeon Moon'
Bumsub Ham
Yonsei University
Articron
(jhmn, bumsub. haam)yyoneei.ac
ABSTRACT
Post-training quantization (PTQ) enables efficient deployment of deep networks using a small set of data. Its application to visual autoregressive models (VAR), however, remains relatively unexplored. We identify two key challenges for ap- plying PTQ to VAR: (i) large reconstruction errors attention-value products, especially coarse scales where high attention scores occur more frequently; and (ii) a discrepancy between the sampling frequencies of codebook entries and their predicted probabilities due to limited calibration data. To address these chal- lenges, we propose a PTQ framework tailored for VAR. First, we introduce a shift-and-sum quantization method that reduces reconstruction errors by aggre- gating quantized results from symmetrically shifted duplicates of value tokens. Second, present a resampling strategy for calibration data that aligns sam- pling frequencies of codebook entries with their predicted probabilities. Experi- ments on condiicondii image generation, inpainting, outpainting, and class- conditional editing show consistent improvements across VAR architectures, es- tablishing a new state of the art in PTQ for VAR. Project page available at: http:://v/ wttp///vllbb..... yonsei ac. wt.kkr//prrooo htr/projectss -td- /um/
INTRODUCTION
With the advancement of large language models (LLM), researchers have increasingly adopted au- toregressive (AR) modeling predicting the next token in sequence for a variety of gen- erative tasks. Recently, visual autoregressive models (VAR) (Tian et al., 2024) have emerged as a powerful alternative to state-of-the-art generative models (e.g., diffusion models (Ho et al. 2020), GANs (Goodfellow et al.] 2), by replacing conventional next-token generation with a next-seale generation strategy, making them increasingly popular in the field of computer vision. However, the use of multiple transformer blocks and their iterative generation process across multiple scales re- quire significant computational resources (e.g., memory, FL.Pss). As result, network compression techniques (e.g., pruning, quantization) have gained significant attention to enable their deploy- ment practical devices. Network quantization, which converts the weights and activations of model into low-bit representations for efficient inference, is typically classified into two categories: quantization-aware training (QAT) and post-training quantization (PTQ). QAT methods (Rastegari et al. 2016) Esser et al., 2020) apply quantizers to weights and activations and retrain the model using full training dataset, which computationally intensive. In contrast, PTQ methods (Banner et al.
2019 Shang et al. (023) calibrate quantization parameters (e.g., scale, zero-point) using small subset training data only, allowing for rapid and resource-efficient deployment.
VAR consists of VAR transformer and multi-scale vector quantized variational autoencoder (VQ- VAE) (Van Den Oord et al. (007)), referred to as the VAR tokenizer. It employs a hierarchical gener- ation process that begins with a start token at the coarsest scale. At each scale, the VAR transformer takes token maps from previous scales as input and predicts a probability distribution over entries of the VQVAE codebook for each position. At each position, single entry is sampled from the pre- dicted probabilities to construct the token map at current seale. This process iterates across seales, progressively refining the representation higher resolutions. After completing all scales, the final token map is passed to the VQVAF decoder to generate the image.
"Corresponding author.

---

Published as a conference paper at ICLR 2026
3
1.5
VAR-d16 VAR-d20
Probability (x10-))
Rocon ettor bclo
1.0-
Frequency
2+
10
2 3 4
6
8
10
13
16
1950
2000
2050
1950
2000
2050
Feature resolution
Codebook entries
Codebook entries
(a)
(b)
Figure I: (a) Visualization of reconstruction
for the multiplication of attention scores and value tokens
across scales, particular transformer block of VAR-d16 and VAR-d20; (b) Comparison between predicted probabilities and sampling frequencies of codebook entries, aggregated from 256 calibration samples in VAR- d24. Note that we show 100 codebook entries only for visualization. We can see that attention-valle products tend to incur large reconstruction errors coarse scales, and sampling frequencies of codebook entries do not align with their predicted probabilities.
Although PTQ has shown promising results on conventional generative models, c.g., diffusion mod- els (Ho et al. 2020), its application to VAR remains relatively unexplored. We identify two key challenges in quantizing VAR (Fig. D: (1) Multiplication between attention scores and value tokens in VAR transformer incurs significant errors after quantization, especially low resolutions. We find that quantization errors are amplified for value tokens with large attention scores, which occur more frequently at low resolutions due the reduced number of tokens. To this end, introduce a shift-and-sum quantization technique, that duplicates the value tokens associated with large atten- tion scores, applies symmetric shifts around the original value, and averages the quantized results to suppress quantization errors. (2) In the calibration set, the sampling freguency of each codebook entry-i.e., the number of times sampled across all calibration samples-one often diverges from the predicted probabilities aggregated over token positions, due to the extremely limited number of available samples in PTQ. This mismatch biases the quantization parameters toward an inaccurate distribution, degrading quantization performance. To this end, we introduce a calibration data sampling technique that aligns the frequencies with predicted probabilities in the calibration data by reallocating tokens from oversampled to undersampled codebook entries.
Our contributions are summarized as follows:
We identify two key challenges specific to quantizing VAR: (1) significant quantization errors aris- ing from the multiplication between attention scores and value tokens, especially at coarse scales; and (2) a mismatch between the predicted probabilities their sampling frequencies during calibration.
the entries of VQVAE codebook and
We propose a shift-and-sum quantization technique that reduces quantization error in attention- value products by aggregating quantized results from symmetrically shifted value tokens.
We introduce a calibration data resampling technique to align the sampling frequencies of code- book entries with their predicted probabilities in the calibration data.
We establish state of the art on class-conditional image generation across VAR architectures, with consistent improvements on the tasks of image inpainting, outpainting, and class-conditional editing.
2 RELATED WORKS
2.1 VAR
AR models (Van den Oord et al. 2016 Ramesh et al.) 2021 Sun et al. 2024) generate images by predicting each token once at a time in a raster-scan order, conditioning each step on previously generated tokens. The generation process captures fine-grained dependencies between tokens but incurs substantial overhead especially for high-resolution images (Razavi et al., 2019). In contrast, VAR (Tin et al. 2024) adopts coarse-to-fie -ine generation strategy: it generates token maps progres- sively from coarse to fine scales, with each scale conditioned on the token maps of preceding scales. This hierarchical design enhances structural coherence and improves sampling efficiency compared
2

---

Published as a conference paper at ICLR 2026
to conventional AR models. ControlVAR (Li et al. image generation, where the model takes a control map (e.g.. segmentation mask) as input and gen- erates image conditioned on them. Infinity (Han et al. 2025) introduces a bitwise tokenizer that encodes each token as a binary sequence, allowing the number of possible representations to grow exponentially with the sequence length. CoDe (Chen al., 2025) generates token maps using large drafter model at coarse scales and small refiner model at fine scales. Despite their strong generative capabilities, these models remain computationally demanding due to their hierarchial generation process and multiple transformer blocks.
2024) extends this framework conditional
2.2 NETWORK QUANTIZATION
Network quantization reduces the bit-widths of weights and activations in model to enable efficient inference. QAT methods incorporate quantizers into the weights and activations of the model and retrain it using differentiable approximations of the rounding function (e.g.. Esser et al. (2200). Despite their strong performances, QAT requires access full training dataset and incurs substantial computational cost. In contrast, PTQ adjusts quantization parameters using only subset of training data, enabling fast and data-efficient deployment. Early works calibrate quantization intervals by minimizing quantization error (Banner et al. 2019)) or task-specific losses (Nahshan et al. 202T
Another line of research (Nagel et al. 2020 Li et al. 2021 Lee et al. 2023 focuses on optimizing the rounding function for network weights instead of using standard rounding to--ggttt----rss schemes.
To date, a number of PTQ techniques have been developed for transformer architectures. For ex- ample, PTQ4VIT (Yuan et al.
2022) proposes twin uniform quantizers to address the long-tailed distributions of softmax attentions and activations after GELU (Hendrycks & Gimpel| (016) non- linearity. RepQ-ViT (Li al.) 2023b) proposes consider inter-channel scale variations of acti- vations after LayerNorm, and exploits a scale reparameterization technique that adjusts the affine factors of LayerNorm and the weights of subsequent fully-connected (FC) layers. IGQ-VIT (Moon) itt al. 2024) observes that input activations of FC layers vary drastically across input instances, and splits the channels of activation maps dynamically into multiple groups, quantizing the activation values within each group with a shared quantizer. ERQ (Zhong et al. 2025) proposes to adjust the rounding function for weights to minimize the quantization errors induced from activation quantiza- tion.
Recent methods have extended PTQ to diffusion models (Ho et al. 2020), which synthesize images by iteratively applying denoising process over time steps. For example, PTQ4DM (Shang et al.) 2023) proposes to sample more images from later time steps for calibration, which are typically closer to real images. Q-diffusion (Li et al.) (2023a) observes that concatenation of activations from different layers can result in significant inter-channel scale variation, and proposes to quantize ac- tivations before concatenation to mitigate this issue. TFMQ-DM (Huang et al., 2024 shows that quantizing temporal embedding layers can distort temporal information in diffusion models, and proposes to calibrate them separately from other layers. AccuQuant (Lee al.) 2025) groups con- secutive time steps and minimizes the discrepancy between the quantized and ful full-precision outputs within each group, thereby reducing the accumulation of quantization error across time steps.
Despite these advances, applying PTQ to VAR remains largely unexplored. To the best of our knowledge, VAR (Xic et al.) 2024) is the only PTQ method for VAR. It demonstrates that quan- tizing FC layers following the GELU non-linearity leads to substantial performance degradation, and retains them in full-precision. However, this approach introduces considerable computational overhead and requires hardware support for mixed precision arithmetic. In contrast, our approach quantizes all tensors involved in matrix cultipiicatios-incllddiit
the weights and activations for every FC layer, as well as the activations in self-attention mechanisms (i.e., query, key, value, and softmax (ocattoon-usigg the same bit-width, allowing for efficient deployment on conventional hardware.
METHOD
3.1 PRELIMINARIES
Uniform quantizer. Given a floating point value and quantization bit-width b, the uniform quan- tizer maps b-bit integer as follows:
f = clip(
1 2, 0, 2 1),
(1)
3

---

Published as a conference paper at ICLR 2026
where the step size and zero-point are defined as:
max(x) min(a)
2 clppp min(a))
1. 0, - -1.
26
We denote 1.1 as rounding function, and clip(., m, n) clipping function with lower and upper bounds of m and n, respectively. The quantized output is then computed as:
Q(1; s( - 2).
(3)
Log2 quantizer. The log2 quantizer discretizes with bit-width of follows:
= clip( log2 0, 2 1),
(4)
where the scale is typically set to max(x)) The quantized output is obtained as:
Q(r;s) = s2
(5)
where both the logarithm and exponential base-2 operations be efficiently implemented using bit-shift operations. Following (Lin et al. [2022] Li et al., (023b), we exploit log2 quantizers for softmax attentions, and uniform quantizers for all other weights and activations.
3.2 PROPOSED METHOD
In this section, we introduce our PTQ framework for VAR, consisting of two components: (1) Shifi- and-sum quantization (Sec. (.2.1). which reduces reconstruction errors attention-value products of VAR transformer; and (2) calibration data resampling (Sec. B.2.2), which alleviates the mis- match between sampling frequencies codebook entries and their predicted probabilities during calibration.
3.2. SHIFT-AND-SUM QUANTIZATION
Reconstruction error across scales. VAR exploits autoregressive, multi-scale generation pro- cess that begins with coarse token maps and progressively generates ones at finer scales. We observe that multiplication between attention scores and value tokens exhibits particularly large reconstruc- tion errors at coarse scales (Fig. [a)). Errors introduced at these early stages are especially detri- mental, since they propagate through subsequent scales, become amplified in the final outputs, and ultimately lead to significant performance degradation after quantization.
We hypothesize that the large quantization tokens, which we define as value tokens associated with high attention scores. To formalize this, we consider the multiplication between attention scores a R and value tokens V Rg where we denote by T and d the number of value tokens and hidden dimensions, respectively. Both are subject to quantization noise, with IR' for the attention scores and e" E Txxd for the value tokens. The quantized output can then be represented as:
coarse scales arises from the presence of attentive
Σ (a: + G))(v +.))
(6)
where we denote a; ER and V, V E R as the i-th attention score and value token, respectively,
and et E R and ea Idd are their corresponding quantization noises. Note that we use log2 quantizers for attention scores. Since they discretize values on logarithmic scale, the quantization noise e increases w.r.t. the magnitude of a, (Kofman 2009). The reconstruction error can then be expressed as:
8 I( (as + av2 + a.v, = Σ a. (E + ह. +
+ E
where we define 25 c//a- Assuming that {25 [T, and ( are independent, zero-mean random variables with variances o² and o3 respectively, the variance of the reconstruction error can be obtained as:
T
Var = Σ a + :0 o.))
(8)
4

---

Published as a conference paper at ICLR 2026
#al12
1.0
0.5
00
Attentive tokens (%)
80
#=01
Quantized value
Ouantizatin conttization arror
0.5
0.4
60
0.3
0.0
40-
0.2
20
-0.5
000
0.1
-1.0
3
4 5
10 13 16
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
Peature resolution
Figure 2: Percentage of tokens with average attention scores exceeding a threshold in VAR-d16 (left): Visu- alization of quantization kernel fn(v; t.) across different kernel orders n (middle); Quantization error of the kernel (i.e.. je- (e ta across different kernel orders (right). We present the results for multiple values of A, and we set 1, t1 1/4, and te 1/8. We visualize the upper bounds of quantization errors with horizontal stripes of corresponding colors.
A detailed derivation of Eq. 8 is provided in Appendix B We observe that the variance of the recon- struction error increases in the presence of high attention scores, given that their sum is constrained I. We observe in Fig. left that high attention scores are more common at coarse scales, where they are distributed on fewer tokens, leading to large quantization errors.
Shift-and-sum quantization for attentive tokens. We propose shift-and-sum quantization frame- work to mitigate reconstruction errors in the multiplication between attention scores and value to- kens. To this end, we introduce quantization kernel that effectively reduces quantization errors at the cost of extra operations (e.g., bit-shift, summation). We then apply the kernel only to at- tentive tokens, where the kernel orders are assigned adaptively according to their attention scores. Quantization kernel. For scalar value we define the kernel as follows (Fig. mmiddee))::
n-1
fa(u;
Σ
( + (2k- + 1)(m),
2n
where we define and in as the order and shift factor for the kernel. The upper bound of the quantization error for tv; is inversely proportional to n, as shown below. Theorem 1. The quantization error of fn (x) defined in Eq. 9 satisfies
| (n (.))
(10)
4n
for In = s/An, where 8 is the step size of Q(.). Furthermore, this bound is light: for arbitrary choices of shift factor t,., the error bound cannot be smaller than s/An.
We provide the proofs in Appendix C and visualize the quantization error of fn(u;; in Fig. rrightt. Following Theorem ! we set in to s/4n for a kernel of order n. For vector v, the kernel is applied independently to each component.
Application attention- value products. Since the quantization kernel in Eq. introduces additional computations, we apply the kernel only to attentive tokens, which contribute most to the reconstruc- tion error. practice, the kernel can be efficiently implemented by duplicating attentive tokens, shifting their values symmetrically, and aggregating the quantized results. Specifically, we compute the average attention score for each value token as follows:
Score(v:) = - ||a::l1.
(11)
T
where T" is the number of query tokens, and a; = A.. is the i-th column of the attention matrix A E xT Based on these scores, we identify attentive tokens and quantize them with the kernel, while applying standard quantizers to the remaining tokens:
xT"
T AV Σ a.. ~ (a.
Su
+
(12)
4n
NEH
i
where (-) and (.) are the quantizers for attention scores and value tokens, and and s, are the set of attentive tokens and the step size for Qo(+), respectively. The first term can be further
5

---

Published as a conference paper at ICLR 2026
T+1
sum
A
<
A
V
A
2
Pr
Di +
5,74
5//4
LUNS
mean
duplicate
bit-shift
003 U.2. 0.0
(core(P))
(v. Q. (v(a:/2)0. +s/44)+ (4122)0.0:- s.y44
Figure 3: Visualization of attention-valus products using shift-and-sut quantization for implement the quantization kernel Eq. by duplicating attentive tokens, and aggregating quantized results from their symmetrically shifted counterparts.
decomposed as:
n-1 Σ
Qalas) 4n
Qa
Qv (2k: +) Se )T)
(13)
2n
We illustrate in Fig. the multiplication of attention scores and value tokens for 1. We restrict the kernel order n to powers of two (c.g.. 1,2,4), since dividing attention scores by such values could be implemented as bit-shift operations, enabling efficient computation of (a;a/221)
Adaptive kernel order. We apply quantization kernels only to tokens whose average attention scores exceed a threshold 6, chosen satisfy BOP constraint (see Sec. 4.1, For such tokens, we choose the smallest kernel order that brings the average attention scores below 0. Specifically, for tokens with (vore(v.) > we set their orders as 28.-1 where û, defined as:
Score(v:)
à loga
(14)
0
where denote by [-] the ceiling operation. We can from Eq. 13 that quantizing v; with kernel of order n scales the attention scores os; by a factor of 1 1/2m; with a kernel order of 2 the average attention score is reduced below 0 as follows:
Scoro(vi)
S 0,
(15)
T"
2
2
Note that 2dh 2 Score(v:)/0 by the definition of n..
Comparison to mixed-precision quantization. An alternative to our method is to allocate higher bit-widths attentive tokens, which also reduce reconstruction
in attention-value prod- ucts. Consider matrix multiplication between attention scores {ax (or))1 quantized with a step size of Sa, and value tokens {= quantized with step sizes of & and & for attentive tokens and non non-attentive ones. The matrix multiplication be represented follows tzero-points omitted for clarity):
T ~ (a(as) +
Σas
aav
Σ Qa())) i
ieHl
(16)
Σ 8.S SEH
V. +
Σ 8,8, 2 as V.T
where we denote (-) (.) as a higher-bit quantizer for attentive tokens. &; and V; are integer values obtained by applying Eq. to a; and Eq. to v,, respectively. Computing Eq. [6] the sum- mation of two floating-point matrices, which could only be implemented with specialized hardware support. In contrast, method does require multiple quantizers for a single matrix, thereby avoiding floating-pott summations between different attention-vall products.
3.2.2 CALIBRATION DATA RESAMPLING
Mismatch between probabilities and frequencies. We construct token maps by randomly sam- pling codebook entries according to their predicted probabilities during calibration. Since PTQ

---

Published as a conference paper at ICLR 2026
resampting
random sampling
taken map
frequency
(+3)
VQVAE
3 (-2)
:4 (-1)
mismatch
Pa
P2
mean
oversampled:
target
undersampled:
frequency
Figure 4: Illustration of the calibration data resampling technique, which reassigns tokens from oversampled codebook entrics to undersampled ones, thereby aligning the sampling frequencies with the predicted probabil- ities aggregated over calibration samples.
typically allows only a small number of calibration samples, this random sampling process incurs a high variance: the sampling frequencies of codebook entries often deviate significantly from their predicted probabilities aggregated over calibration samples (Fig. [b)). This problem is exacerbated by the large vocabulary size of VQVAE codebooks (e.g., 4096 entries in VAR-d16), which makes it infeasible that limited calibration data can accurately reflect the underlying probability distribution.
Probability matching for calibration data generation. We propose probability matching strat- egy to align the sampling frequencies codebook entries with their predicted probabilities (Fig. Specifically, first generate calibration samples by randomly sampling token maps according to the predicted probability distributions. We then compute the mean probability each codebook entry, aggregated over token positions, as follows:
N
P
(i.).)),
(17)
NT ==
where we denote by pa (i,j, the predicted probability of selecting the k-th codebook entry for the j-th token in the i-th calibration sample, and N and T are the number of calibration samples and tokens per sample, respectively. Based Pk. define the target frequency of k-th entry as:
t NTPp.
(18)
We define the sets of oversampled and undersampled entries as
O {k Sk t 2 1}, U (k to 1},
(19)
corresponding to entries sampled more or less frequently than their target frequency. We denote by $3. the sampling frequency of the k-th codebook entry. For tokens assigned to entries in O we reassign them entries in U, where the probability for reassignment is defined follows:
Pk (i,
keu U,
(t.).) =
Ever (M(i,))
(20)
0.
k $ U.
This process is repeated until le tokens are reassigned for the k-th codebook entry, thereby enforcing consistency between its frequency and probability.
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We evaluate our method on both VAR (Tian et al.) 2024) and its text-to-image extension, Infin- ity (Han et al.) 2025). For conventional VAR models, we consider four tasks: class-conditional image generation, inpainting, outpainting, and class-conditional editing. For class-conditional im- age generation, we use VARs varying depths (16, 20, 24, and 30) for evaluating our method. We generate 50 images for each class in ImageNet (Deng et al.) 2009), which includes 1.2M images for training, and 50K images for validation. We evaluate the generated images using standard bench- marks (e.g., .. IS (Salimans al. 2016), FID (Heusel et al. 2017)). To further validate our method,

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative comparison of quantizing VAR with various methods for the task of conditional image generation on ImageNet (Deng et al., 2009). We denote by W/A the bit- widths of weights (W) and activations (A), respectively. We report IS, FID, and FID2FPI6 for VARs of varying depths. Note that we have re- implemented VAR (Xic et al. 2024) to ensure consistency with the evaluation settings provided in the work of (Tian et al. 2024).
#BiLS (W/A)
Methods
VAR-I6
VAR-420
VAR-d24
VAR-130
18
FID FID2FP16
IS
FID FID2PP16
IS
FID FID2FP16
IS
FID FID2FP16
16/16
Full-precision
274.8
3,41
302.9
2.73
313.2
2.17
303.5
1.97
BRECQ al 2021
202.9
5.56
4.29
239.5
3.23
2.72
261
2.73
2.44
250.9
3.38
2.55
VAR (Xee et
2024
212.1
5.17
4.05
247.8
3.13
2.50
261.8
2.76
2.43
258.5
3:27
2.48
6/6
Ours
213.5
4.46
3.14
249.5
3,01
2.34
264.8
2.59
2.30
262.7
2.88
2.09
Ours+Lite
226.1
4.08
2.64
253.3
2.89
2.09
2.38
1.97
2.66
1.85
BRECQ Li al. 2021
203.8
6.07
4.76
226.2
4.21
3.82
261.7
2.90
2.23
270.4
2.91
1.96
4/8
VAIR Xie et 2024
217.8
5.56
3.85
242.6
3.58
2.93
270.4
2.72
2.19
273.4
2.85
2.03
Ours
212.5
5.71
4.05
241.0
3,71
2.98
275.8
2.60
1.91
276.8
2.71
1.81
227.8
4.83
3,00
256.4
3.32
2.32
285.6
2.50
1.73
279.4
2.60
1.76
BRECQ et al. 2021
145.6
11:16
10.57
189.1
6.74
6.62
215.8
4.89
4.82
221.4
5.09
4.17
1/6
LiteVAR Xie ct 2024
158.1
9,93
9.09
205.6
5.25
5.06
225.4
4.23
4.04
218.9
5.41
4.60
162.1
9.20
8:44
202.2
5,36
5.09
230.7
3.96
3.90
4.59
3.74
OurR+LiceVA
189.3
6.62
5.28
219.2
4.46
3,98
248.1
3.32
3,01
235,3
4.43
3,56
BRECQ Li al. 2021
67.6
33.03
32.82
94.1
23.03
22.42
119.0
16.89
17.07
125.3
15.71
14.66
VAR Xie et al. 2024
66.9
35.87
36.67
100.7
21.71
21.10
128.6
15.75
16.81
149.3
12.25
11.11
4/4
Ours
90.7
24.57
24.35
116.8
17.74
17.28
148.0
12.04
11.71
177.5
9.26
8.68
Ours+LievVAR
110.9
18.92
18.32
145,4
12.43
12.03
172.7
8.85
8.24
180.6
9.12
8.48
report FID2FP16 (Tang al. 2024), which measures the FID score between outputs of full-
precision VAR and its quantized counterpart. For Infinity (Han et al. on text-to-image generation with IewgeReward (Xu et al.) 2023). 1PSv2.1 (Wu et al. 2023). and
2025), we evaluate our method
GenEval Ghosh et al.
2023), following the experimental protocols in (Han et al. 2025).
evaluate inpainting, outpainting, and class-conditional editing directly on images with the validation split of ImageNet. For image anpainting/ouppanningg, follow- measure average LPIPS (Zhang al. 2018) between generated and original images, where the generated images are produced by filling ran- domly masked region covering 25% of the image area. For class-conditional editing, we convert the ImageNet class label into text prompt ("a photo of [classname] """ and compute CLIP scores (Rad-) ford et al. 202I) between the text prompt and generated image, following (Radford et al., 2021).
Following (Tian et al.
2024
ing (Zhuang et al., 2024 Suvvrrvv et al. 2022),
We built our method on top of BRECQ (Li et al. (021) to quantize VAR, calibrating the quantization parameters using 256 images obtained either by random sampling or by our calibration data resam- pling method described in Sec. 3.2.2] We initialize the step size and zero-point using Percentile method (Li et al.] (2019) for weight quantizers. Following VAR (Xie et al., 2024), we exploit dynamic min-max quantizers for activations. Given a BOP constraint, we determine the minimum value of that satisfies the constraint through a grid search over [0, 1] with a step size of 1e-4. We impose an additional BOP constraint equal to 1% of the total operations required for full inference, unless stated otherwise.
4.2 RESULTS
Class-conditional image generation. We show in Table [ the quantitative comparison of our method with prior approaches for the task of class-conditional image generation on ImageNet (Deng al. 2009). We can see that our method consistently outperforms BRECQ et 2021) across all bit-widths and architectures, demonstrating the effectiveness of the proposed shift-and-sum quan- addition, our method achieves quantiza- tion performance comparable to LiR (Xie et al.) [2024), although it retains FC layers after GELU (Hendrycks & Gimpel 2016) non-linearity in full precision, which is suboptimal in terms of efficiency. Finally, applying our approach in combination with VAR (Xie et al. 2024) yields further improvements, suggesting that the two approaches are complementary. We visualize the generated images using each method in Appendix G.1
tization and calibration data resampling techniques.
Text-to-image generation. We provide in Table 2 the quantitative comparison of our method with Wate-offthe-art methods for the task of text-to-image generation. We can see that our method con-
8

---

Published as a conference paper at ICLR 2026
Table 2: Quantitative comparison of quantizing Infinity- 2B (Han et al. denote by W/A bit-widthe of weights (W) and activations (A), respectively.
2025) for text-to-image generation. We
#Bts (WA)
Methods
ImageReward
HPSv2.1 1 GenEval
16/16
Full precision
0.932
32.23
0.736
BRECQ Li al 2021
0.908
32.01
0.719
LiR al.
6/6
Ours
0.914
32.03
0.722
VAR
0,929
32,04
0.725
BRECQ ei al
2021
0.845
31.79
0,712
4/8
LiteVAR Xie et a.
2024
0.866
32.03
0.719
Ours
0.861
31.94
0.716
OAR
BRECO Li 2021
LiteVAR Xie
2024
0.838
31.61
0,702
4/6
Ours
0.842
31.70
0,706
Ours+Lite VAR
0,880
32.04
0.714
BRECQ Li al 2021
0.346
27.92
0.563
4/4
LiteVAR Xie et
2024
0.407
28.98
0.626
Oursel VAR
0.748
29.57
0.672
Table 3: Quantitative comparison of quantizing VAR-d d30 with various methods for the tasks of image inpaint- ing, outpainting, and class-condene conditional editing.
#Bits (W/A)
LPIPS (inpaint)
LPIPS (outpaint)
CLIP scores (editing)
Methods
16/16
Full precision
0.2827
0.2119
0.2480
BRECQ Li 2021
0.2852
0.2144
0.2455
LiteVAR Aie 2024
0.2861
0.2143
0.2458
Ours
0.2843
0.2141
0.2459
iteVAR
0.2835
0,2134
0.2465
BRECQ ei al 2021
0.2912
0.2207
0,2289
LiteVAR Xie et 2024
0.2907
0.2208
0,2318
4/4
Ours
0.2888
0.2181
0,2358
Ouss-LieeAR
0.2879
0.2172
0.2363
sistently outperforms BRECQ (L al. 2021) and VAR (Xie et al.) (224) for all benchmarks, demonstrating its efficiency. Applying our method on top of VAR further improves perfor- mance; however, Lite VAR retains the FC layers after GELU (Hendrycks & Gimpel| (0166) full precision, which introduces additional computational inefficiency. We provide the generated images for a number of text prompts Appendix G.2
Image inpainting, outpainting, and class-conditional editing. We show in Table 3 quantitative comparison of quantizing VAR-d30 using BRECQ (Li et al., (20T), LiteVAR (Xie al. 2024), and our method on image inpainting, outpainting. and class-conditional editing. The overall perfor- mance differences are relatively small, since the bascline already performs close to the full- full-precision model on these editing tasks, leaving only narrow margin for further improvement. Within this limited range, our method consistently achieves the best LPIPS and CLIP scores across bit-widths. Moreover, as shown in Fig. D (Appendix (.3), our quantized model produces fewer artifacts and better preserves semantic content compared to BRECQ and LiteVAR.
4.3 DISCUSSION
Ablation study. We show in Table 4 an ablation analysis on different components of our method. We find that shift-and-sum quantization consistently improves performance for both VAR-d16 and VAR-d20, indicating that alleviating reconstruction errors in attention-valus multiplications is crit- ical for effective quantization. We can also see that calibration data resampling further enhances quantization performance, suggesting that discrepancies between predicted probabilities of code- book entries and their sampling frequencies in calibration data degrade quantization performance.
Analysis on shift-and-sum quantization. We visualize in Fig, a(abb) the reconstruction error of iitention-value products across scales. We can see that shift-and -nm-sum quantization effectively re- duces reconstruction errors in attention-value products, especially coarse scales where high atten-

---

Published as a conference paper at ICLR 2026
Table 4: Ablation analysis for different components of method. We report the results of conditional im- age generation on ImageNet (Deng et al. 2009) under a 4/6-bit setting. We denote by "Resample" the shift-and-sum quantization and calibration data resampling techniques, respectively.
and
Shift-and-sum Resample
18
FID
FID2FP16
Shift-and-sum Resample
IS
FID
FID2FP16
145.6
11.16
10.57
189.1
6.74
6.62
155.4
10.15
9.55
194.8
6.10
6.03
152.8
10.19
9.74
195.3
5.86
5.61
162.1
9.20
8.44
202.2
5.36
5.09
(a) VAR-d16
(b) VAR-d20
BRECO
Ricon error scio")
BRECO
10
VAR-0L6
Dars
Ours
200
VAR-420
Rloon orru
135
100
F1D
160
WAR-L6
VA3-000
(VAIR-421)
0
BOP overtteed
ovecheed
(a)
(b)
(c)
(d)
Figure 5: (a-b) Comparison of reconstruction error for the multiplication between attention scores and value
tokens, using BRECQ (Li et al. 2021) and our method. (c-d) Visualization of IS and FID scores for VAR-d16
and VAR-d20 w.r.t the BOP budget for conditional image generation on ImageNet (Deng et al. 2009). We
visualize the reconstruction errors performances VAR-d16 and VAR-d20 under 4/6-bit setting.
15
1.0
mrquenne
10
Probabiliny
10
0.5
1950
2000
2050
1830
2000
20500
1950
2,000
2050
Codebbok entries
Codebook entries
Codebook entries
Figure 6: Visualization for the predicted probabilities of codebook entries (left), and their sampling frequencies before and after calibration data resampling (middle-right)
tion scores occur frequently. We show in Fig, 5(c-d) the IS and FID scores of generated images under varying BOP constraints. We observe that quantization performance consistently improves for both VAR-d16 and VAR-d20 as the BOP budget increases, indicating that higher kernel orders products. We also find that the im-
more effectively reduce reconstruction errors in attention- provement saturates the BOP constraint reaches 1% of the total inference cost, suggesting that shift-and-sum quantization alleviates reconstruction errors with only a marginal overhead.
Analysis calibration data resampling. We compare in Fig. 6 the predicted probability of each codebook entry aggregated over calibration samples, and its sampling frequency before and after ap- plying calibration data resampling technique. We can that our resampling technique effectively aligns sampling frequencies of codebook entries with their probabilities. allowing the calibration data to better reflect the underlying probability distribution.
5 CONCLUSION
We have observed that applying PTQ to VAR suffers from large reconstruction errors in attention- value multiplications, which are amplified in the presence of high attention scores. To this end, we have introduced shift-and-sum quantization, which mitigate these errors by applying quantization kernels to attentive tokens. In addition, we have found that a mismatch often arises between the predicted probabilities of codebook entries and their sampling frequencies due to limited calibration samples in PTQ. Based on this, we have proposed calibration data resampling technique that aligns sampling frequencies with the underlying probability distribution. We have demonstrated the effec- tiveness of our method through extensive experiments on class-conditional generation, inpainting, outpainting, and class-conditional editing across VAR architectures.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported in part by Institute of Information & Communications Technology Plan- ning Evaluation (IITP) grant funded by the Korea government (MSIT) (No.RS-2022-001435544 Development of Fundamental Technology and Integrated Solution for Next-Generation Automatic Artificial Intelligence System, SARS-2025509942295
Al Semiconductor Innovation Lab (Yonsei University)), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (BS-2023-02216328)
REFERENCES
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. NeurIPS, 2019.
Zigeng Chen, Xinyin Ma, Gongfan Fang, and Xinchao Wang. Collaborative decoding makes visual auto-regressive modeling efficient. In CVPR, pp. 23334-23333 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale
hierarchical image database. In CVPR, 2009.
Steven K Esser, Jeffrey McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S Modha. Learned step size quantization. ICLR, 2020.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 2015.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. NewrIPS, 2023.
lan J Goodfellow, Jean Pouget-Abatee Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurlPS, 2014.
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In CVPR, 2025.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu. Tfmq-dm: Temporal feature maintenance quantization for diffusion models. In CVPR, 2024.
Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In CVPR, 2023.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In CVPR, 2019.
Ernesto Kofman. Relative error control in quantization based integration. Latin American applied research, 2009.
Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. Flexround: Learnable rounding based on element-wise division for post-training quantization. In ICML, 2023.
Seunghoon Lee. Jeongwoo Choi, Byunggwan Son, Jachyeon Moon, Jeimin Jeon, and Bumsub Ham. Accuquant: Simulating multiple denoising steps for quantizing diffusion models. NeurIPS, 2025.

---

Published as a conference paper at ICLR 2026
Rundong Li, Yan Wang. Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan. Fully quantized network for object detection. In CVPR, pp. 2810-2819, 2019.
Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling. arXiv preprint arXiv:2406.097755 2024.
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and
Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In ICCV, 2023a.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. ICLR,, 2021.
Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq-vit: Scale reparameterization for post- training quantization of vision transformers. In ICCV, 2023b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr
Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. Fq-vit: Post-training quanti- zation for fully quantized vision transformer. IJCAI, 2022.
Jaehyeon Moon, Dohyung Kim, Junyong Cheon, and Bumsub Ham. Instance-aware group quanti- zation for vision transformers. In CVPR, 2024.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In ICML, 2020.
Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex Bron- stein, and Avi Mendelson. Loss aware post-training quantization. Machine Learning. 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICMI, 2021.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. NeurlPS, 2019.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NeurIPS, 2016.
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In CVPR, 2023.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv, 2024.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In WACV, 2022.
Siao Tang, Xin Wang, Hong Chen, Chaoyu Guan, Zewen Wu, Yansong Tang, and Wenwu Zhu. Post-training quantization with progressive calibration and activation relaxing for text-to-image diffusion models. In ECCV, 2024.
Keyu Tian, Yi Jiang. Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NeurIPS, 2024.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Condi- tional image generation with pixelenn decoders. NeurlPS, 2016.
12

---

Published as a conference paper at ICLR 2026
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. NeurIPS, 2017.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv, 2023.
Rui Xie, Tianchen Zhao, Zhihang Yuan, Rui Wan, Wenxi Gao, Zhenhua Zhu, Xuefei Ning, and Yu Wang. Litevar: Compressing visual autoregressive modelling with efficient attention and quantization. NeurIPSW, 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. NewrIPS, 2023.
Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun. Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization. In ECCV, 2022.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Yunshan Zhong, You Huang, Jiawei Hu, Yuxin Zhang, and Rongrong Ji. Towards accurate post- training quantization vision transformers via error reduction. TPAMI, 2025.
Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. In ECCV, 2024.
13

---

Published as a conference paper at ICLR 2026
Table A: Magnitude of pearson correlation coefficients between quantization noises. We evaluate the correlation
between Es and ej across 1,000 randomly sampled images using VAR-d30.
Noise Pair
Pearson Correlation Coefficient
(eg Duae (C) ( (eig, (ky)ss
2.5 10 6
1.7 10-7
2.3 10-6
A CORRELATION BETWEEN QUANTIZATION NOISES
Eq. S assumes that the quantization errors of attention scores, and those of value tokens are indepen- dent across tokens. Although this assumption is introduced only simplify the derivation of Eq. we verify here that it closely matches actual behavior in practice.
We show in Table A the magnitude of Pearson correlation coefficients between ε and eij 1,000 randomly sampled Tmagges using VAR-d30. In addition, we measure the magnitudes of correlations between quantization eπrоrs across different attention rows and different value tokens. We can see that all measured correlations are extremely small (typically an order of 10 10-)), indicating that the error terms are uncorrelated in practice.
B PROOF OF EQ.
We obtain the reconstruction error in Eq. 7 for the multiplication between attention scores a E IR." and value tokens V E Txd as follows. For the j-th dimension, the reconstruction error is
T
8,
a. (
+ 2 + +
(21)
where we denote Ui E IR and E E R as the j-th component of the i-th value token, and the corresponding quantization noise.
Since the quantization noises ε` and E are independent across tokens, the variance of 6, can be written as:
T a Varl + +E + +
Var[o,] = Var Σ aa + E + b5))
Σ
(22)
1=1
We can expand the variance term as follows:
Varl 27 + ctyl Varlefeul + Vark Var[ey]
(23)
+2Covl2 27 2Cov| 4tjl + 2Cov Gy)-
Based on the assumption that { and {e are independent zero-mean noises, the covari- ance terms vanish follows:
Covlejey, G = vy ((EE) - by Bل enj = 0,
(24)
CovlEf <gj = vi Covle: +5] = 0,
(25)
CovlE 1[E" E - [ = O.
(26)
Thus Eq. 23 simplifies
Var + 2 + + = 03 +0 +
(27)
Substituting into Eq. 22 we obtain
T Σ
Var[5,]
d (o - +0?0
(28)
i=1
14

---

Published as a conference paper at ICLR 2026
Finally, summing across dimensions j l...., gives the total variance
Var [8]
Var [را]
a
aa
biy + d(o200 +o))
(29)
which corresponds to Eq. 8 and completes the proof.
C PROOF OF THEOREM
We will prove the following in sequence: (i) using i, s/4n provides upper bound of s s/nn, and (ii) no upper bound smaller than s/ can be obtained for arbitrary shift factor tn Note that we omit zero-points for clarity.
(i). Suppose we set tn = s/4n. this case, we will first show that the quantization kernel in Eq. 2ne reduces to 2n We define
2ne
g(v)
(u
2n
8
4n
2ne
n-
Q u (2k
+ 12
(30)
2n
8
2n
k=-n n
Adding s/2n to v, can obtain:
g( u
2n
2n
2nv
8
+1
+
2n
1 n-1
Σ
o( (2k + +3) 8
4n
(31)
We can see that g(v) is periodic as follows:
2nw
Q U (2k + +3) +3)
2m
2n
n-1
(32)
2nv
8
1
+
Q+ 1 (2k + +1)
+
2n
2n
2n
4n
g(r)-
Furthermore, g(v) = 0 for v I-s/4n, s/4n); by periodicity this implies g(v) = 0 for all v. Hence the quantization error of fa (v: s/4n) is bounded as
8
2nw
fn(u; 4n
17
2n
2nz
8
2nw
(33)
2n
8
<
4n
which establishes part (i).
(ii). We now prove tightness of the bound. Suppose, for contradiction, that there exists shift factor
ty such that
max IV- - fn <
(34)
4n
Since Q(-) outputs values on the quantization grid {ms m E 2}, the averaging operation in
fa(u; f.) produces outputs confined to the finer grid
ms
m E 2
(35)
2n
15

---

Published as a conference paper at ICLR 2026
Table B: Analysis of computational overheads (i.e., memory, BOPs) introduced by shift-and-se aund-sum quantization, quantization performances VAR-d16 under a 4/6-bit setting. Note that we not use calibration data resampling in this analysis.
0.05
0.1
0.2
Memory
BOPs
Memory
BOPs
Memory
BOPs
Baseline
1.03G
3.72T
1,03G
3.72T
1.03G
3.72T
Eq. 11
0.22M
1.17G
0.22M
1.77G
0.22M
1.17G
Duplicate
23:21M
0.28G
9.58M
3,94M
Eq
33.35G
11.19G
3.94G
Total
1.05G
3.75T
1.0GG
3.73T
L03G
3.72T
(a) Computational overhcads
Method
FID
FID2HPI6
Buseline
145.6 11.16
10.57
Shift-and-sum 0.05)
155.4 10.15
9.55
Shift-und-sum (0 0.1)
155.2
10.25
9.57
Shift and-sun (0 = 0.2) 154.3 10.33
9.71
(b) Quantization performances
Now suppose set /n. For this choice, obtain
fm
tx
8
188
2
(36)
4n
4n
4n
2n
4n
where in is an integer. Therefore, conclude that
max v (u; 2
(37)
4n
contradicting Eq. 34 Hence no choice of in can achieve a bound smaller than 9/4n, which estab- lishes part (ii).
D COMPUTATIONAL OVERHEAD
Compared to baseline, shift-and-sum quantization introduces additional memory usage and BOP overhead during inference. We can see from Fig. 3 that shift-and-sum quantization requires (i) computing average attention scores for each value token (Eq. [T]. (ii) duplicating and shifting to- kens with high attention scores, and (iii) aggregating the quantized results from their symmetrically shifted counterparts (Eq. 13).
Consider matrix multiplication between attention matrix A E T XT and value matrix V E R'xd where we apply a quantization kernel with an order of n to single token V. Computing the aver- age attention scores (Eq. [1] requires additional 167 bits for storage and 16T'T BOPs. Duplicating and shifting attentive tokens, together with their corresponding attention scores, introduces addi- tional 16(2n 2n 1)" + d) extra bits and 2n(T" + 16d) BOPs. Finally, aggregating the quantized outputs (Eq. 13 incurs (2m dT" additional BOPs, where we denote b as the bit-widths of activations.
In practice, shift-and-sum guantization is applied only to tokens whose average attention scores exceed the threshold 0, and the order n remains small for most tokens. Consequently, the com- putational overhead for shift-and-sum quantization is relatively marginal compared to the whole inference process. We report in Table a] the memory and BOP overheads under 4/6-bit set- ting for VAR-d16 with varying threshold 0. We show in Table B[b) the quantization performances of VAR-d16 for varying 0. We can that method improves all three metrics (IS, FID, and FID2FP16) consistently, while introducing only minor computational overhead.
E THROUGHPUT
We compare in Table C the throughputs of VAR-d16 under different quantization methods, where convert the precision of weights and activations into bits using ONNX Runtime. All quantized
16

---

Published as a conference paper at ICLR 2026
Table C: Throughputs (image/sec) of VAR-d16 under different quantization methods, measured using ONNX
Runtime an Intel Xeon Gold 6226R CPU.
Method
Bits (W/A)
Throughput (imago/sec)
Full-procision
16/16
0.1208 0.0010
BRECQ eLi al. 2021
0.2195 0.0046
VAR (Xie 2024
0.0028
Ours
0.2147
0.0042
OurR+LiteVAR
0.1719 0.0026
Table D: Quantitative comparison of quantizing OneFormer (Jain et al. 2023 ods for the tasks semantic, instance, and panoptic segmentation on COCO.
with various quantization meth-
Bits (W/A)
Method
mloU
AP
PQ
16/16
Full-precision
67.4
49.0
57.9
BRECQ Li al
2021
66.5
47.8
55.8
LiteVAR
Xic al.
2024
66.9
48.2
57.4
6/6
Ours
66.8
48.3
57.6
Ours+Litc VAR
67.0
48.6
57.8
BRECQ Li al. 2021
63.7
37.9
51.0
Lit Xie
2024
64.4
39.2
51.5
4/4
Ours
64.6
39.1
51.6
Ours+Lite VAR
65.0
40.6
52.2
models exported to the ONNX format and executed with 8-bit integer operations on Intel Xcon Gold 6226R CPU, allowing us to report actual images-per-secone throughput rather than estimates based on simulated quantization in floating-poin arithmetic. We also include variant where the FC layer after GELU non-linearity is kept in full precision, following the quantization strategy adopted that our method is only about 2% slower than BRECQ (Li et al. 2021]. while being considerably faster than VAR, highlighting the practical efficiency of our approach for real deployment scenarios.
in Lite VAR (Xie al., 2024). We
APPLICATION TO ONEFORMER
We show in Table D the results for quantizing OneFormer (Jain 2023) for the tasks of semantic, instance, and panoptic segmentation on COCO (Lin et al. 2014). We measure mloU (Everingham et al. 2015), AP (Lin al. 2014]. and PQ (Kirillov et al. (019) for semantic, instance, and panoptic segmentation, respectively. We observe that our method outperforms BRECQ (Li et al. 202 l) con- sistently, while having comparable performance with Lite (Xie al. 2024). despite Lite keeping the FC layers after the GELU (Hendrycks & Gimpel 2016) non-linearity in full precision, leading to computational inefficiency. Finally, combining our method with Lite VAR yields further improvements across all three tasks.
17

---

Published as a conference paper at ICLR 2026
(a) Full-pree preciiieciion
(b) BRECQ
(c) LiteVAR
(d) Ours
(e) Ours+Lite VAR
Figure Visualization of generated images full-precision VAR-d16 its counterparts quantized with BRECQ (Li et al. 2021), LiteVAR (Xie et al. 2024), and our method under 6/6-bit setting.
(a) Full-precision
(b) BRECQ
(c) LiteVAR
(d) Ours
(e) Ours+Lite
Figure B: Visualization of generated images for full-precision VAR-d16 and its counterparts quantized with BRECQ (Li et al., [202T), LiR (Xie et al. 2024). and our method under 6/6-bit setting. Note that we teacher-force the token maps sampled from the full-precision model for the first two stages VAR, unlike Fig.
18

---

Published as a conference paper at ICLR 2026
"chicken"
coffee mug cardboard"
"a car chilling the beach, sunset"
"a beautiful portrait of'a
beautiful womun in the jungle surrounded by pink
flowers, shamanism,
matte painting. fantasy
art"
(a) Full-Precision
(b) BRECQ
(c) Lite
(d) Ours
(c) Ours+1 LiteARR
Figure C: Comparison generated images using Infinity-2B (Han al. 2025 and its quantized counterparts using different methods.
G QUALITATIVE RESULTS
G.1 CONDITIONAL IMAGE GENERATION
show in Fig. A a qualitative comparison images generated by VAR-d16 quantized with BRECQ (i al. 2021. LiR (Xie al. 2024), and our method. For each row, present results generated from the same random seed and ImageNet label to enable direct visual comparison across methods.
Since VAR samples token maps from the predicted probability distribution at each stage, even slight perturbations coarse scales cause different codebook entries be chosen, resulting in dras- tically different output images. To address this, we additionally report results in Fig. B where apply teacher-foreing align the sampled codebook entries with the full-precision model in early stages, while allowing later stages to proceed autoregressively. This ensures consistent global structure across methods and highlights the impact of different quantization methods on fine-scale details. We can see that our method generates images that are more realistic and closer to those of full-precision model compared to other approaches.
G.2 TEXT-TO-IMAGE GENERATION
We show in Fig. C qualitative comparison of images generated by Infinity-21 quantized with BRECQ (Li et al. 2021). LAR (Xie et al. (024), and our method. We can see that our method consistently produces images with better fidelity compared other methods.
G.3 IMAGE INPAINTING, OUTPAINTING, AND CLASS-CONDITIONAL EDITING
We present in Fig. D qualitative comparison of VAR-d30 model quantized using BRECQ LLi et al. 2021). VAR (Xie et al.] 2024). and our method on the tasks of inpainting (1st-3rd rows), outpainting (4th-6th rows), and class- cass-conditional editing 77th-8th rows). We observe that method consistently produces better global structure and fewer artifacts compared to other meth- ods.
19

---

Published as a conference paper at ICLR 2026
(a) Original
(b) Full- dell--peeiisoon
(c) BRECQ
(d) VAR
(c) Ours
(f) Ours+LiteVAR
Figure D: Qualitative comparison of results for the tasks of image inpainting (1st-3rd rows), outpainting
(4th 6th rows), and class-conditional editing (7th 8th rows) using VAR-d30.
H BIT-WIDTH VS. PERFORMANCE
We provide in Fig. E qualitative comparisons of images generated by VAR-d30 under different bit- width settings using our method. The results show that generation quality remains largely stable at 6/6- and 4/8-bit configurations, while significant degradation appears at the more aggressive 4/4-bit setting. This visualization highlights how our method performs under varying compression levels and directly illustrates the trade-off between bit-width and perceptual quality.
20

---

Published as a conference paper at ICLR 2026
(a) Full- call-precision
(b) W6A6
(c) W4A8
(d) W4A6
(e) W4A4
Figure E: Visualization images generated by full-precision VAR-d30 and quantized counterparts under different bit-width settings using our method.
W4A8
W4A4
250
W4AG
W6A6
WGAG
10
W4A4
6
200
W444
W444
W4A6
WSA6
W8A6
150
W4A8
W4A8
20
30
90
100
20
30
90
100
BOPs
ROPs (T)
BRROQ
LIleARR
Ours
Ourse pors-lllatan HtvVAR
BRBCQ
LIWVAR
Ours
OursATAtaAR
Figure F: Visualization of IS and FID scores for VAR-d30 w.r.t. the BOP hudget for conditional image genera- tion on ImageNet (Deng et al. 2009).
In addition, we present in Fig. F the IS and FID scores of VAR-d30 quantized with various bit-width settings. Our method consistently provides more favorable quality-efficinnc trade-off compared to BRECQ Li et al., 202T) and VAR (Xie et al. 2024). For settings with tight BOP budgets, our approach yields noticeably larger performance gains achieving higher IS and lower FID than both methods-whiee which indicates that the proposed method preserves generation quality substantially better when computation is highly constrained. This demonstrates that our method is more robust to quantization and maintains higher fidelity across wide range of compression levels, thereby offering a superior balance between computational cost and generation quality.
We also observe that in some regions of the trade-off curves, a configuration with lower BOP budget can unexpectedly outperform one with higher budget (e.g., W4A8 and W6A6). This occurs because activation quantization is typically more critical to the final generation quality than weight quantization. As a result, a configuration with fewer operations but higher- precirrirriiiioon activations can sometimes outperform another with more operations but lower-precision activations.
21

---

Published as a conference paper at ICLR 2026
I LIMITATIONS & FUTURE WORK
Although we demonstrate the applicability of our method beyond VAR using OneFormer (Jain et al., 2023), evaluating a broader range of transformer architectures remains an interesting direction for fu- ture work. In addition, the effectiveness of our method degrades when the precision is pushed below bits. In such extremely low- low-precision regimes, quantization noise could dominate the generation process, leading to severe performance degradation across all evaluated methods. Addressing these regimes likely requires fundamentally different quantization strategies or training- aware approaches, which we leave for future work.
J LARGE LANGUAGE MODEL (LLM) USAGE
In this paper, we use LLMs solely to aid in polishing the writing. No LLMs were used for ideation, retrieval of related work, or experimental design.
22


Reviews:
[{"review_id": "JYeAHH73JJ", "reviewer": "Reviewer_8oiH", "summary": "This paper identifies two major challenges when applying post-training quantization (PTQ) to Visual Autoregressive Models (VAR): (1) large reconstruction errors arising from quantizing the multiplication between attention scores and value tokens, especially at coarse scales where high attention scores are more common; and (2) a mismatch between codebook-entry probabilities and their sampling frequencies during calibration due to limited calibration data. To address these issues, the authors propose two components: a shift-and-sum quantization technique that duplicates and symmetrically shifts attentive tokens (those with high attention scores) to reduce quantization errors, and a calibration data resampling method that reassigns codebook entries to better match predicted probabilities. Experiments across multiple VAR depths and tasks—including class-conditional generation, inpainting, outpainting, and conditional editing—show that the proposed methods consistently outperform prior PTQ methods while maintaining low computational overhead. The approach achieves state-of-the-art PTQ performance on VAR and is complementary to existing methods like LiteVAR.", "questions": "- Could the proposed method be evaluated on other transformer-based models to verify whether it generalizes and improves performance? Although applying it to plain autoregressive generation may be less straightforward, architectures that use multi-scale representations might benefit significantly.\n  - The following models might be worth exploring:\n    - OneFormer: One Transformer to Rule Universal Image Segmentation, CVPR 2023\n    - VGGT: Visual Geometry Grounded Transformer, CVPR 2025\n- In BRECQ, the main PTQ evaluation table compares W4A4 and W2A4 settings. It would be interesting to see how the proposed method performs under these quantization settings compared to BRECQ. Can the authors provide results or insights on W4A4 and W2A4 performance?\n- How does the proposed method perform quantitatively on inpainting, outpainting, and class-conditional editing tasks? Since the current version mainly focuses on quantization for VAR, it would be valuable to include numerical performance metrics for these tasks, rather than relying solely on qualitative visual comparisons.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The two components proposed in the paper are well-designed to address the specific problems that arise in VAR quantization, and the paper clearly explains how they solve these issues.\n- The proposed method appears broadly applicable beyond VAR, with potential usefulness in visual generation and autoregressive modeling in general.", "weaknesses": "- Given the nature of quantization research, more generic and widely applicable methods are preferable. However, the proposed approach is validated only on VAR, making the research scope narrow and potentially limiting its impact.\n- Recent PTQ research on transformer quantization is not discussed; the related work mainly covers older studies. Similarly, LiteVAR also focuses specifically on VAR quantization, which suggests that the overall scope of related work is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A9qDZZMNPv", "reviewer": "Reviewer_dUD4", "summary": "This paper addresses the challenge of efficient deployment of Visual Autoregressive Models (VAR) by focusing on Post-Training Quantization (PTQ), a technique that enables deep network compression using a small subset of calibration data. While PTQ has shown promise in conventional diffusion models generative models, its application to VAR remains underexplored, primarily due to two critical issues:\n\n* First, significant reconstruction errors arise from the multiplication of attention scores and value tokens in the VAR transformer, especially at coarse scales (low resolutions) where high attention scores are more concentrated—these errors propagate across subsequent finer scales and degrade final image quality. \n* Second, limited calibration data leads to a mismatch between the sampling frequencies of VQVAE codebook entries and their predicted probabilities, biasing quantization parameters and reducing quantization performance.\n\nTo tackle these challenges, the paper proposes a PTQ framework tailored for VARs, consisting of two core components: Shift-and-Sum Quantization and Calibration Data Resampling.\n\nExtensive experiments validate the framework on ImageNet across four tasks: class-conditional image generation, image in-painting, out-painting, and class-conditional editing. Evaluations on VARs of varying depths (16, 20, 24, 30 layers) and different bit-widths show consistent improvements over baseline methods (e.g., BRECQ, LiteVAR) in metrics like IS, gFID, and FID2FP16.", "questions": "None", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "In fact, I have a good understanding of autoregressive models, but I am not an expert in the field of quantization. Please correct me if there are any issues with my descriptions.\n\n1/ This paper mainly focuses on optimizing Post-Training Quantization for Visual Autoregressive Models. There is relatively little related work on autoregressive models, so this research is undoubtedly worthy of encouragement .\n\n2/ This work has achieved promising results on ImageNet-256. It outperforms LiteVAR, and the performance improvement is even more significant when combined with LiteVAR.\n\n3/ The analysis of \"Reconstruction error across scales\" is quite interesting. The authors found that quantization errors are more significant at early (coarse) scales, and based on this observation, they designed the Shift-and-Sum Quantization technique.", "weaknesses": "1/ I have a major question: Since the main purpose of this work is to improve the efficiency of generative models for deployment, why are there no experiments in the paper showing the speed performance or throughput performance of the VAR model after quantization?\n\n2/ Currently, the experiments on VAR are only conducted at a resolution of 256. I am curious whether the results are consistent at higher resolutions. For example, at a resolution of 1024—admittedly, VAR itself has no experiments at 1024 resolution, but Infinity (the text-to-image model of VAR) has experiments at the 1024 resolution version, and it would be valuable to observe the experimental phenomena there.\n\nreference:\nInfinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gQEJcsJm20", "reviewer": "Reviewer_TesS", "summary": "This work analyzes the post training quantization of VAR models and points out two VAR problems: 1. significant quantization errors from the multiplication between attention scores and value tokens. 2. a mismatch between the predicted probabilities over the entries of VQVAE codebook and their sampling frequencies during calibration (Line 71). The authors propose shift-and-sum quantization which can reduce error with $O(s/4n)$ bound (Theorem 1), and calibration data resample that can resolve the mismatch. The author also provide empirical validations to show that the method improves over BRECQ and is competitive with LiteVAR under W/A bit-widths.", "questions": "$\\bullet$ Can authors explain the practical validity of the assumption used in Eq. 8? \n\n$\\bullet$ If the Eq. 8 assumption is removed, does it affect the main result, or does it merely complicate the derivation of the bounds?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "$\\bullet$ Theoretical Contribution: this work formalizes coarse-scale attention make post-training quantization error worse, and the authors derived a variance expression (Eq. 8). \n\n$\\bullet$ shift-and-sum kernel has tight error bound: the error bound is tight $|v-f_n(v;t_n) | \\leq s/(4n)$\n\n$\\bullet$ the calibration fix is simple but effective: I think the resampling technique (probablity-matching) is easy to add to the top of existing post training quantization pipeline. \n\n$\\bullet$ Experiments are comprehensive: the authors conduct experiments on multiple VAR depths, multiple bit-settings, standard metrics (IS/FID/etc.), and qualitative tasks (in/out- painting, editing)", "weaknesses": "Eq. 8 relies on an unrealistic assumption:  $\\\\{ \\tilde{\\epsilon\\_i^a} \\\\}\\_{i=1}^T$ and $\\\\{ \\mathbf{\\epsilon}\\_i^v \\\\}\\_{i=1}^T$ are independent, zero-mean random variables with variances $\\sigma_a^2$ and $\\sigma_v^2$ respectively. I checked the proof of Eq. 8 briefly, and I found the assumption is used at Eq. 22, where $\\\\mathrm{Var}[ \\\\sum\\_{i} a\\_i X\\_i ] = \\\\sum\\_{i} a\\_i^2 \\mathrm{Var}[X\\_i] $ (covariance set to 0). If the assumption is dropped, it will not get the same closed form as introducing the covariance term. Furthermore, this assumption seems unrealistic to me, and one proof to break this assumption can be the following: We define $a_t$ and $\\\\hat{a}_t$ as the exact attention score and quantized attention score at timestep t respectively. Then, the quantization error is defined as $e_t := a_t-\\\\hat{a}_t$. By the definition of Softmax, $\\\\sum_i {a_i} = 1$ and $\\\\sum_i {\\\\hat{a}_i} = 1$ (softmax of quantized logits), thus we have $\\\\sum_i {e_i} = \\\\sum_i {a_i}  - \\\\sum_i {\\\\hat{a}_i}  = 0$. We suppose $\\\\{ e_1, \\\\dots, e_t \\\\}$ are independent and at least one had nonzero variance. Then we have $\\\\mathrm{Var}(\\\\sum_i e_i) = \\\\sum_i \\\\mathrm{Var}(e_i) > 0$. Since $\\\\sum_i {e_i}  = 0$, we have $\\\\mathrm{Var}(\\\\sum_i e_i)  = 0$. This leads to a contradiction. This proof has shown the independence assumption is unrealistic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uEcCCCY4Oo", "reviewer": "Reviewer_ND1G", "summary": "This paper applied post-training quantization (PTQ) for visual autoregressive models (VAR), including a shift-and-sum quantization method to reduce calibration data and a resampling strategy for calibration data to align sampling frequencies of codebook entries with their predicted probabilities.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper fills a the gap by explicitly identifying VAR-specific quantization challenges, i.e., the attention-value error amplification at coarse scales and the codebook frequency-probability mismatch. \n- The theoretical analysis is sufficient, e.g., the theoretical analysis (Theorem 1) that proves the error bound for the proposed shift-and-sum quantization.", "weaknesses": "- Insufficient Analysis of Computational Overhead: The proposed shift-and-sum quantization introduces additional operations, such as shift, duplication, and aggregation, which may increase inference time and memory consumption. However, the paper lacks a thorough analysis or empirical evaluation of these computational costs. A detailed study on the overhead introduced by these operations is necessary to fully assess the practicality of the method.\n- Qualitative Results Show Noticeable Degradation: The qualitative results presented demonstrate a clear degradation compared to full-precision models. To better illustrate the trade-off between compression rate and generation quality, the authors should provide a comprehensive comparison of generated results across different bit-widths. Additionally, it would be beneficial to include trade-off curves comparing the proposed method with other baseline approaches.\n- Limited Evaluation Metrics: The paper primarily adopts FID and IS as evaluation metrics, which mainly assess the generation quality for inpainting and outpainting tasks. However, these metrics do not adequately capture the semantic alignment between the generated results and the conditional guidance. The authors should consider incorporating additional metrics or evaluation protocols to assess semantic consistency and alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)