QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsification
Authors: Weilun Feng
Chuanguang Yang
Haotong Qin
Mingqiang Wu
Yuqi Li
Xiangqi Li
Zhulin An
Libo Huang
Yulun Zhang
Michele Magno
Yongjun Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Video Generation
Model Quantization
Attention Sparsification
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Diffusion transformers exhibit remarkable video generation capability, yet their prohibitive computational and memory costs hinder practical deployment. Model quantization and attention sparsification are two promising directions for compression, but each alone suffers severe performance degradation under aggressive compression. Combining them promises compounded efficiency gains, but naive integration is ineffective. The sparsity-induced information loss exacerbates quantization noise, leading to amplified attention shifts. To address this, we propose **QuantSparse**, a unified framework that integrates model quantization with attention sparsification. Specifically, we introduce *Multi-Scale Salient Attention Distillation*, which leverages both global structural guidance and local salient supervision to mitigate quantization-induced bias. In addition, we develop *Second-Order Sparse Attention Reparameterization*, which exploits the temporal stability of second-order residuals to efficiently recover information lost under sparsity. Experiments on HunyuanVideo-13B demonstrate that QuantSparse achieves 20.88 PSNR, substantially outperforming the state-of-the-art quantization baseline Q-VDiT (16.85 PSNR), while simultaneously delivering a **3.68$\times$** reduction in storage and **1.88$\times$** acceleration in end-to-end inference.

Published as a conference paper at ICLR 2026
QUANTSPARSE: COMPREHENSIVELY COMPRESSING VIDEO DIFFUSION TRANSFORMER WITH MODEL QUANTIZATION AND ATTENTION SPARSIFICATION
Weilun Feng Chuanguang Yang'; Haotong Qin Mingqiang Wul.2, Yuqi Li', Xiangqi Li Zhulin A Libo Huang', Yulun Zhang', Michele Magno,, Yongjun Xu
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences Univeerityy of Chinese Academy of Sciences *ETH Zürich
*City College of New York, City University of New York, USA SShanghai Jiao Tong University {fengweil 11w224,,,,,, yangchuanguang, 1wx.angqi24s,, anzhulin, xycteict..c.............c....tta ac.en fhaoteng. qin, michele.... magno Jépll. wth. ch, wumingeiiang25emal ucas syuqi11010600 www huangl
wwa.saa......................... ac. en,
QuantSparse HuryuanVideo
Model Storage
x
3.68x
Wan2.1
Quant Quant Sporee
Hunyuan
Quant Quam Spon
Latency
1.74x
188x
Prompt: serene underwater scene featuring sea turtle swimming through coral
Won2.1
Wan
Quant QuantSp Sparss
Humpen
QuantSparse
VQA-Score
Quant-
90.79 90.73 21 89.19
Won2.1 QuantSparse Sporse
Humyzon
Quant
Quant
Prompt: A soaring drone footage coptures the majestic beauty of coastal cliff
Figure 1:
QuantSparse effectively quantizes 1an2.1-14B (Wan al., 2025) and Hunyuan- Video (Kong et al., 2024) to W4A8 with 15% attention density without compromising visual quality.
ABSTRACT
Diffusion transformers exhibit remarkable video generation capability, yet their prohibitive computational and memory costs hinder practical deployment. Model quantization and attention sparsification pression, but each alone suffers severe performance degradation under aggres- sive compression. Combining them promises compounded efficiency gains, but naive integration is ineffective. The sparsity-induced information loss exacer- bates quantization noise, leading to amplified attention shifts. To address this, we propose QuantSparse, a unified framework that integrates model quantiza- tion with attention sparsification. Specifically, we introduce Multi-Scale Salient Attention Distillation, which leverages both global structural guidance and local salient supervision to mitigate quantization-induce bias. In addition, we develop Second-Orde Order Sparse Attention Reparameterization, which exploits the tempo- stability second-order residuals to efficiently recover information lost un- der sparsity. Experiments Hunyuan Video- 13B demonstrate that QuantSparse
promising directions for
"Corresponding authors: Chuanguang Yang, waaggggunngun@@iiicc.....m Zhulin An, anzhulin @ict.ac.cn

---

Published as a conference paper at ICLR 2026
achieves 20.88 PSNR, substantially outperforming the state-of-the-ar quantiza- tion baseline Q-VDiT (16.85 PSNR), while simultaneously delivering a 3.68 reduction in storage and x acceleration in end-to-end inference. Our code will be released in https: http//gtttttttteeoo com/wl http:///wweeero000ogoooooaaaaaar QuantsSparse.
INTRODUCTION
Recently, Diffusion Transformer (DiT) (Peebles Xie, 2023) has attracted significant attention due to its outstanding capability in visual generation, particularly in video generation (Liu et al., 2024c; Sun et al., 2024a; HPC-AI, 2024; An et al., 2025). Despite the remarkable progress, state- of-the-art models such as Wan2.1-14B (Wan et al., 2025) still demand extraordinary computational resources: generating a single high-resolution video clip can consume more than 20GB of GPU memory and take nearly one hour of inference time. Such prohibitive memory and latency require- ments fundamentally limit the deployment of cnffusiio-baseed video generation models in real-world applications, especially under resource-constrained scenarios.
Model quantization (Jacob et al., 2018; Gholami et al., 2022; Ma et al., 2023a;b) and attention sparsification (Xi al., 2025; Yuan et al., 2024) have emerged two promising directions for com- pression and acceleration. Quantization reduces memory footprint and computation by representing weights and activations in compact integer formats, while attention sparsification prunes redun- dant computations by removing negligible attention scores. However, pushing either technique to the extreme inevitably causes severe degradation. For instance, binary quantization (Zheng et al., 2024b;a) collapses representational capacity, while aggressive sparsification (Xi et al., 2025; Zhang et al., 2025d) discards crucial context information.
Since quantization and sparsification are fundamentally orthogonal, natural idea is to combine them for compounded efficiency gains while maintaining complementary benefits. Ideally, such in- tegration could approach Pareto frontier between performance and efficiency. Yet, our empirical analysis shows that naïvely combining quantization and sparsification leads to severe perfor- mance degradation. We attribute this amplified attention shift: while sparsification removes dow-magitudde attention weights, quantization introduces systematic perturbations to the remaining attention products. These effects reinforce each other, producing compounded distortions in at- tention distributions and severely impairing fine-grained dependency modeling in video generation.
To overcome this challenge, we propose QuantSparse, unified compression framework that syner- gistically integrates model quantization and attention sparsification as shown in Fig. 2. QuantSparse introduces two novel techniques. First, Multi-Scale Salient Attention Distillation (MSAD). We de- sign memory-efficien distillation scheme that balances global and local supervision. Specifically, we employ global guidance by distilling attention patterns on downsampled token sequences to capture coarse structural topology, while local guidance focuses high-resolution supervision on small set of salient tokens that dominate the attention distribution. Second, Second-Orler Sparse Attention Reparumeterization (SSAR). We exploit the temporal stability of second-order residuals to recover information lost due to sparsity. Furthermore, we introduce singular value decomposition (SVD) projection onto dominant principal components, enabling lightweight yet accurate correc- tion mechanism that restores fine-grat mne-graieed attention outputs at negligible computational overhead.
Our contributions can be summarized as follows:
1. We provide formal analysis of the amplified attention shift problem, showing that naive integration of quantization and sparsification severely damages video generation quality.
2. We propose QuantSparse, a unified compression framework that seamlessly combines model quantization and attention sparsification, breaking the traditional trade-off between efficiency and performance.
3. We introduce two key techniques: Multi-Scale Salient Attention Distillation for robust tention alignment and Second-Orde Sparse Attention Reparameterization for temporally stable correction for efficient yet accurate approximation of full-attention outputs.
4. Extensive experiments on large-scale video generation models ranging from 1.3B to 14B parameters demonstrate that QuantSparse achieves superior efficiency-qualie trade-offs,
2

---

Published as a conference paper at ICLR 2026
Multi-Scale Salient Attention Distillation
Second-Order Sparse Attention Reparametterization
Local Attn.
Stage 1:
Key
Init.
Second-Order
Salient Query
D
QKT
Cache
Residual
Residual
Distil
Q
Dense Attn.
PCA Dnolsing
Local Guidance
Safancee Order
Inft
X
Latent Token
Post Softinax Attention
Counterpart
Attn.
Order f Init.
Second-Drder
AvG
Pooling
Distiil
Sporse Attn.
First-Order Residuel
Cache
Residual
Global Guidance
Global
Query
Stage
Sparse
Global Key
Global
Attn.
Inference
Frrtt-Drder
F Second-Order
Refined
Figure 2: Overview of proposed QuantSparse. Left:: During calibration, we apply two parallel attention distillation branch for efficient and robust attention alignment. Right: During inference, we apply accurate attention approximation using temporal stable second-order residual.
outperforming both quantization-only and sparsification-only baselines, while preserving state-of-the-ard performance.
2
RELATED WORKS
2.1 SPARSE ATTENTION IN DIFFUSION MODELS
Sparse attention has been extensively explored in transformer-base models to accelerate attention computation (Lu al., 2025; Yuan al., 2025; Lou al., 2024; Gao et al., 2024; Zhang et al., 2025b). In large language models, common designs include sliding-window (Xiao et al., 2024a;b; Zhang et al., 2023) and sink-based patterns (Fu al.; Xiao et al., 2023b). For diffusion-based vi- sual generation, spatial window masks (Yuan et al., 2024; Zhang et al., 2025c; Ren et al., 2025) and spatial-temporal masks (Xi et al., 2025) have been proposed. Other approaches dynamically generate masks via sampling (Zhang et al., 2025b) or low-resolution attention (Zhang et al., 2025d), though higher computational cost. However, these works mainly focus on preserving the original attention pattern, while the adaptation to other acceleration techniques that alter attention distribu- tions, such as quantization, remains underexplored.
2.2 QUANTIZATION IN DIFFUSION MODELS
Quantization (Gholami et al., 2022; Chitty- Venkata al., 2023; Pilipović et al., 2018; Ma et al., 2024b;c; Feng et al., 2025b; Ding et al., 2024; Feng et al., 2025d) reduces model precision to im- prove efficiency and has been applied to diffusion-based visual generation (Shang et al., 2023; Li et al,, 2024bb; He et al., 2024; Huang et al., 2024a; He et al., 2023; Feng et al., 2025a; Wu et al., 2024; Zheng al., 2024a;b; Li al., 2024a). For video generation, some works target the attention mod- ule (Zhang et al., 2024b;a; 2025a), but often keep linear operations in high precision. Other methods focus on quantizing linear layers: Q-DiT (Chen al., 2024) uses automatic granularity allocation; ViDiT-Q (Zhao et al., 2024) adopts static dynamic strategy: Q-VDiT (Feng et al., 2025c) intro- duces temporal distillation. These methods primarily pursue acceleration via quantization, without exploring its synergy with sparse attention. In this work, we integrate the two orthogonal compres- sion techniques enhance the efficiency and practicality of video generation models.
METHODS
3.1 PRELIMINARY
3.1.1
POST-TRAINING QUANTIZATION (PTQ)
Model Quantization (Gholami al., 2022: Chitty-Venkata et al., 2023) reduces weights/activationss from floating-pott (FP32) to low-bit integers (e.g., INT8). Given an floating-pons tensor X e Rd with dimension d, quantization maps X to discrete representation Xq E (0, 1,......, .2 1)
3

---

Published as a conference paper at ICLR 2026
X clip (1=)
X
2, +2,0,2 Q(x) (Xx 2).
with scale 8, zero-point 2, and bit-width Q(X) denotes the de-quantized value. Post-training Quantization (PTQ) (Wei et al., 2024; Wu et al., 2024) calibrates (s, z) on a small dataset by mini- mizing reconstruction error:
Lquam
min 8,5
Σ
X -Xx
XiEDol
Notably, PTQ avoids retraining the model weights, thus being computationally efficient.
3.1.2 SPARSE ATTENTION
Sparse attention (Zhang et al., 2025b; Xi et al., 2025; Yuan et al., 2024) prunes token pairs via a mask M E [0,1)4x4, reducing complexity from O(L2) near-linear (L is the sequence length).
Given X E RLxdd and query, key, value projection matrices W,, WA. W, E Rdux sparse attention computes:
Q Q=XW - =
QK (dr ΦM V.
(3)
SparseAtttenionn( Q, K, V:M) = softmax
where @ denotes element-wise multiplication.
Probabilty
60-
Full Aztn.
Medin:
50
Global
0.6
..... 90th percentile; 1.9172
Distil
505
60
Mamory
20.
MSE: 0.117 (b) w/o distillation.
0.006 (c) distillation.
Ven-1.30 Hu
Ven-1400 40
(a) Token saliency distribution.
(d) Memory consumption.
Figure 3: The motivation and effect of Multi-Scale Salient Attention Distillation. (a): Token saliency distribution of Wan2.1-1.3B (Wan et al., 2025) block19 headl. Only less than 10% tokens are salient. (b)(c): Visualization of attention difference between quantized model and FP model.
(d): Memory consumption of different attention distillation.
3.2 MULTI-SCALE SALIENT ATTENTION DISTILLATION
The combination PTQ and sparse attention offers promising route toward efficient video genera- tion. However, naively integrating these techniques results in severe performance degradation.
Proposition 3.1. Quantization injects noise into the QK dot product QK yielding systematic bias 8:
Q Q(X)Q(W,,)) T, K =
(4)
QK QK +e, where ||e|| S 8.
The parallel error caused by quantization and sparse attention further leads to a compounded shift:
A total
Aspars
Aquant
O([]e]] M
(5)
Proposition 3.1 indicates that the joint of quantization and sparse attention introduces amplified attention shift (see Fig. 3b), resulting in notable attention degradation. straightforward mitigation strategy is to perform attention distillation during PTQ. However, for large-scale video generation
4

---

Published as a conference paper at ICLR 2026
models (e.g., with L > 10s for Hunyuan Video (Kong et al., 2024)), storing the full attention matrices is prohibitively expensive as shown in Fig. 3d, incurring (L2)) memory and compute overhead.
To address this, propose Multi-Scale Salient Attention Distillation (MSAD), memory-cfficien framework that distills attention across multiple resolutions, preserving both global structure and local saliency without excessive resource consumption. MSAD employs complementary guid- ance mechanisms: global guidance for high-level structural supervision, and local guidance for fine-grained detail preservation.
Global Guidance. Our approach exploits the intrinsic locality of video data: patially adjacent tokens exhibit high similarity due temporal smoothness and spatial continuity (Ren et al., 2025; Xi al., 2025; Yuan et al., 2024; Feng et al., 2024). To efficiently capture global attention patterns, we downsample Q and K via average pooling with stride s, producing low-resolution features Q K E R fxdd where L L/s² < L. The global distillation is computed as:
QK
Allobal = softmax
Cglobal = MSE
(A
EP global
quant global
(6)
vdl
where MSE denotes the Mean Square Error. This approach requires only O(i2) complexity which is s² times cheaper than full attention.
Local Guidance. While global guidance ensures structural fidelity, fails capture the fine-grained details crucial for high-quality video synthesis. We further observe that the attention saliency in video models is highly skewed: only small subset of tokens dominates the attention mass (see Fig 3a). Formally, we define the token saliency as:
A = softmax (QK) (dd))RRLL 8y = Σ An,ids h,
where h denotes the attention head, denotes the key token index, and 8j measures the aggregate attention received by token j. Empirically, follows a heavy-tailed distribution, with fewer tokens accounting for the majority of attention mass (we provide more analysis in Appendix Sec. F). We exploit this by selecting the top-k queries I = {j 8y is top-k} from the FP model and computing high-resolution attention only for these salient queries:
(Q.KK] dd
Alocal softmax
Liocal MSE
Iocal
||A
local
where QI E kxdx Local distillation focuses supervision on high-impas regions at minimal cost.
Integration and Optimization. We combine both guidance terms into unified distillation object:
Ldistill
Eqquant + İglobal Liloball + Moocal Ltocal:
where Àglobal and Atoca balance the two guidance component. During PTQ calibration, optimize the quantization parameters over Deal to minimize Caisill- aligning the quantized attention with its FP counterpart. As shown in Fig 3b and Fig 3c, MSAD substantially reduces attention shift, enabling robust integration of quantization and sparse attention in video generation.
3.3
SECOND-ORDER SPARSE ATTENTION REPARAMETERIIZTIIN
While the proposed MSAD mitigates the countization-inducce attention shift during calibration phase by aligning attention maps, the intrinsic bottleneck of sparse attention (i.e., the unavoidable discard of low-mage wwwgaiint yet non-trivial attention connections) still exacerbates the amplified atten-
tion shift, especially under high sparsity rates (Xi et al., 2025; Zhang al., 2025b). We formalize this deviation at denoising timestep t in the diffusion process as: (1) !1) - A sparses where A and Aguaee denote the full-attention and sparse attention. We define this deviation (+) as the
(1)
first-order residual. This residual is intrinsic sparsity and cannot be recovered through attention
5

---

Published as a conference paper at ICLR 2026
One Order Order
200
Mean Value Variance
223
No Cache Order
253
Two Order
SVD
Timostop
Singular
Timestop
(a) Residual temporal difference. (b) Singular value distribution
(c) Atlention error comparison.
timesteps.
Figure 4: The motivation and effect of Second-Order Sparse Attention Reparameterization. The results are from Hunyuan Video-13B (Kong et al., 2024) lingle_Iransfore bllockk W4A8. We provide more visualization and analysis in Appendix Sec. G.
_loock. 10 under
distillation alone. Prior work (Yuan et al., 2024) exploits temporal coherence in video generation by assuming that residuals are invariant across timesteps:
A
A A (1) vt,
(10)
Under this assumption, one can cache a reference residual A (tar) from a chosen timestep and reuse it across the successive timesteps, yielding firrst-order sparse attention reparameterization:
A
(t) A (t) full
R3 A
(fre) fall
(frer) sparas
= A (far)
A (f)
(Puer)
sparse
sparse
(11)
cuched
Proposition 3.2. Let A! denote the quantized sparse atlention output. The quantizatioonindduee perturbation ctt) (as defined in Eq. 4) modifies the one-order residual to:
(e)
A (6)
A
A (1) +O(1 [M]]b)),
(12)
(t') quant
(t) for t' tt. quant:
quant varies with (()) due to the quantization noise (Wu et al., 2024; Zhao et al., 2024; He et al., 2023) which violating Eq. 10. We visualize this variance of (4) (t-1) in Fig. 4a. This temporal variance undermines the accuracy of Eq. 11, causing non-negligible attention errors when first-order reparameterization applied after quantization.
Proposition 3.2 indicates that, unlike A A (1)
Proposition 3.3. Although quaa is unstable, we observe that the second-order residual quant
(1-1) exhibits significantly higher temporal stability:
quant quant
E,
||
quaan
A
(t') quani
E4
A
(4) quami
(t') quant
for |t S-11ST..
(13)
F
Fl
We visualize the empirical analysis results in Fig. 4a. This stability arises because quantization noise (t) follows slow-varying stochastic process in diffusion process (Ma al., 2024a; Liu et al., 2024a): adjacent timesteps share similar distributions, rendering (!)) - (!f-1) approximately stationary. Leveraging this property, we propose second-order sparse attention reparameterization:
(A
(8) fill
A (t)
(A (fae)
-A (tar)
(A
(tei)
A (tar))
(A (1/e)
Asg
(Mart quant>
sq
A (1)
(4)
A (bar)
A (tert)
) A (Bver)
8.9
quant
(14)
A (6)
(fat)
(fnr)
cached

---

Published as a conference paper at ICLR 2026
Table I: Text-to- Video generation results on W-n2.1-1.3B. Density the attention density. Full Prec. denotes Full Precision model. Bold: the best result.
Quality
BBits (W/A)
Method
Density
Video Quality Metries
FP Diff Metrics
CLIPSIM-
VQA,
AFSCore
PSNR,
SSIM,
LPIPS
6an 1.3B (CPG 6.0. 480 832p,
Frames
80)
Full Prec.
16/16
100%
0.191
73.12
0.000
PTQ4DIT
6/6
100%
0.182
36.79
2.287
10.20
0.343
0.598
Q-DiT
6/6
100%
0.183
39.21
2.125
10,36
0.351
0.577
SmoothQuant
0.184
40.57
2,008
10.44
0.353
QuaRot
6/6
100%
0.190
42.81
1.754
10.71
0.379
0.571
ViDiT-Q
6/6
100%
0.190
50.85
1.253
11,02
0.385
0,526
Q-VIDIT
6/6
100%%
0.191
75.20
0.982
12,06
0.405
0.474
QuaRot=DFT
0.183
36.79
2.297
0.546
QuaRot=Jenga
6/6
40%
0,184
38.78
2.165
11.32
0.329
0.543
QuaRot-SVG
6/6
40%
0.183
41.93
1,940
11.43
0.331
0.541
D-V
0.188
47.33
1.377
11.06
0.345
g-VDiT+Jenga
6/6
40%
0.189
53.52
1.087
11.21
0.345
0.583
Q-VDiT+VVG
6/6
40%
0.191
55.92
0,942
11.61
0.384
0,508
QuantSparse
6/6
40%
0.193
78.35
0.055
15.51
0.511
0.324
PTQ4DIT
4/8
100%
0.181
30.26
2.574
10,00
0.318
0.603
Q-DiT
0.182
32.57
2.767
0.320
0.594
SmoothQuant
4/8
100%
0.182
34.82
2.174
10,20
0.327
0,569
QuaRot
4/8
100%
0.185
65.15
1.870
11.72
0.349
0.514
ViDiT-Q
0.186
63.21
1.698
11.24
Q-VIFT
4/8
100%
0.190
56.45
2,240
11.01
0.394
01566
QuaRot=DFT
4/8
40%
0.187
32.23
2.329
10.32
0.360
0.583
QuaRot-Jenga
4/8
40%
0.191
32.83
2.148
10.33
0.346
0.578
QuaRot-SVG
40%
32.48
2.088
10.58
0.370
0.576
D-V
4/8
40%
0,185
45.60
2,907
10.03
0.331
0.594
O-VDiT+Jenga
4/8
40%
0.185
47.61
3.000
10.04
0.334
0.596
S-VDiT+SVG
51.84
10.07
0.342
0.592
QuantSparse
4/8
40%
0.193
81.09
0.576
15.22
0.502
0.338
Theorem 3.4. When Proposition 3.3 holds, the expected upproximation error of sparse attention
satisfies:
E2
(0) Jull
( B.G
S
E
(0) Fall
A ( || for 11-2155
F
(15)
Theorem 3.4 indicates two-order guaranteeing tighter full-attention approximation than the first- order method. Also A (twr) + Auu
quant in Eq. 14 can be jointly cached, without any additional storage burden compared with one-order residual. We further reduce the temporal variance of Aquam by projecting it onto its most stable subspace. Empirically, the top-r principal components from the singular value decomposition (SVD) of Aquant capture the dominant, temporally stable patterns (see Fig. 4b). Critically, the dominant principal component exhibit exceptional temporal stability, which inspired us project residuals onto the top-r extracted stable components:
SVD(Áquam)) quant) = SUVT. Å quant S U srar VT 1o0r
A (4)
(t)
+
A (tart) quant
A Andt
(16)
quant
suched
We apply the sparse attention for inference with fixed cache-refreshing interval (5 experiments) for full-attention calculation. As visualized in Fig. 4c, SVD-based second-order reparameterization further suppresses temporal variance, yielding accurate full-attention approximation results.

---

Published as a conference paper at ICLR 2026
Table Video generation on large video generation models. Bold: the best result. Underline: the second best result.
Quality
Latency & Speed
Method
(W/A)
Density.
Video Quality Metrics
EP Dift. Metries
DiT Time
Speedupt
CLIPSIM,
VQA,
ATSCore
PSNR,
SSIM, LPIPS
Hunyuan Vidco 13B (CFG
6.0, 720 x 1280p, Trames 00)
Full Prec.
16/16
100%
0.184
81.23
0.000
1264s
1,00x
SmoothQuant
42.21
1.194
15.44
1149s
1.10x
QuaRot
4/8
100%
0.180
42.89
0.708
15.46
0.502
0.528
1149
1.10x
VIDII-Q
1,254
1.10x
Q. VDIT
4/8
100%
0.182
67.95
1.168
16.85
0.605
0.461
1155s
1.09
Qua
15.39
1.73x
Q VDIT+SVG
4/8
25%
0.182
70.99
1.379
16.71
0.595
0.458
7436
1,70x
Qu
6718
Q VDD+++VG
4/8
15%
0.182
76.30
1,393
16.66
0.591
0,460
6876
1.84X
QuantSparse
0.183
20.86
7318
1.73
QuantSparse
4/8
15%
0.184
81.19
0.016
20.88
0.678
0.273
6718
1.88
Man2.1
(CPG 5.0.
1280p, Er Eramea
80)
Full Prec.
16/16
100%
0.182
90.79
0.000
4031s
1,00X
SmoothQuant
4/8
100%
0.180
73.11
0.875
13.70
0.423
0.510
3425s
1.18x
QuaRet
100%
85,91
0.753
13.79
0.431
3425s
1,18x
VIDIT-Q
4/8
100%
0.182
83.13
0,496
15.12
0.487
0.425
3425s
1.18x
Q-VDIT
83.76
0.343
15.85
0.512
0.398
3457s
1.17
SVG
4/8
25%
0.182
85.66
0.134
13.70
0.427
0.487
2594s
1.55
O-VDIT+SVG
87.89
0.310
15.48
2635s
1,53x
SVG
4/8
15%
0.182
81.93
0.152
13.40
0.415
0.494
2315%
1.74x
O-VDIT+SVG
0.181
82.31
0.411
0.493
2372s
1,70x
QuantSparse
4/8
25%
0.183
91.98
0.056
18.72
0.630
0.240
2594s
1.55
QuantSparse
0.042
18.22
0.605
0.272
2315s
1,74x
3.4
OVERALL PIPELINE
Our proposed QuantSparse framework consists of two component shown in Fig. 2: MSAD for attention distillation during calibration and SSAR for dynamic attention reparameterization during inference. The detailed overall pipeline is provided in Appendix Algorithm 1.
EXPERIMENTS
4.1 EXPERIMENTAL AND EVALUATION SETTINGS
Evaluation Settings. We apply QuantSparse VinyuanVddeo-133 Video-13B (Kong et al., 2024), Wan2.1- 1.3B and 14B (Wan et al., 2025) with 50 sampling steps. We employ two types of metrics: (1) Multi-aspects metries evaluation: including CLIPSIM (Wu et al., 2021), VQA (Wu al., 2023), FlowScore (Liu et al., 2024b), PSNR, SSIM, and LPIPS (Zhang et al., 2018). All metrics are evalu- ated on the prompt select 8 major dimensions from Vbench (Huang et al., 2024b) following prior works (Zhao et al., 2024: Chen et al.. 2024; Feng al., 2025c). For bit setting, we use W6A6 and W4A8 following prior work (Zhao et al., 2024; Chen et al., 2024; Wu et al., 2024), since they can bring more compression effects and ensure the performance.
used in (Zhao et al., 2024; Feng et al., 2025c) (2) Benchmark evaluation: We
Baseline Methods. We select PTO4DIT (Wu et al., 2024), Q-DIT (Chen et al., 2024), ViDIT- Q (Zhao et al., 2024), and Q-VDiT (Feng et al., 2025c) for diffusion baseline. We also compare with strong LLM baseline SmoothQuant (Xiao et al., 2023a) and QuaRot (Ashkboos et al., 2024). For sparsification, we compare with DiTFastAttn (DFT) (Yuan et al., 2024) saache-bsed), Jenga (Zhang al., 2025d) (dynamic pattern), and Sparse VideoGen (SVG) (Xi et al., 2025) (static pattern..
Implementation Detail. Same with prior works (Zhao et al., 2024; Ashkboos et al., 2024; Feng et al., 2025c), adopt channel-wist weight quantization and dynamic token- wise activation quan- tization. We follow block-wise post-training strategy used in (Wu et al., 2024; Chen et al., 2024; Sun al., 2024b) for calibration. More details be found in Appendix C.
4.2 MAIN RESULTS
8

---

Published as a conference paper at ICLR 2026
We present multi-aspects metrics evalua- tion results on Hunyuan Video (Kong et al., 2024) and Wan2.1-14B (Wan et al., 2025) in Tab. 2. can be seen that the existing SOTA quantization methods have signif- icant performance degradation after apply- ing sparse attention. But QuantSparse still maintains high generation performance even high sparsity. It is worth mention- ing that QuantSparse even surpasses the existing quantization-only methods under the low-bit settings of W6A6 and W4A8. Compared with the mance. For example, for Hunyuan Video under W6A6, QuantSparse achieved a VQA score of 82.26 with only 15% attention density, far exceeding current SOTA method Q-VDiT (Feng et al., 2025c) of 73.68, and even surpassing the FP model of 81.23. We present more baseline methods compar- ison in Appendix Sec. D, and comprehensive VBench evaluation results in Appendix Sec. E. We also observed that QuantSparse slightly outperforms Full Precision model certain metries. This slight outperformance of QuantSparse can be attributed to its focus on task-critical tokens and reduced attention noisy irrelevant tokens, as shown our saliency analysis. Additionally, the SSAR module stabilizes sparse attention, reducing quantization noise and improving temporal consistency. These effects, combined with targeted compression, allow QuantSparse to maintain near-lossless quality while offering substantial compression and acceleration. We also visualized the generated videos in Fig. 5. Compared with FP model, QuantSparse achieves almost lossless generation performance while other methods have notable quality degradation. We provide more visual comparison results in Appendix Sec. M.
Table Ablation results of each component.
Method
VQA
PSNR; SSIM;
LPIPS)
Distillation
Analysis
None
81.92
14.35
0.486
0.425
Global
85.26
16.01
0,547
0,349
Local
86.95
16.82
0.561
0,325
MSAD
91.98 10.00
18.72 4.37
0.630 0.144
0.240 0.186
Cache Analysis
None
68.00
14.16
0.470
0.445
First
70.82
17.08
0.572
0.285
Second
89.73
18.68
0.616
0.258
SSAR
91.98, 233
18.72 4.56
0.630 -0.100
0.240 0. 20G
(FP) model, QuantSparse even maintains almost lossless perfor-
FP16 Full Attn. (100%)
SmoothQuontt (100%)
VIDIT-Q (100%)
Quarot-SVG (25%)
Q-VDiT-5VG (25%)
QuantSparse (25%)
QuantSparse (15%)
Figure 5: Visual comparison on Wan2.1-14B under W4A8 quantization setting. We uniformly
sample two frames for visualization. '(xx%)' denotes the attention density.
4.3 ABLATION STUDY
We conduct ablation study on proposed Multi-Scale Salient Attention Distillation (MSAD) and Second- Sparse Attention Reparameterization (SSAR) on Wan2.1-14B under W4A8 in Tab. 3.
Effect of attention distillation. Compared with no distillation, both proposed attention guidance can enhance the model performance. The combined MSAD further improves PSNR from 14.35 to 18.72, demonstrating the effect of attention distillation.
Effeet of attention reparameterization. Compared with naive sparse attention, first-order residual can reduce the attention error, demonstrating the effectiveness of attention reparameterizztion. Our proposed SSAR achieves the best approximation performance by reducing both the quantization- induced error and temporal variance.
Effect of cache-interval. We also supplement the ablation and the results are shown in Tab. 5. While shorter intervals yield higher PSNR and SSIM, indicating better performance, they also result in reduced speedup (1.65x and 1.69x respectively). For instance, interval=3 achieves the highest PSNR (18.86) but sacrifices a noticeable amount of the potential speedup (9%). Longer intervals increasing the interval to 6 provides slightly higher speedup (1.76x). However, this comes the cost of a degradation in performance (PSNR drops to 17.72). We choose interval=5 based on its optimal balance between model performance and inference speedup. But highlight that this

---

Published as a conference paper at ICLR 2026
Table 4: Detailed efficiency comparison.
#Bits (W/A)
Model Overload
Latency & Speed
Method
Density.
Model Storage,
Memory Consumption,
DiT Time;
Speedup;
Vineu 13B (CFG 6.0, 720 1280p. Irames 60)
Full Prec.
16/16
100%
23.88GB
35.79GB
1264s
1.00x
QuaRot
4/8
100%
6.49GB
24.34GB
1149s
1.10x
Q-VDIT
4/8
100%
6,50GB
24.89GB
1155s
1.09x
DFT
16/16
25%
23 B3.8GBB
40.1|GB
792s
1.60x
Jenga
16/16
25%
23 88GB
36,92GB
846s
1.49x
SVG
16/16
25%
23 88GB
40.10GB
786s
1.61
SVG
16/16
15%
23.88 88GB
40.10GB
707s
1.79x
QuantSparse
4/8
25%
6,49GB
27.02GB
731s
1.73x
QuantSparse
4/8
15%
6.49GB
27.02GB,
671s
1.88x
Wan2.1 14B
(CFG
5.0, 720 x 1280p. Frames = 80)
Full Prec.
16/16
100%
26.6IGB
42.48GB
4031s
1.00x
QuaRot
4/8
100%
7,00GB
26.04GB
3425s
1.18x
Q-VDiT
4/8
100%
7.02GB
26,73GB
3457s
1.17x
DFT
16/16
25%
26.61GB
44.86GB
3015s
1.34x
Jenga
16/16
25%
26.61GB
42.62GB
3087s
1.31x
SVG
16/16
25%
26.61GB
44.07GB
2987s
1.35x
SVG
16/16
15%
26.61GB
44.07GB
2661s
1.51
QuantSparse
4/8
255
7.00GB
28. 14,14GB,, 1.114GBB1...
2594s
1.55x
QuantSparse
4/8
15%
7.00GB.a.se
28, 141GB,
2315s
1.74x
trade-off based computational resource and all interval settings offer reasonable results and notable acceleration.
More ablation study about pooling stride 8, salient token k, weight factor X, and SVD rank in Eq. and Eq. 16 in Appendix Sec. H.
4.4 EFFICIENCY ANALYSIS
We present the deployment efficiency Tab, 4. All the experiments are conducted on single NVIDIA A800 80G GPU with CUDA 12.4. We use CUTLASS (Thakkar et al., 2023) on top of PyTorch for performing INT matrix multiplica- tion. Existing quantization methods can bring higher model compression, but the effect in- ference acceleration is limited. Sparse attention brings significant acceleration, but has almost model compression, and even brings more memory consumption. QuantSparse combines the advantages of both quantization and sparse attention, bringing significant model compres- sion and acceleration. For 1-n2.1-14B (Wan et al., 2025), QuantSparse (15% density) brings 3.80x storage compression, 1.51 memory sav-
Table 5: Ablation study of cache-fresh interval
and attention density on W4A8 1-14B.
PSNR, SSIM,
LPIPS
Speedup,
Interval Analysis
Interval=3
18.86
0.631
0.243
1.65
Interval=4
18.48
0.617
0.260
1.69
Interval=5
18.22
0.605
0.272
1.74x
Interval=6
17.72
0.566
0.321
1.76x
Density Analysis
Den
18.72
0.630
0.240
1.55
Density=205:
18.45
0.622
0.252
1.63
Density=15%
18.22
0.605
0.272
1.74x
Density=10%:
17.73
0.589
0.288
1.80x
ing, and 1.74x end-to-end acceleration. We further report the calibration resource consumption in Appendix Sec. I and report the performance combined with other acceleration methods in Appendix Sec.
CONCLUSION
In this paper, we propose QuantSparse, unified compression framework that effectively combines model quantization and sparse attention. To address the amplified attention shift, propose Multi- Scale Salient Attention Distillation to efficiently align the attention shift. To address the intrinsic sparsity loss, propose Second-Oree second-order residual for attention approximation. Extensive experiments shown that QuantSparse achieves lossless performance while bringing significant model compression and acceleration.
Sparse Attention Reparameterizztion utilize decomposed
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGEMENTS
This work is supported by the National Natural Science Foundation China under Grant Number 624063 12 and 62476264, the Beijing Natural Science Foundation under Grant Number 4244098, the Science Foundation of the Chinese Academy of Sciences, and the Swiss National Science Founda- tion (SNSF) project 200021E 219943 Neuromorphic Attention Models for Event Data (NAMED).
ETHICS STATEMENT
This research strictly adheres to the ICLR Code of Ethics with no ethics-related risks: uses public open-source video-generation models (Wan2.1 (Wan et al., 2025) and Hunyuan Viddu (Kong et al., 2024)) and focuses on algorithmic innovation for inference acceleration and compression, without involving scenarios endangering public safety, infringing privacy, or producing discrimination.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, experimental configurations, method details, and evaluation metrics thoroughly described Sec. 4.1 and Appendix Sec. C. Experimental results of comparative methods sourced from public literature, and experiments strictly follow the same configurations as baseline methods for fair comparison. The key codes and the presented video source files are also attached in the supplementary materials. For the theorem used in the paper, we also provided detailed proof in Appendix Sec. A.
REFERENCES
Zhulin An, Xinqiang Yu, Chu Wang, Yinlong Zhang, and Chunhe Song. bodied intelligence: Recent advances and future perspectives, 2025. ISSN 3105-
Em-
8515. URL https: //wp://wwwwww/////// the- innoviniionnnnommm 68da755beeeed99441222000
Salch Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi. Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint h0l::2004.004566 2024. 8, 17, 18, 19, 20, 26
Lei Chen, Yuan Meng, Chen Tang, Xinzhu Ma, Jingyan Jiang, Xin Wang. Zhi Wang, and Wenwu Zhu. Q-dit: Accurate post-training quantization for diffusion transformers. arXiv preprint arXiv:2406. 17343, 2024. 3, 8, 18
Krishna Teja Chitty-Venkata. Sparsh Mittal, Murali Emani, Venkatram Vishwanath, and Arun K Somani. A survey of techniques for optimizing transformer inference. Journal of Systems Archi- fecture, pp. 102990, 2023.
Yifu Ding, Weilun Feng, Chuyan Chen, Jinyang Guo, and Xianglong Liu. Reg-ptq: Regression- specialized test-training quantization for fully quantized object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16174-16184, 2024.
Weilun Feng, Chuanguang Yang, Zhulin An, Libo Huang, Boyu Diao, Fei Wang, and Yongjun Xu. Relational diffusion distillation for efficient image generation. In Proceedings the 32nd ACM International Conference on Multimedia, pp. 203-213, 2024. 5
Weilun Feng, Haotong Qin, Chuanguang Yang, Zhulin An, Libo Huang, Boyu Diao, Fei Wang, Renshuai Tao, Yongjun Xu, and Michele Magno. Mpq-dm: Mixed precision quantization for extremely low bit diffusion models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, pp. 16595-16603, 2025a.
Weilun Feng. Haotong Qin, Chuanguang Yang, Xiangqi Li, Han Yang, Yuqi Li, Zhulin An, Libo Huang, Michele Magno, and Yongjun Xu. vdtt Accurate quantized video diffusion trans- former with salient data and sparse token distillation. arXiv preprint arXiv: 250.00600 2025b.
11

---

Published as a conference paper at ICLR 2026
Weilun Feng, Chuanguang Yang, Haotong Qin, Xiangqi Li, Yu Wang, Zhulin An, Libo Huang, Boyu Diao, Zixiang Zhao, Yongjun Xu, et al. Q-vdit: Towards accurate quantization and distillation of video-generation diffusion transformers. In International Conference Machine Learning, pp- 16956-16976. PMLR, 2025c. 3, 8, 9, 16, 17, 18, 20, 26
Weilun Feng, Chuanguang Yang, Haotong Qin, Yuqi Li, Xiangqi Li, Zhulin An, Libo Huang, Boyu Diao, Fuzhen Zhuang, Michele Magno, et al. Mpq-dmv2: Flexible residual mixed precision quan- tization for low-bit diffusion models with temporal distillation. arXiv preprini arXiv:2507.04290, 2025d.
Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, et al. Moa: Mixture of sparse attention for automatic large language model compression, 2024. URL htpps//arxii
org/ubs/2406. 14909.
Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi. Junjie Lai, Hayden Kwok-Hay So, Ting Cao, Fan Yang, et al. Secrattention: Learning intrinsic sparse attention in your llms. arXiv preprint arXiv:2410. 13276, 2024.
Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pp. 291 -326. Chapman and Hall/CRC, 2022. 2,
Yefei He, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Efficientdm: Efficient quantization- aware fine-tuning of low-bit diffusion models. arXiv preprint arXiv:2310.03270, 2023. 3, 6
Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Ptqd: Accurate post- training quantization for diffusion models. Advances in Neural Information Processing Systems, 36, 2024. 3
HPC-AL Open-sora, 2024. URL htwph:tpp//eee /tttt::://githute wtt Sora. 2
Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu. Tfmqq-m: Temporal fea- ture maintenance quantization for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7362-7371, 2024a. 3
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024b. 8, 17,
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient ainger-arithmmetc-only inference. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018. 2
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03660 2024. 1, 5, 6, 8, 9, 11, 20, 21,
Muyang Li, Yujun Lin, Zhekai Zhang. Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun- Yan Zhu, and Song Han. Svdqunat: Absorbing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:2411.05007 2024a. 3
Yanjing Li, Sheng Xu, Xianbin Cao, Xiao Sun, and Baochang Zhang, Q-dm: quantized diffusion model. Advances in Neural Information Processing Systems, 36, 2024b. 3
An efficient low-bit
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine- -raiind chinese understanding. arXiv preprint arXiv:2405.0877 2024c. 26
Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei. Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It's time to cache for video diffusion model, arXiv preprint 111199241..19108, 2024a. 6, 24, 25
12

---

Published as a conference paper at ICLR 2026
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22139-221444 2024b. 8
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, al. Sora: review on background, technology, limitations, and opportunities of large vision models. arXiv preprint 11X11:2402.17177, 2024c. 2
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv: 2017. 18
Chao Lou, Zixia Jia, Zilong Zheng, and Kewei Tu. Sparser is faster and less is more: Efficient sparse attention for long-ram rangrrage transformers. arXiv preprint arXiv:2406.16677, 2024.
Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong. Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, al. Moba: Mixture of block attention for long-context Ilms. arXiv preprint Xrxiv:2502.1311 13189, 2025. 3
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15762 -1572,, 2024a.
Yuexiao Ma, Taisong Jin, Xiawu Zheng, Yan Wang, Huixia Li, Yongjian Wu, Guannan Jiang, Wei Zhang, and Rongrong Ji. Ompq: Orthogonal mixed precision quantization. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 9029-9037, 2023a.
Yuexiao Ma, Huixia Li, Xiawu Zheng. Xuefeng Xiao, Rui Wang, Shilei Wen, Xin Pan, Fei Chao, and Rongrong Ji. Solving oscillation problem in post-training quantization through theoreti- cal perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7950-7959, 2020...
Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. arXiv preprint XrXiv:2403. 12544, 2024b. 3
Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Outlier-aware slicing for post-training quantization in vision transformer. In Forty-first International Conference on Machine Learning, 2024c.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023. 2
Ratko Pilipović, Patricio Bulié, and Vladimir Risojević. Compression of convolutional neural networks: A short survey. In 2018 17th International Symposium INFOTEH-JAHORI (IN- FOTEH), pp. 1-6. IEEE, 2018.
Sucheng Ren, Qihang Yu, Ju He, Alan Yuille, and Liang-Chieh Chen. Grouping first, attending smartly: Training-free acceleration for diffusion transformers. arXiv preprint arXiv:2505. 14687, 2025. 5, 16
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffuuion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-364944 2022. 26
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1972-1981, 2023.
Rui Sun, Yumin Zhang, Tejal Shah, Jiahao Sun, Shuoying Zhang, Wenqi Li. Haoran Duan, Bo Wei, and Rajiv Ranjan. From sora what we can see: A survey of text-to-video generation. arXiv preprint ar 4005 2024a.
13

---

Published as a conference paper at ICLR 2026
Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzi Yu, Lu Hou, Chun Yuan, et al. Flatquant: Flatness matters for Ilm quantization. arXiv preprint arXiv:2410.09426, 2024b. 8, 18
V. Thakkar et al. CUTLASS, 2023. URL
10
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Cei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.200 20314, 2025. 2, 4, 9, 10, 11, 18, 20, 21, 23, 24
Lu Wei, Zhong Ma, Chaojie Yang, and Qin Yao. Advances in the neural network quantization: A comprehensive review. Applied Sciences, 14(17):7445, 2024. 4
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and d Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104. 14806, 2021. 8
Haoning Wu, Erli Zhang. Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
Junyi Wu, Haoxuan Wang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Ptq4dit: Post-training quantization for diffusion transformers. arXiv preprint 1rXiv:2405.16005, 2024. 3, 6, 8, 18
Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. Sparse videogen: Accelerating video diffusion transformers with
spatial-temporal sparsity, arXiv preprint arXiv:2502.01776, 2025. 2, 3, 5, 8, 16, 19, 20
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infilm: Training- long-context extrapolation for Ilms with an effi- cient context memory. Advances in Neural Information Processing Systems, 37:1 19638 1211119686666111 2024a. 3
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pp. 38087-38099, PMLR, 2023a. 8, 18
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309 17453, 2023b. 20233,
Guangxuan Xiao, Jiaming Tang. Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. arXiv preprint arXiv:2410. 10819, 2024b. 3
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11080 2025. 3
Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. Ditfastattn: Attention compression for diffusion transformer models. Advances in Neural Information Processing Systems,
1219, 2024. 2, 3, 4, 5, 6, 8, 16
Jintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. Sageatten- tion2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. arXiv preprint 1xXi:2241111095: 2024a.
Jintao Zhang, Jia Wei, Haofeng Huang, Pengle Zhang, Jun Zhu, and Jianfei Chen. Sageat- tention: Accurate 8-bit attention for plug-and-play inference acceleration. arXiv preprint
arXiv:2410.02367. 2024b. 3, 24,
Jintao Zhang, Jia Wei, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Haoxu Wang, Kai Jiang, Jun Zhu, and Jianfei Chen. Sageattention3: Microscaling fp4 attention for inference and an exploration of b- training. arXiv preprint 2oXiv:5555/11944 2025a.
14

---

Published as a conference paper at ICLR 2026
Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. arXiv preprint arXiv:2502.18137. 2025b. 3, 4,5
Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, lon Stoica, Zhenghong Liu, and Hao Zhang. Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.04507, 2025c.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018. 8
Yuechen Zhang, Jinbo Xing, Bin Xia, Shaoteng Liu, Bohao Peng, Xin Tao, Pengfei Wan, Eric Lo, and Jiaya Jia. Training-frin free efficient video generation via dynamic token carving. arXiv preprint arXiv:2505. 16864, 2025d. 2, 3, 8, 16
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient gen- erative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710, 2023.
Tianchen Zhao, Tongcheng Fang, Enshu Liu, Wan Rui, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, al. Vidit-q: Efficient and accurate quantiza- tion of diffusion transformers for image and video generation. arXiv preprint arXiv:2406.022 02540, 2024. 3, 8, 16, 17, 18
Xingyu Zheng, Xianglong Liu, Yichen Bian, Xudong Ma, Yulun Zhang, Jiakai Wang, Jinyang Guo, and Haotong Qin. Bidm: Pushing the limit of quantization for diffusion models. arXiv preprint arXiv:2412.05922 2024a. 2,3
Xingyu Zheng, Haotong Qin, Xudong Ma, Mingyuan Zhang, Haojie Hao, Jiakai Wang. Zixiang Zhao, Jinyang Guo, and Xianglong Liu. Binarydm: Towards accurate binarization of diffusion model, arXiv preprint arXiv:2404.05660 2024b. 2,
15

---

Published as a conference paper at ICLR 2026
A PROOF OF THEOREM 3.4.
Proof of Theorem 3.4.
(t) For Agg. we have:
(t)
(A (bar))
A ((trt)
A quant
full
A
(4)
A
(A (fret)
A (mre)
A.q
Å
quant
(17)
A (1)
A (frrr
quant
A
quane
Given this, we further have:
A
full
A
(A(()
(+)
A
quant
(furr)
A (t)
quant)
A()
(A(()
(+)
S.
quant
(tur)
quant)
(1)
(A!
(t)
+ A
quant
+ A
quaat
(18)
A
(0)
(hat)
quant
quant
Similarly, for A we also have:
A full
A
(A (3)
+ A (t) (t)
quant)
(A
(t)
+ quant
(4) (A(t)
(t)
+ A
(hur)) quant
(19)
(fart
quant
quant
Based on Proposition 3.3, have:
E2
A ( tull
(4)
E,
|
Auaat
(har
< E2
A (#)
A (Bar)
E2
A (9 full
quant
quamt
quani
second-order
first- order
(20)
Therefore, Theorem 3.4 holds.
B DETAILS OF SELECTED EVALUATION METRICS
MULTII-SPECTS METRICS EVALUATION
This evaluation suite includes absolute quality of videos and relative difference metrics that quantify the difference between FP16 generation.
Absolute Quality. Consistent with prior quantization works (Zhao et al., 2024; Feng et al., 2025c), we adopt CLIPSIM, VQA, and FlowScore to measure text-video alignment, quality, and temporal consistency, respectively.
Relative Difference Metrics. Following prior sparse attention works (Xi al., 2025; Yuan et al., 2024; Ren et al., 2025; Zhang et al., 2025d), we adopt Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) for pixel- space differences, structural similarity, and high-level patch similarity, respectively.
All the evaluations conducted high-resolution generation tasks. Due the computational overhead, we use the OpenSORA prompt sets used (Zhao et al., 2024; Feng et al., 2025c) for video generation.
16

---

Published as a conference paper at ICLR 2026
Algorithm QuantSparse: Calibration to Inference Pipeline
Require: Pre-trained video diffusion transformer M (FP16), calibration dataset Deal, target bit-
width (W/A), denoising steps T. cache interval т
Ensure: Quantized-sparse model Mos. generated video Y
1: Calibration Phase:
2: Initialize quantization parameters {s, for weights (W) and activations (A)
3: Input X E Dett to
4: Compute token saliency 8j using Eq. for FP model M
5: Select top-k salient tokens - [ 8j is top-k)
6:
Global Guidance Distillation:
7:
Calculate Litabal using Eq. 6
8:
Local Guidance Distillation:
9:
Calculate Local using Eq. 8
Optimize quantization parameters using Eq. with Liloball and Litcal
11:
Obtain quantized model Mquai with optimized (s, =}
12:
Inference Phase:
13:
Load M Mquant and input prompt P.
14:
Input P into Mquant and initialize cached residuals {A
quant Å
(brr) quami
for in T''
16:
Compute quantized sparse attention:
SparscAttention(22 (QaatiiioonQquu..... K Kquants Vquant' M)
17:
if f - fre T
18:
Reuse cached residuals: Acarr A
(fre) quant
+ A (fwe)
quant
19:
else
20:
Update fae recompute and cache residuals
21:
endif
22:
Refine attention using Eq. 16
23:
endfor
Generate video Y return Y
B.2
BENCHMARK EVALUATION
To further provide benchmark evaluation, follow previous works (Feng et al., 2025c; Zhao al., 2024). We select 8 major dimensions from Vbench (Huang et al., 2024b), including frame-wise quality, temporal quality, and semantic evaluation.
For Frame-wise Quality, we select Imaging Quality and Aesthetic Quality for distortion assess- ment and artistic and beauty evaluation. For Temporal Quality, Smoothness, Subject Consistency, and Background Consistency for degree of dynamics, physical law smoothness. subject's 's appearance consistent, and temporal consistency of the background, spectively, For Semantic Evaluation, we use Scene and Overall Consistency for text prompt scene consistency and overall video-text consistency.
Dynamic Degree, Motion
The evaluation follows the suite provided by VBench (Huang et al., 2024b). We generate one video for each prompt, same as previous works (Zhao et al., 2024; Feng et al., 2025c). Due to the large prompt sets used in VBench, we slightly decrease the resolution for computational efficiency. In addition, this experimental setup also provides an additional evaluation of multi-resolution video generation performance, which proves the generalization and effectiveness of our method in different application scenarios.
C EXPERIMENT SETTINGS
Same with prior works (Zhao et al., 2024; Ashkboos et al., 2024; Feng et al., 2025c), we adopt channel-win weight quantization and dynamic token-wise activation quantization. And we use uniform symmetry quantization for both weight and activation for better hardware acceleration and memory saving. For fair comparison, apply the same quantization granularity for all quantization
17

---

Published as a conference paper at ICLR 2026
Table 6: Text-to-Vides generation experiments on more huge models.
Quality
#Bits (W/A)
Method
Density
Video Quality Metrics
FP Diff. Metries
CLIPSIM;
VQA,
AFSCore
PSNR,
SSIM,
LPIPS,
Hunyuan Video 13B (CFG = 6.0, 720 x 1280p. Frames
60)
Full Prec.
16/16
100%
0.184
81.23
0.000
SmoothQuant
6/6
100%
0.180
69.55
1.406
15.91
0.553
0.411
QuaRot
6/6
100%
0.182
72.28
0.546
16.99
0.590
0.378
ViDiT-Q
6/6
100%
0.182
72.36
0.937
18.24
0,623
0.335
Q-VDIT
6/6
100%
0.182
73.68
1.232
21.02
0.675
0.264
Sua
6/6
25%
0.181
72.57
0.718
16.85
0.581
0.385
Q SVV
6/6
25%
0.181
72.59
1.405
20.38
0.658
0.284
Qua
15%
0.181
72.60
0,997
16.85
0.578
0.394
SVV
15%
0.181
72.04
1.763
19.94
0.644
0.307
QuantSparse
6/6
25%
0.183
81.17
0.435
22.71
0.720
0.221
QuantSparse
6/6
15%
0.183
82.26
0.328
22.68
0.720
0.224
Wan 14B
5.0,
1280p,
Frames 80)
Full Prec.
16/16
100%
0.182
90.79
0.000
SmoothQuant
6/6
100%
0.178
62,25
0.363
13.06
0.404
0.656
QuaRot
6/6
100%
0.180
66.56
0.313
13.59
0.409
0.566
ViDiT-Q
100%
0.180
71.26
0,251
15.30
0.513
0.376
Q-VDIT
100%
0.180
89.10
18.13
0.610
0,264
QuaRot+SVG
25%
0.179
67.64
0.336
13.60
0.407
0.555
Q-VDIT+SVG
6/6
25%
0.179
88.29
0.091
16.69
0.563
0.323
QuaRot+SVG
6/6
15%
0.180
60.14
0.396
13.55
0.399
0.567
Q- SVVG
6/6
15%
0.179
85.26
0.182
15.94
0.532
0.367
QuantSparse
6/6
25%
0.182
89.96
0.002
18.67
0.622
0.240
QuantSparse
6/6
15%
0.181
92.87
0.060
18.67
0.616
0,277
methods. We adopt channel-wise scale used in (Xiao et al., 2023a; Wu et al., 2024; Zhao al., 2024; Feng et al., 2025c) and rotation-based matrix used in (Ashkboos et al., 2024; Zhao et al., 2024; Sun al., 2024b) for quantization. We follow block-wise post-training strategy used in (Wu al., 2024; Chen et al., 2024; Sun et al., 2024b) for calibration. All the experiments are conducted on a single NVIDIA A800 GPU.
During calibration, we set channel-wise scale, rotation matrix, and quantization scale as learnable following (Feng et al., 2025c; Sun al., 2024b). We use 20 random generated samples and train 15 epoch for each transformer block. We apply the same calibration samples and epochs for all methods for fair comparison. rate scheduler. For the channel-wise scale and rotation matrix, we use a learning rate of 5e -3. For the learnalle quantization scale, we use a learning rate 5e -2 For distillation, we use r = 128 for global distillation pooling, 256 for salient query selection, and Aglobal = Lee Aglobal
use AdamW (Loshchilov Hutter, 2017) optimizer and cosine learning
le-4 for Wan2.1-1.39 Wan2.1-14DB and Aglobal 1.0, Agtobal le for Hunyuan Video, respectively. The selection distillation balancing factor is based on the order of magnitude of the loss. For sparse attention,
fixed cache refreshing interval of and use k = 16 for SVD.
For deployment, we quantize the weight and absorb all the quantization parameters following (Zhao et al., 2024; Sun al., 2024b; Feng et al., 2025c; Ashkboos et al., 2024). For activation, use dynamic online quantization same as (Feng et al., 2025c; Sun et al., 2024b; Zhao et al., 2024).
D MORE EVALUATION RESULTS ON WAN2.1-1.38
We present comprehensive evaluation Wan2.1 1-1.3B (Wan et al., 2025) in Tab. 6. Since Wan2.1- 1.3B has less computation budget and we find that it will suffer from serious performance degrada- tion under high sparsity, uniformly adopt 40 density in sparse attention ensure its performance.
Different quantization methods have obvious performance degradation, especially under W4A8. Among them, the quantization method specially designed for video model Q-VDiT (Feng et al.,
18

---

Published as a conference paper at ICLR 2026
Table 7: Performance of text-to-video generation under VBench evaluation benchmark suite. We evaluate on Imaging Quality (IQ), Aesthetic Quality (AQ), Motion Smoothness (MS), Dynamic Degree (DD), Background Consistency (BC), Subject Consistency (SuC), Scene Consistency (ScC), and Overall Consistency (OC). Higher (1) metrics represent better performance. Bold: the best result. Underline: The second best result.
Method
(W/A)
Density
TQ
AQ
MS
DD
BC
SuC
ScC.
OC
Wan2.1 1.3B (CPG
6.0, 480 x 832p, Erames
80)
Full Prec.
16/16
100%
64.05
$7.86
97.03
87.50
94,94
93.00
16.72
13.16
Qua
6/6
40%
62.53
$2.16
95.48
81.94
93.65
89.20
12.43
22.42
V-VDT++VVG
6/6
40%
64.01
53.89
96.25
81.94
94.23
91.78
17.81
22.90
QuantSparse
6/6
40%
64.96
56.44
96.68
83,33
94.84
92.56
18.46
23.12
SGG
4/8
40%
54.45
43.60
96.29
73.61
94.99
87.02
8.14
18.88
Q-VDiT+SVG
4/8
40%
56.08
48.
97.27
61.11
95.86
89.72
10.32
19.89
QuantSparse
4/8
40%
64.41
58.00
97.35
87.50
94.99
93.02
18.24
23.31
Hunyuan Video
13B (CFG
6.0,
512
768p, f araes
Full Prec.
16/16
100%
62.30
62.49
99.00
56.94
98.08
95.30
33.36
26.85
QuaRot+SVG
6/6
25%
56.82
57.23
97.93
40.00
97.75
95.10
23.98
25.63
Q- SVG
6/6
25%
59.22
58.77
97. .6
40.00
97.60
95.68
26.80
25.87
QuaRor+SVG
15%
53.95
56.43
97.84
38.89
97.48
94.40
23.36
25.57
Q- SVG
15%
57.43
58.02
97.84
38.61
97.07
95.20
24.27
25.74
QuantSparse
60.24
59.06
99.01
43.06
98.33
28.42
26.62
QuantSparse
15%
59.54
58.87
98.95
40.28
98.08
26.63
QuaRot+SVG
4/8
25%
45,81
56.59
98.26
22.22
98.18
95.78
21.00
24.64
SVG
44.94
$6.62
98.36
23.61
97.98
18.53
24.81
SVG
43.51
55.35
98.21
20.83
97.21
95.15
18.31
24.50
VDG++SVG
4/8
15%
42.16
55.32
98.32
20.83
97.96
95.48
16.64
24.68
QuantSparse
4/8
25%
59.85
59.37
99.08
38.89
98.32
96.41
29.80
26.92
QuantSparse
4/8
15%
59.27
59.20
99.04
40.28
98.21
96.18
30.31
26.92
2.. 14B (CPG
5.0, 480
832p, Frames
80)
Full Prec.
16/16
100%
63.38
59.56
96.73
86.
96.71
90.84
28.13 13
25.68
QuaRot+SVG
6/6
25%
61.77
$4.13
95.89
45.83
94.78
90.20
17.59
23.37
Q Q-VD-++VVG
6/6
25%
60.92
57.53
96.44
82.50
95.48
89.34
27.76
25.46
QVa
6/6
15%
42
$4.09
95.78
45.83
94.70
89.95
16.50
23.08
Q- SVVG
15%
59.77
56.56
96.31
82.50
95.68
89.25
27.05
25.28
QuantSparse
25%
63,89
58.77
96.77
84.72
96.48
90.91
29,80
25.80
QuantSparse
63.87
$8.32
96.69
90.28
96.29
90.85
29.14
26.07
QuaRot+SVG
4/8
25%
62.53
$7.24
96.52
87.50
95.40
89.77
22.69
25.11
Q SVG
4/8
25%
60.92
58.53
96.44
87.50
95.48
89.34
22.76
25.46
QVa
61.16
$6.71
96.32
87.50
95.33
22.14
25.05
-VV
15%
59.77
$7.56
96.31
87.50
95.68
89.25
22.04
25.08
QuantSparse
4/8
25%
63.55
59.59
96.82
87.50
96.69
90.69
27.76
25.81
QuantSparse
4/8
1.%
63.81
58.86
96.75
87.50
96.56
90.55
26.09
25.93
2025c) and the strong LLM quantization method Quarot (Ashkboos et al., 2024) show relatively stronger performance. For a broader and fair comparison, we add existing sparse attention methods to Q-VDiT and Quarot to verify the effect of naive combination of model quantization and sparse attention. We find that when combining Q-VDiT and Quarot with different sparse attention meth- ods, the performance decreases to varying degrees, and the performance of SVG (Xi et al., 2025) decreases the least. Therefore, we chose SVG as our baseline sparse attention in all other experi- ments.
Compared with all existing methods, QuantSparse achieves SOTA performance under all bit settings, and is almost lossless compared with the FP model. It is worth mentioning that QuantSparse surpasses all quantization-onl methods. not only achieves better compression effect, but also has better performance, which fully demonstrates the effectiveness of method.
19

---

Published as a conference paper at ICLR 2026
E VBENCH EVALUATION RESULTS
We present the VBench (Huang et al., 2024b) evaluation results in Tab. 7. Under the comprehensive evaluation of all 8 dimensions, the naive combination of Q-VDiT (Feng et al., 2025c), Quarot (Ashk- boos et al., 2024) and SVG (Xi et al., 2025) all show significant performance degradation, which fully demonstrates the disadvantage of simply combining existing quantization and sparse atten- tion methods. While QuantSparse achieves comprehensive SOTA performance in all bit settings of all models, and almost lossless compared with model, even better in some dimensions. For W- (Wan et al., 2025) under W4A8, QuantS parse achieves 63.55 and 63.81 under 25% and 15% attention density, respectively, surpassing 63.38 of FP model.
Procel By feeily B-l dil 2,300
Peaesity With prrond 1390
Redeniine Eieely
Terial Mry Berely
IEB perzo LIBE
BDD plncoed 1.CA3
35
15
5
Block14 head1. (b) Block15 head1. (c) Block16 head1. (d) Block17 head1. (c) Block18 head
Figure 6: More token saliency distribution of Wan2.1-1.30 (Wan et al., 2025).
PIONU CHNNE pINCIN LE973
Modie day
PROA BC DRAAR
(a) Block14 headl. (b) Block15 headl.
(c) Block16 headl.
(d) Block17 headl. (c) 18ock18 head
Figure 7: More token saliency distribution of HunyuanVideo-D Video-13B (Kong et al., 2024).
F MORE ANALYSIS OF MULTI-SCALE SALIENT ATTENTION DISTILLATION
We present more analysis of the proposed Multi-Scale Salient Attention Distillation (MSAD) here.
We conducted quantitative experiments to test the impact of quantization and sparsification on at- tention shift by measuring the attention Mean Square Error (MSE). The results are collected from 1000 random samples on 1-1.3B (Wan et al., 2025) under W4A8 and 40% attention density. The results presented in Tab. 8. The attention shift caused by the simple combination of quanti- zation and sparsification methods is much greater than the sum of individual shifts. This proves the joint effect of quantization and sparsification on attention error, and our core motivation "amplified attention shift".
Table 8: Quantitative experiment on attention shift caused by different compression techniques.
Method
Attention Shift
Quantization (QuaRot (Ashkboos et al., 2024)) Sparsification (SVG (Xi et al., 2025))
0.216
0.4
Quantization-Sparsiiccttton
0.685
We supplement 4 additional attention map comparisons in Fig. 8 and Fig. 9, showing the attention distribution difference between the FP model and quantized model. The results collected from W-n2.1-1.3B under W4A8.
Each column in Fig. 8 and Fig. 9 corresponds to the attention difference between the same attention map before and after the proposed distillation MSAD. This indicates that our MSAD effectively alleviates the attention shift.
20

---

Published as a conference paper at ICLR 2026
MSE: 0.625
MSE: 0.463
MSE: 2.153
MSE: 0.708
Figure 8: Attention differences between FP model and quantized model without distillation.
MSE: 0.319 Figure Attention differences between model and quantized model with distillation.
MSE: 0.238
MSE: 0.291
MSE: 1.064
We present more visualization of heavy-tail token saliency distribution in Fig. 6 and Fig. 7. It can be seen that a significantly heavy-tailed token saliency phenomenon appears in different blocks of Wan2.1 (Wan al., 2025) and Hunyuan Video (Kong et al., 2024), which fully shows that salient local distillation is meaningful.
To further prove the effect of top-k salient queries lection, we compare with random selection methods and present the results in Tab. 9, Compared with ran- dom selection, our top-k salient selection further im- proves the PSNR from 15.49 to 16.82, fully demon- strating the effectiveness of our local distillation.
Table 9: Ablation results of local distillation.
Method
VQAr
PSNR:
SSIM
IPPISS)
None
81.92
14.35
0.186
0.425
Rondom
83.17
15.19
0.523
0.372
16.82.2.47
0..61_p.0r5
0.100
Tometer
(a) block. I..
(b) block. 12.
(c) block. 13.
(d) block. 14,
(c) block. 15,
Figure 10: More residual temporal difference distribution of Hunyuan Video 13B (Kong al., 2024).
G MORE ANALYSIS OF SECOND-ORDER SPARSE ATTENTION REPARAMETERIZATION
We present more analysis the proposed Second-Order Sparse Attention Reparameterization (SSAR) here. We present more visualization of residual temporal difference in Fig. 10. It can be seen that after the introduction quantization, the numerical difference of the first-order resid- uals of adjacent time steps cannot be simply ignored. However, the numerical difference of the second-order residual significantly smaller than that of the first-order residual, so the use of the second-order residual has better approximation effect.
21

---

Published as a conference paper at ICLR 2026
(a) block. 11.
(b) block. 12.
(c) block. 13.
(d) block. 14.
(e) block. 15.
Figure 11: More singular value distribution of all timesteps of Hunyuan Video- 13B (Kong et al,, 2024).
To verify the motivation of using the temporal-stable component of the second-order residual, visualize more singular value distribution of all timesteps in Fig. 11. It can seen that in different blocks of different models, the second-order residuals different time steps show considerable sta- bility. Therefore, the second-order residual after SVD can retain the characteristics of time stability, further reduce the variance caused by different time steps, and have better approximation effect.
We further visualize more attention error comparison in Fig. 12. It can be seen that the residual mechanism significantly reduces the attention error, which proves the importance of sparse atten- tion reparameterization. At the same time, compared with the first-order residual, the second- eedee residual further reduces the attention error, which proves the necessity of introducing the second- order residual after quantization. Also, the second-residus after using SVD can further reduce the attention error, which proves that we have indeed extracted the temporally stable component and achieved the best attention approximation effect.
Datle
Calle
Dalee
Cario
oolor
2010
2012
2013
(a) block. 11.
(b) block.12.
(c) block.
(d) block. 14.
(e) block. 15.
Figure 12: More attention error comparison of Hunyuan Video- 13B (Kong et al., 2024).
H MORE ABLATION STUDY
Here, we provide more ablation study about the proposed Multi-Scale Salient Attention Distillation (MSAD) and Second- Sparse Attention Reparameterization (SSAR).
We first study the pooling stride s used in Eq. 6 and salient token &
Table 10: Ablation and used in attention distillation.
in Eq. 8 to verify the hyperparameter selection of both global and lo- cal distillation. We present the results Tab. 10. It can be seen that different hyperparameters can improve the distillation performance. This shows that our distillation method is both effective and robust, which is insensitive to hyperparameters. This also demonstrates that the eeemory-efficient distillation are effective enough and we do have to use the giant complete attention map to supervise the attention module. Higher 8 and lower k can reduce memory, but typically harm performance. Yet we identify that decreasing increasing k also brings little improvement. Since efficient as shown Fig. 3d, we choose this balanced selection.
Value Mose
VQ4+
PSNR;
SSIM,
LMMPS,
31.92
14:35
0.486
0.425
pooling stride
5-64
25.19
16.05
0.543
0.348
s-128
85.26
1601
0.547
0.349
5-256
85.12
15.93
01545
0.355
salient solen
k=128
86.21
16.72
0.551
0.349
k-256
36,95
16.82
0.561
0.325
1-512
86.93
16.95
01561
0.324
128 and A
256 are both effective and
We then study the top-r components in SVD used in Eq. 16, and present the results in Tab. 11. Com- pared with the original second-ords -oddr residual, it can be seen that the different selection of r in SVD can improve the temporal stability of the second-orde residual and bring better performance. In our experiment, we chose r = 16 as achieves good performance. We further explore higher-order residual effectiveness attention approximation. Compared with the Second-Order residual, Third-
22

---

Published as a conference paper at ICLR 2026
Table 11: Ablation on SVD used in SSAR.
Method
+QA,
PSNR,
SSIM,
LPIPS
None
68.00
14.16
0,470
0,445
First
70.82
17.08
0.572
0.285
Second
89.73
18.68
0.616
0.258
Third
89.71
18.70
0.620
0.263
1op-8
91.12
18.69
0.621
0.253
top-16 16
91.98
18.72
0.630 +00
0.240 00200
top-32
91.75
18.72
0.628
0.242
Order residual only slightly improves PSNR from 18.70 to 18.68 and decreases the performance on VQA. SSIM, and LPIPS. This indicates that the stability brought by higher-order residuals will grad- ually saturate, and we attribute to the additional noise brought by longer time series information on higher-order residuals. The second-order residual not only stabilizes the first-order residual, but SVD can further reduce spatiotemporal noise.
We then study the weight factor used in Eq. 9 to verify the dis- tillation robustness hyperparameters. We present the results Tab. 12. The values selected by controlling the distillation term to be of the same order of magnitude as Lquant- It can be seen that different weight factors improve the model performance. This shows that our distillation method is not only effective but also in- sensitive to the choice of hyperparameters, indicating its general- ization and effectiveness. Since Agtotal le 4 and Aiocal le 4 are good enough, and the hyperparameter selection robust, we do not further fine-tune the hyperparameter selection.
Table 12: Ablation on A. used in Eq.
PSNR:
SSIM;
LPIPS:
None
81.92
14.35
0.186
0.425
84.76
15.70
0.362
0.351
local
3e:3
85.73
16.80
0.447
0.336
1c-4
86.95
16.32
0.561
IL325
Se:4
86.54
16.72
0.562
0.328
We further compare our Multi-Scale Salient Attention Distillation (MSAD) with full-attention distil- lation (using the complete FP attention map as the target) on W-n2.1-1.3B (Wan et al., 2025) under W4A8 quantization. The results are shown in Tab. 13. MSAD achieves nearly identical performance to full-attention distillation. The results highlight MSAD's efficiency advantages while maintaining comparable performance.
Table 13: Ablation study on full-attention distillation.
Method
Resolution
LPIPS
Attention Memory (GB))
Calibration Time (Hours).
War.2.1
1.3B
Full Attention
(17472, 17472)
13.25
0.338
6.82
1.86
MSAD
(273,
15.23
0.66
MSAD (s 128)
(137, 137)
15.22
0.338
0.14
0.64
MSAD
Effect of attention density. We conduct an ablation study on attention density, analyzing the trade- off between performance and inference speed. The results are presented Tab. 5. As shown, a 25% density offers a good balance, achieving a significant 1.55x speedup with minimal performance degradation (PSNR 18.72). 15% density further boosts the speedup to 1.74x while maintaining acceptable performance (PSNR of 18.22). Based on these results, we selected 25% and 15% density for the experiments presented the main paper. The 25% density provides a strong baseline for high performance with good acceleration, while the 15% density demonstrates the potential for even greater inference speedup at a slightly decreased performance trade-off.
I CALIBRATION COMPUTATION RESOURCE
We study the calibration computation resource of each of our proposed methods and the overall pipeline. As Second- Order Sparse Attention Reparameterizzttoon (SSAR) is used for only inference, for calibration, we only add Multi-Scale Salient Attention Distillation (MSAD) compared to naive Post- Post-Training Quantization (PTQ). We present the calibration resource Tab. 14, Compared with
23

---

Published as a conference paper at ICLR 2026
Table 14: Calibration computation resource report. PTQ denotes naive Post-Training Quantization
without attention distillation.
Calibration Overload
Performance
Method
GPU Memory (GB),
GPU Time (Hours),
PSNR
LPIPS
Kam2.1 1.3B
PTQ
16.21
0.62
10.57
0.587
GGoobal
16.27.004%
0.63 00.25%
13.27
0.452
Local
16.28.0.4%
0.63
13.85
0.421
QuantSparse
16.34.0.8%
0.64 4%%
15.22..4.00
0.338 -0.30 249
Hunyuan Video 13B
PTQ
39.22
5.08
16.27
0.472
Global
00.3%
5.10
18.42
0.357
Locol
39.32.+0.5%
5.11....%
18.96
0.342
QuantSparse
39.41+0
5.13+1.0%
20.86..
0.272
Kan2.1 14B
PTQ
47.39
2.57
14.35
0.425
+Global
47.54
2.59
16.01
0.349
Locol
47.50,0.2%
2.58 .0.4%
16.82
0.325
QuantSparse
47.65
2.60
18.72..1.37
0.240
naive PTQ, our Global Distillation only brings an average of 0.8% extra time burden and almost additional memory consumption because its efficient low-resolution attention operation. Also, our Local Distillation only needs to calculate the token saliency distribution once before each block calibration and reuse the salient token index in each optimization iteration, which is also very ef- ficient. These two distillation methods are not only efficient but also can effectively alleviate the attention shift caused by quantization and improve the video generation effect. QuantSparse has sig- nificantly improved the model performance by combining two distillation methods, while ensuring high efficiency.
To further prove the effectiveness of proposed Second-Ordes Sparse Attention Reparameterization (SSAR), we present the inference burden brought by SSAR in Tab. 15. Compared with Non- Reparameterization, the cache-based method only requires one additional matrix addition opera- tion for the sparse attention output, which is very efficient. Therefore, the cache-based method will only bring little additional latency and memory burden. Furthermore, the second-order residual can store and calculate the second-order term and the first-order term together. Therefore, compared with the first-order residual, the second-order residual only requires an additional second order term calculation, but significantly improves the sparse attention performance under quantization, and im- proves the PSNR from 17.08 to 18.68 under Wan2. (Wan et al., 2025). In addition, using SVD extract the temporally stable component second-order residuals brings almost no additional consumption, but can further improve the effect of second-order residuals, which further decreases LPIPS from 0.258 to 0.240 under 1an2.1-14B.
COMBINATION WITH OTHER ACCELERATION TECHNIQUES
To further validate the integration ability of QuantSparse with other acceleration techniques, we combined it with existing attention quantization techniques SageAttention (Zhang al., 2024b) and cache techniques TeaCache (Liu et al., 2024a), and presented the results in Tab. 16. All the experiments are conducted on Wan2.1-14B (Wan et al., 2025) under W4A8 quantization setting. We apply Attentitn by quantizing attention into 8-bit. For TeaCache, we set the threshold as 0.16 to ensure performance.
It can be seen that, despite retaining only 15% attention density under W4A8 quantization, the combination of QuantSparse and SageAttention still incurs almost no performance loss. This in- dicates that QuantSparse is highly friendly to sparsification and quantization, fully demonstrating the necessity of attention distillation and second-order reparameterizztton. Although further adding TeaCache may result in slight performance decrease, it can bring significant additional inference acceleration. This provides a further trade-off between performance and inference speed, and also demonstrates the effectiveness of combining QuantSparse with cache-based methods.
24

---

Published as a conference paper at ICLR 2026
Table 15: Sparse
attention reparameterization
resource
report.
'None'
denotes Non-
Reparameterizattion.
Inference Overload
Performance
Method
GPU Memory (GB),
DiT Time (s),
PSNR,
LPIPS
Mam2.. 1.3B
None
5.44
312
10.57
0.587
First
5.84
313.40
12.76
0,493
-Sconnd
5.93 ..9
313 +0.3%
13.55
0.427
QuantSparse
5.93.w%
313,00.0
15.22
0.338
Hunyuan Video 13B
None
24.34
725
16.27
0,472
-First
26.51
729
18.25
0.381
+Second
27.02 -11%
730 10.7%
19.03
0.317
QuantSparse
27.02.+13%
731 +0.09%
20.86
0.272 0.220
Wan2.1 14B
None
26.04
2589
14.16
0,445
+First
27.86 77%
2593 00.2%
17.08
0.285
+Second
28. ..8%
2594 .0.2%
18.68
0.258
QuantSparse
28.14: 8%%
2594 0..2%
18.72 4.56
0.240
0.205
We further provide more visualization results in Fig. 13. can be seen that the combination of QuantSparse and other acceleration techniques not only shows almost decrease in metrics but also maintains good visual effects without producing any decrease in visual quality.
Table 16:
More efficiency comparison under W4A8 quantization setting. Sage. denotes SageAtten- tion (Zhang et al., 2024b). Tea. denotes TeaCache (Liu et al., 2024a).
Method
Quality
Latency & Speed
Density
QuantSparse
SageAttention
TeaCache
CLIPSIM-
VQA,
AFSCore
DiT Time
Speedup:
Wan2.1 I 14B (CFG 5.0, 720 1280p, Erames
80)
Full Prec.
100%
0.182
90.79
0.000
4031s
1.00x
0.183
91.98
0.056
259ds
1.55x
25%
0.181
91.70
0.240
2480s
1.63x
0.180
84.01
0,211
1802s
2.24x
0.182
90.73
0.042
2315s
1,74x
f
15%
0.180
90.58
0.046
2201s
1.83x
0.179
86.24
0.249
1629s
2.47x
FP16 Full Attn.
QpantSparse (15%)
QuantSparse Sage. (15%)
QuantSparse Sage.
Tea.
Figure 13: Combining with other acceleration techniques visualization Wan2.1-14B under W4A8 quantization setting.
25

---

Published as a conference paper at ICLR 2026
K IMAGE GENERATION EXPERIMENT
QuantSparse is designed as a general framework for Diffusion Transformers (DiTs) and is not lim- ited to video generation. To validate its generalizability, conducted an experiment on Hunyuan- DiT (Li et al., 2024c), a 1.5B parameters model targeting image generation. We evaluate on Draw- Bench (Saharia et al., 2022) under W4A8 quantization and present the results in Tab. 17. Even for image-generation DiTs, QuantSparse outperforms SOTA quantization baselines QuaRot (Ashkboos al., 2024) and Q-VDiT (Feng et al.. 2025c) while using only 40% attention density. This con- firms that our framework generalizes to DiT-based visual generation tasks and not limited to video generation.
Table 17: Image generation experiment results on HlnnyunnDDil
Method
Density,
PSNR
SSIM,
LPIPS,
Hunyuan-Dii
QuaRot
100%
17.30
0.627
0.460
Q- DDiT
100%
19.32
0.658
0.347
QuantSparse
40%
20.34
0.692
0.289
L THE USE OF LARGE LANGUAGE MODELS (LLMS)
In this paper, Large Language Models are only used as general-puree purpose auxiliary tools, primarily for document-leww auxiliary tasks such grammar checking and expression refinement. LLMs did not participate in the core conceptualization, method derivation, or experimental design of this research, nor did they contribute to any core writing content.
M MORE VISUAL COMPARISON
In the following pages, we provide more visual comparisons of different-scale video-generation models. 'Full Prec.' denotes the Full Precision model. (xx%) denotes the attention density.
We also provide the used text prompt for each figure:
1. Fig. 14: A soaring drone footage captures the majestic beauty of a coastal cliff. its red and yellow stratified rock faces rich in color and against the vibrant turquoise of the sea. Seabirds can be seen taking flight around the cliff's s precipices. As the drone slowly moves from different angles, the changing sunlight casts shifting shadows that highlight the rugged textures of the cliff and the surrounding calm sea. The water gently laps at the rock base and the greenery that clings the top of the cliff, and the scene gives sense of peaceful isolation at the fringes of the ocean. The video captures the essence of pristine natural beauty untouched by human structures.
2. Fig. 15: A serene night scene in a forested area. The first frame shows tranquil lake reflecting the star-filled sky above. The second frame reveals a beautiful sunset, casting a warm glow over the landscape. The third frame showcases the night sky, filled with stars and a vibrant Milky Way galaxy. The video is a time-lapse, capturing the transition from day night, with the lake and forest serving as constant backdrop. The style of the video is naturalistic, emphasizing the beauty of the night sky and the peacefulness of the forest.
3. Fig. 16: A serene underwater scene featuring a sea turtle swimming through a coral reef. The turtle, with its greenish-browe shell, is the main focus of the video, swimming grace- fully towards the right side of the frame. The coral reef. teeming with life, is visible in the background, providing vibrant and colorful backdrop to the turtle's 's journey. Several small fish, darting around the turtle, add a sense of movement and dynamism to the scene. The video is shot from slightly elevated angle, providing comprehensive view of the turtle's surroundings. The overall style of the video is calm and peaceful, capturing the beauty and tranquility of the underwater world.
26

---

Published as a conference paper at ICLR 2026
4. Fig. 17: The video captures the majestic beauty of waterfall cascading down cliff into serene lake. The waterfall, with its powerful flow, is the central focus of the video. The sur- rounding landscape is lush and green. with trees and foliage adding the natural beauty of the scene. The camera angle provides a bird's s eye view of the waterfall, allowing view- ers appreciate the full height and grandeur of the waterfall. The video is stunning representation of nature's 's power and beauty.
5. Fig. 18: The dynamic movement of tall, wispy grasses swaying in the wind. The sky above is filled with clouds, creating dramatic backdrop. The sunlight pierces through the clouds, casting a warm glow on the scene. The grasses are a mix of green and brown, indicating change in seasons. The overall style of the video is naturalistic, capturing the beauty of the landscape in a realistic manner. The focus is on the grasses and their movement, with the sky serving secondary element. The video does not contain any human or animal elements.
6. Fig. 19: The video captures the majestic beauty of a waterfall cascading down a cliff into a serene lake. The waterfall, with its powerful flow, is the central focus of the video. The sur- rounding landscape is lush and green, with frees and foliage adding to the natural beauty of the scene. The camera angle provides a bird's eye view of the waterfall, allowing view- ers to appreciate the full height and grandeur of the waterfall. The video is stunning representation of nature's power and beauty.
27

---

Published as a conference paper at ICLR 2026
Full Prec. (100%)
SmoothQuant (100%)
QuaRot (100%)
Q-VDiT (100%)
QuantSparse (25%)
QuantSparse (15%)
Figure 14: Hunyuan Video- results.
Full Prec. (100%)
SmoothQuant (100%)
QuaRot (100%)
Q-VDiT (100%)
QuantSparse (25%)
QuantSparse (15%)
Figure 15: Hunyuan VlanyuanVideo1133 Video-13B results.
28

---

Published as a conference paper at ICLR 2026
Full Prec.
(100%)
SmoothQuant (100%)
QuaRot (100%)
Q-VDIT (100%)
QuantSparse (25%)
QuantSparse (15%)
Figure 16: Wan2.1-141 1-14B results.
Full Prec. (100%)
SmoothQuant (100%)
QuaRot (100%)
Q-VDIT (100%)
QuantSparse (25%)
QuantSparse (15%)
Figure 17: Wan2.1-14B results.
29

---

Published as a conference paper at ICLR 2026
Full Prec. (100%)
SmoothQuant (100%)
QuaRot (100%)
Q-VDIT (100%)
QuantSparse (40%)
Figure 18: Wan2.1-1.3B results.
Full Prec. (100%)
SmoothQuant (100%)
QuaRot (100%)
Q-VDiT (100%)
QuantSparse (40%)
Figure 19: Wan2.1-1.3 results.
30


Reviews:
[{"review_id": "SlmGJ6CfwY", "reviewer": "Reviewer_TKrA", "summary": "In this work, QuantSparse is proposed for compressing video diffusion transformers, with both attention sparsification and quantization applied. The authors point out that, when both techniques are naïvely applied, the error introduced by quantization will induce attention shift, which negatively impacts model performance. In QuantSparse, MSAD is used to efficiently perform attention distillation, leveraging both global and local guidance, and SSAR is proposed to exploit the temporal stability of the second-order residual to enhance sparse attention. The results show that diffusion transformers compressed by the proposed method can achieve SOTA performance while reducing both storage and latency simultaneously.", "questions": "- Is the proposed method applicable to other transformer models that are not designed for video generation?\n- How does the attention distillation perform when using full attention as the target?\n- Could you clarify the calculation of the saliency? In Eq. (7), it is written as $s_{i}=\\sum_{j}A_{j, i}$, but in the figure 2 it is $\\sum_{h, i}A_{h, i, j}$.\n- Given the large pooling stride used, what are the attention maps resolution? How do they impact the distillation time?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a detailed analysis of the problem caused by the combined use of attention sparsity and quantization. The proposed method is rational, easy to implement, and has low overhead, while demonstrating promising performance.\n- The results are impressive. It improves the inference efficiency of SOTA video generation models with minimal degradation in generation quality.", "weaknesses": "- There are some clarity issues in the paper. Some symbols are reused for different hyperparameters. For example, the pooling stride is denoted as $s$ in Eq. (6) and the SVD rank is denoted as $r$ in Section 4.3, but the pooling stride becomes $r$ in Section H. See also the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NTD8JIlYwE", "reviewer": "Reviewer_5UCB", "summary": "This paper proposes QuantSparse, a unified framework that jointly applies model quantization and attention sparsification to compress video diffusion transformers efficiently. To address the performance drop caused by their naive combination, the authors introduce: Multi-Scale Salient Attention Distillation (MSAD) for aligning quantized and sparse attention using global–local supervision, and Second-Order Sparse Attention Reparameterization (SSAR) for recovering information lost under sparsity via temporally stable residuals.\nExperiments on HunyuanVideo and Wan2.1 models show that QuantSparse achieves up to 3.8× compression and 1.7× speedup with minimal quality loss, outperforming prior quantization and sparsification baselines.", "questions": "For more details, see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper demonstrates strong methodological innovation by proposing the first unified framework that integrates model quantization and attention sparsification, along with a systematic analysis of the “amplified attention shift” problem, showcasing both theoretical depth and engineering value. The proposed MSAD and SSAR modules effectively mitigate performance degradation from the perspectives of distillation calibration and dynamic reconstruction, balancing global and local features with good generality and scalability. Extensive experiments on multiple video generation models validate the approach, showing that QuantSparse significantly outperforms existing baselines in metrics such as PSNR, VQA, and LPIPS, while achieving notable model compression and inference acceleration.", "weaknesses": "1. Figure 2 serves as the core illustration of the QuantSparse framework; however, the manuscript does not provide sufficient explanation of each module’s function, their interrelations, or the meaning of the symbols used. It is recommended that the authors include detailed annotations and descriptions of key steps and variables in the main text or appendix to enhance the clarity and reproducibility of the method.\n\n2. The main objective of this paper is to achieve efficient video diffusion model deployment on consumer-grade GPUs through model quantization. Therefore, the specific experimental hardware configurations (e.g., GPU model, memory capacity, and system environment) are crucial for assessing the practical feasibility of the proposed method. The authors are encouraged to explicitly include these hardware details in the main text.\n\n3. In the Introduction, the authors state that directly combining quantization and sparsification causes an “amplified attention shift,” leading to performance degradation. However, only theoretical analysis is provided without corresponding experimental data or visual evidence. It is suggested that the authors include quantitative comparisons or visualizations to more convincingly demonstrate the existence and impact of this phenomenon.\n\n4. The experiments employ a 15% attention sparsity ratio, yet the rationale or tuning process behind this choice is not explained. The current setup appears to be empirical and lacks systematic hyperparameter analysis. The authors are advised to clarify the basis for selecting this sparsity level and to provide comparative experiments, such as results at 10%, 20%, and 30% sparsity levels, showing the trade-off between model performance and inference acceleration in the main text or appendix to strengthen the persuasiveness of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5hF4LtU55F", "reviewer": "Reviewer_N8bA", "summary": "This paper presents **QuantSparse**, a unified framework for compressing large video diffusion transformers (DiTs) by combining model quantization and attention sparsification. The core problem it addresses is that naively integrating these two compression techniques leads to compounded errors and severe performance degradation. The authors propose a two-part solution: 1) **Multi-Scale Salient Attention Distillation** (MSAD), a post-training alignment method to mitigate quantization-induced bias , and 2) **Second-Order Sparse Attention Reparameterization** (SSAR), an inference-time technique to recover information lost to sparsity. The method achieves impressive results, demonstrating significant reductions in model storage and latency while maintaining the generation quality of the original full-precision models.", "questions": "I find it surprising that the compressed model not only matches but, in several instances, **slightly improves upon the full-precision (FP) baseline** on key video quality metrics (e.g., VQA scores in Table 1 2).\n\nThis is an interesting result. Do you have insights on why this occurs?\n\n* Is this improvement simply an artifact of the post-training alignment (MSAD), which may act as a light fine-tuning step, effectively regularizing the model or cleaning up minor noise from the original FP checkpoint?\n* Or, does this hint at a more fundamental property of these large video DiTs? For example, could it be that the models are **significantly over-parameterized**, and that the aggressive, aligned compression from QuantSparse acts as a powerful regularizer, forcing the model to rely on its most robust features and leading to a more generalizable output?\n* Or is it the metric VQA itself that has some special property for DiTs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Alignment of Non-Trivial Techniques:** The main strength of this work is the **synergistic alignment** of two orthogonal and error-introducing acceleration techniques (quantization and sparsification). The paper correctly identifies that the key challenge lies in managing the compounded errors, and it proposes a dedicated solution for each error source (MSAD for quantization bias, SSAR for sparsity loss) within a single, unified pipeline. This method could also be combined with other methods like SageAttention and TeaCache.\n\n2.  **Excellent Empirical Results:** The method delivers on its promise. The experiments show a significant reduction in model size (e.g., **3.80x** for Wan2.1-14B) and inference latency (e.g., **1.74x** to **1.88x**) across multiple large-scale models. Crucially, this efficiency gain comes with almost no loss—and in some cases, a slight improvement—in video quality metrics compared to the full-precision baselines.\n\n3.  **Good Presentation:** The paper is well-written and clearly presented. The problem statement is well-motivated, and figures like Figure 2, 3, and 4 effectively illustrate the proposed mechanisms and the empirical phenomena that motivate them (e.g., saliency distribution, residual stability).", "weaknesses": "1.  **Complex, Engineering-Heavy Pipeline:** One possible weakness is the **high complexity of the overall system**. The QuantSparse pipeline is highly engineered and involves multiple, intricate components:\n    * MSAD is a two-branch distillation process requiring both global average pooling and local top-k salient token selection.\n    * SSAR is a multi-stage inference process involving caching, calculating second-order residuals, and then further applying SVD-based projection to denoise these residuals.\n\n    This makes the solution appear less like a simple, generalizable algorithm and more like a complex, bespoke system, which may hinder its broader adoption.\n\n2. **Post-Training Alignment Requirement:** The method relies on a post-training calibration/alignment step (MSAD). This adds overhead compared to a zero-shot compression method, though this is also necessary for PTQ. This alignment step allows the authors to identify and exploit key internal patterns of the DiT, such as the heavy-tailed token saliency distribution shown in Figure 3a, which is a useful finding.\n\n3.  **Theoretical Rigor:** The main weakness lies in the theoretical justification, which is **not tight**. The \"theory\" section feels more like a *post-hoc* justification for the engineering choices rather than a *first-principles* derivation.\n    * **Proposition 3.3 as an Assumption:** The entire SSAR method hinges on **Proposition 3.3**, which states that the second-order residual is more temporally stable than the first-order. This is the most critical claim, yet it is presented as a \"proposition\" when it is, in fact, an **empirical assumption** justified only by Figure 4a. The paper attributes this to quantization noise being a \"slow-varying stochastic process\", which is another unproven assumption.\n    * **Tautological Theorem:** **Theorem 3.4** is presented as a theoretical guarantee but is effectively **tautological (a circular argument)**. The proof in Appendix A essentially demonstrates that *if* one assumes Proposition 3.3 is true (i.e., the second-order error is smaller), *then* the second-order approximation method has a smaller expected error. This theorem does not provide any new insight beyond restating its own assumption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bogOQKpqgc", "reviewer": "Reviewer_Ff2L", "summary": "This paper proposes QuantSparse, a framework to jointly compress large video diffusion transformers using model quantization and attention sparsification. The authors argue that naively combining these two techniques leads to severe performance degradation due to \"amplified attention shifts,\" where quantization noise exacerbates sparsity-induced information loss. To solve this, they introduce two techniques: 1. Multi-Scale Salient Attention Distillation (MSAD): A post-training quantization (PTQ) calibration method that uses a memory-efficient distillation loss. It combines (a) global guidance from downsampled attention maps to preserve coarse structure and (b) local guidance focused on high-saliency tokens to preserve fine-grained details. 2. Second-Order Sparse Attention Reparameterization (SSAR): An inference-time technique that aims to recover information lost to sparsity. It posits that the second-order residual (the change in the attention residual between timesteps) is more temporally stable than the first-order residual, especially under quantization. It caches this second-order residual (projected via SVD) to apply a more accurate correction at subsequent steps. Experiments on large models (Hunyuan-13B, Wan2.1-14B) show that QuantSparse achieves significant storage (3.8x) and latency (1.88x) reductions while claiming to maintain or even exceed the quality of the full-precision models.", "questions": "See weakness. I hope the authors could clarify these issues, and I am willing to reevaluate based on the response.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Compressing large-scale video generation models is a critical bottleneck for their practical deployment. The paper identifies the interaction between quantization and sparsification as the key challenge, which is a more nuanced and accurate problem statement than addressing each in isolation.\n2. The two proposed components are well-motivated.\n3. The method achieves good performance, substantially outperforming existing quantization-only (Q-VDIT) and sparsification-only (SVG) methods, as well as their naive combinations (e.g., Q-VDIT+SVG). \n4. The authors test on multiple, very large models and use a comprehensive suite of evaluation metrics, including VBench and standard video quality metrics (PSNR, SSIM, LPIPS, VQA).", "weaknesses": "1. Unexplained \"Better-than-FP\" Results: The most significant concern is that the compressed model (QuantSparse) outperforms the 16-bit Full Precision (FP) baseline on key quality metrics.\n\nIn Table 5 (Hunyuan-13B, W6A6), QuantSparse (15%) achieves a VQA score of 82.26, while the FP model scores 81.23.\n\nIn Table 6 (Wan2.1-14B, W4A8), QuantSparse (15%) achieves an \"Imaging Quality\" score of 63.81, while the FP model scores 63.38.\n\nWhile quantization can occasionally act as a regularizer, such outperformance on large-scale generative models is an extraordinary claim that requires an explanation. The authors do not acknowledge, investigate, or explain this phenomenon. This could suggest an undertrained FP baseline, a flaw in the evaluation metrics, or a fundamental misunderstanding of the model's behavior. Without a thorough discussion, these results cast doubt on the validity of the entire evaluation.\n\n2. Inconsistent/Confusing Ablation Study: The paper's narrative motivates MSAD as the primary solution to the core \"amplified attention shift\" problem. However, the ablation study in Table 3 seems to tell a different story and is presented in a highly confusing manner.\n\nAssuming the \"Cache Analysis\" block's \"None\" row (68.00 VQA) represents an MSAD-only model (with no cache) and the \"Distillation Analysis\" block's \"None\" row (81.92 VQA) represents an SSAR-only model (with no distillation).\n\nIf this interpretation is correct, it implies that SSAR-only (81.92 VQA) provides a vastly larger performance gain over the \"None (Cache)\" baseline (68.00 VQA) than MSAD-only does. In fact, it suggests SSAR is the dominant contribution, and MSAD provides a smaller, secondary boost (from 81.92 VQA to 91.98 VQA).\n\nThis directly contradicts the paper's core motivation, which focuses on the quantization/sparsity interaction (MSAD's job). The authors must clarify the baselines in Table 3 and reconcile this apparent contradiction.\n\n3. The SSAR method introduces a key hyperparameter: the \"cache-refreshing interval\" (set to 5). This parameter dictates the trade-off between the accuracy of the cached residual and the computational overhead of re-calculating the full-attention map. This is a critical parameter that is not ablated in the paper or appendix.\n\nMinor:\n4. I wish the author could reorganize the tables to combine the results table with the Efficiency comparison table to better illustrate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)